# LobsiteBot

LobsiteBot fetches a page when someone checks its link preview with Lobsite's Social Share Preview. What it reads, how often, and how to allow or block it.

User agent: `LobsiteBot/1.0 (+https://lobsite.com/bots/lobsitebot)`

Page: https://lobsite.com/bots/lobsitebot

LobsiteBot is the fetcher behind Lobsite's free [Social Share Preview](https://lobsite.com/tools/social-share-preview). When someone enters an address there and presses **Check**, LobsiteBot fetches that one page, the way Facebook's, X's and Slack's link preview crawlers do, and reads the tags that decide how a shared link looks. It isn't a search engine crawler: it never follows links, and it only visits a page a person asked to check.

## What it fetches

- **The page** at the address that was checked, with a `GET` request. It follows up to 5 redirects, reads at most 2 MB of HTML and gives up after 8 seconds.
- **Up to two images:** the page's `og:image`, and its `twitter:image` when that's a different file. At most 8 MB each, also within 8 seconds.

It reads the `<title>`, the `<meta>` tags and the `rel="canonical"` and icon links. It doesn't run JavaScript, keep cookies, sign in, submit forms or download anything else. Nothing it fetches is stored for longer than two minutes.

## How often it visits

Only when someone checks a page, and never more than **60 requests a minute to one host**, from everyone using the checker together. Results are kept for two minutes, so the same address checked again in that time doesn't reach your server at all.

## How to recognise it

The user agent is:

```
LobsiteBot/1.0 (+https://lobsite.com/bots/lobsitebot)
```

The requests come from Cloudflare's network, where Lobsite runs, so there's no fixed list of IP addresses. Cloudflare adds a `CF-Worker: lobsite.com` header to them. Match `LobsiteBot` in rules and log searches: it also covers the older user agent, `Mozilla/5.0 (compatible; LobsiteBot/1.0; +https://lobsite.com/tools/social-share-preview)`.

## Allow LobsiteBot

If your firewall or bot protection turns it away, the checker says your page answers 403 (or 401 or 429), and can't show its previews. Facebook, X and Slack may still be let in, since most bot filters know their crawlers by name. To let LobsiteBot in too:

- **Cloudflare:** Security → WAF → Custom rules: a rule with `(http.user_agent contains "LobsiteBot")` and the action **Skip**, skipping the bot protection and managed rules that block it.
- **Wordfence, Sucuri and other WordPress firewalls:** add `LobsiteBot` to the allowed user agents.
- **Your own server:** make sure rules that block unknown bots don't match `LobsiteBot`.

## Block LobsiteBot

LobsiteBot doesn't read `robots.txt`. Like other link preview fetchers, it acts on one request from a person, for one page. To block it, turn the user agent away:

- **Cloudflare:** a custom rule with `(http.user_agent contains "LobsiteBot")` and the action **Block**.
- **nginx:** `if ($http_user_agent ~* "LobsiteBot") { return 403; }` in the `server` block.
- **Apache:** `RewriteCond %{HTTP_USER_AGENT} LobsiteBot [NC]` followed by `RewriteRule .* - [F,L]`.

People checking your pages then see that your site turned the checker away.

## Frequently asked questions

### Why did LobsiteBot visit my site?

Someone checked how a link to your page looks when it's shared on X, Facebook, LinkedIn, Slack, Discord, WhatsApp or iMessage. Often that's you or your team, after editing the page's tags; sometimes it's someone about to share your link.

### Does LobsiteBot index or copy my content?

No. It reads the head of one page and its preview images, shows them to the person who asked, and keeps the result for two minutes. Nothing is indexed, published or used to train anything.

### Is LobsiteBot safe to allow?

It only makes `GET` requests for public pages and images, at most 60 a minute per host, and never follows links. A user agent can be faked, so if you allow it in a firewall, allow it only past the bot checks, not past rules that protect logins or admin pages.

### How do I report a problem with LobsiteBot?

Write to abuse@lobsite.com with the time, the address it fetched and your site's response. We read every message.