AI Crawler Check.
A free Chrome extension that reads any site’s robots.txt and tells you which AI crawlers it lets in, which it turns away, and what each refusal costs.
Your robots.txt was written before any of these existed.
Most robots.txt files on the web were not written so much as inherited. Copied from a template, generated by a CMS, or typed once by whoever built the site years ago, and never opened again. For a long time that was fine, because the file only ever governed search engines and the rules that mattered were the ones about Googlebot.
The same file now decides something else entirely. It decides whether ChatGPT, Claude, Perplexity, Gemini and Apple Intelligence can read your site at all. A single Disallow: / under a wildcard user agent turns every one of them away at once, and a rule written in 2019 to keep a staging site out of Google is doing exactly that on a great many sites right now.
What makes it expensive is that nothing tells you. Search Console reports on Googlebot and has nothing to say about the rest of the file. Analytics counts visits that happened, so a crawler that was refused at the door leaves no trace to find. There is no warning, no error, no drop in a chart. The first real signal is a buyer mentioning that they asked an AI for a recommendation and got somebody else.
Ten seconds, on any site, from the toolbar.
We built the check we kept running by hand. It reads both files, resolves the rules against every agent on the list the way a crawler would, and reports the answer in the language of consequences rather than syntax.
One deliberate refusal. When a site answers with a 401, 403 or 429, which bot protection does routinely, the extension reports unknown rather than allowed. On this question a false pass is worse than no answer, so it does not guess.
The Chrome Web Store listing is in review.
The install link lands here the moment it clears. It will be free, with no account and nothing to configure. If you want the check run across your site properly in the meantime, that is the work below.
All 16 agents, what each one is for, and what blocking it costs.
| Agent | What it is for | What blocking it costs |
|---|---|---|
| OpenAI | ||
| GPTBot | Collects pages that go into training future OpenAI models. It is not the crawler behind ChatGPT search, and the two are blocked separately. | Your content stays out of what the models learn. Nothing disappears from ChatGPT search, because that runs on a different agent. |
| OAI-SearchBot | Builds the index ChatGPT search reads from. OpenAI documents it as search only, not training. | You stop appearing in ChatGPT search results and in the sources it links. This is the one that costs you buyers. |
| ChatGPT-User | Fetches a single page live, at the moment a person in ChatGPT asks about it or follows a link to it. | ChatGPT cannot open your page when someone asks about you by name. It answers from whatever it already has, which may be years old. |
| Anthropic | ||
| ClaudeBot | Collects pages for training Claude. | Your content stays out of future training runs. Live fetches and search are unaffected. |
| Claude-User | Fetches a page live when a Claude user asks about it or pastes the link. | Claude cannot read your page on request, so it answers about you without ever seeing you. |
| Claude-SearchBot | Indexes pages so Claude can find them and cite them when it searches. | You drop out of what Claude can surface and cite. |
| Perplexity | ||
| PerplexityBot | Builds the index Perplexity answers from, and the citation list under each answer. | No citations. Perplexity shows its sources on every answer, so this is a visible absence. |
| Perplexity-User | Fetches a page because a specific user question sent Perplexity to it. Perplexity documents this as user triggered rather than crawling. | Perplexity treats a user request as a visit rather than a crawl, so a disallow here may not stop the fetch. Read their published position before you rely on it. |
| Googlebot | The crawler behind Google Search. The same crawl feeds AI Overviews and AI Mode, which are Search features rather than separate products. | You leave Google Search. This is the most expensive line in the file, and it is almost never blocked on purpose. |
| Google-ExtendedSwitch, not a crawler | Not a crawler. A switch that controls whether pages Googlebot already has can be used to train and ground Gemini. | Gemini stops using your content. Google states this does not affect Search ranking or AI Overviews, because those are Search. |
| Apple | ||
| Applebot | Crawls for Siri, Spotlight, Safari suggestions and Apple search features. | You disappear from Siri and Spotlight results. |
| Applebot-ExtendedSwitch, not a crawler | Not a crawler. A switch over pages Applebot already has, covering their use in training Apple foundation models. | Apple stops training on your content. Siri, Spotlight and Safari suggestions are untouched. |
| Meta | ||
| meta-externalagent | Collects pages for Meta AI training and for AI features across Meta products. | Your content stays out of Meta AI, on a platform where a lot of buyers ask their first question. |
| Other | ||
| CCBot | Common Crawl. A free public archive of the web that has been a training input for a long list of models, and for a lot of academic research. | You are absent from a corpus you cannot join retroactively. A block today does not remove what was archived before it. |
| Bytespider | ByteDance. Feeds TikTok search and ByteDance models. | Nothing on the AI answer side, and a quieter server. It has a reputation for crawling hard, which is usually why it is blocked. |
| Amazonbot | Crawls for Alexa, Rufus and Amazon answer features. | You drop out of what Amazon assistants can answer with. |
And then there is llms.txt.
llms.txt sits at the root of your site next to robots.txt, and it does the opposite job. Robots.txt says what a machine may not fetch. llms.txt is a markdown file that says here is what this site is, here are the pages worth reading, and here is a line on each of them. The extension checks for one on every site it looks at and tells you whether it is there, which is why it earns a section here.
Be clear about its status. It is a proposal, not a standard, and no engine has publicly committed to reading it. Anyone telling you it is a ranking factor is selling something. What it is, is the one file where you describe your own site in your own words instead of leaving a model to infer your business from your navigation, and it costs about an hour to write. We publish ours, generated from the CMS so it cannot go stale.
It is not a substitute for the file above it. Robots.txt is enforced; llms.txt is an offer. An llms.txt on a site that disallows every AI crawler is an invitation posted to a locked door, which is the order of operations worth getting right: open the door first.
The extension collects nothing, because there is nothing to collect.
AI Crawler Check is entirely client side. It reads two public files from the site you are already looking at and shows you what they say. That is the whole program, and the list below is the complete account of what it does with data.
activeTabis the entire permission list. One entry in the manifest, no host permissions, nothing else. It grants temporary access to the active tab’s origin at the moment you click the icon, which is what allows the two fetches, and it lapses again afterwards. The extension ships no content script, so nothing it runs touches the page you are on: it reads two files from that page’s origin, and that is the whole of its access. GOJI’s site-wide privacy policy covers goji.agency itself, and nothing in the extension reports to it.
Knowing you are blocked is the first ten seconds of the job.
The extension answers one question: can they get in. It cannot tell you whether you are being named in the answers once they can, which engines are already recommending your competitors, or which pages would have to exist before any of them had something to cite. That is measurable, and it is what our Search and AI Visibility practice does.
We run search across Google and the AI answer engines, and we publish what it costs.