Technical

Check whether AI crawlers can read your site

Most sites are missing from AI answers for an unglamorous reason: a crawler was turned away at the door. This checks that first, then the page it would have read.

AI Visibility
Checks robots.txt and llms.txt at the root, then the page itself.
Only needed when a host refuses bot requests for robots.txt. Leave blank to fetch it.
Results appear here. Nothing is stored.

How AI Visibility Is Actually Decided

01

What AI Visibility Means Here

An AI answer cites a page only if a crawler was allowed to read it, and only if the page said something specific enough to quote. This tool checks the first half and reports the evidence for the second.

It does not ask ChatGPT about your brand. That would be a different product with a different price, and it is covered further down.

02

The Crawlers That Decide It

CrawlerWhat it feeds
OAI-SearchBotThe index behind ChatGPT search answers
GPTBotOpenAI's wider crawl, training and retrieval
ClaudeBot, Claude-SearchBotAnthropic's crawl and Claude's web search
PerplexityBotPages Perplexity can cite in an answer
GooglebotSearch, which AI Overviews and AI Mode draw on
BingbotBing, which Copilot answers are built from
CCBotThe open Common Crawl dataset many models train on
03

Why Robots.txt Comes First

A blocked crawler never sees the page, so no amount of content work reaches that assistant. It is the only part of AI visibility that is a binary switch, and it is the part most sites have never looked at.

  • Some hosts, Cloudflare among them, offer a one click block for AI crawlers. It takes effect at the edge, so robots.txt still looks open.
  • An allow rule needs to sit above the wildcard group, or the wildcard still wins.
  • Blocking one crawler does not block the rest. Each user-agent group is read separately.
04

Google-Extended Is Not Googlebot

Google-Extended controls Gemini and Vertex training. It has no effect on Search, and no effect on AI Overviews, because those are drawn from the normal index that Googlebot builds.

Sites that block Google-Extended to keep their content out of Gemini often believe they have opted out of AI answers. They have not.

05

What llms.txt Is And Is Not

llms.txt is a proposed convention: a Markdown file at the root that summarises a site for language models. It is not a standard, Google has said it does not use it, and no major assistant requires it.

It costs one file and does no harm. Treat it as a tidy summary for anyone reading programmatically, not as a ranking or citation mechanism.

06

Why This Tool Will Not Score Your Brand

  • A prompt panel costs money per run, so a free version would have to sample a handful of prompts and call that a score.
  • Answers change per model, per location and per day. One number cannot hold that.
  • Most tools in this category measure whether one vendor's model names you, then present it as visibility everywhere.
07

What Actually Moves AI Citations

  • Crawl access, because nothing else matters if the page is refused.
  • Specific, checkable claims: a number, a date, a named method, in a sentence that stands alone.
  • Entity clarity: who publishes this, what it is about, marked up so a machine can tell.
  • Third-party mentions, because assistants repeat what other sites already say about you.
08

A Sensible Order Of Work

  • Allow the crawlers you actually want to be cited by.
  • Remove noindex from pages that should be quotable.
  • Add author, publisher and dates to the markup.
  • Rewrite the two or three strongest pages so each key claim is stated in one quotable sentence.
  • Then look at mentions elsewhere, starting with the sites your buyers already read.

Common Questions

Does This Tell Me If ChatGPT Mentions My Brand?

No, and no free tool honestly can. Prompt tracking costs money per run and changes per model and per day. This checks whether ChatGPT could read the page at all, which is the part you can fix today.

Why Is Google-Extended Separate From Googlebot?

Google-Extended controls Gemini and Vertex training only. AI Overviews and AI Mode are built from the normal Googlebot index, so blocking Google-Extended does not remove you from AI answers.

Everything Is Allowed But I Am Still Not Cited

Access is necessary, not sufficient. A page also needs a specific claim worth quoting, a clear publisher and mentions elsewhere. The guide panels below cover that half.

Should I Block AI Crawlers?

That is a legitimate choice, and some publishers make it. Be aware that blocking GPTBot and OAI-SearchBot removes you from ChatGPT's retrieval as well as its training, which usually costs more visibility than it protects.

Why Does It Say robots.txt Is Missing?

A site without one allows everything by default, so nothing is broken. If it says unreadable, the host refused our request or returned an HTML page instead: open robots.txt in your browser and paste it into the optional box.

More Tools

Get in touch

Tell us what you want to rank for

Custom quote in one business day. Human outreach, monthly proof.

The form did not load. That is our fault, not yours.

Either reload the page, or email contact@buildingbacklinks.io and we will reply within one business day.

Use the contact page