Question 1

What is an AI crawler?

Accepted Answer

An AI crawler is a bot that visits websites to collect content for training large language models (LLMs). Unlike search crawlers that index pages for ranked results, AI crawlers build training datasets. GPTBot trains ChatGPT, ClaudeBot trains Claude, Google-Extended trains Gemini, and so on. Each major AI company runs its own.

Question 2

Why do AI crawlers matter for my brand?

Accepted Answer

If your site blocks AI crawlers, your content never enters the training data those models learn from. This directly reduces the chance that AI assistants recommend, mention, or cite your brand when users ask relevant questions. It is a fast-growing blind spot for most marketing teams.

Question 3

Should I block AI crawlers from my site?

Accepted Answer

Blocking AI crawlers is a legitimate choice if you want to protect proprietary content. The trade-off is reduced AI visibility. For most brands trying to build citation presence, allowing crawlers is the better strategy. The exception: if your content is a competitive moat, blocking selectively makes sense.

Question 4

What is Google-Extended and how is it different from Googlebot?

Accepted Answer

Googlebot indexes your site for traditional Google Search. Google-Extended is separate and controls whether your content is used to train Gemini and other Google AI products. You can block Google-Extended without affecting your Google Search ranking. They are completely independent signals.

Question 5

What does Default status mean in this checker?

Accepted Answer

Default means there is no explicit rule in your robots.txt for that specific crawler. The crawler inherits whatever your wildcard (User-agent: *) rule says. If you have no robots.txt at all, Default is equivalent to Allowed since the absence of restrictions means full access.

Question 6

How do I fix a blocked crawler?

Accepted Answer

Add a specific User-agent block to your robots.txt file with Allow: /. For example: User-agent: GPTBot then Allow: /. This overrides any wildcard Disallow rule for that specific crawler. You can add multiple User-agent blocks, one per crawler you want to allow.

Question 7

What is llms.txt?

Accepted Answer

llms.txt is an emerging standard (similar to robots.txt but for LLMs) where you place a plain-text file at /llms.txt describing what your site is about, who you are, and what you want AI models to know. It is not yet adopted by all AI platforms, but it is gaining traction fast.

Can AI Read Your Site?

What we check and why

Fetch your robots.txt

Parse 10 AI crawlers

Show your exposure

Common questions