AI crawlers and robots.txt

Allow Claude, Gemini, ChatGPT, and Perplexity to discover OpenQuok public pages when Cloudflare managed robots.txt is enabled.

4 min read

Connect your agent today

Draft from chat, review in your calendar, and publish only what you approve.

Start for $0

Overview

OpenQuok serves /robots.txt from the web app and publishes /llms.txt plus /llms-full.txt for documentation discovery. Marketing pages, docs, and channel hubs are meant to be crawlable; auth and workspace routes stay disallowed.

If a directory or AI visibility tool reports that Claude or Gemini “has not found you”, check production /robots.txt first. On OpenQuok, the usual cause is Cloudflare managed robots.txt (“block training in robots.txt”), not the SvelteKit route alone.

Why “Training → Allow” is not enough

Cloudflare has two separate controls:

Control	What it does	Claude / Gemini
Training → Allow (do not block) under Configure AI bot policies	Stops Cloudflare from HTTP-blocking training crawlers at the edge	Necessary, but not sufficient
Set your preference to block training in robots.txt (managed robots.txt)	Prepends Disallow: / for ClaudeBot, Google-Extended, GPTBot, …	This is what PeerPush reads

Cloudflare’s Training UI even points at the robots preference (“To exclude such crawlers, set your preference here”). Having Training on Allow while managed robots.txt stays on is exactly the state that produces: crawlers are not WAF-blocked, but /robots.txt still tells them the site is off-limits — and coverage tools treat that as “Claude / Gemini hasn’t found you.”

Confirm with curl, not the Training toggle

If curl -sS https://www.openquok.com/robots.txt still shows # BEGIN Cloudflare Managed content with ClaudeBot / Google-Extended and Disallow: /, managed robots.txt is still on — regardless of Training Allow.

Fix in Cloudflare (production)

Open Security Settings → Bot traffic

In the Cloudflare dashboard, select the zone that serves www.openquok.com. Open Security → Settings, filter by Bot traffic (or follow the here link from the Training policy copy).

Turn OFF managed training blocks in robots.txt

Find Set your preference to block training in robots.txt (managed robots.txt) and turn it off. Do not stop at Training → Allow (do not block) — that is a different setting.

That is the control that prepends Disallow: / for ClaudeBot, Google-Extended, GPTBot, and related training crawlers. With it off, crawlers see only the OpenQuok origin file (Content Signals + auth disallows + explicit AI Allow groups). OpenQuok still emits ai-train=no via Content-Signal, so you keep a soft training preference without a site-wide crawl block.

Or flip it via API

Create a Cloudflare API token with Bot Management Write for the marketing zone, then:

export CLOUDFLARE_API_TOKEN='…'
export CLOUDFLARE_ZONE_NAME='openquok.com' # or set CLOUDFLARE_ZONE_ID
pnpm --filter ./web run fix:ai-robots

This sets is_robots_txt_managed to false on the zone (see Update Zone Bot Management Config).

Keep Training Allow (already correct if set)

Under Security → Settings → Configure AI bot policies, leave Training on Allow (do not block). That only affects edge HTTP blocking.

Optional: AI Crawl Control Allow

Under Security → AI Crawl Control, set Action to Allow for crawlers you want (at least ClaudeBot and Google-Extended). This also controls WAF blocking.

Verify the live file

pnpm --filter ./web run verify:ai-robots
# or
curl -sS "https://www.openquok.com/robots.txt"

Pass criteria:

No Cloudflare managed section with ClaudeBot / Google-Extended and Disallow: /
Origin suffix still lists Sitemap: and Allow: / for those bots

After robots is fixed: Al crawlers may still lag

Once pnpm —filter ./web run verify:ai-robots passes, the crawl gate is open. PeerPush’s “AI engine coverage map” is not only a robots check — the percentages (e.g. ChatGPT 73% / Copilot 24% / Perplexity 3%) are a visibility mix across engines that have already retrieved or attributed your product.

So Claude hasn’t found you / Gemini hasn’t found you after a successful Cloudflare change usually means:

That is no longer fixed by more robots.txt edits. Keep /llms.txt, pricing, compare, and docs public; watch Cloudflare AI Crawl Control for ClaudeBot / Google-Extended request logs; and re-check PeerPush after a rescan.

What the web app emits

The route web/src/routes/robots.txt/+server.ts builds:

User-agent: * — Content-Signal (search=yes, ai-input=yes, ai-train=no) plus disallow auth, editor, admin, and OAuth paths only
Per–AI-bot groups with Allow: / plus the same auth/admin disallows (including Claude-SearchBot / Claude-User)
Meta crawler allows (Facebook / Instagram app verification)
Comments pointing to /llms.txt and /rss.xml