28 October 2026 · 5 min read
When Cloudflare overwrote my robots.txt
SEO 92 with clean markup: the culprit was a Content-Signal directive injected at the edge.
The symptom
Lighthouse scored SEO 92 for this site while accessibility and best practices were 100. The failing audit: `robots-txt`. The origin robots.txt was three lines — `User-agent: *`, `Disallow:`, and `Sitemap`. Nothing wrong.
What the public received was different: dozens of lines of Content-Signal comments, `Content-Signal: search=yes,ai-train=no,use=reference`, then a list of blocked crawlers. Cloudflare was prepending all of it to my file.
Why it happened
A feature called Bot Preference Sync reads the AI Search, AI User, and AI Training policies from the dashboard and translates them into robots.txt. Because my Training policy was set to `disallow`, Cloudflare decided it had to write that rule into the file — overriding the origin version.
Lighthouse does not recognise the `Content-Signal` directive, flags it as an invalid line, and SEO drops eight points. Nothing could be fixed in the code: HTML, metadata, canonical, and sitemap were all correct.
The fix and the lesson
I disabled Bot Preference Sync through the Bot Management API, confirmed `is_robots_txt_managed` was false, then purged the `/robots.txt` cache. The next response was `cf-cache-status: MISS` with the original content, and SEO returned to 100.
Lesson: when one score category drops while others hold, compare what the edge serves with what the origin serves. Curling both the domain and the Render address is often enough to spot the difference.