AUTO-UPDATED

Google’s Illyes: Robots.txt Covers Crawling, Not AI Training via @sejournal, @MattGSouthern

Google’s Gary Illyes clarified that the robots.txt protocol only governs URL access and cannot restrict how AI models use data once it has been successfully crawled.

Key Points

  • The RFC 9309 standard defines robots.txt as an access preference protocol, not a tool for controlling post-crawl data usage by AI systems.
  • To block specific AI crawlers, website owners must explicitly target the crawler’s unique user-agent token within their robots.txt file.
  • Google-Extended is a specific token used to manage whether Google crawls content for training Gemini models or for grounding in Vertex AI.
  • An IETF Internet-Draft co-authored by Illyes proposes a new "Content-Usage" rule, though it is not yet an official standard and lacks industry consensus.
  • Cherry Prommawin recommends using percent-encoded paths for non-English characters in robots.txt to ensure compatibility with older or non-Google web crawlers.

Why it Matters

Website owners currently lack a universal standard to prevent AI training while still allowing search engine indexing, forcing them to rely on specific bot-by-bot exclusions. This technical limitation highlights the growing gap between existing web protocols and the complex data-usage requirements of modern generative artificial intelligence.
Search Engine Journal Published by Matt G. Southern
Read original