Google’s Gary Illyes clarified that the robots.txt protocol only governs URL access and cannot restrict how AI models use data once it has been successfully crawled.
Key Points
- The RFC 9309 standard defines robots.txt as an access preference protocol, not a tool for controlling post-crawl data usage by AI systems.
- To block specific AI crawlers, website owners must explicitly target the crawler’s unique user-agent token within their robots.txt file.
- Google-Extended is a specific token used to manage whether Google crawls content for training Gemini models or for grounding in Vertex AI.
- An IETF Internet-Draft co-authored by Illyes proposes a new "Content-Usage" rule, though it is not yet an official standard and lacks industry consensus.
- Cherry Prommawin recommends using percent-encoded paths for non-English characters in robots.txt to ensure compatibility with older or non-Google web crawlers.