The Internet Engineering Task Force is currently debating new technical standards that could allow website operators to restrict automated web scraping and prioritize paid access over open information.

Key Points

• The IETF is evaluating proposals from its AI Preferences and Web Bot Auth working groups regarding automated data collection.
• Proposed standards could enable websites to use cryptographic authentication to block specific bots or require licensing fees for data access.
• Critics argue these changes threaten the ability of researchers, journalists, and non-profits like the Internet Archive to crawl and preserve public web content.
• The Electronic Frontier Foundation is actively opposing these standards, citing concerns that they undermine the fundamental openness of the internet.
• While some proposals aim to prevent infrastructure strain from aggressive bots, others could grant publishers broad veto power over how their content is accessed.

Why it Matters

These proposed changes could fundamentally alter the internet by shifting it from a public, accessible resource to a gated ecosystem controlled by licensing agreements. If adopted, these standards may stifle innovation and limit the ability of independent organizations to conduct research, archive history, or provide public-interest tools.