Cloudflare updates robots.txt for the AI era - but publishers still want more bite against bots

"Cloudflare's new Content Signals Policy effectively upgrades the decades-old honor system and adds a way for publishers to spell out how they do (and perhaps more importantly - how they don't - want AI crawlers to use their content once it's scraped.) For publishers, that distinction matters because it shifts the robots.txt file from a blunt yes-or-no tool into a way of distinguishing between search, AI training and AI outputs."

"Several publishers Digiday has spoken to over the last several months have at one point or another described the current robots.txt as "unfit for purpose." And while this upgrade still doesn't ensure AI compliance, it does at least set a new precedent for better transparency and means publishers can spell out, in black and white, how they want AI crawlers to use their content - a move many publishers have welcomed as long overdue."

"It includes the option to signal that AI systems shouldn't use their material for things like Google's AI Overviews or inference. Cloudflare's policy distinguishes between different ways AI systems use content: 'search, where material might be pulled into something like an AI Overview with the potential for attribution or referral; 'train,' where content is ingested to build the model itself, often without compensation; and 'crawl,' where bots systematically scrape pages."

Cloudflare released a Content Signals Policy that upgrades robots.txt, allowing publishers to specify how AI systems may access and use scraped content. The policy differentiates search, training, and crawling uses and enables signals to block material from AI Overviews or inference. Publishers gain clearer options to indicate permissions, attribution expectations, and potential monetization pathways. Many publishers welcomed the added transparency and the ability to distinguish AI uses after criticizing robots.txt as unfit for purpose. The policy does not include enforcement mechanisms, so platforms could still ingest content without compensation, leaving compliance dependent on platform cooperation and legal or commercial remedies.

Read at Digiday