Stop AI Looting: The Industry Blueprint For 'Do Not Scrape'

"Publishers and creators are at a critical inflection point where the future of the digital media industry hangs in the balance. Publishers that once worried about algorithms burying their headlines are now grappling with a more existential threat as AI companies siphon content outright without permission, payment or attribution. What used to be a battle over traffic has evolved into a fight for survival-and it's one we can't afford to fight alone."

"Over the past year, more creators and publishers have taken legal or licensing action against AI companies to rein in unauthorized data scraping. But case-by-case deals can't scale. Standards do. The future of the open web demands something stronger, proactive and enforceable. That's why now is the time to set industry-wide standards for "Do Not Scrape" policies that are respected by default, not disregarded by design."

"For decades, open-web infrastructure rewarded publishers for participating in discovery. Crawlers indexed articles, search engines delivered traffic and monetization followed. But in the era of generative AI, that social contract is broken. AI crawlers now consume content not to drive discovery but to train models and serve up synthetic responses, cutting off the creators who made the content in the first place in an almost 180-degree change of procedure from years past."

Publishers and creators face an existential threat as AI companies siphon content without permission, payment or attribution. Legal and licensing actions have increased, but individual deals cannot scale. Industry-wide "Do Not Scrape" standards that are proactive and enforceable are necessary. Effective self-regulation requires large participation from publishers, artists, platforms and policymakers. Shared technical standards, unified legal frameworks and incentives for responsible licensing are needed. The prior open-web contract that rewarded publishers with discovery and monetization has broken. AI crawlers now consume content to train models and generate synthetic responses, undermining traffic and creator compensation. Enforcement of rules like robots.txt remains flimsy.

Read at Forbes