#robotstxt tag - Briefly

Cloudflare updates robots.txt for the AI era - but publishers still want more bite against bots

Cloudflare's Content Signals Policy upgrades robots.txt to let publishers specify permissions for search, AI training, and crawling, improving transparency but lacking enforceability.

Privacy technologies

fromSearch Engine Roundtable

1 week ago

Cloudflare Tries To Give Sites A Way To Block Google AI Overviews

Cloudflare's Content Signals Policy adds robots.txt directives allowing site owners to block AI Overviews use of their content while keeping standard Google Search inclusion.

Online marketing

fromSitePoint Forums | Web Development & Design Community

1 month ago

Website sitemap is not detecting by ubersugget

Ubersuggest can fail to detect live sitemaps due to access, format, robots.txt blocks, firewall/password protection, or lack of Google Search Console submission.

Intellectual property law

fromArs Technica

3 weeks ago

Pay-per-output? AI firms blindsided by beefed up robots.txt instructions.

RSL enables publishers to declare licensing terms and require compensation from AI crawlers and agents via an automated robots.txt-based protocol.

Artificial intelligence

fromThe Verge

3 weeks ago

The web has a new system for making AI companies pay up

Really Simple Licensing (RSL) lets web publishers specify licensing and royalty terms in robots.txt and other content to require payment for AI training-data scraping.

Tech industry

fromAdExchanger

1 month ago

Amazon Gets Scraped, Too; LinkedIn Loves Video | AdExchanger

AI companies are crawling Amazon for shopping data, LinkedIn is expanding invite-only video revenue-sharing, and platform competition is generating disputes between Google and Fox.

Intellectual property law

fromTheregister

1 month ago

Asahi, Nikkei sue Perplexity AI for copyright infringement

Perplexity faces a copyright lawsuit from Japan's Nikkei and Asahi alleging unlawful scraping, robots.txt violations, and seeking injunctions plus ¥2.2 billion damages per firm.

E-Commerce

fromDigiday

1 month ago

Amazon quietly blocks AI bots from Meta, Google, Huawei and more

Amazon is blocking AI companies' web crawlers via robots.txt to prevent scraping of its e-commerce data and protect its marketplace and ad business.

Web development

fromSearch Engine Roundtable

1 month ago

Good Web Crawler Attributes

Good web crawlers support HTTP/2, identify via user-agent, respect robots.txt, follow caching and redirects, back off on slow servers, and expose crawl details.

Information security

fromTheregister

1 month ago