AI Giants Offer Publishers One-Click Scraping Opt-Out
OpenAI, Google, Microsoft and Anthropic will let publishers block AI training scraping with one shared setting, replacing separate rules for every bot.
OpenAI, Google, Microsoft and Anthropic have pledged to give publishers a single way to stop their content being scraped to train AI models. The four companies announced the shared opt-out on Friday, October 30, after years of complaints from news outlets that blocking each AI bot one by one had become unworkable.
What changes for publishers
Today, a publisher that wants to block AI training scraping must add a separate blocking rule for every bot that crawls the web. If one AI company runs several scrapers, each scraper needs its own entry. The new proposal replaces all of that with a single, standardized instruction in the site's robots.txt file, the file websites already use to tell automated crawlers what they may access. Any AI company that adopts the protocol would recognize and honor that one instruction. The four companies have already submitted a draft of the proposal to the Internet Engineering Task Force, the nonprofit that maintains the Robots Exclusion Protocol, the standardized format behind robots.txt files. The setting would cover scrapers that collect data for training generative AI models. It would not apply to models that perform exclusively non-generative tasks.
When each company will switch it on
The four companies are moving at different speeds, so the single setting will not work everywhere at once. Anthropic, the maker of Claude, said it has already started rolling it out. OpenAI, the maker of ChatGPT, said it will roll it out in the coming months. Microsoft will implement it beginning February 1, 2027, and Google plans to follow sometime in 2027. Microsoft also announced a separate control called AI Grounding, for the case where a model fetches a web page to answer a user's question rather than to gather training data. That control would let site operators stop Microsoft's services from sending such requests, including through third-party AI tools built on Microsoft technology. OpenAI plans similar controls. Meta, the owner of Facebook, expressed support for the idea but has not committed to implementing it yet.
What the opt-out does not do
The new setting has clear limits, and publishers should not mistake it for a full block. It will apply only to the future training of models. It will not otherwise prevent bots from crawling websites, and content scraping will be allowed by default until a publisher changes the setting. It also does not settle the money question. The announcements set out future implementation plans, not licensing or payment arrangements, so publishers that want to be paid for their content still need separate deals. And there is a trade-off to weigh: Microsoft and Google said in separate blog posts that opting out of AI training could reduce the chances that a publisher's content will surface in AI apps and experiences.
Why publishers pushed for this
The announcement comes as publishers face a surge in bots scraping their articles to feed AI models, and as some have taken the fight to court. In December 2023, The New York Times sued OpenAI and Microsoft for allegedly using millions of its articles without authorization or payment to train ChatGPT. The two companies have invoked an exception to copyright law known as fair use. Other defenses have already appeared. Internet infrastructure provider Cloudflare has blocked AI bots by default on new customer sites since July 2025, and a standard called Really Simple Licensing already allows publishers to charge bots that visit their pages. A single shared opt-out, if it is widely honored, could become the simplest tool of all, especially for smaller publishers that cannot fight each AI company separately. For now, the promise is just that: a promise with staggered start dates. The real test will be whether the single signal is honored in practice once all four companies have switched it on, and whether other AI firms join them.
Sources
- The Hindu - https://www.thehindu.com/sci-tech/technology/ai-giants-promise-publishers-new-way-to-block-some-scraping/article71648159.ece - Inquirer.net - https://technology.inquirer.net/150113/ai-giants-promise-publishers-new-way-to-block-some-scraping/amp - SiliconANGLE - https://siliconangle.com/2026/10/30/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard