OpenAI, Google, Microsoft, and Anthropic back shared AI scraping opt-out standard
The biggest names in AI are backing a new way to opt out of scraping
- Who
- OpenAI
- What
- announced it will honor a shared robots.txt signal for website operators to communicate AI training preferences to participating providers
- Quote
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable .
What happened
OpenAI, Google, Microsoft, and Anthropic say they will support a shared policy token in robots.txt that lets publishers express AI scraping preferences across companies. Anthropic has already adopted the signal, Google and Microsoft plan support next year, and OpenAI says implementation will come in the coming months. Microsoft will distinguish between AI training and AI grounding preferences while keeping websites eligible to appear in Bing search results.
Read next
See the full storyOpenAI, Google, Microsoft, and Anthropic Back Shared robots.txt Signal for AI Scraping PreferencesReported by 2 sources
The story so far
OpenAI, Google, Microsoft, and Anthropic have backed a shared robots.txt signal for publishers to express AI scraping preferences across participating companies; Anthropic has already adopted it, while the others plan to add support. OpenAI says it will implement the signal in the coming months, and Google and Microsoft plan support in 2027. The shared signal gives website operators a way to communicate their preferences to multiple providers. OpenAI says it will honor the signal for AI training. Microsoft says it will offer separate controls for opting out of AI training or AI grounding, and that websites will remain eligible to appear in Bing search results regardless of those preferences.
Keep up with AI, one email a day
Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue · 2026-10-30