Skip to content
K = 0.731Subscribe

OpenAI, Google, Microsoft, and Anthropic Back Shared robots.txt Signal for AI Scraping Preferences

2 reports 2 reporting sources 40 minutes agoNew

Understand the event

AI overview

OpenAI, Google, Microsoft, and Anthropic have backed a shared robots.txt signal for publishers to express AI scraping preferences across participating companies; Anthropic has already adopted it, while the others plan to add support. OpenAI says it will implement the signal in the coming months, and Google and Microsoft plan support in 2027. The shared signal gives website operators a way to communicate their preferences to multiple providers. OpenAI says it will honor the signal for AI training. Microsoft says it will offer separate controls for opting out of AI training or AI grounding, and that websites will remain eligible to appear in Bing search results regardless of those preferences.

Generated by AI from reports · Updated 10 minutes ago

Latest development10/31 04:26 (UTC+8)
OpenAI, Google, Microsoft, and Anthropic back shared AI scraping opt-out standardThe Verge · AI

Read next

More changes in today’s briefingMistral releases improved Mistral Large 4 open weights for coding and cybersecurity

Report timeline

Follow the reports to explore different sides of the event.

10/31
  1. The Verge · AI
    OpenAI, Google, Microsoft, and Anthropic back shared AI scraping opt-out standard

    OpenAI, Google, Microsoft, and Anthropic say they will support a shared policy token in robots.txt that lets publishers express AI scraping preferences across companies. Anthropic has already adopted the signal, Google and Microsoft plan support next year, and OpenAI says implementation will come in the coming months. Microsoft will distinguish between AI training and AI grounding preferences while keeping websites eligible to appear in Bing search results.

10/30
  1. OpenAI News
    OpenAI will honor a shared robots.txt signal for AI training preferences

    OpenAI will honor a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers.

Keep up with AI, one email a day

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue

Event heat trend

There is not enough continuous observation data to plot a trend yet.