OpenAI, Google, Microsoft, and Anthropic Back Shared robots.txt Signal for AI Scraping Preferences
Understand the event
OpenAI, Google, Microsoft, and Anthropic have backed a shared robots.txt signal for publishers to express AI scraping preferences across participating companies; Anthropic has already adopted it, while the others plan to add support. OpenAI says it will implement the signal in the coming months, and Google and Microsoft plan support in 2027. The shared signal gives website operators a way to communicate their preferences to multiple providers. OpenAI says it will honor the signal for AI training. Microsoft says it will offer separate controls for opting out of AI training or AI grounding, and that websites will remain eligible to appear in Bing search results regardless of those preferences.
Generated by AI from reports · Updated 2 days ago
Read next
More changes in today’s briefingBeekeeping tips and wrestling advice can train AI models to sabotage oversightReport timeline
Follow the reports to explore different sides of the event.
- OpenAI Might Rebrand Its Data Centers as “AI Compute Campuses”
OpenAI is considering calling its data centers “AI compute campuses” as part of proposals to shape the AI debate, chief global affairs officer Chris Lehane told Axios. The proposals also include developing AI “speed limits” and helping people see AI as empowering them in daily life. Lehane described the debate over data centers as “good and healthy.”
- TechCrunch Suggests Last-Minute Halloween Costumes for Tech Nerds
TechCrunch offers last-minute tech-themed Halloween costume ideas, including dressing as recursive self-improvement, a cute AI agent, an AI usage limit notification, or a PHP vulnerability. The article also suggests costumes inspired by Mark Zuckerberg’s T-shirts and Bryan Johnson’s “GLP-Bryan” post.
- OpenAI, Google, Microsoft, and Anthropic back shared AI scraping opt-out standard
OpenAI, Google, Microsoft, and Anthropic say they will support a shared policy token in robots.txt that lets publishers express AI scraping preferences across companies. Anthropic has already adopted the signal, Google and Microsoft plan support next year, and OpenAI says implementation will come in the coming months. Microsoft will distinguish between AI training and AI grounding preferences while keeping websites eligible to appear in Bing search results.
- OpenAI will honor a shared robots.txt signal for AI training preferences
OpenAI will honor a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers.
Keep up with AI, one email a day
Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue
Event heat trend
- Current within comparable coverage
- 3
- Peak within comparable coverage
- 1710/31 05:00 (UTC+8)
- 24-hour change
- -72%
Trends compare the same continuously observed participants, which may cover fewer sources than current heat. Point or click to inspect hourly heat, or use left and right arrow keys.