Skip to content

A Three-Month Study Tracked 936 AI Agents Producing 936 Papers

Baoyu · XRelayed· @dotey·· 21 hours agoPicks

1.3K views · 1 likes · 0 reposts (at collection)

Original

What happened

A report published by alphaXiv with MIT, Stanford, and other institutions tracked 936 AI agents across 35 labs from August to November as they produced 936 papers. In blind reviews of 202 AI and human papers, two Gemini models rated the AI papers 2 points higher for experimental rigor and 0.4 points lower for novelty on a 10-point scale. The authors caution that the overall AI score advantage came entirely from the earliest lab, and the public dataset excludes raw data and model weights.

Why it matters

In blind reviews, AI papers scored higher on experimental rigor but lower on novelty than human papers; the overall score advantage came entirely from the earliest lab, and the dataset excludes raw data and model weights.

Body · English edition pending (original content)

936 个 AI 自己做了三个月研究,发了 936 篇论文;AI 评委打分,它们比人类论文严谨,但新意不如人类

论文网站 alphaXiv 联合 MIT、斯坦福等机构 11 月 19 日发布报告:8 月到 11 月,35 个实验室接入 936 个 AI 智能体,人只给算力、定大方向,选题、实验、代码和论文全由 AI 完成。同一天,alphaXiv 开放了专门收 AI 论文的“AI Track”,有账号就能投,AI 可以自己上传。

三个月里,它们做了这些事:

一个智能体在频道里宣布方法提升 13.2 个百分点,不到两小时自己发帖撤回,因为发现对照组没调好。

有实验室不到三分钟传了 12 篇占位论文,另一个智能体看不下去,办起了 AI 期刊:四个 AI 盲审,编辑还用作者的代码把数字重算一遍。

只读文字的 AI 审稿人分不清改好的和没改好的论文;把审稿意见写成代码一跑,156 条重大意见里有 77 条推翻了论文结论。编辑读代码时还抓到三篇代码和论文对不上的投稿,其中一篇直接调用了标准答案。

它们也复现了 GPT-6 Astra 直接改 Qwen 权重那项研究:分数确实涨了,但原作者说的原理不对。

和同期人类论文比,两个 Gemini 模型盲评 202 对论文,AI 论文实验严谨高 2 分(10 分制),新颖性低 0.4 分。

这些结果要打个折扣:评委是 AI,只读文字,AI 的总分优势全部来自最早那个实验室,其余实验室和人类论文差不多。“没人写代码”靠参与者自己申报。公开数据集不含原始数据和模型权重,数据集说明也提醒,每条结论都当作未经核实。

论文:alphaxiv.org/abs/2611.society…

Quoted postSamuel Schmidgall@SRSchmidgall

Leave one AI agent alone to do research and it often cuts corners. Leave nearly 1,000 AI agents together for three months and they invent peer review.

Across 35 labs (Stanford, MIT, ETH Zurich, UCSF, and more), agents published 936 papers with full code and data on a shared server built for AI. No human wrote code or text.

Unprompted, they debated study designs in chat, retracted their own mistakes, found real errors in human papers, and started a journal that caught other agents cheating in code.

In blind reviews against human arXiv papers, humans still won on original ideas, agents won on rigor.

alphaxiv.org/abs/2611.society…

View the quoted post on X

Read next

More changes in today’s briefingHugging Face and Thinking Machines Lab Release Inkling 2 Mini
This story is in 11/21 AI DailyRead the whole issue