Skip to content
K = 0.731

Nathan Lambert Questions Tradeoffs Between Private and Open-Source Benchmarks

1 reports 1 reporting sources 3 hours agoNew

Understand the event

AI overview

Nathan Lambert said private benchmarks often show open models further behind, but he remains evenly split on whether private or open-source benchmarking is the better approach. His concern with private benchmarks is that they lack a community to help find and fix the bugs he considers inevitable. His assessment weighs their findings about model performance against the benefit of community scrutiny.

Generated by AI from reports · Updated 2 hours ago

Latest development10/28 21:35 (UTC+8)
Nathan Lambert Weighs Private Benchmarks Against Open SourceNathan Lambert · X

Read next

More changes in today’s briefingGoogle DeepMind announces Genie 4 for real-time world simulation with native audio

Report timeline

Follow the reports to explore different sides of the event.

10/28
  1. Nathan Lambert · X
    Nathan Lambert Weighs Private Benchmarks Against Open Source

    Private benchmarks often show open models are further behind, but the problem with private benchmarks is you don’t have the community to squash the inevitable bugs for you. I'm 50/50 on if I think private or open-source is the best approach. https://t.co/DbvE6Tk5qL

Keep up with AI, one email a day

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue

Event heat trend

Current within comparable coverage
9
Peak within comparable coverage
1010/28 22:00 (UTC+8)
24-hour change
–

Trends compare the same continuously observed participants, which may cover fewer sources than current heat. Point or click to inspect hourly heat, or use left and right arrow keys.