Nathan Lambert Weighs Private Benchmarks Against Open Source

What happened

Private benchmarks often show open models are further behind, but the problem with private benchmarks is you don’t have the community to squash the inevitable bugs for you. I'm 50/50 on if I think private or open-source is the best approach.

Read next

• More changes in today’s briefingGoogle DeepMind announces Genie 4 for real-time world simulation with native audio

The story so far

Nathan Lambert said private benchmarks often show open models further behind, but he remains evenly split on whether private or open-source benchmarking is the better approach. His concern with private benchmarks is that they lack a community to help find and fix the bugs he considers inevitable. His assessment weighs their findings about model performance against the benefit of community scrutiny.