CWE-bench v2 Expands Public Security Benchmark to 100 Tasks
CWE-bench v2: More coverage, quality, and transparency
- Who
- Collinear AI
- What
- released a public set of 100 CWE-bench v2 security tasks
- Quote
CWE-bench v2 opens up a public set of 100 tasks that anyone can run and evaluate their models on.
What happened
Hugging Face and Collinear AI have released CWE-bench v2, a public set of 100 tasks spanning 91 CWEs for evaluating agents that find and fix weaknesses in real open-source repositories. Its deterministic checks are curated and reviewed to test realistic instructions, valid fixes, and resistance to memorized solutions. Among 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1; the dataset, grader, and harness are open.
Why it matters
The benchmark pairs a reproducible evaluation setup with task curation criteria and results, giving readers a concrete view of both its coverage and the remaining performance gap.
Read next
See the full storyCollinear AI Releases Public CWE-bench v2 With 100 Security TasksThe story so far
Collinear AI and Hugging Face released CWE-bench v2, a public benchmark of 100 tasks covering 91 CWE categories for evaluating agents that find and fix weaknesses in real open-source repositories. The dataset is gated; users must accept its access conditions. Evaluating on the held-out set requires contacting [email protected]. The benchmark uses curated, reviewed deterministic checks designed to assess realistic instructions, valid fixes, and resistance to memorized solutions. Among 11 evaluated models, Gemini 4 Argon had the top pass@1 score at 67.8%. The dataset, grader, and harness are open.
Keep up with AI, one email a day
Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue · 2026-11-03