Skip to content
K = 0.731Subscribe

Collinear AI Releases Public CWE-bench v2 With 100 Security Tasks

1 reports 1 reporting sources Updated 18 hours ago

Understand the event

AI overview

Collinear AI and Hugging Face released CWE-bench v2, a public benchmark of 100 tasks covering 91 CWE categories for evaluating agents that find and fix weaknesses in real open-source repositories. The dataset is gated; users must accept its access conditions. Evaluating on the held-out set requires contacting [email protected]. The benchmark uses curated, reviewed deterministic checks designed to assess realistic instructions, valid fixes, and resistance to memorized solutions. Among 11 evaluated models, Gemini 4 Argon had the top pass@1 score at 67.8%. The dataset, grader, and harness are open.

Generated by AI from reports · Updated 43 minutes ago

Latest development11/3 08:00 (UTC+8)
CWE-bench v2 Expands Public Security Benchmark to 100 TasksHugging Face Blog

Read next

More changes in today’s briefingOpenAI’s updated Model Spec explained through its five core values

Models in this story

API prices of all models

Report timeline

Follow the reports to explore different sides of the event.

11/3
  1. Hugging Face BlogPicks
    CWE-bench v2 Expands Public Security Benchmark to 100 Tasks

    Hugging Face and Collinear AI have released CWE-bench v2, a public set of 100 tasks spanning 91 CWEs for evaluating agents that find and fix weaknesses in real open-source repositories. Its deterministic checks are curated and reviewed to test realistic instructions, valid fixes, and resistance to memorized solutions. Among 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1; the dataset, grader, and harness are open.

Keep up with AI, one email a day

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue

Event heat trend

There is not enough continuous observation data to plot a trend yet.