Vals AI Benchmarks Show Open Models Closing the Cyber Capability Gap
The gap between open and closed AI models in cyber capabilities keeps shrinking
- Who
- Vals AI
- What
- reported benchmark results AI agents’ performance on simulated corporate network cyberattacks
- Quote
The Vals CyberOffense benchmark simulates realistic corporate networks to test this.
What happened
Vals AI’s CyberOffense benchmark finds that Deepseek V4.1 Flash 1017 approaches GPT-6 Sol Daybreak in the Nova test and completes 21 of 31 objectives in the harder Lighthouse scenario, compared with Daybreak’s 27. The report says Deepseek costs about $3 per Nova run versus roughly $22 for Daybreak, and that its Lighthouse progress came mostly after 100 million tokens. The results align with an AI Security Institute analysis that put the open and closed model gap at four to seven months.
Why it matters
The benchmark compares open and restricted models across two cyber ranges, with results showing how performance changes by scenario and token budget while costs differ substantially.
Read next
See the full storyVals AI Finds Deepseek V4.1 Flash Near GPT-6 on a Cyber BenchmarkThe story so far
Vals AI reported that Deepseek V4.1 Flash 1017 nearly matched GPT-6 Sol Daybreak on the Nova cyber benchmark and completed 21 of 31 objectives in the harder Lighthouse scenario, while Daybreak completed 27. CyberOffense simulates attacks on corporate networks to compare AI agents’ capabilities. The Decoder reported that a Nova run cost about $3 with Deepseek versus roughly $22 with Daybreak. Deepseek’s Lighthouse progress came mostly after 100 million tokens. The results are consistent with an AI Security Institute analysis that estimated a four-to-seven-month gap between open and closed models.
Keep up with AI, one email a day
Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue · 2026-11-07