Skip to content
K = 0.731Subscribe

Artificial Analysis Releases AA-Omniscience v1.1 With Refreshed Questions and Public Test Set

1 reports 1 reporting sources 2 hours agoNew

Understand the event

AI overview

Artificial Analysis released AA-Omniscience v1.1, updating its knowledge and reasoning benchmark; it said model rankings largely match v1, while average Net Scores are 1.9 points lower on the refreshed dataset. The update removes saturated questions and fixes ambiguities in the 3,000-question dataset. It also introduces a newly calibrated 300-question public set on Hugging Face, which Artificial Analysis said matches the private set’s difficulty. The company reported Pearson and Spearman correlations of 0.99 between public and private Net Scores.

Generated by AI from reports · Updated 1 hours ago

Latest development10/31 08:01 (UTC+8)
Artificial Analysis Releases AA-Omniscience v1.1 Benchmark UpdateArtificial Analysis · X

Read next

More changes in today’s briefingConfigure the ChatGPT desktop app with Amazon Bedrock in AWS GovCloud (US)

Report timeline

Follow the reports to explore different sides of the event.

10/31
  1. Artificial Analysis · X
    Artificial Analysis Releases AA-Omniscience v1.1 Benchmark Update

    Artificial Analysis released AA-Omniscience v1.1, refreshing its knowledge and reasoning benchmark with a 3,000-question dataset and a newly calibrated 300-question public set. Public and private Net Scores have 0.99 Pearson and Spearman correlations. Model rankings largely match v1, while average Net Scores are 1.9 points lower on the refreshed set.

Keep up with AI, one email a day

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue

Event heat trend

There is not enough continuous observation data to plot a trend yet.