Artificial Analysis 将端点准确率指数更新至 v1.1

Launching Artificial Analysis Endpoint Accuracy Index v1.1: a minor version upgrade alongside expanded coverage for GLM-5.3 Flash and Muse Spark Open. Across the 100+ serverless endpoints we benchmarked, nearly 1 in 4 score below the reference

Providers trade off accuracy to optimize for speed and cost. They quantize weights, write custom kernels, tune their inference stacks and cap output lengths, and sometimes they simply ship bugs. The Endpoint Accuracy Index measures how much of a model’s accuracy each endpoint preserves, so developers can pick providers on accuracy, not just price and speed.

We score each endpoint as a percentage of our self-hosted reference deployment of the official weights, across tool calling (BFCL-500), scientific reasoning (HLE-250) and long context recall (AA-LCR-25), equally weighted.

What’s changed:

➤ Upgraded AA-LCR-25 to AA-LCR v1.1 and increased repeats from 10 to 25: AA-LCR v1.1 corrects errors and ambiguities in answer keys and moves grading to GPT-5.6 Luna (medium). Running each question 25 times, up from 10, reduces the average 95% confidence interval from ±4.7 to ±2.6 points

➤ Expanded coverage to six open weights models: We add GLM-5.3 Flash (max) and Meta’s Muse Spark Open (xhigh), alongside DeepSeek V4.1 Flash 1017 (max), GLM-5.3 (max), Kimi K3 (max) and gpt-oss-120b (high)

Key takeaways for GLM-5.3 Flash:

➤ 16 of 22 endpoints are statistically in line with the reference, and one scores above it (104.2)

➤ Output token caps limit the model on complex workloads: Two endpoints capped at ~32K output tokens cut off about half of HLE-250 responses before an answer, scoring 22% and 29% against 55% for the reference

➤ Endpoints that reason less give lower quality answers: Two endpoints produce ~30% fewer reasoning tokens than the reference on HLE-250, with no truncation, and score 48% against 55%

Key takeaways for Muse Spark Open:

➤ Three of five endpoints match the reference: They score 99.5 to 99.9, including one serving FP4 weights

➤ The other two lose accuracy only on tool calling: Both return tool arguments that the tool defines as text, such as booking IDs, as numbers, scoring 50% and 51% on BFCL-500 against 59%. On HLE-250 and AA-LCR-25 they are within 2 points of the reference