FinanceBenchmark
What it measures and how
Separate task sets v2 and v3 and runtime versions. Retain the condition of three attempts per task. The official Tasks passed means at least one success but is also called Pass@1; it cannot be mixed with average single-attempt accuracy. Finance Index is another aggregate metric.
How this evidence is used
Under observation: could add financial evidence with fixed answers and code verification. Metric definitions need review. The official Sol result JSON download returned 404 during this review. Only the source is listed; no automated collection or ranking yet.
Limitations and attribution
At least one success in three attempts is not a single-attempt success rate. Intervals must match the correct metric, and rolling model aliases require version verification. Public tasks and download buttons do not establish that raw results are available.
Data license: Official repository code is MIT; the website requires a FinanceBenchmark citation. Usage terms for complete live score exports remain unconfirmed. Code licensing is not substituted for leaderboard data licensing.
Results are published by FinanceBenchmark. Raw scores and the e/acc consensus index use different scales and cannot be added directly.