SWE-rebench
What it measures and how
Fix the task window, ReAct environment and text/tools protocol. Use average resolution rates and standard errors from repeated runs. Complete Agent products are not treated as base models.
How this evidence is used
Candidate under observation: real repository tasks could add useful evidence. Verifiable run dates, a stable score protocol and display terms are pending. Automated collection and ranking are not enabled.
Limitations and attribution
Rolling tasks, runtime environment and task windows affect comparability. Recent website maintenance does not mean all models were retested.
Data license: The official Hugging Face CC BY 4.0 dataset contains tasks. Stable exports and display terms for final website scores remain unconfirmed.
Results are published by Nebius. Raw scores and the e/acc consensus index use different scales and cannot be added directly.