Ai2 Introduces FairStress to Test Demographic Fairness Beyond LLM Decisions

Ai2 introduced FairStress, a benchmark that tests demographic differences in how language models maintain decisions, explain them and represent them in internal activations; its evaluation of 15 models found disparities in how readily they abandoned correct answers. The benchmark uses an equality-centered definition of fairness and expands 300 scenarios into more than 13 million test cases, with a 50,000-item FairStress-Core subset. Ai2 reported that demographic gaps in abandoning correct answers averaged 6.9 percentage points when identity was implied and 9.3 percentage points when identity labels were explicit, with the differences favoring minority-coded candidates. In experiments across three model families, counterfactual chain-of-thought—a reasoning approach that considers alternative demographic identities—reduced robustness gaps while largely preserving accuracy, but did not eliminate every asymmetry.

Generated by AI from reports · Updated 2 hours ago

There is not enough continuous observation data to plot a trend yet.