Microsoft Research: why AI evaluations can fail unexpectedly outside traditional benchmarks
Jennifer Neville of Microsoft says models can fail unexpectedly when tested beyond traditional benchmarks. Evaluation is essential to pushing the performance boundaries of current AI systems and aligning them with user needs. She recommends paying close attention to data when using AI systems, since the data itself is often the first thing to examine when results differ from expectations.

