Hugging Face Proposes Model Integration Contracts for Transformers
Transformers Model Integration Contracts
What happened
Hugging Face proposes transformers-mic, a standard for testing Transformers models with small fixtures, runnable usage examples, and human reviewed output baselines. Replaying frozen contracts across five months of commits showed that a UMT5 fixture caught an SDPA regression from July 30 until its fix on September 28; the selected T5 contract's runner step took 13.4 seconds on CPU. As of November 1, the CI job reports failures without blocking pull requests.
Why it matters
The post traces a UMT5 attention regression through replayed commits and explains how pinned, CPU runnable contracts could catch similar failures on pull requests, while outlining the remaining coverage and review questions.
Read next
See the full storyHugging Face Proposes Model Integration Contracts for TransformersThe story so far
Hugging Face proposed transformers-mic, a standard for testing Transformers models with small fixtures, runnable usage examples, and human-reviewed output baselines; as of November 1, its CI job reports failures without blocking pull requests. Replaying frozen contracts across five months of commits showed that a UMT5 fixture caught an SDPA regression from July 30 until the fix on September 28. The selected T5 contract's runner step took 13.4 seconds on CPU.
Keep up with AI, one email a day
Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue · 2026-11-03