GitHub’s model for secret detection: Finding more secrets with less noise
GitHub explains how its secret scanning pipeline combines regex candidate detection with a fine-tuned ModernBERT classifier to distinguish real credentials from lookalikes using surrounding context. Compared with the previous LLM pipeline, offline evaluation showed 13% higher precision and 19% higher recall; a customer A/B experiment showed 11.5% higher repository owner-weighted acceptance. Median post-push scan execution time fell 86%, and serving costs dropped more than 98%, making it practical to extend generic secret detection to push protection.
Why it matters: GitHub’s pipeline illustrates a transferable design pattern for narrow classification tasks, combining deterministic candidate generation with a compact contextual model trained on realistic hard negatives.