GitHub details ModernBERT secret detection model and gains over its previous LLM pipeline

GitHub replaced its previous LLM-based secret detection pipeline with a fine-tuned ModernBERT classifier that uses surrounding context to distinguish real credentials from lookalikes, reporting better detection and lower operating costs that make extending generic secret detection to push protection practical. The pipeline first uses regular expressions to identify potential secrets, then applies the classifier to assess them in context. GitHub said offline evaluation showed 13% higher precision and 19% higher recall than the previous pipeline, while a customer A/B experiment showed 11.5% higher repository owner-weighted acceptance. GitHub also reported an 86% reduction in median post-push scan execution time and a drop of more than 98% in serving costs.

Generated by AI from reports · Updated 3 days ago