Skip to content
K = 0.731
Original
TutorialsGitHub Blog · AI & MLFrom the source· Batu Ozturkler·· 3 days agoPicks

GitHub’s model for secret detection: Finding more secrets with less noise

Who
GitHub
What
explained the design, training, and evaluation of its context-aware secret classifier
Quote
Here, we go behind the scenes of the context-aware classifier powering that effort: how we designed, trained, and evaluated it to find more real secrets with less noise.

What happened

GitHub explains how its secret scanning pipeline combines regex candidate detection with a fine-tuned ModernBERT classifier to distinguish real credentials from lookalikes using surrounding context. Compared with the previous LLM pipeline, offline evaluation showed 13% higher precision and 19% higher recall; a customer A/B experiment showed 11.5% higher repository owner-weighted acceptance. Median post-push scan execution time fell 86%, and serving costs dropped more than 98%, making it practical to extend generic secret detection to push protection.

Why it matters

GitHub’s pipeline illustrates a transferable design pattern for narrow classification tasks, combining deterministic candidate generation with a compact contextual model trained on realistic hard negatives.

The story so far

GitHub replaced its previous LLM-based secret detection pipeline with a fine-tuned ModernBERT classifier that uses surrounding context to distinguish real credentials from lookalikes, reporting better detection and lower operating costs that make extending generic secret detection to push protection practical. The pipeline first uses regular expressions to identify potential secrets, then applies the classifier to assess them in context. GitHub said offline evaluation showed 13% higher precision and 19% higher recall than the previous pipeline, while a customer A/B experiment showed 11.5% higher repository owner-weighted acceptance. GitHub also reported an 86% reduction in median post-push scan execution time and a drop of more than 98% in serving costs.

Microsoft / CopilotDeployment engineeringSafety and alignmentTutorials#Tutorials and practice#Safety and alignment#Deployment and engineering#GitHub#Microsoft

Keep reading

Latest daily · Issue 24NVIDIA and AI4Bharat Introduce Nemotron-pretraining-Indic-v1 and an Open Data Generation Pipeline

Trending now

  1. 1Meta Releases Muse Spark Open With Mac and Cloud Support, Topping US Open-Weight Index
  2. 2Google Launches Gemini 4 Argon Globally, Expands Paid Access as Benchmark Results Vary
  3. 3OpenAI Updates GPT-6 Astra Pro in ChatGPT With Parallel Agents and Faster Responses
  4. 4Meta’s New Business Terms Raise Dispute Over Website Crawling and AI Training Rights