Hugging Face releases tokenizers v1: benchmarks show major encoding and decoding speedups
tokenizers v1: encode, decode and scaling, measured
What happened
Hugging Face has released tokenizers v1, producing exactly the same token IDs as v0.23 while running up to dozens of times faster. The project splits a single crate into a workspace and introduces bitcannon, which replaces regex tokenization with SIMD bit operations, along with an allocation-free merge loop, thread-local word cache, and native multithreaded parallelism.
Hugging FaceOpen sourceDeployment engineeringProduct updates#Product update#Open source#Deployment and engineering#Hugging Face
Keep reading
Latest daily · Issue 24NVIDIA and AI4Bharat Introduce Nemotron-pretraining-Indic-v1 and an Open Data Generation PipelineTrending now
- 1Meta Releases Muse Spark Open With Mac and Cloud Support, Topping US Open-Weight Index
- 2Google Launches Gemini 4 Argon Globally, Expands Paid Access as Benchmark Results Vary
- 3OpenAI Updates GPT-6 Astra Pro in ChatGPT With Parallel Agents and Faster Responses
- 4Meta’s New Business Terms Raise Dispute Over Website Crawling and AI Training Rights
Tomorrow at 08:00, a briefing like this arrives in your inbox.
The same kind of briefing as the one above: a few stories worth reading. Free. Mail starts after you confirm, and every letter has a one-click unsubscribe. Read a past issue · 2026-10-17