Hugging Face releases tokenizers v1: benchmarks show major encoding and decoding speedups
Hugging Face has released tokenizers v1, producing exactly the same token IDs as v0.23 while running up to dozens of times faster. The project splits a single crate into a workspace and introduces bitcannon, which replaces regex tokenization with SIMD bit operations, along with an allocation-free merge loop, thread-local word cache, and native multithreaded parallelism.

