METASTONE launches TokenFleet inference scheduler, reporting 32.4% lower P95 time to first token

What happened

METASTONE released TokenFleet, a distributed inference scheduling framework combining cache-aware request routing, capacity coordination across data centers, and mixed-workload management. Company-provided GLM5.3 tuning results report 32.4% lower P95 and 71.3% lower P50 time to first token under tested conditions. A separate illustrative calculation estimates about 29.4% higher output throughput meeting service requirements, versus about 1.5% higher total output throughput.

The story so far

METASTONE released TokenFleet, a distributed inference scheduling framework, and reported lower time to first token in company-provided GLM5.3 tuning results. Under the tested conditions, P95 time to first token fell 32.4% and P50 fell 71.3%. TokenFleet combines cache-aware request routing, capacity coordination across data centers, and mixed-workload management to improve the effective output of existing computing resources. A separate illustrative calculation estimated about 29.4% higher output throughput meeting service requirements, compared with about 1.5% higher total output throughput; those estimates were distinct from the reported GLM5.3 tuning results.

Keep reading

Trending now

1. 1Mistral Releases Voxtral 3 Speech Models, Voice Agents API and LiveKit-Based SDK
2. 2DeepSeek Open-Sources DeepInfer and API Components, With Model and Hardware Limits
3. 3OpenAI Acquires Orchestra Bio to Expand AI Tools for Drug Development
4. 4Google Rolls Out Gemini Personal Intelligence to Android Auto, With Google Built-In Cars to Follow