How SenseTime Improves Inference Efficiency While Serving 6 Trillion Tokens Daily
日均6万亿Token服务背后，商汤大装置如何提升推理效能？
- Who
- SenseTime 大装置 CTO 宣善明
- When
- Quote
商汤大装置副总裁、CTO宣善明在主论坛发表题为《Agentic时代AI基础设施实践与趋势》的主题演讲，围绕Agentic时代Token增长趋势、异构推理集群实践及AI基础设施新技术探索等话题，分享了商汤大装置在大规模推理服务领域的技术探索与实践。
What happened
SenseTime describes how its AI infrastructure uses prefill-decode separation and heterogeneous scheduling to improve inference efficiency while serving an average of 6 trillion tokens daily. In remarks republished by QbitAI, CTO Xuan Shanming reports 5x overall token throughput under a specific test configuration using the same number of flagship GPUs as the baseline. The article also explains failure recovery through resource reassignment and explores cross-region prefill-decode separation and attention-FFN separation.
The story so far
SenseTime outlined how it uses prefill-decode separation and heterogeneous scheduling for AI inference, with infrastructure CTO Xuan Shanming reporting fivefold overall token throughput in a specific test configuration using the same number of flagship GPUs as the baseline. SenseTime said its infrastructure serves an average of 6 trillion tokens daily; the fivefold gain applies to the reported test configuration. In remarks republished by QbitAI, Xuan also described failure recovery through resource reassignment and discussed exploration of cross-region prefill-decode separation and attention-FFN separation.
Keep reading
Latest daily · Issue 26VAST open-sources EdenLive world model with over 24FPS generation at 720PTomorrow at 08:00, a briefing like this arrives in your inbox.
The same kind of briefing as the one above: a few stories worth reading. Free. Mail starts after you confirm, and every letter has a one-click unsubscribe. Read a past issue · 2026-10-19