Shanghai Jiao Tong University professor Yan Junchi presented JTok and xHC, two approaches to expanding Transformer capacity, at CNCC 2026. The report says JTok-M reached the same loss with about 35% less training compute, while xHC improved downstream scores and used about 33% less training compute than a standard Transformer.
JTok uses a learnable parameter table to modulate model updates for each token. xHC expands the residual stream, using dense reads and sparse writes. Both methods carry costs in inference latency, memory bandwidth, and throughput.
AI 根据报道生成 · 2 小时前更新
