Yan Junchi’s Team Explores Two Ways to Extend Transformer Architectures
Transformer 架构还能咋改？严骏驰团队提供两条思路 | CNCC 2026
- Who
- Yan Junchi
- What
- presented research on JTok and xHC architecture extensions
- Quote
在题为《大模型新架构探索：JTok & xHC》的报告中，严骏驰介绍了两项工作：JTok 为 token 引入可学习的参数表，通过查表和逐元素运算调制模型更新；xHC 则扩展残差流，以稠密读取、稀疏写回的方式提高模型容量。
What happened
At CNCC 2026, Shanghai Jiao Tong University professor Yan Junchi presented JTok and xHC, two approaches to adding model capacity within the Transformer framework. The report says JTok-M reduced training compute by about 35% at the same loss, while xHC improved downstream scores and reduced training compute by about 33% versus a standard Transformer. It also describes inference latency, memory bandwidth, and throughput costs for both methods.
Read next
See the full storyYan Junchi Presented JTok and xHC Transformer ExtensionsThe story so far
Shanghai Jiao Tong University professor Yan Junchi presented JTok and xHC, two approaches to expanding Transformer capacity, at CNCC 2026. The report says JTok-M reached the same loss with about 35% less training compute, while xHC improved downstream scores and used about 33% less training compute than a standard Transformer. JTok uses a learnable parameter table to modulate model updates for each token. xHC expands the residual stream, using dense reads and sparse writes. Both methods carry costs in inference latency, memory bandwidth, and throughput.
Keep up with AI, one email a day
Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue · 2026-11-04