Google Research Introduces Sleep Framework for Continual Learning in LLMs

Google Research introduced a “sleep” framework that alternates active interaction with offline memory consolidation in large language models, reporting near-zero performance loss in sequential translation evaluations and near-perfect scaling up to 10M tokens on BABILong. The framework aims to turn information held in short-term context into lasting knowledge stored in model parameters. Within a fixed architecture, it unlocks pre-allocated mixture-of-experts (MoE) memory blocks, distills short-term information into parameters, and trains on selected examples generated by the model itself. Google Research presents this periodic learning cycle as a way for LLMs to retain new knowledge and improve reasoning.

Generated by AI from reports · Updated 1 hours ago

There is not enough continuous observation data to plot a trend yet.