Pre-training large language models (LLMs) is a costly endeavor, and even slight improvements in efficiency can yield substantial savings. Nous Research has unveiled Token Superposition Training (TST), a novel approach that accelerates pre-training while maintaining the existing model structure, optimizer, tokenizer, and data. At the 10B-A1B mixture-of-experts scale, TST achieved a final training loss lower than a comparable baseline, requiring only 4,768 B200-GPU-hours compared to 12,311, representing a 2.5x reduction in pre-training time. TST operates in two phases: the first phase, Superposition, involves grouping tokens into bags, allowing the model to process more text per compute unit. The second phase, Recovery, reverts to standard token prediction. Experiments across various model sizes demonstrated that TST consistently outperformed traditional methods in terms of efficiency and performance, confirming its effectiveness as a dual-mechanism approach that enhances throughput without compromising model integrity.
Nous Research Introduces Token Superposition Training to Enhance LLM Pre-Training Efficiency
Token Superposition Training (TST) offers significant reductions in pre-training time for large language models without altering their architecture or training methods.
3 min readTechnology