Tilde Research has introduced Aurora, a cutting-edge optimizer designed for training neural networks, which effectively addresses a significant flaw found in the popular Muon optimizer. This flaw results in a considerable number of neurons in multi-layer perceptrons (MLPs) becoming inactive during training. Aurora has been validated through a pretraining experiment involving 1.1 billion parameters and has achieved state-of-the-art results on the modified nanoGPT speedrun benchmark, with open-source code available.
To grasp the significance of Aurora, it's essential to first understand Muon, which gained traction in the machine learning community for its superior performance over AdamW in terms of convergence speed. Muon employs a unique algorithmic approach that computes the polar factor of the gradient matrix, enabling efficient weight updates. However, it was found that Muon inadvertently leads to a phenomenon where a substantial portion of neurons become inactive, particularly in tall weight matrices.
Aurora rectifies this issue by employing a new mathematical framework that ensures uniform updates across all neurons while maintaining the advantages of orthogonalization. The optimizer has demonstrated remarkable efficiency, achieving 100 times data efficiency on open-source datasets and outperforming larger models in various evaluations, all while being a near drop-in replacement for Muon with minimal overhead.