Nvidia Research Finds AdamW Optimizer Degrades at Batch Sizes Exceeding 100M Tokens

A new Nvidia research paper finds that the AdamW optimizer degrades beyond batch sizes of 100M tokens for next-token prediction, while alternative optimizers SOAP and Muon maintain training stability and model quality. The team addressed SOAP’s previous large-batch instabilities using per-step QR orthogonalization and improved preconditioning, testing the methods on multi-billion-parameter models trained over trillions of tokens.

Higher-order optimizers have long promised faster training convergence but have historically been limited by computational costs and numerical instability at scale. The researchers integrated a layer-wise distributed optimizer compatible with Megatron-LM to balance memory usage and minimize communication overhead without approximating the math, preserving convergence benefits across the systems layer.