Blog
Your batch size should have a schedule, too
TL;DR: Let your learning-rate schedule determine your batch-size schedule.
Demystifying Manifold Constraints in LLM Pre-training
An interactive explanation of our MACRO work, exploring how manifold constraints interact with RMSNorm, weight decay, and low-precision LLM training.