schedule-free optimization

A family of optimizers that matches or beats tuned learning-rate scheduleslearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in. such as cosine annealingcosine annealing, n.: A popular plan for shrinking a model's step size during training: steps start large, shrink slowly, then faster, then level off near zero, following the shape of half a wave. without decaying the learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls. and without knowing the training length in advance. Schedule-Free SGD and Schedule-Free AdamW maintain two sequences — a fast base iterate updated by ordinary gradient steps and a running average of iterates, which is the model actually evaluated — and compute gradients at an interpolation between them. The averaging does the work that decay does in a schedule, yielding an 'anytime' model that is good whenever training stops; apart from the usual momentum settings no new hyperparameters are needed (warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing. is still used). The method grew out of a theory connecting learning-rate schedules with averaging over iterates, a line that goes back to Polyak–Ruppert averaging (see exponential moving average of weightsexponential moving average of weights, n.: A smoothed copy of a model kept during training, which each update nudges only slightly toward the latest weights. The calmer copy often performs better than the jittery original.). In the 2024 AlgoPerf algorithmic-efficiency competition, the winning entry in the self-tuning track was built on Schedule-Free AdamW.