AlgoPerf
In plain wordsA public contest that measures which methods for training neural networks reach a set level of accuracy fastest on identical hardware, across several kinds of tasks, without being hand-tuned for each one.
A benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. and competition, run by the MLCommonsMLCommons, prop. n.: An industry and university consortium that runs MLPerf, the standard tests of how fast computers train and run AI models, as well as benchmarks of AI safety. Algorithms working group, that compares neural-network training algorithms (chiefly optimizersoptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment., learning-rate scheduleslearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in. and related update rules) by time-to-result: the wall-clock time needed to reach a fixed validation target on a set of workloads spanning image classification, speech recognition, machine translation, molecular property prediction, MRI reconstruction and click-through prediction, all on the same hardware. Scores aggregate across workloads with performance profiles, so an algorithm must be fast and robust everywhere rather than excel on one task, and randomized held-out workload variants penalize overfitting to the visible set. Two rulesets separate methods given a small, workload-agnostic hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking. budget from fully hyperparameter-free ('self-tuning') ones. In the first competition, which drew 18 submissions from 10 teams, a Distributed ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed. submission won the external-tuning ruleset and Schedule-Free AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models. the self-tuning ruleset.
The new optimizer looked great on one ImageNet run, but on AlgoPerf it missed the target on two workloads.
Origin
Proposed in 'Benchmarking Neural Network Training Algorithms' (2023) by George Dahl, Frank Schneider, Zachary Nado and some two dozen co-authors, who argued that claimed optimizer speedups were rarely comparable because tuning protocols differed. Priya Kasimbeg and colleagues analyzed the inaugural competition at ICLR 2025.
Usage note
AlgoPerf measures speed to a fixed quality, not final quality, and its fixed budgets and hardware mean results may not transfer to frontier-scale language-model training. Distinct from MLPerfMLPerf, prop. n.: A standard set of speed tests for AI hardware and software, where chip makers and cloud companies compare how fast their systems train and run AI models. Training, which benchmarks hardware and software systems running prescribed models.
Related
- optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment.
- ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed.
- schedule-free optimizationschedule-free optimization, n.: A way of training that needs no plan for slowing learning down over time, so training can be stopped or extended at any point and still produce a well-settled model.
- AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models.
- hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
- MLCommonsMLCommons, prop. n.: An industry and university consortium that runs MLPerf, the standard tests of how fast computers train and run AI models, as well as benchmarks of AI safety.
- MLPerfMLPerf, prop. n.: A standard set of speed tests for AI hardware and software, where chip makers and cloud companies compare how fast their systems train and run AI models.
- learning rate schedulelearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in.
- MLPerfMLPerf, prop. n.: A standard set of speed tests for AI hardware and software, where chip makers and cloud companies compare how fast their systems train and run AI models.
Sources
- Dahl, G. E., Schneider, F., Nado, Z. et al. (2023). Benchmarking Neural Network Training Algorithms. arXiv preprint.
- Kasimbeg, P. et al. (2025). Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition. ICLR 2025. arxiv.org/abs/2502.15015v1