YOLO run
In plain wordsSlang for launching a big, expensive training run without the usual careful small-scale tests first — changing many things at once and trusting experience and instinct that it will work. YOLO means 'you only live once'.
A large training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months. launched with little or no systematic de-risking — few or no ablations, hyperparameter sweepshyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking. or scaling-lawscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments. extrapolations — often changing several variables at once and relying on researchers' intuition. YOLO runs trade the safety of the standard recipe (many small proxy runs, then a carefully extrapolated main run) for speed and lower experimentation cost; they are associated with compute-poor startups and deadline pressure, and their success depends heavily on a team's prior experience narrowing the search space. The phrase is also used for a lab's decision to commit most of its compute to an untested configuration at the frontier.
We had budget for one shot at the 20B model, so it was a YOLO run with the new tokenizer and optimizer.
Origin
Practitioner slang built on 'YOLO' ('you only live once'). It was popularized by Yi Tay, co-founder of Reka, whose blog post of 6 March 2024 on training models at a startup said the team 'had to work with many Yolo runs' guided by gut feeling, and by a July 2024 Latent Space podcast episode with him; earlier informal use in labs is likely but undocumented.
Related
- training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
- GPU poorGPU poor, adj.: Slang for researchers, startups or countries without access to the huge numbers of expensive chips needed to build the most advanced AI, as opposed to the GPU-rich big labs.
- compute budgetcompute budget, n.: The total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees.
Sources
- Tay, Y. (2024). Training great LLMs entirely from ground zero in the wilderness as a startup. Personal blog, 6 March 2024. yitay.net/blog/training-great-llms-entirely-from-ground-zero-in-the-wilderness
Mentioned in
training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months., YOLOYOLO, prop. n.: A family of fast object-detection models that look at an image once and immediately output boxes and labels for everything they recognize, quick enough for live video., YOLO modeYOLO mode, n.: Slang for letting an AI coding agent run any command without asking you first — fast and convenient, but risky, since nothing stops a harmful action.