Skip to content
Last Dictionary
Plane
Training & Post-trainingDeep LearningHardware & Compute

critical batch size

nounAlso: CBS, gradient noise scale, maximal useful batch size

In plain wordsThe largest number of examples a model can usefully learn from in each training step. Below it, bigger batches let many chips work in parallel and finish sooner; above it, the extra chips are mostly wasted.

  1. The batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step. beyond which further increases give sharply diminishing reductions in the number of optimization steps needed to reach a target loss. Below it, doubling the batch (with a suitably scaled learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.) roughly halves the steps, so data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical. buys wall-clock speed at little extra compute; above it, steps barely fall while total compute rises. Sam McCandlish and colleagues (2018) showed that it is predicted by the gradient noise scale — the ratio of per-example gradient variance to the squared norm of the mean gradient — which grows as loss falls, so the useful batch size rises during training and is larger for better models. Jared Kaplan and colleagues (2020) modeled it as a power of the loss, and Hanlin Zhang and colleagues (2024) found it grows mainly with data size rather than model size. It caps how many accelerators can usefully train one model synchronously and motivates batch-size ramp-up schedules.

    Past the critical batch size, adding another 4,096 GPUs barely shortened the run.

Bsimple=tr⁡(Σ)∥G∥2B_{\text{simple}} = \frac{\operatorname{tr}(\Sigma)}{\lVert G \rVert^{2}}

Origin

Siyuan Ma, Raef Bassily and Mikhail Belkin (2017) described a critical mini-batch size separating linear-scaling and saturation regimes of SGD; McCandlish, Kaplan, Dario Amodei and the OpenAI Dota team (December 2018) tied it to the gradient noise scale. Priya Goyal and colleagues' one-hour ImageNet training (2017) had popularized the linear learning-rate scaling rule for large batches.

Usage note

In the formula, Σ is the covariance of per-example gradients and G the full-batch gradient; this 'simple' noise scale is a practical estimate of the critical batch size.

Related

See also
  • batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step.
  • data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical.
  • learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.
  • distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.
  • scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
  • stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.

Sources

  1. McCandlish, S., Kaplan, J., Amodei, D. & OpenAI Dota Team (2018). An Empirical Model of Large-Batch Training. arXiv:1812.06162. arxiv.org/abs/1812.06162
  2. Ma, S., Bassily, R. & Belkin, M. (2018). The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning. ICML 2018. arXiv:1712.06559. arxiv.org/abs/1712.06559
  3. Zhang, H., Morwani, D., Vyas, N. et al. (2024). How Does Critical Batch Size Scale in Pre-training? arXiv:2410.21676. arxiv.org/abs/2410.21676

Mentioned in

batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step., data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical., distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks., goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough.