mini-batch
In plain wordsA small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once.
A subset of the training settraining set, n.: The examples a machine-learning model learns from, like the worked problems a student practices on, kept separate from the examples later used to check how well it learned. — typically from a few to a few thousand examples, or for language models up to millions of tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. — over which the loss and its gradient are averaged to produce one parameter update in stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset. and its relatives. Mini-batches trade off two extremes: a single example gives very noisy gradients and uses hardware poorly, while the full dataset gives exact gradients at prohibitive cost per step. Processing a mini-batch as one tensor lets a GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. parallelize the work, which is why a batch dimension appears in nearly every deep-learning array. Examples are usually reshuffled each epochepoch, n.: One complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once. so that batches approximate i.i.d.i.i.d., abbr.: Short for 'independent and identically distributed': every data point is produced the same way, and none influences any other — like repeated rolls of one fair die. samples. Some layers, notably batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably., compute statistics across the mini-batch, coupling the examples within it; a micro-batch is the slice one device processes at once when a batch is split.
Each mini-batch holds 256 images, so one epoch over the dataset is about 5,000 steps.
Origin
Neural-network researchers of the 1980s contrasted 'online' (per-example) with 'batch' (whole-dataset) updating; the intermediate mini-batch scheme was in use by at least the 1990s and became the default once GPU training took hold around 2010.
Usage note
In casual use 'batch' almost always means mini-batch, which clashes with the older 'batch learning' (whole dataset). 'Batch' in batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less. or continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish. refers to grouping requests at serving time, not training.
Related
- batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step.
- stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.
- epochepoch, n.: One complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once.
- training steptraining step, n.: One round of learning: the model makes predictions on a small batch of examples, measures its error and adjusts its settings once. Training runs consist of thousands to millions of steps.
- gradient accumulationgradient accumulation, n.: A way to train with large batches on limited hardware: the computer processes several small batches one after another, adds up what each says, and only then makes a single update.
- batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably.
- batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less.
Mentioned in
gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down., batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably., batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step., data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical., epochepoch, n.: One complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once., gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., gradient accumulationgradient accumulation, n.: A way to train with large batches on limited hardware: the computer processes several small batches one after another, adds up what each says, and only then makes a single update., stochasticstochastic, adj.: Involving chance, so the same starting point can lead to different outcomes. AI chatbots are stochastic: ask the same question twice and you may get two different answers., stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset., training looptraining loop, n.: The repeating cycle at the heart of training software: grab some examples, let the model guess, measure the error, adjust the model, and do it again — often millions of times., training steptraining step, n.: One round of learning: the model makes predictions on a small batch of examples, measures its error and adjusts its settings once. Training runs consist of thousands to millions of steps., variancevariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves., pipeline parallelismpipeline parallelism, n.: Splitting a model's layers into consecutive stages on different chips, like stations on an assembly line, and streaming small batches of data through so that every station stays busy.