kernel

Kernel methods

A function k(x, x′) that measures the similarity of two inputs as an inner product in some, possibly infinite-dimensional, feature space without computing that space explicitly; by Mercer's theorem, any symmetric positive semidefinite function qualifies. Substituting a kernel for dot products, the kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space., lets linear methods such as support vector machinessupport vector machine, n.: A classic method that separates two groups of examples with the widest possible gap — like laying the broadest possible road between them. A mathematical shortcut lets it draw curved boundaries too., kernel ridge regression and kernel PCA learn nonlinear patterns. Common choices are polynomial and Gaussian (radial basis function) kernels. Gaussian processesGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them. use kernels as covariance functions, and the neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior. describes the training of very wide neural networks.

Convolutional networks

The small array of learnable weights, for example 3 × 3 per input channel, that a convolutionconvolution, n.: An operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location. slides across an input to compute a feature mapfeature map, n.: The grid of numbers produced when one learned pattern detector is scanned across an image, showing where, and how strongly, that pattern appears.; also called a filterfilter, n.: In an image-recognition network, a small grid of learned numbers that acts as a pattern detector, for vertical edges, a patch of fur or a wheel, scanned across the whole picture.. The kernel size sets the local receptive fieldreceptive field, n.: The patch of the original input, such as a region of an image, that a particular neuron inside a network can see and respond to. of each output unit.

GPU computing

A small program that an accelerator such as a GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. runs as thousands of parallel threads, each handling a slice of the data: for example a matrix multiplication or an attention computation. Hand-optimized kernels, written in CUDACUDA, prop. n.: Nvidia's software system for programming its graphics chips to do general calculations. Most AI software is built on it, which is a big reason Nvidia chips dominate AI. or TritonTriton, prop. n.: A free language from OpenAI for writing the fast inner routines of AI programs in Python-style code; its compiler handles hardware details that Nvidia's own tools leave to experts., largely determine the speed of training and inference; FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip. and kernel fusionkernel fusion, n.: Merging several small calculation steps into one, so a chip finishes them in a single pass instead of saving and reloading data in between, like cooking at the counter without returning ingredients to the fridge after each step., which merges several operations into one kernel to save memory traffic, are examples.

Rewriting the attention kernel cut inference latency by a third without touching the model.

Other senses

In statistics, a weighting function used in kernel density estimation and smoothing. In operating systems, the core program that manages hardware and processes; in Jupyter notebooks, the process that executes code; in linear algebra, the null space of a matrix, meaning the vectors it maps to zero.