computational graph
In plain wordsA map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward.
A directed acyclic graph that represents a computation, with nodes for operations (matrix multiplication, addition, activation functions) or for the values they produce, and edges for the data dependencies between them. Executing the graph from inputs to outputs is the forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction.; traversing it in reverse while applying each node's local derivative is reverse-mode automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically., that is, backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.. Deep-learning frameworks differ in when the graph is built: 'define-and-run' (static) systems such as Theano and early TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems. construct and compile the whole graph before execution, enabling global optimization, while 'define-by-run' (dynamic or eager) systems such as Chainer and PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips. record the graph as ordinary code runs, easing debugging and data-dependent control flow. Modern stacks combine the two by capturing eager code into graphs for compilers such as XLAXLA, abbr.: A compiler from Google that turns AI programs into fast code for chips such as TPUs and GPUs, mainly by merging many small steps into fewer, bigger ones.. Graph structure also enables memory planning, kernel fusionkernel fusion, n.: Merging several small calculation steps into one, so a chip finishes them in a single pass instead of saving and reloading data in between, like cooking at the counter without returning ingredients to the fridge after each step., gradient checkpointinggradient checkpointing, n.: A memory-saving trick: instead of storing every intermediate result needed for learning, the computer keeps only a few and recalculates the rest when needed, trading extra computing time for memory. and parallel scheduling.
Origin
Graph representations of arithmetic for analyzing rounding errors and derivatives appear in numerical analysis by the 1970s, for example in Friedrich L. Bauer's 'Computational graphs and rounding error' (1974). In deep learning the term became everyday vocabulary through Theano (2010) and TensorFlow (2015); Chainer (2015) popularized the define-by-run alternative that PyTorch adopted.
Related
- automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically.
- backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
- forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction.
- backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change.
- PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips.
- AI compilerAI compiler, n.: Software that translates an AI model into efficient instructions for a specific chip, rearranging and combining its calculations so the hardware is used as fully as possible.
Sources
- Bauer, F. L. (1974). Computational graphs and rounding error. SIAM Journal on Numerical Analysis 11(1), 87–96.
Mentioned in
backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction., automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically., backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change., feedforward neural networkfeedforward neural network, n.: A neural network laid out like an assembly line: each stage hands its result only to later stages, never back, so every answer depends on the current input alone and nothing is remembered between inputs., forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction., partial derivativepartial derivative, n.: How much a quantity changes when you nudge just one of its many inputs while holding all the others fixed., PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips., TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems., AI compilerAI compiler, n.: Software that translates an AI model into efficient instructions for a specific chip, rearranging and combining its calculations so the hardware is used as fully as possible., Seppo LinnainmaaSeppo Linnainmaa, prop. n.: A Finnish computer scientist who in 1970 first published the method for computing all of a program's derivatives in one backward pass, the mathematical core of backpropagation., TheanoTheano, prop. n.: One of the first popular free software tools for deep learning, built at a Montreal university lab; development ended in 2017, but it shaped the toolkits that replaced it., XLAXLA, abbr.: A compiler from Google that turns AI programs into fast code for chips such as TPUs and GPUs, mainly by merging many small steps into fewer, bigger ones.