Skip to content
Last Dictionary
Plane
Interpretability & SteeringVision, Speech & PerceptionDeep Learning

feature visualization

nounAlso: activation maximization, feature viz

In plain wordsMaking pictures that show what a part of an image-recognition network is looking for, by repeatedly adjusting an image until that part responds as strongly as possible.

  1. A family of techniques that reveal what a unit, channel, direction or layer of a neural network responds to, chiefly by optimization: starting from noise, an input is adjusted by gradient ascent to maximize the chosen activation (activation maximization), yielding an image that exemplifies the unit's preferred pattern. Unconstrained optimization produces high-frequency, adversarial-looking noise, so practical methods add regularizers — robustness to jitter, rotation and scaling, frequency penalties, optimization in a decorrelated Fourier parameterization — or learned image priors. Diversity terms expose the separate facets of polysemanticpolysemanticity, n.: When a single artificial neuron responds to several unrelated things — say, bird beaks, bicycle wheels and a particular typeface — so that no single label describes what it does. units. Feature visualization is normally paired with dataset examples, the real inputs that most excite a unit, because each can mislead alone. It was central to the Circuits work on image models, including its curve detectors and multimodal neuronsmultimodal neuron, n.: A unit inside an AI vision model that responds to one idea however it appears — a photo, a drawing or the written name — much like the 'Halle Berry neuron' found in a human brain.. For language models, discrete inputs make optimization hard and dataset examples dominate.

    The feature visualization for this channel shows fur texture with dog snouts emerging from it.

Origin

Dumitru Erhan, Yoshua Bengio, Aaron Courville and Pascal Vincent introduced activation maximization for deep networks (2009); Karen Simonyan and colleagues (2013) and Matthew Zeiler and Rob Fergus (2014) extended visualization to convolutional networks, and Google's DeepDream (2015) popularized it. Chris Olah, Alexander Mordvintsev and Ludwig Schubert's 'Feature Visualization' (Distill, 2017) systematized the techniques under that name.

Related

See also
  • multimodal neuronmultimodal neuron, n.: A unit inside an AI vision model that responds to one idea however it appears — a photo, a drawing or the written name — much like the 'Halle Berry neuron' found in a human brain.
  • network dissectionnetwork dissection, n.: A method for finding out what individual units inside an image-recognition network respond to, such as dogs, windows, grass or stripes, by comparing where each unit lights up with pictures labeled by people.
  • polysemanticitypolysemanticity, n.: When a single artificial neuron responds to several unrelated things — say, bird beaks, bicycle wheels and a particular typeface — so that no single label describes what it does.
  • convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects.
  • mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
  • circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
Contrast with
  • saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.

Sources

  1. Erhan, D., Bengio, Y., Courville, A. & Vincent, P. (2009). Visualizing Higher-Layer Features of a Deep Network. Technical Report 1341, Université de Montréal.
  2. Olah, C., Mordvintsev, A. & Schubert, L. (2017). Feature Visualization. Distill.

Mentioned in

Chris OlahChris Olah, prop. n.: A Canadian researcher who pioneered ways of looking inside neural networks to see what they have learned, and a co-founder of the AI company Anthropic., filterfilter, n.: In an image-recognition network, a small grid of learned numbers that acts as a pattern detector, for vertical edges, a patch of fur or a wheel, scanned across the whole picture., saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye., DeepDreamDeepDream, prop. n.: A 2015 Google program that exaggerates whatever patterns an image-recognition network detects in a picture, producing hallucinatory images full of eyes, dog faces and swirls., DistillDistill, prop. n.: An online scientific journal (2016–2021) that published clear, interactive explanations of machine learning research, including founding work on understanding what happens inside neural networks., guided backpropagationguided backpropagation, n.: A way of producing sharp, detailed pictures of which parts of an image a vision network responded to, by passing only positive signals backward through the network., multimodal neuronmultimodal neuron, n.: A unit inside an AI vision model that responds to one idea however it appears — a photo, a drawing or the written name — much like the 'Halle Berry neuron' found in a human brain., network dissectionnetwork dissection, n.: A method for finding out what individual units inside an image-recognition network respond to, such as dogs, windows, grass or stripes, by comparing where each unit lights up with pictures labeled by people.