Interpretability & Steering

Looking inside models: features, circuits, steering and explanations. 123 entries live here, and 55 from other domains touch it.

Essential

explainabilityHow well the reasons behind an AI system's outputs can be put into terms that people can understand, check and, if needed, challenge.
explainable AIThe field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application.
interpretabilityThe effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
mechanistic interpretabilityA branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.

All other entries

119

ablationRemoving or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network.
abliterationA technique for removing a chat model's ability to refuse requests by finding the internal direction responsible for refusals and permanently erasing it from the model's weights, with no retraining.
accumulated local effectsA chart of how a model's predictions change with one input, built from small local comparisons so it stays reliable when inputs are related to each other, unlike simpler average-based charts.
activationThe numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment.
activation additionA simple way to steer a language model: run two contrasting prompts such as 'Love' and 'Hate', take the difference between the model's internal states, and add it while the model writes.
activation patchingAn experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
activation spaceThe imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions.
activation steeringChanging an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model.
activation vectorThe list of numbers a neural network produces at one layer for one input — a snapshot of its internal state. In steering, also a direction added to that state to shift behavior.
algorithmic recourseGiving a person who received an unfavorable automated decision a realistic set of actions they could take to get a better result next time, such as paying down a particular debt.
AnchorsA method that explains a prediction with a simple if-then rule that locks in the result — for example, if an email contains both 'invoice' and 'urgent', the filter almost always flags it.
assistant axisThe main direction in a chatbot's internal activity that measures how much it is acting as its usual helpful-assistant self, as opposed to some other character it has slipped into.
attention patternA grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map.
attention-based explanationExplaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output.
attribution graphA diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next.
attribution patchingA fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part.
automated circuit discoverySoftware methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand.
automated interpretabilityUsing AI models to help explain other AI models — for instance, having a language model read examples that excite an internal unit and write a description of what it detects.
causal abstractionA mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped.
causal mediation analysisA statistical method for splitting a cause's effect into the part that flows through some go-between and the part that does not; in AI it is used to find which internal parts carry a behavior.
causal scrubbingA strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before.
causal tracingA test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled.
centered kernel alignmentA score from 0 to 1 measuring how similar two neural networks' internal representations are, by comparing how each arranges the same set of inputs relative to one another.
circuitA small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
circuit tracingMethods that map, step by step, which internal concepts an AI model used to produce a particular answer and how they fed into one another, like tracing wires on a circuit board.
concept bottleneck modelA model made to first predict human-understandable concepts — such as 'bone spur present' on an X-ray — and then decide only from those concepts, so people can inspect and correct them.
concept erasureRemoving a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it.
concept injectionAn experiment that plants a specific concept directly into an AI model's internal activity, then asks the model whether it notices anything unusual, to test whether its reports about itself are genuine.
contrast-consistent searchA method for finding where a language model internally tracks whether statements are true, using only the logical rule that a statement and its opposite cannot both be true.
contrastive activation additionA way to make a steering signal more reliable by averaging the difference in a model's internal state across hundreds of example answers that do and don't show a behavior, such as flattery.
control vectorA steering signal saved as a small file that can be loaded alongside an open AI model to turn a trait — say, cheerfulness or laziness — up or down while it generates text.
counterfactual explanationAn explanation that tells someone the smallest change that would have produced a different outcome — for example, that the application would have been accepted if it had listed two more years of work experience.
crosscoderA concept-finding tool that looks at several layers, or several versions of a model, at once, so that one shared concept can be tracked wherever it appears.
DeepLIFTA method that explains a neural network's output by comparing each internal signal with its value on a neutral reference input, then passing the differences back to the inputs as credit.
developmental interpretabilityThe study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model.
dictionary learningA method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
direct logit attributionA quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions.
directional ablationErasing one specific direction from an AI model's internal signals while it runs, so whatever that direction encodes — such as the tendency to refuse — can no longer influence what the model does.
distributed alignment searchA method that searches, by training, for a hidden direction inside a neural network that plays the role of one variable in a human-readable explanation of the network's behavior.
enumerative safetyA proposed safety strategy: list every concept an AI model represents internally, then check the list for dangerous ones, such as deception, before trusting the model.
explainable boosting machineA prediction method for spreadsheet-style data that draws each input's influence as its own readable graph and adds those effects up to make every prediction, usually losing little accuracy compared with opaque methods.
faithfulnessHow accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given.
feature absorptionA flaw in concept-finding tools where a general concept goes missing in some cases because a more specific concept swallows it — like a 'starts with S' detector staying silent on the word 'snake'.
feature attributionGiving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
feature (interpretability)A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.
feature splittingWhat happens when a concept-finding tool is given more room: a broad concept it found before breaks into several narrower ones, much as 'animal' might split into 'dog', 'cat' and 'bird'.
feature steeringSteering an AI model by turning up or down specific concepts that interpretability tools have found inside it — for example, making a model obsessed with a bridge by maxing out its 'bridge' concept.
feature visualizationMaking pictures that show what a part of an image-recognition network is looking for, by repeatedly adjusting an image until that part responds as strongly as possible.
function vectorA compact internal signal, found inside a language model, that stands for a whole task — such as 'translate into French' or 'give the opposite word' — and can trigger that task when added to the model.
Gemma ScopeA free collection of ready-made concept-finding tools released by Google DeepMind for its open Gemma language models, so outside researchers can study their insides without training the tools themselves.
global explanationAn explanation of how an AI model behaves overall — which factors matter most across all its decisions, and in what way — rather than why it made one particular decision.
Golden Gate ClaudeA one-day public demo in 2024 in which Anthropic turned up a single internal concept inside its Claude chatbot — the Golden Gate Bridge — so it steered almost every conversation back to the bridge.
Grad-CAMA technique that shows, as a coarse heat map, which regions of an image a vision network relied on when choosing a particular label.
guided backpropagationA way of producing sharp, detailed pictures of which parts of an image a vision network responded to, by passing only positive signals backward through the network.
in-context vectorA steering signal distilled from a few worked examples shown to a language model, so the model can follow the examples' pattern without having them repeated in every prompt.
indirect object identificationA fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model.
individual conditional expectationA chart that traces, for each example separately, what a model would predict as one input is swept across its range, exposing differences between examples that an overall average would hide.
induction headA pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt.
inference-time interventionA technique that makes a language model answer more truthfully by slightly shifting the outputs of a few internal parts that seem to track truth, each time the model produces a word.
influence functionA mathematical shortcut for estimating how much a single training example shaped a model's behavior, for instance how a prediction would change if that example had been left out, without retraining the model.
integrated gradientsA way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way.
interpretability illusionA convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow.
interpretable modelA model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed.
iterative nullspace projectionAn early method for removing a trait such as gender from a model's internal representations by repeatedly training a simple detector for it and then flattening away whatever the detector relied on.
knowledge neuronAn internal unit of a language model claimed to be responsible for storing a particular fact, such that turning it up or down makes the model more or less likely to state that fact.
layer-wise relevance propagationA method that traces a neural network's decision backward, layer by layer, handing each part of the input its share of the credit until every pixel or word has a relevance score.
LEACEA one-step mathematical method for scrubbing a concept, such as gender, from an AI model's internal data so that no simple detector can find it, while changing everything else as little as possible.
LIMEA method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example.
linear probeThe simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly.
linear representation hypothesisThe idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.
local explanationAn explanation of why an AI model gave one particular answer for one particular case — this patient, this loan — rather than how it behaves in general.
local learning coefficientA number measuring how complex a trained neural network effectively is around its current settings — usually far less than its raw parameter count suggests — used to spot changes as the network learns.
logit lensA quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time.
MEMITA method for rewriting thousands of facts stored in a language model at once, by spreading small, calculated weight changes across several of its layers.
model biologyThe study of what AI models actually do inside — the strategies and hidden steps they use — approached the way biologists study living things, by observation and experiment rather than from a blueprint.
model diffingComparing the insides of two closely related AI models — such as a model before and after extra training — to find exactly which internal concepts and mechanisms changed.
model editingChanging one specific fact or behavior inside a trained AI model — for example, who leads a country — by directly adjusting a small part of it, without retraining the whole thing.
model-agnosticDescribes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside.
monosemanticityThe property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several.
multimodal neuronA unit inside an AI vision model that responds to one idea however it appears — a photo, a drawing or the written name — much like the 'Halle Berry neuron' found in a human brain.
network dissectionA method for finding out what individual units inside an image-recognition network respond to, such as dogs, windows, grass or stripes, by comparing where each unit lights up with pictures labeled by people.
NeuronpediaA free website where anyone can browse, search and test the internal concepts that researchers have found inside AI models, with examples of the text that switches each one on.
occlusion sensitivityFinding out which parts of an image matter to a model by covering them up one patch at a time and watching how much the model's answer changes.
Othello-GPTA small language model trained only on lists of moves from the board game Othello, which researchers found had built an internal picture of the board — evidence that such models can form world models.
parameter decompositionAn approach that tries to understand a neural network by splitting its weights — rather than its internal activity — into many simple pieces, each responsible for one mechanism.
partial dependence plotA chart showing how a model's average prediction changes as one input is varied while everything else stays as it is in the data — for example, predicted house price against floor area.
PatchscopesA family of tricks for getting a language model to describe its own hidden states in words, by transplanting an internal state into a new prompt designed to make the model talk about it.
path patchingA refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior.
persona vectorA direction inside a language model's internal activity linked to a character trait, such as being flattering, malicious or prone to making things up, used to watch for or control that trait.
platonic representation hypothesisThe idea that bigger, more capable AI models, even ones trained on different things such as images and text, end up organizing information in increasingly similar ways, as if all were mapping the same underlying world.
plug and play language modelAn early way to steer what a language model writes — its topic or mood — by letting a small add-on classifier nudge the model's internal state at each word, without retraining the model.
polysemanticityWhen a single artificial neuron responds to several unrelated things — say, bird beaks, bicycle wheels and a particular typeface — so that no single label describes what it does.
post hoc explanationAn explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable.
privileged basisA property of a layer in a neural network where the individual numbered slots, not just blends of them, carry special meaning, because something in the design treats each slot separately.
probingTesting what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.
prototypical part networkAn image classifier that explains each decision by pointing to parts of the picture that resemble typical examples it learned — this part looks like that part of a known bird.
QK and OV circuitsA way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees.
refusal directionA single direction inside a chat model's internal activity that largely decides whether it refuses a request. Erasing it makes the model comply with almost anything; adding it makes the model refuse harmless requests.
replacement modelA more understandable copy of an AI model, built from simple labeled parts, that behaves almost like the original, so researchers can study the copy's workings instead.
representation engineeringAn approach to understanding and controlling AI models by finding where high-level ideas such as honesty or danger show up in their internal activity, then reading or adjusting those signals directly.
representation fine-tuningA way of adapting a language model to a new task by training small, targeted edits to its internal signals instead of changing its weights, often with far fewer adjustable numbers than other methods.
residual streamThe running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
ROMEA method for rewriting a single fact stored in a language model — say, moving the Eiffel Tower to Rome — by making one small, precisely calculated change to one layer's weights.
saliency mapA heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.
SHAPA popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players.
singular learning theoryA mathematical theory of learning for models like neural networks, whose many redundant settings break standard statistics; it explains how such models generalize and how their learning unfolds.
SmoothGradA trick for making explanation heat maps less speckled: compute the map for many noisy copies of the input and average them, so random flicker cancels out.
sparse autoencoderA small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
sparse feature circuitA map of how a language model performs a task, drawn in terms of understandable concepts found by concept-finding tools, rather than hard-to-read neurons or whole layers.
steeringGuiding what an AI model does — its topic, tone or behavior — while it runs, by changing its instructions, its internal signals or how its next word is chosen, rather than by retraining it.
steering vectorA direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it.
superpositionThe way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.
surrogate modelA simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply.
testing with concept activation vectorsA method for asking whether a human idea, such as 'wheels', matters to an image model's decision, such as 'car', by finding the direction for that idea inside the model and measuring its effect.
toy modelA deliberately tiny, simplified AI model trained on an artificial task, small enough for researchers to understand completely and use to test ideas about how big models work.
transcoderA stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes.
TransformerLensA free software library that lets researchers open up language models and record, inspect or alter what happens inside them at every step while the model runs.
tuned lensAn improved version of the logit lens that first learns a small translator for each layer, so a language model's halfway-through states can be read as predictions more accurately.
universality hypothesisThe idea that different AI models trained on similar tasks tend to develop the same internal concepts and mechanisms, so what is learned about one model may carry over to others.

Also relevant

AI lie detectionMethods for telling when an AI is saying something it 'believes' is false — either by questioning it cleverly or by reading signals inside the model.
AI transparencyOpenness about AI: telling people when they are dealing with one, how a system was built and tested, and why it reached a particular decision.
alignment auditA systematic investigation of an AI model to find out whether it has hidden goals or bad tendencies — such as deceiving its users — that ordinary testing might miss.
attention headOne of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.
attention sinkA quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need.
baselineA simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value.
black boxA system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer.
causal inferenceWorking out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time.
chain-of-thought faithfulnessWhether the reasoning an AI writes out really reflects how it reached its answer, rather than being an after-the-fact story that leaves out what actually drove the decision.
chain-of-thought monitoringHaving a program — often another AI — read the step-by-step reasoning an AI writes before it acts, to catch signs that it intends to cheat, deceive or cause harm.
circuit breakersA defense that changes an AI model's inner workings so that, whenever it starts heading toward a harmful answer, its internal activity is knocked off course and it cannot produce coherent harmful content.
counterfactualA statement about what would have happened if something had been different — 'had I left earlier, I would have caught the train'. AI uses counterfactuals to explain decisions and to measure causes.
data attributionWorking out which training examples were responsible for a particular behavior or answer of an AI model — useful for explaining, debugging and crediting the data behind it.
DeepDreamA 2015 Google program that exaggerates whatever patterns an image-recognition network detects in a picture, producing hallucinatory images full of eyes, dog faces and swirls.
disentangled representationAn internal representation in which each part tracks one separate, meaningful factor of the data, such as one number for a face's angle, another for its lighting and another for hair color.
DistillAn online scientific journal (2016–2021) that published clear, interactive explanations of machine learning research, including founding work on understanding what happens inside neural networks.
eliciting latent knowledgeThe unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe.
featureOne piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself.
feature importanceA score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect.
feed-forward networkThe part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge.
generalized additive modelA prediction model that adds up separate, possibly curved effects of each input, so each input's influence can be drawn as its own graph and read off directly.
GoodfireA San Francisco company, founded in 2024, that builds tools to look inside AI models, understand what they have learned and adjust how they behave.
grokkingWhen a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right.
in-context learningA language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round.
information bottleneckThe idea that a good internal summary of data keeps exactly the details needed for the task and discards everything else, as a good note-taker does.
inner alignmentMaking sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training.
introspectionLooking inward to observe your own thoughts and feelings. For AI, the question is whether a model can accurately report what is actually happening inside it, rather than guessing or making it up.
Johnson–Lindenstrauss lemmaA surprising fact about high-dimensional data: a random projection can squash a large set of points into far fewer dimensions while distorting every distance between them only slightly.
Kolmogorov–Arnold networkA 2024 alternative to the standard neural network: instead of fixed rules inside neurons, every connection learns its own small curve, which can make the result easier to read as a formula.
latent adversarial trainingA safety training method that attacks a model's internal signals, rather than its inputs, during training, so the model learns to behave safely even when its inner workings are pushed toward bad behavior.
latent reasoningGetting an AI to do its step-by-step thinking inside its internal numbers rather than writing it out in words — potentially faster and more flexible, but much harder for people to inspect.
loss landscapeA way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys.
Marr's levels of analysisA way of explaining any information-processing system on three levels: what problem it solves and why, what step-by-step method it uses, and how that method is physically built.
mechanistic anomaly detectionFlagging cases where an AI reaches its output by an unusual internal route — for instance because it is being tricked, or is tricking us — even when the output itself looks normal.
mesa-optimizationWhen training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for.
model organism of misalignmentAn AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes.
monitorabilityHow easy it is to watch an AI's reasoning and spot when it is planning something bad — a property that could be lost if future AIs stop thinking in readable language.
Neel NandaA British AI researcher who leads mechanistic interpretability at Google DeepMind and built TransformerLens, a widely used tool for looking inside language models.
natural abstraction hypothesisThe idea that the world naturally sorts into a limited set of useful concepts — like 'tree' or 'temperature' — that any capable mind, human or AI, would end up learning.
neural collapseA striking tidiness that appears when classifiers are trained for a long time: the network's internal summaries of each class shrink to single points arranged as evenly and as far apart as possible.
neuraleseA nickname for the private 'language' of numbers that AI systems could use to think or to talk to each other, instead of words that people can read.
Chris OlahA Canadian researcher who pioneered ways of looking inside neural networks to see what they have learned, and a co-founder of the AI company Anthropic.
orthogonalityThe mathematical version of being at right angles. Two directions are orthogonal when moving along one tells you nothing about the other, like north and east on a map.
parametric knowledgeWhat an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.
Rashomon effectThe common situation in which many quite different models fit the same data almost equally well — like witnesses telling different but equally believable stories about the same event.
right to explanationThe idea that a person affected by a decision made or shaped by an algorithm — a refused loan, a rejected job application — is entitled to a meaningful account of why.
seahorse emojiA viral meme in which chatbots, asked whether a seahorse emoji exists, insisted it did and then spiraled through wrong emojis trying to produce one — though there was no such emoji.
Shapley valueA rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order.
Buck ShlegerisAn AI-safety researcher who runs Redwood Research and helped found 'AI control', the study of keeping powerful AI safe even if it is secretly trying to cause harm.
steerabilityHow easily an AI model's behavior — its tone, style, persona or rules — can be adjusted by the people using or deploying it, for example through instructions, without retraining.
structural causal modelA precise description of what causes what in a system — a diagram of arrows plus equations — that predicts what would happen if you intervened, not just what tends to occur together.
task arithmeticEditing an AI model by adding or subtracting the changes that fine-tuning made to it, as if skills were arrows that can be added together or reversed.
Max TegmarkA Swedish-American physicist at MIT who co-founded the Future of Life Institute, which organized the 2023 open letter calling for a pause in training the most powerful AI systems.
Transformer Circuits ThreadAnthropic's online publication for research on how language models work inside, presented as long interactive web articles rather than conventional papers.
unembeddingThe last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores.