tuned lens
In plain wordsAn improved version of the logit lens that first learns a small translator for each layer, so a language model's halfway-through states can be read as predictions more accurately.
A method for decoding intermediate states of a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. language model into next-token predictions by training, for each layer, a small affine translator that maps the residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document. at that layer into the space expected by the model's final unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores.. The translators are fitted, with the model frozen, by minimizing the KL divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. between each layer's decoded distribution and the model's actual output. In its authors' tests the resulting readouts were steadier from layer to layer and less skewed toward particular tokens than those of the logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time., worked on model families where the logit lens fails, relied on directions the model demonstrably uses, and could expose malicious inputs, such as prompt injections, through unusual prediction trajectories. Being a trained probeprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere., it could in principle read in information the model itself does not use.
The tuned lens shows the model settling on the answer three layers earlier than the logit lens suggested.
Origin
Introduced by Nora Belrose and colleagues, including Stella Biderman and Jacob Steinhardt, in 'Eliciting Latent Predictions from Transformers with the Tuned Lens' (arXiv, March 2023), released with an open-source library through EleutherAI.
Related
- logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time.
- probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.
- residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
- linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly.
- PatchscopesPatchscopes, prop. n.: A family of tricks for getting a language model to describe its own hidden states in words, by transplanting an internal state into a new prompt designed to make the model talk about it.
Sources
- Belrose, N., Ostrovsky, I., McKinney, L., Furman, Z., Smith, L., Halawi, D., Biderman, S. & Steinhardt, J. (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv:2303.08112. arxiv.org/abs/2303.08112
Mentioned in
logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time., PatchscopesPatchscopes, prop. n.: A family of tricks for getting a language model to describe its own hidden states in words, by transplanting an internal state into a new prompt designed to make the model talk about it.