context distillation

Fine-tuning a language model so that, without a given context — a long system promptsystem prompt, n.: Background instructions given to a chatbot before the conversation starts, setting its role, rules, tone and tools. Users usually never see it, but it shapes every reply., behavioral guidelines or worked examples — it reproduces the outputs or token distributions it would produce with that context. The same model, conditioned on the context, acts as teacher; the student is trained on the teacher's outputs, or on the KL divergence between the two distributions, over prompts that omit the context. This moves knowledge or behavior from the context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out. into the weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself., saving inference cost and context space, and makes the behavior harder to override with a competing prompt. Anthropic first used it to instill a helpful, honest and harmlesshelpful, honest and harmless, phr.: A three-part goal for AI assistants, popularized by Anthropic: be genuinely useful, tell the truth, and avoid causing harm — shorthand for what an aligned chatbot should be. assistant persona before preference training; later uses include internalizing safety specifications (as in the supervised stage of deliberative alignmentdeliberative alignment, n.: A training method in which an AI is taught the actual text of its safety rules and learns to think through which rules apply before it answers a sensitive request.), learning from instructions or scratchpad reasoning, and shaping model character. The distilled behavior can generalize differently from the prompted version and is less transparent, since the rules no longer appear in the input.

We context-distilled the 6,000-token style guide, so the deployed model follows it with an empty system prompt.