supervised fine-tuning
In plain wordsTeaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.
Fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way. a pretrained model with ordinary supervised learning on input–output pairs; for language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., on demonstrations consisting of a prompt (often a multi-turn conversation in a chat templatechat template, n.: The exact text layout a chat AI was trained on for marking who is speaking — system, user or assistant. Software must reproduce it precisely, or the model performs noticeably worse.) and a desired response written by people or generated by stronger models. The objective is the cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. loss of next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers., usually computed only on the response tokens, so SFT is a form of imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error. or behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation.. In the standard post-training pipeline it comes first: it teaches format, instruction following, persona and tool-call syntax, and supplies the starting policy (and usually the reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started.) for preference optimization or reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time.. Quality and diversity of demonstrations matter more than volume; a few thousand curated examples can set style, while reasoning SFT uses long chains of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate., often distilled from a stronger model.
We ran SFT on 50,000 distilled reasoning traces before starting RL, to give the policy a cold start.
Origin
Supervised fine-tuning of pretrained networks long predates the abbreviation. 'SFT' as the name of the demonstration stage of the language-model alignment pipeline was popularized by OpenAI's InstructGPT paper (Long Ouyang and colleagues, 2022), which labeled its stages SFT, RM (reward model) and PPO; the label spread with ChatGPT-era open replications.
Usage note
Strictly, any supervised fine-tuning counts, but in LLM writing 'SFT' means the demonstration stage. Some argue it mainly teaches style rather than knowledge (the superficial alignment hypothesissuperficial alignment hypothesis, n.: The idea that a language model learns nearly all its knowledge and skills from its initial bulk training, and that later tuning mainly teaches it the style and format to use when talking to people.). Comparisons suggest SFT drifts further from the base model and forgets more than on-policy reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., though it remains the cheapest way to install a behavior.
Related
- fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
- instruction tuninginstruction tuning, n.: Training a language model on many tasks written as plain instructions with correct answers, so it learns to do what a request asks instead of just continuing the text.
- instruction datasetinstruction dataset, n.: A collection of example requests paired with good answers, used to teach a pretrained language model to follow instructions and act as a helpful assistant.
- behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner.
- knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
- reinforcement fine-tuningreinforcement fine-tuning, n.: Improving a trained AI model by trial and error instead of examples to copy: it tries answers, a grader scores them, and the model shifts toward whatever earned high scores.
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
Sources
- Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022. arXiv:2203.02155. arxiv.org/abs/2203.02155
- Shenfeld, I., Pari, J. & Agrawal, P. (2025). RL's Razor: Why Online Reinforcement Learning Forgets Less. arXiv:2509.04259. arxiv.org/abs/2509.04259
Mentioned in
fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way., knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run., post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner., pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow., reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation., chat modelchat model, n.: A language model trained to hold a back-and-forth conversation, keeping track of who said what and replying as a helpful assistant. It is the engine inside chat apps., continued pretrainingcontinued pretraining, n.: Taking an already-trained AI model and resuming its broad, read-everything training on new material — a specialty such as medicine or law, a new language, or fresher data — rather than starting from scratch., imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error., InstructGPTInstructGPT, prop. n.: A 2022 OpenAI language model trained with human feedback to follow instructions; people preferred its answers to those of a model 100 times larger, and its recipe led directly to ChatGPT., instruction datasetinstruction dataset, n.: A collection of example requests paired with good answers, used to teach a pretrained language model to follow instructions and act as a helpful assistant., instruction tuninginstruction tuning, n.: Training a language model on many tasks written as plain instructions with correct answers, so it learns to do what a request asks instead of just continuing the text., instruction-tuned modelinstruction-tuned model, n.: A language model given extra training on examples of instructions paired with good responses, so it does what you ask instead of merely continuing your text., mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents., reinforcement fine-tuningreinforcement fine-tuning, n.: Improving a trained AI model by trial and error instead of examples to copy: it tries answers, a grader scores them, and the model shifts toward whatever earned high scores., rejection samplingrejection sampling, n.: Generating many candidate answers and keeping only those that pass a check — a scoring model, a test or a correct final answer — often so the AI can then be trained on its own best work., safety trainingsafety training, n.: The part of an AI's training that teaches it to avoid harmful behavior — declining dangerous requests, resisting manipulation and following its developer's rules — on top of being generally helpful., Scale AIScale AI, prop. n.: A company that pays large numbers of people to label examples, write model answers and grade chatbot replies, producing the data AI developers train and test on. Meta bought nearly half of it in 2025., shoggothshoggoth, n.: A tentacled monster from H. P. Lovecraft's horror fiction, used as a meme for large AI models: a vast, alien system hidden behind a small, friendly mask that training adds to make it polite., sleeper agentsleeper agent, n.: An AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated., supervised learningsupervised learning, n.: Teaching a computer by example: you give it many inputs paired with the right answers, such as photos labeled 'cat' or 'dog', and it learns to answer correctly for new inputs., teacher forcingteacher forcing, n.: Training a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one.