instruction dataset
In plain wordsA collection of example requests paired with good answers, used to teach a pretrained language model to follow instructions and act as a helpful assistant.
A dataset of instructions, optionally with inputs, paired with target responses — or of full multi-turn conversations — used for instruction tuninginstruction tuning, n.: Training a language model on many tasks written as plain instructions with correct answers, so it learns to do what a request asks instead of just continuing the text. and supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples. of a base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering.. Sources include existing NLP tasks rewritten as natural-language instructions through templates (FLAN; Super-NaturalInstructions, with more than 1,600 tasks), demonstrations written by paid annotators or volunteers (the data behind InstructGPT, Databricks' Dolly 15k, OpenAssistant Conversations), and responses generated by stronger models (Self-InstructSelf-Instruct, n.: A technique in which a language model writes its own practice tasks and answers, starting from a handful of human examples; the results are filtered and used to train it to follow instructions., Stanford's Alpaca and many distilled sets). Quality and diversity matter more than size: about a thousand carefully chosen examples can produce a capable assistant, while noisy or homogeneous data teaches bad habits. Modern pipelines filter generated data with reward modelsreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. or verifiers, mix in safety and refusal examples, and format examples with a chat templatechat template, n.: The exact text layout a chat AI was trained on for marking who is speaking — system, user or assistant. Software must reproduce it precisely, or the model performs noticeably worse..
We cut the instruction dataset to 20,000 high-quality examples and the model got better.
Origin
Took shape in 2021–2022 with Swaroop Mishra and colleagues' Natural Instructions (2021), Google's FLAN (Jason Wei and colleagues, 2021), BigScience's T0 and its P3 collection (2021) and OpenAI's InstructGPT (2022). The 2023 wave of open chat models, beginning with Alpaca, made model-generated instruction data the norm.
Usage note
Distinct from preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like., which records judgments between alternative responses for reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. or direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.. Data generated with commercial models may carry terms of service restricting its use to train competing models.
Related
- instruction tuninginstruction tuning, n.: Training a language model on many tasks written as plain instructions with correct answers, so it learns to do what a request asks instead of just continuing the text.
- supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.
- Self-InstructSelf-Instruct, n.: A technique in which a language model writes its own practice tasks and answers, starting from a handful of human examples; the results are filtered and used to train it to follow instructions.
- synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars.
- chat templatechat template, n.: The exact text layout a chat AI was trained on for marking who is speaking — system, user or assistant. Software must reproduce it precisely, or the model performs noticeably worse.
- knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
- preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
Sources
- Wei, J. et al. (2022). Finetuned Language Models Are Zero-Shot Learners. ICLR 2022.
- Wang, Y. et al. (2022). Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. EMNLP 2022.
- Zhou, C. et al. (2023). LIMA: Less Is More for Alignment. NeurIPS 2023.
Mentioned in
data-centric AIdata-centric AI, n.: An approach to building AI that focuses on improving the data — fixing labels, adding missing cases, cleaning inconsistencies — rather than endlessly tweaking the model., instruction tuninginstruction tuning, n.: Training a language model on many tasks written as plain instructions with correct answers, so it learns to do what a request asks instead of just continuing the text., instruction-tuned modelinstruction-tuned model, n.: A language model given extra training on examples of instructions paired with good responses, so it does what you ask instead of merely continuing your text., preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like., pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat., supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples., Self-InstructSelf-Instruct, n.: A technique in which a language model writes its own practice tasks and answers, starting from a handful of human examples; the results are filtered and used to train it to follow instructions.