knowledge cutoff
In plain wordsThe date after which an AI model knows nothing from its training, because its training data stops there. Without a web search tool, it cannot know about later events.
The latest date represented in a model's training datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat., after which it has no parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book. of events, publications, prices or software versions. Developers usually state a cutoff in model documentation, but the effective cutoff varies by source and topic: the final months before the cutoff are thinly covered at training time, web crawls contain much older material, and models are often unsure of their own cutoff. Models without tools may present outdated facts as current or assume the present date is close to their cutoff; deployments mitigate this by putting the current date in the system promptsystem prompt, n.: Background instructions given to a chatbot before the conversation starts, setting its role, rules, tone and tools. Users usually never see it, but it shapes every reply. and by adding web search and retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from.. Because data collection, training and testing take time, cutoffs typically precede public release by several months to a year; continued pretrainingcontinued pretraining, n.: Taking an already-trained AI model and resuming its broad, read-everything training on new material — a specialty such as medicine or law, a new language, or fresher data — rather than starting from scratch. can move a cutoff forward without training from scratch.
Its knowledge cutoff is early 2025, so give it the new API documentation in the prompt.
Origin
The phrase became common after ChatGPT's release in November 2022, when users met its stock disclaimers about not knowing events after its training cutoff in 2021. Jeffrey Cheng and colleagues' 'Dated Data' (2024) showed that effective cutoffs often differ from those developers report, partly because of old material in web crawls and imperfect deduplication.
Usage note
Not the same as the model's release date or its context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out. limit. A model does not reliably know its own cutoff unless told; asking it is not a good test.
Related
- parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from.
- groundinggrounding, n.: Connecting an AI's words to something real — trusted documents, a live search, images or the physical world — so that its answers rest on evidence rather than only on what it absorbed in training.
- continued pretrainingcontinued pretraining, n.: Taking an already-trained AI model and resuming its broad, read-everything training on new material — a specialty such as medicine or law, a new language, or fresher data — rather than starting from scratch.
- hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true.
Sources
- Cheng, J., Marone, M., Weller, O., Lawrie, D., Khashabi, D. & Van Durme, B. (2024). Dated Data: Tracing Knowledge Cutoffs in Large Language Models. arXiv:2403.12958. arxiv.org/abs/2403.12958
Mentioned in
continual learningcontinual learning, n.: The ability of an AI to keep learning new things over time from a stream of experience, without forgetting what it already knew and without being retrained from scratch., distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested., contamination-resistant benchmarkcontamination-resistant benchmark, n.: A test built so that its questions are unlikely to have been seen by an AI during training — kept secret, refreshed often, or written after the model was made., parametric knowledgeparametric knowledge, n.: What an AI model 'knows' from training, stored in the numbers of its network rather than looked up in documents — like facts you have memorized rather than ones you check in a book.