blurry JPEG of the web

A metaphor characterizing a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. as a lossy compression of its training text, like a heavily compressed JPEG image: it preserves statistical regularities and the gist of much of the web but cannot reproduce exact details, and fills the gaps by plausible interpolation. On this view hallucinationshallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true. are compression artifacts, and fluent paraphrase creates a misleading impression of understanding, much as a blurry image looks acceptable at a glance. The essay also predicted that training models on their own output would degrade them, like re-saving a JPEG repeatedly, anticipating concerns about model collapsemodel collapse, n.: The gradual decline of AI models trained on the output of earlier AI models: rare details vanish first, then quality and variety erode, like a photocopy of a photocopy.. Critics reply that compression and understanding are not opposites — compressing text well may require modeling its underlying structure — a view tied to arguments that next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers. can induce world modelsworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real..

If the model is a blurry JPEG of the web, retrieval is how you get the sharp original back.