streaming
In plain wordsSending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end.
- 1
Model output
Delivery of a generative model's output incrementally, as each tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. or small group of tokens is produced, rather than as one complete response. LLM APIsmodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves. usually stream over HTTP with server-sent events (SSE) or over WebSockets, sending a series of 'delta' messages carrying text fragments, partial tool-calltool call, n.: A single request from an AI model to use one of its tools — naming the tool and filling in the details — which the surrounding software then carries out and reports back on. arguments and finally usage and stop-reason metadata. Streaming does not make generation faster, but it cuts perceived latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system. to the time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear., lets users read or cancel early, and lets downstream components such as text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps. or incremental parsers of structured outputsstructured outputs, n.: Getting an AI model to answer in an exact, machine-readable format — like a form with fixed fields — so other software can use the answer without guesswork. start work at once. It complicates moderation, error handling and billing, because a response can fail or be withheld midway.
With streaming on, users see the first words in 300 milliseconds even though the full answer takes eight seconds.
- 2
Continuous input
Incremental processing of an unbounded input, such as live audio, video or an endless conversation, emitting outputs with bounded delay and memory, as in streaming speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video. or simultaneous translation. For transformers this requires bounded attention state; StreamingLLM (2023), for example, keeps a few initial attention sinkattention sink, n.: A quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need. tokens plus a sliding window of recent keys and values.
Origin
An older computing word for continuous data delivery ('streaming media' spread in the 1990s). Token streaming became familiar to the public through ChatGPT's word-by-word display from November 2022; OpenAI's completions API already offered a stream option built on server-sent events, a web standard from the HTML5 era.
Usage note
Unrelated to the residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document. of interpretability research.
Related
- time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear.
- latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system.
- tokens per secondtokens per second, n.: A speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant.
- model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves.
- attention sinkattention sink, n.: A quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need.
- speech-to-speech modelspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice.
- batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less.
Mentioned in
model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves., latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system., time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear.