tokens per second
In plain wordsA speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant.
A rate of tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. generation or processing, quoted in three easily confused ways. Per-request output speed is the number of tokens one response streams per second during the decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far., the reciprocal of time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow.; it decides how quickly an answer or a long chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate. completes. Aggregate output tokens per second sums generation across all concurrent requests on a server and measures throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second. and cost efficiency. Input or prefill tokens per second, usually far higher because prompt tokens are processed in parallel, measures how fast prompts are read. Per-request speed is bounded largely by memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed. divided by the bytes read per step, which is why quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality. and chips with large on-chip SRAMSRAM, abbr.: The fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it. raise it. Figures depend on model, hardware, batch size, prompt length and tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text., so cross-model comparisons are approximate.
On my laptop the 8-billion-parameter model runs at about 40 tokens per second, plenty for chat.
Origin
A long-standing informal unit in text generation that became a headline metric in 2023–2024, when local-inference hobbyists compared hardware by tok/s and independent benchmarking sites and specialized-chip vendors began publishing output-speed leaderboards for hosted models.
Usage note
A token is roughly three-quarters of an English word on common tokenizers, but the ratio varies by language and tokenizer, so tokens per second is not directly words per second.
Related
- throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second.
- time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow.
- latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system.
- decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far.
- memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed.
- speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality.
- FLOPSFLOPS, abbr.: Floating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips.
Mentioned in
inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text., tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., inference providerinference provider, n.: A company that runs AI models on its own computers and sells access to them over the internet, charging by use, so customers need not buy or operate the hardware themselves., latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system., speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., streamingstreaming, n.: Sending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end., throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second., time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear., time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow.