sleep-time compute
In plain wordsLetting an AI think about your documents or project while it would otherwise sit idle, before you ask anything, so it can answer later questions faster and more cheaply.
Extra computation a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. performs on a body of context while no one is asking it anything (a codebase, a document collection, a long-running conversation), working out in advance what is likely to be asked and saving useful intermediate conclusions for later. Moving this share of test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter. into idle time makes eventual answers faster and cheaper, and the up-front work can be reused by every later question about the same material. The introducing paper found that the advantage tracks how foreseeable the questions are. On math benchmarks rebuilt so that a context precedes each question, the method reached the same accuracy with roughly a fifth of the answer-time compute and raised accuracy by up to 13–18% when more offline compute was spent; when related questions shared a context, the average cost per question fell by a factor of 2.5.
Overnight sleep-time compute let the agent pre-digest the repo, so morning questions came back in seconds.
Origin
Introduced by Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica and Joseph E. Gonzalez (Letta and the University of California, Berkeley) in 'Sleep-time Compute: Beyond Inference Scaling at Test-time', posted to arXiv on 17 April 2025.
Related
- test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.
- agent memoryagent memory, n.: The ways an AI assistant or agent keeps information — your preferences, past conversations, lessons from earlier tasks — and brings it back later, beyond what fits in a single conversation.
- prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper.
- background agentbackground agent, n.: An AI agent you hand a task to and then leave alone: it works by itself in the cloud, often for many minutes or hours, and reports back with finished work for you to review.
- inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
- KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
Sources
- Lin, K., Snell, C., Wang, Y., Packer, C., Wooders, S., Stoica, I. & Gonzalez, J. E. (2025). Sleep-time Compute: Beyond Inference Scaling at Test-time. arXiv:2504.13171. arxiv.org/abs/2504.13171