on-device AI
In plain wordsAI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device.
The execution of AI models on an end-user device such as a phone, laptop, tablet, wearable, car or headset, rather than on remote servers. Benefits include privacy (inputs need not leave the device), offline operation, low and predictable latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system., and no per-request inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text. for the provider; the limits are memory, battery, heat and compute, which cap model size and speed. On-device language models are therefore usually smallsmall language model, n.: A language model built to be compact enough to run cheaply and quickly, even on a laptop or phone, trading some knowledge and skill for speed, privacy and lower cost. (one to a few billion parameters, more on high-end laptops), heavily quantizedquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., and run on the device's NPUNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud., GPU or CPU through runtimes such as Core ML, LiteRT, ExecuTorch, ONNX Runtime, llama.cppllama.cpp, prop. n.: A free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models. and MLX. Commercial assistants commonly use hybrid designs that handle simple requests locally and send harder ones to cloud models, which raises questions about when data leaves the device. Hobbyists running open-weight modelsopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private. on their own computers call the practice local AI.
Transcription now happens on-device, so the recording never leaves your phone.
Origin
On-device machine learning grew with smartphone neural accelerators such as Apple's Neural Engine (2017) and with on-device speech recognition and keyboards. The phrase became a consumer category when Google's Gemini Nano (December 2023) and Apple Intelligence, with an on-device model of about three billion parameters (June 2024), brought generative models to phones; tools such as llama.cpp (2023) turned local AI into a hobbyist movement.
Usage note
Overlaps with edge AIedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers., which also covers nearby servers, gateways and industrial devices; on-device AI usually means the user's own device. 'On-device' claims deserve scrutiny when features silently fall back to the cloud.
Related
- edge AIedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers.
- small language modelsmall language model, n.: A language model built to be compact enough to run cheaply and quickly, even on a laptop or phone, trading some knowledge and skill for speed, privacy and lower cost.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- NPUNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud.
- llama.cppllama.cpp, prop. n.: A free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models.
- OllamaOllama, prop. n.: A free app that makes it easy to download and run AI language models on your own computer with a single command, keeping your data on your machine.
- TinyMLTinyML, n.: Machine learning shrunk to run on tiny, cheap, battery-powered chips inside sensors, wearables or appliances, using so little power that a coin-sized battery can last for months or years.
- inference providerinference provider, n.: A company that runs AI models on its own computers and sells access to them over the internet, charging by use, so customers need not buy or operate the hardware themselves.
Mentioned in
CPUCPU, abbr.: The main general-purpose chip in a computer, good at running many different programs step by step. In AI systems it usually coordinates the work while specialized chips do the heavy math., edge AIedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers., GemmaGemma, prop. n.: Google's family of free-to-download AI language models: smaller relatives of its Gemini systems, compact enough to run on a single computer and, in some versions, on a phone., llama.cppllama.cpp, prop. n.: A free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models., model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy., NPUNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud., OllamaOllama, prop. n.: A free app that makes it easy to download and run AI language models on your own computer with a single command, keeping your data on your machine., small language modelsmall language model, n.: A language model built to be compact enough to run cheaply and quickly, even on a laptop or phone, trading some knowledge and skill for speed, privacy and lower cost., VRAMVRAM, abbr.: The memory on a graphics card. For running AI at home it is the key number: the model has to fit in it, so more memory means bigger models., wake wordwake word, n.: A word or phrase, such as 'Alexa' or 'Hey Google', that a smart speaker or phone listens for constantly and that switches it into full listening mode., AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them., BitNetBitNet, prop. n.: A family of AI language models from Microsoft in which most of the learned numbers are limited to minus one, zero or one, so they need far less memory and energy to run., Core MLCore ML, prop. n.: Apple's software framework that lets apps on iPhones, iPads and Macs run AI models directly on the device, using its own chips instead of sending data to the cloud., ExecuTorchExecuTorch, prop. n.: Meta's free software for running PyTorch AI models directly on phones, wearables and other small devices, without needing an internet connection or a data center., GGUFGGUF, abbr.: A file format that packs an AI language model, often in a shrunken low-precision version, into one file that ordinary computers can load and run quickly; used by llama.cpp and Ollama., LiteRTLiteRT, prop. n.: Google's free software for running AI models on phones, small gadgets and microcontrollers; it was called TensorFlow Lite until 2024 and is used inside many Android and iPhone apps., MLXMLX, prop. n.: A free Apple software toolkit for running and training AI models efficiently on Mac computers with Apple's own chips, popular for running language models locally on a laptop., MobileNetMobileNet, prop. n.: A family of compact image-recognition networks from Google, designed to run quickly on phones and other small devices with little loss of accuracy., offloadingoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run., small AIsmall AI, n.: Cheap, compact AI tools built for one practical job — like advising farmers or screening patients — that poorer countries can afford to run, instead of giant all-purpose models., TinyMLTinyML, n.: Machine learning shrunk to run on tiny, cheap, battery-powered chips inside sensors, wearables or appliances, using so little power that a coin-sized battery can last for months or years., TOPSTOPS, abbr.: Trillions of operations per second: the speed rating usually given for AI chips in phones and laptops, often counting simple whole-number math rather than precise fractions.