ONNX
In plain wordsA shared file format for trained AI models, so that a model built with one toolkit can be moved to and run by another, much as a PDF opens on any computer.
A vendor-neutral file format (Open Neural Network Exchange) that describes a trained model as a computation graph of standardized operators with typed inputs and outputs, saved together with the model's weights using Protocol Buffers serialization. Operators are versioned in operator sets, or opsets, so a model declares which version it needs and runtimes can check support. Models are exported from training frameworks — PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips. through torch.onnx, TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems. and scikit-learnscikit-learn, prop. n.: A free, widely taught Python library of classic machine-learning methods — such as decision trees, regression and clustering — that all work in the same simple way. through converters — and executed by runtimes and compilers that optimize them for target hardware, most prominently Microsoft's ONNX Runtime, whose execution providers target CPUs, GPUs and NPUsNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud. and which also runs in browsers and on phones; vendor toolkits such as TensorRT and OpenVINO also accept ONNX. The format decouples training from deployment, but export can fail for models with dynamic control flow or unusual operators, and large language models are more often distributed in safetensorssafetensors, n.: A file format for storing an AI model's learned numbers that cannot hide harmful programs and loads very quickly, replacing older formats that could run code when opened. or GGUFGGUF, abbr.: A file format that packs an AI language model, often in a shrunken low-precision version, into one file that ordinary computers can load and run quickly; used by llama.cpp and Ollama..
We export the classifier to ONNX and run it with ONNX Runtime on the edge boxes, with no PyTorch install.
Origin
Announced by Facebook and Microsoft in September 2017 to let models move between PyTorch, Caffe2 and Microsoft's Cognitive Toolkit; Amazon Web Services, chip makers and others soon joined. The project moved to the Linux Foundation's AI foundation (now LF AI & Data) in November 2019.
Usage note
ONNX is the format; ONNX Runtime is one engine that executes it. The two names are often used loosely for each other.
Related
- PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips.
- TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems.
- model deploymentmodel deployment, n.: Putting a trained AI model to work in a real product so that people or other software can use it — and doing so carefully enough that the change can be undone if something goes wrong.
- edge AIedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers.
- Core MLCore ML, prop. n.: Apple's software framework that lets apps on iPhones, iPads and Macs run AI models directly on the device, using its own chips instead of sending data to the cloud.
- LiteRTLiteRT, prop. n.: Google's free software for running AI models on phones, small gadgets and microcontrollers; it was called TensorFlow Lite until 2024 and is used inside many Android and iPhone apps.
- safetensorssafetensors, n.: A file format for storing an AI model's learned numbers that cannot hide harmful programs and loads very quickly, replacing older formats that could run code when opened.
Mentioned in
model deploymentmodel deployment, n.: Putting a trained AI model to work in a real product so that people or other software can use it — and doing so carefully enough that the change can be undone if something goes wrong., AI compilerAI compiler, n.: Software that translates an AI model into efficient instructions for a specific chip, rearranging and combining its calculations so the hardware is used as fully as possible., Core MLCore ML, prop. n.: Apple's software framework that lets apps on iPhones, iPads and Macs run AI models directly on the device, using its own chips instead of sending data to the cloud., GGUFGGUF, abbr.: A file format that packs an AI language model, often in a shrunken low-precision version, into one file that ordinary computers can load and run quickly; used by llama.cpp and Ollama., model hubmodel hub, n.: An online library where trained AI models are published so that others can find, download and reuse them instead of training their own from scratch., safetensorssafetensors, n.: A file format for storing an AI model's learned numbers that cannot hide harmful programs and loads very quickly, replacing older formats that could run code when opened., TensorRTTensorRT, prop. n.: NVIDIA's software for making a trained neural network run as fast as possible on NVIDIA graphics chips, by merging and simplifying its calculations and testing which way of doing each one is quickest.