TensorRT
In plain wordsNVIDIA's software for making a trained neural network run as fast as possible on NVIDIA graphics chips, by merging and simplifying its calculations and testing which way of doing each one is quickest.
An inference optimizer and runtime from NVIDIANvidia, prop. n.: An American chip company whose graphics processors became the main hardware for training and running AI, making it one of the most valuable companies in the world. that compiles a trained neural network into an 'engine' tuned for a particular NVIDIA GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. and range of input shapes. Models usually arrive as ONNXONNX, prop. n.: A shared file format for trained AI models, so that a model built with one toolkit can be moved to and run by another, much as a PDF opens on any computer. graphs or through framework integrations such as Torch-TensorRT. The builder fuses layers and operations (kernel fusionkernel fusion, n.: Merging several small calculation steps into one, so a chip finishes them in a single pass instead of saving and reloading data in between, like cooking at the counter without returning ingredients to the fridge after each step.), eliminates unneeded computation, selects memory layouts and lower numerical precisions — FP16, BF16, INT8 with calibration, FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits. and, on recent GPUs, FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory. — and times candidate kernel implementations ('tactics') on the target hardware to choose the fastest. The serialized engine then runs with low latency and little overhead, but by default it is tied to the GPU model and TensorRT version used to build it. TensorRT is common in production vision, speech and recommendation services and in embedded and automotive systems such as NVIDIA's Jetson and DRIVE platforms, and it is the basis of TensorRT-LLMTensorRT-LLM, prop. n.: NVIDIA's free software library for running large language models as fast as possible on NVIDIA's own graphics chips, used by many companies that host AI models. for language models.
Exporting the detector to ONNX and building an INT8 TensorRT engine more than halved its latency on the Jetson.
Origin
Introduced by NVIDIA in June 2016 as the GPU Inference Engine (GIE) and renamed TensorRT that September. Later versions added INT8 calibration, an ONNX parser, Python interfaces and FP8 and FP4 support for the Hopper and Blackwell GPU architectures; the language-model library TensorRT-LLM followed in 2023.
Usage note
Not a training framework: it runs models trained elsewhere. Not to be confused with NVIDIA's Tensor Corestensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can. (matrix-multiply units) or Google's TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems., with which it was integrated as TF-TRT. Comparable tools include ONNX Runtime, Apache TVM and the compilers behind PyTorch's torch.compile.
Related
- TensorRT-LLMTensorRT-LLM, prop. n.: NVIDIA's free software library for running large language models as fast as possible on NVIDIA's own graphics chips, used by many companies that host AI models.
- inference engineinference engine, n.: The part of an AI system that does the working-out: in older AI, the program that applies rules to facts; in today's AI, the software that runs a trained model to produce answers.
- kernel fusionkernel fusion, n.: Merging several small calculation steps into one, so a chip finishes them in a single pass instead of saving and reloading data in between, like cooking at the counter without returning ingredients to the fridge after each step.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- ONNXONNX, prop. n.: A shared file format for trained AI models, so that a model built with one toolkit can be moved to and run by another, much as a PDF opens on any computer.
- CUDACUDA, prop. n.: Nvidia's software system for programming its graphics chips to do general calculations. Most AI software is built on it, which is a big reason Nvidia chips dominate AI.
- post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.
- TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems.
Sources
- NVIDIA. TensorRT Developer Guide. NVIDIA documentation.
- NVIDIA (2016). Production Deep Learning with NVIDIA GPU Inference Engine. NVIDIA Developer Blog, 19 June 2016 (updated September 2016). developer.nvidia.com/blog/production-deep-learning-nvidia-gpu-inference-engine