NVIDIA TensorRT is an inference-optimization toolkit and runtime for getting trained deep-learning models to run as fast and as cheaply as possible on NVIDIA GPUs. A model that is correct after training is not necessarily efficient to serve; TensorRT takes that trained model and compiles an optimized version specialized to the target hardware.
It does this with several techniques. Quantization converts the model to lower-precision number formats, such as FP8, INT8, or INT4, which the documentation notes significantly cuts latency and memory bandwidth while usually preserving accuracy. Layer and tensor fusion merges many small operations into fewer, larger ones so the GPU spends less time launching kernels and moving data. Kernel auto-tuning picks the fastest implementation of each operation for the specific GPU. A separate Model Optimizer adds pruning, sparsity, and distillation. NVIDIA reports large speedups, including roughly 36x over CPU-only inference in some cases, and TensorRT models are commonly served through engines like Triton Inference Server.
Why a business reader should care: inference is where the recurring cost of an AI product lives, and tools like TensorRT can cut that cost substantially by squeezing far more throughput out of the same GPUs.