What is AI inference?

Inference is running a trained AI model to produce outputs: answers, classifications, decisions, or actions. Training builds the model once; inference is every request after that. In production, inference dominates AI cost and latency, which is why teams optimize it with routing, caching, quantization, and purpose-built infrastructure.

Every time an agent answers a customer, classifies a document, or drafts a report, that is one inference call. A production company might run millions per week, so a 30% cost or latency improvement at the inference layer is worth more than almost any prompt change.

Inference platforms sit between your product and the models. They handle routing, metering, failover, and policy so your application code stays a single stable API call while everything behind it can change.

Related terms