AllocateThe inference glossary
The vocabulary behind training and running your own models, defined in plain language with the numbers that matter.
Inference
Inference is running a trained AI model to produce outputs: answers, classifications, decisions, or actions.
ReadFine-tuning
Fine-tuning continues a model’s training on your own examples so it learns your domain, formats, and decision patterns.
ReadOpen-weight models
Open-weight models publish their trained parameters so anyone can run, inspect, and fine-tune them under the model’s license.
ReadModel routing
Model routing is directing each AI request to the best model for it through one stable endpoint.
ReadTokens
Tokens are the units AI models read and write: fragments of words, roughly four characters of English each.
ReadContext window
A context window is the maximum amount of text a model can consider at once, measured in tokens.
ReadMixture of experts
A mixture-of-experts (MoE) model contains many specialist sub-networks but activates only a few per token, so Qwen 3.5 carries 397 billion total parameters but activates about 17 billion per token.
ReadQuantization
Quantization compresses a model by storing its weights at lower numeric precision, such as 8-bit or 4-bit instead of 16-bit.
ReadRetrieval-augmented generation
Retrieval-augmented generation (RAG) fetches relevant documents from your data at request time and gives them to the model as context, grounding answers in your actual sources instead of the model’s memory.
ReadEmbeddings
Embeddings are numeric representations of text, images, or other data where similar meaning produces nearby vectors.
ReadAI agents
An AI agent is a model given tools, context, and a goal, running in a loop until the work is done: reading systems, taking actions, and escalating to humans when policy requires.
ReadTraining signal
Training signal is any recorded outcome that teaches a model what good looks like: an approved claim, an accepted draft, a human correction.
ReadModel weights
Model weights are the billions of learned numbers that constitute a trained model: the artifact itself.
ReadInference latency
Inference latency is how long a model takes to respond.
ReadModel distillation
Distillation trains a smaller, cheaper model to imitate a larger one by learning from its outputs.
ReadReinforcement learning
Reinforcement learning (RL) trains a model by scoring its outputs and updating it to earn higher scores, rather than imitating fixed examples.
ReadData residency
Data residency is the requirement that data stays within a defined jurisdiction or boundary: a country, a cloud region, or a private environment.
ReadAI compliance
AI compliance means running models the way regulators expect systems of record to run: isolated data, logged decisions, human escalation paths, and the ability to explain any output after the fact.
ReadOpenAI-compatible API
An OpenAI-compatible API implements the same request and response format as OpenAI’s chat completions endpoint, so any existing client, SDK, or framework works by changing only the base URL and key.
ReadGPU utilization
GPU utilization measures how much of expensive accelerator hardware does useful work rather than sitting idle.
ReadLoRA
LoRA (low-rank adaptation) fine-tunes a model by training small adapter matrices on top of frozen weights instead of updating every parameter.
ReadQLoRA
QLoRA is LoRA fine-tuning run on top of a 4-bit quantized base model.
ReadQuantization formats (GGUF, AWQ, FP8)
GGUF, AWQ, and FP8 are common formats for storing quantized model weights.
ReadKV cache
The KV cache stores the attention keys and values for every token a model has already read, so each new token only attends over cached state instead of reprocessing the whole prompt.
ReadPaged attention
Paged attention manages the KV cache the way an operating system manages RAM: in small fixed-size blocks allocated on demand, instead of one contiguous slab per request.
ReadSpeculative decoding
Speculative decoding speeds up generation by letting a small draft model propose several tokens ahead, which the large model then verifies in a single forward pass.
ReadContinuous batching
Continuous batching lets an inference server add and remove requests from a running batch at every generation step, instead of waiting for a whole batch to finish together.
ReadTime to first token
Time to first token (TTFT) is the delay between sending a request and receiving the first token of the response.
ReadTokens per second
Tokens per second measures generation speed: how fast a model streams output once it starts.
ReadPrompt caching
Prompt caching reuses the computed state of a prompt prefix that repeats across requests, such as a system prompt or tool definitions, so those tokens are not reprocessed and bill at a deep discount.
ReadContext caching
Context caching is the explicit form of prompt caching: you register a large block of context, a policy book, a codebase, a document set, once, and reference it across many requests.
ReadBatch inference
Batch inference processes large sets of requests without a latency requirement: classify a backlog, embed an archive, score every record in a table.
ReadFunction calling
Function calling lets a model respond with a structured request to invoke a function you defined, arguments included, instead of prose.
ReadTool use
Tool use is a model acting through capabilities you grant it: searching, querying systems, editing files, sending messages.
ReadMoE routing
Inside a mixture-of-experts model, a small learned router picks which expert sub-networks process each token, so only a fraction of the model computes at a time.
ReadRLHF
RLHF (reinforcement learning from human feedback) post-trains a model against human preferences: annotators rank outputs, a reward model learns those rankings, and the model is optimized to score higher.
ReadDPO
DPO (direct preference optimization) trains a model on preference pairs, a chosen answer and a rejected one, by directly adjusting the model to favor the chosen response.
ReadReranking
Reranking re-orders search results by scoring each candidate against the query with a model that reads both together.
ReadVector database
A vector database stores embeddings and answers nearest-neighbor queries: given a query vector, return the most similar stored vectors in milliseconds, even across millions of items.
ReadAgent sandbox
An agent sandbox is an isolated execution environment where an AI agent runs: its own filesystem, network policy, and compute, separated from your production systems.
ReadLLM gateway
An LLM gateway is one endpoint in front of every model: your code calls a single OpenAI-compatible API, and the gateway handles authentication, routing, metering, failover, and policy across providers.
ReadStructured outputs
Structured outputs constrain a model to emit valid JSON matching a schema you provide, instead of free text that your code must parse and hope over.
ReadInference cost per token
Inference cost per token is the provider’s price per million tokens, split by direction: input tokens (your prompt) and output tokens (the response), with output typically 2 to 6 times dearer.
ReadGPU memory bandwidth
Generating a token requires streaming every active model weight from GPU memory through the compute units, once per token.
Read