AI inference

An AI inference call has three stages:

• Prefill: the model ingests the prompt and any retrieved context, computing a key-value cache. Cost scales with input token count.

• Decode: the model generates output tokens one at a time, each conditioned on everything that came before. Latency scales with output token count and model size.

• Stream or batch: tokens stream back over Server-Sent Events for chat experiences, or arrive as a single JSON response for batch use cases.

Frontier inference also includes optional steps: retrieval (for grounded answers), tool calls (for agents), and structured-output validation. Each adds latency but also reliability.

Training builds the model. Inference uses it.

Training is a one-time (or periodic) operation that consumes massive compute — thousands of GPUs running for weeks. Inference happens every time a user sends a prompt and consumes a small but non-trivial slice of GPU time per call.

The economic asymmetry matters: a frontier model might cost tens of millions of dollars to train but pennies per inference call. Volume tips the balance — at scale, inference dominates the AI compute bill.

Inference is where AI products live or die. Latency, cost, and reliability all live in the inference loop:

• Latency determines whether a chat experience feels magical or sluggish. Sub-second first-token latency is the modern bar.

• Cost scales with both input and output token volume. Caching, batch APIs, and routing simple requests to cheaper models are the main levers.

• Reliability depends on grounding, structured output validation, and retry logic. Hallucination at inference time is the most common failure mode.

For brands optimizing for AI search, inference is also where citation decisions get made. Grounded inference pulls from the index at runtime — meaning a page added yesterday can be cited today, without waiting for the next training run.

Five practices for production inference:

1. Cache prompts. Anthropic's prompt caching, OpenAI's cached inputs, and similar features cut cost 5–10× on repeated context.

2. Stream responses. First-token latency drops dramatically when you stream — even if total generation time is unchanged, the user perceives the product as faster.

3. Route by complexity. Send simple classification to Haiku / Flash / GPT-4o-mini, send hard reasoning to Sonnet / Opus / GPT-4o. Quality improves and cost drops simultaneously.

4. Bound output length. Set max_tokens aggressively and use structured outputs to keep generations terse. Most products waste tokens on filler the user never reads.

5. Validate before showing. Citation validation, schema validation, and tool-call argument validation catch the rare hallucinations that slip through. The cost of a re-roll is small compared to the cost of a wrong answer.