AI Inference

AI inference is the act of running a trained model on new input to produce output. Training builds the model once; inference is every use of it afterwards. Each ChatGPT reply, AI Overview, coding agent step and API call is an inference run.

At consumer scale, inference never stops. OpenAI said on October 7, 2026 that more than 1.2 billion people use ChatGPT each week, and every message, web search and agent action consumes accelerator time. Providers price that work per token. In October 2026 OpenAI lists GPT-6 Luna at $0.10 per million input tokens and GPT-6 Astra at $10, a hundredfold spread between its smallest and largest current models, while Google prices Gemini 4 Argon at an introductory $2 per million input tokens. Output tokens cost several times more than input, and cached input much less.

Reasoning models made the cost of each answer variable. They spend extra test-time compute thinking before they respond, and those thinking tokens are billed as output. Products expose this as effort settings: Claude Opus 5.5 controls thinking depth through an effort parameter, Gemini 4 Argon offers low, medium and high thinking levels, and GPT-6 in ChatGPT can start answering while it keeps thinking. OpenAI says GPT-6 Instant begins answering questions that need web search 44% sooner on average than GPT-5.6 Instant.

Serving costs fall through batching many requests together, prompt caching, quantization, speculative decoding, mixture-of-experts models that activate only part of their weights per token, and specialized hardware. NVIDIA licensed Groq's inference chip technology in December 2025 and now pairs the resulting Groq 3 LPX racks with its Vera Rubin platform for low-latency token generation. Long prompts carry their own cost. A context window of a million tokens is expensive to fill even before the model writes a word, which is why providers charge more above a threshold; OpenAI's long-context rates start above 272,000 input tokens.

Inference budgets shape AI search answers. Every web search, page fetch and extra reasoning step costs the provider money, so platforms limit how many sources a quick answer reads, while deep research modes that run for minutes can read far more. That is one reason the same question often gets fewer citations in a fast answer than in a research report. Pages that state the key fact early, in clean HTML, are cheap to extract and more likely to make the cut when a model skims only a few results. Inference settings also differ between an AI API and the consumer apps built on the same model, which is one reason API-based tracking and in-app answers can disagree. More detail on the GPT-6 rollout is in OpenAI's announcement.