Definition
Test-time compute is the computation a language model spends at inference time, while it produces an answer, as opposed to the compute spent training it. Spending more of it, through longer internal reasoning, several candidate answers or explicit verification, lets the same model solve harder problems at the cost of more latency and more tokens.
For most of the history of large language models, capability gains came from scaling training: more data, more parameters, more GPU hours. A trained model then answered every prompt with roughly the same effort. OpenAI's o1, released in September 2024, made a different axis visible: accuracy on math, coding and science problems kept improving the longer the model was allowed to think. The research case was made around the same time by Snell et al. (August 2024), who found that allocating test-time compute adaptively per prompt was more than 4x more efficient than a best-of-N baseline, and that on easier and intermediate problems a smaller model with extra inference compute could outperform a 14x larger model in a FLOPs-matched comparison.
The budget is spent in a few ways. The model can generate hidden chain-of-thought reasoning before the visible answer. It can sample several answers in parallel and pick one by majority vote or with a verifier. It can revise a draft step by step. In agentic settings it spends compute on tool calls: running searches, reading pages and executing code. OpenAI's reasoning guide notes that reasoning tokens are not returned through the API but still occupy the context window and are billed as output tokens, so a short answer can carry a large bill.
By October 2026 test-time compute is a setting on mainstream models rather than a separate product line of reasoning models:
- OpenAI's GPT-6 models take a reasoning effort parameter; GPT-6 Astra accepts low, medium, high, xhigh and max, and ChatGPT exposes the same idea as a Thinking slider.
- Anthropic's Claude 5.5 models use adaptive thinking, where the model decides whether and how much to think, steered by an effort parameter from low to max. Fixed thinking budgets are no longer accepted on Claude 4.7 and later.
- Google's Gemini 4 Argon, released October 13, 2026, has thinking levels low, medium and high, and Google offers Deep Think as a separate specialized reasoning mode.
For AI search, much of the test-time budget goes into retrieval. A model answering a research-style prompt issues several searches (query fan-out), reads the results, and reconciles what they say before writing. Deep research modes run that loop for minutes. Two practical consequences follow. First, more sub-queries create more chances for a page to be retrieved, especially pages that answer a narrow sub-question well. Second, a model that spends tokens comparing sources can notice when pages disagree, which favors pages with specific, checkable claims over vague ones. Answers can also change with the effort level and model, so visibility tracking should record which model and mode produced each answer before comparing results over time.
Examples of Test-Time Compute
- A developer runs GPT-6 Astra at high reasoning effort for a contract-review agent and at low effort for a ticket-tagging job, because tagging gains little from extra reasoning tokens but still pays for them as output tokens.
- A team moves a research agent from Claude Opus 5.5's default medium effort to xhigh and raises max_tokens at the same time, since thinking and visible answer text share the same output limit.
- An evaluation harness samples eight answers per math problem and keeps the majority answer. Accuracy improves over a single sample, and inference cost rises roughly eightfold.
- A GEO analyst runs the same vendor-comparison prompt in ChatGPT at Instant and at Extra High and stores the results as separate series, because the two settings can retrieve and cite different pages.
