Definition
Tokens are the units of text that a large language model reads and generates. A token can be a whole word, part of a word, a punctuation mark or a space-prefixed fragment, and a tokenizer converts text into a sequence of token IDs before the model processes it. Every limit and price in an LLM API is counted in tokens, not words.
Tokenizers are learned from data, usually with a subword method such as byte-pair encoding, so common words become a single token and rare words split into pieces. OpenAI's rule of thumb for English is that one token is about four characters or three quarters of a word, so 100 tokens is roughly 75 words. Spacing and capitalization change the split, and so does the model: each model family uses its own vocabulary. Anthropic's models overview says that 1 million tokens holds about 555,000 words on the tokenizer introduced with Claude Opus 4.7, against about 750,000 words on earlier Claude models, so the same document costs more tokens on newer Claude models. Languages other than English, code and dense numeric tables usually use more tokens per word.
Tokens matter in three practical places:
- Context limits: the context window caps input plus output tokens per request. In October 2026 Claude Opus 5.5 and Sonnet 5.5 offer 1 million tokens with 128,000 output tokens, GPT-6 Sol a 1,050,000-token window, and Gemini 4 Argon a unified 1 million token input and output limit.
- Pricing: AI APIs charge per million input and output tokens, with output priced higher. GPT-6 Sol lists $2 input and $10 output, Claude Opus 5.5 $4 and $20, and Gemini 4 Argon an introductory $2 and $10 through November 30, 2026. Cached input is discounted, and reasoning tokens from test-time compute are billed as output even though they are hidden.
- Rate limits and quotas, which providers express as tokens per minute or per day.
For AI search, tokens explain why retrieval systems rarely read a whole page in one piece. Pages are extracted, split into passages (content chunking) and converted to embeddings, and only the passages that score well for a sub-query are placed into the model's prompt, alongside passages from other sites. Navigation menus, cookie banners and repeated boilerplate in the extracted text take up tokens without adding anything. There is no evidence that a lower token count on its own earns citations. The practical point is that the passage answering a question should be self-contained and sit in clean, server-rendered HTML, so it survives extraction and chunking intact. Promptwatch's free page token inspector shows how a page's text breaks into tokens.
Examples of Tokens
- A writer pastes a 1,200-word English article into OpenAI's Tokenizer and gets about 1,600 tokens, in line with the three-quarters-of-a-word rule of thumb.
- A team moves a summarization job from an older Claude model to Claude Opus 5.5 and sees token counts rise for the same documents, because the newer tokenizer produces more tokens per word. It updates the cost forecast before switching.
- A RAG pipeline gives each source a 2,000-token budget. On one product page, the extracted text starts with 1,500 tokens of navigation and legal notices, so the specification table that answers the question is cut off.
- An agent call on GPT-6 Sol returns a 250-word answer but bills several thousand output tokens, most of them reasoning tokens that never appear in the response.
