Transformer Architecture

The transformer is the neural network architecture behind modern large language models. It processes a sequence of tokens by letting every position weigh its relationship to every other position through self-attention, instead of reading the text one step at a time. It was introduced by a team of Google researchers in the June 2017 paper Attention Is All You Need.

A transformer turns each token into a vector (embeddings), adds information about its position, and passes the sequence through a stack of identical layers. Each layer has two parts. Multi-head self-attention computes, for every token, how much to draw on each other token, which lets the model resolve meaning from context: in 'she sat on the river bank', attention links 'bank' to 'river'. A feed-forward network then transforms each position. Because all positions are processed at once rather than in order, transformers train efficiently on GPUs, which is what made scaling to billions of parameters practical where earlier recurrent networks stalled.

Three layouts are common. Encoder-only models such as BERT read the full input in both directions and are used for understanding tasks like classification, search ranking and embeddings. Decoder-only models predict the next token and power nearly all chat assistants. Encoder-decoder models, the layout of the original paper, map one sequence to another, as in translation.

The main weakness is cost at long lengths. Attention compute grows with the square of the sequence length, and the key-value cache that stores past tokens grows with every token in the context window. That is why million-token contexts are expensive, and it drives most current architecture research. OpenAI, Anthropic and Google do not publish full architecture details for GPT-6, Claude Opus 5.5 or Gemini 4 Argon, but open models show the direction. Mixture-of-experts layers route each token through a few specialist sub-networks so only a fraction of parameters is active. Hybrid designs replace most attention layers with state-space layers such as Mamba-2: NVIDIA's Nemotron 3 Super, released in 2026, has 120 billion parameters with 12 billion active and uses mainly Mamba-2 blocks with a smaller number of attention layers. These hybrids keep some attention, so they are better described as modified transformers than as replacements.

For AI search, transformers sit on both sides of the answer. The generating model is a decoder, and the retrieval stack that picks sources usually relies on transformer encoders: embedding models for semantic search and cross-encoder models for reranking. These score passages by meaning rather than exact keyword overlap, so a passage about 'ESG screening for index funds' can match a query about 'sustainable investing' with no shared words. In practice that rewards text that states its subject and relationships explicitly within a passage, and it makes keyword repetition pointless. Google has used transformer encoders to interpret queries in Search since it rolled out BERT in 2019.