Definition
A small language model (SLM) is a language model compact enough to run on a phone, a laptop or a single modest GPU, usually somewhere between a few hundred million and about 10 billion parameters. There is no formal cutoff. The label describes a design goal: low cost, low latency and the option to run locally, accepting less general knowledge and weaker multi-step reasoning than frontier LLMs.
The 2026 generation shows how far the category has moved. Google released Gemma 4 in April 2026; its E2B and E4B variants use about 2.3 and 4.5 billion effective parameters, accept text, images and audio, support a 128,000-token context and are built for phones and edge devices (Gemma 4 on Google AI Edge). Google says Gemini Nano 4, the model Android exposes to apps on supported phones, is based on Gemma 4. Apple announced its third generation of foundation models in June 2026, including AFM 3 Core, a dense 3 billion parameter on-device model, and AFM 3 Core Advanced, a 20 billion parameter sparse model that activates only 1 to 4 billion parameters per request (Apple Machine Learning Research). Microsoft's Phi-4-mini (3.8 billion parameters) and Alibaba's Qwen3.5 small series (0.8 to 9 billion) are widely used open-weight options.
SLMs reach this quality mostly through better data and training rather than size. Distillation from larger teacher models is common, and synthetic data plays a large role: Microsoft's Phi-4 technical report states that synthetic data made up 40% of its pretraining mix. Quantization (storing weights at 4 bits instead of 16) shrinks memory further so models fit in a phone's RAM.
Where SLMs are used today:
- On-device assistants and OS features: summarizing notifications, rewriting text, extracting details from messages, all without sending data to a server.
- Routing: a small model decides whether a request can be handled locally or needs a larger cloud model. Apple's Foundation Models framework now lets developers swap between the on-device model, Apple's Private Cloud Compute model and third-party cloud models such as Claude and Gemini through one Swift API.
- High-volume pipelines: classification, tagging, reranking and query rewriting inside search and RAG systems, where cost per call matters more than eloquence.
- Domain tasks: a fine-tuned SLM can match a general model on narrow work like support ticket triage.
For AI search and GEO, SLMs matter less as answer engines and more as infrastructure. Small models hold little factual knowledge and are prone to hallucination when asked about specific brands without retrieval, so consumer products typically hand factual questions to a larger model with web access. Where SLMs do touch your content is in the steps around retrieval: rewriting a user's question into search queries, filtering and ranking retrieved passages, and summarizing a page in a browser or on a phone. Pages with a clear structure, short self-contained passages and facts stated plainly survive those compression steps better than long, meandering copy. See open-source LLMs, AI inference and foundation models for the wider model picture.
Examples of Small Language Models (SLMs)
- A note-taking app uses Apple's on-device foundation model to summarize meeting notes offline, and hands longer research questions to a cloud model through the same Foundation Models framework API.
- An Android banking app runs Gemini Nano on supported phones to draft replies to customer messages locally, so message text never leaves the device.
- A support team fine-tunes a 4 billion parameter open-weight model to route tickets into 30 categories. It matches their previous cloud model on accuracy at a fraction of the per-ticket cost.
- A RAG pipeline uses a small model to rewrite each user question into three search queries and to rerank 50 retrieved passages, then passes the top five to a frontier model that writes the final answer with citations.
