Plain-English explanation
Before an LLM can process any text, that text has to be broken into pieces the model can process numerically; those pieces are tokens. A token is often close to a whole word, but common word fragments, prefixes, suffixes, and punctuation are frequently split into their own tokens too, and unusual words can be broken into several smaller pieces. As a rough rule of thumb, a token is a bit shorter than an average English word, so a given piece of text usually contains somewhat more tokens than it has words.
How it works
Converting text into tokens is handled by a tokenizer, trained separately from the model itself on a large body of text to build a fixed vocabulary of common word pieces. A widely used family of approaches, byte-pair encoding (BPE), builds this vocabulary by starting from individual characters and repeatedly merging the most frequently co-occurring pairs into single vocabulary entries, so common whole words end up as single tokens while rarer words get split into smaller, more frequent sub-pieces. Every message sent to a model, and every reply it generates, is a sequence of these token IDs under the hood: the model predicts, and is scored on, one token at a time, not one word or character at a time.
Adult-AI use
Tokens are the practical unit behind two things users directly experience on AI platforms: cost and length limits. Hosted models are commonly billed per token (often with separate rates for input and output tokens), which is why credits systems on companion and chat apps are typically pegged to token usage under the hood even when presented to users as a simpler "messages" or "credits" count. Tokens are also the unit the context window is measured in, so a long roleplay session, a detailed persona description, and any retrieved RAG content are all quietly competing for the same fixed token budget on every single message.
How it differs from context window and temperature
Token is the unit; context window is the total capacity measured in that unit; temperature is a setting that affects how the next token is chosen once the model is already looking at everything in its context window. None of the three do the same job: token is what's counted, context window is the limit on how much can be counted at once, and temperature shapes the selection process token by token.
Limitations and misconceptions
- A token isn't a fixed unit of meaning across languages. The same sentence can tokenize very differently (and use more or fewer tokens) depending on the language and the specific tokenizer a model uses.
- Token count isn't the same as word count or character count. Estimating cost or context usage from word count alone is only an approximation; actual token counts for the same text can vary noticeably between different models' tokenizers.