What is a token?
A token is a piece of text that an AI language model can read, compare and produce. Tokenisation is the step that turns your words, spaces, punctuation and symbols into those pieces before the model works with them. Think of tokens as the LEGO bricks that models use to build sentences and ideas: the model does not handle a prompt as a smooth line of writing, but as a sequence of reusable parts. That sequence shapes how much text the model can consider, how it interprets unusual wording, and how it generates its reply.
How text is split into tokens
A tokenizer is the software that breaks text into tokens. It uses a vocabulary, which is a stored set of text pieces the model knows how to handle. Some tokens look like whole words. Others are parts of words, bits of punctuation, spaces, symbols, or fragments used in code.
This is why tokenisation is not the same as counting words or letters. A common short word may stay together. A longer or less familiar word may be split into smaller parts. A made-up name, a spelling mistake, or a technical term may be broken up in a way that looks odd to a human reader but is useful to the model.
After tokenisation, each token is represented inside the system as an identifier. The model works with those identifiers rather than with the visible text directly. When it replies, it chooses the next token from its vocabulary, then repeats that process until the answer is complete or the system stops generation.
Why tokenisation matters
Tokens matter because they set the practical size of what a model can handle. AI systems have a context window, meaning the amount of tokenised text they can consider during a request. Your prompt, pasted material, conversation history and the model’s reply all draw from that space.
If the text is too large for the available context, the system must deal with that limit. Depending on the tool, it may reject the request, shorten what it includes, or leave out earlier material. This is why a model can seem to “forget” something from a long conversation: the information may no longer be inside the text it can currently see.
Tokens also affect cost and speed in many AI services, because usage is often measured by token volume. Clear, compact writing usually gives the model more useful context to work with.
How tokens affect meaning
Tokenisation can influence how a model reads nuance. Common patterns are easier for the model to recognise because they appear often in its training data. Rare words, unusual spellings, mixed languages, code, emojis and specialist terms may be split into less familiar pieces.
That does not mean the model cannot understand them. It means the route from text to meaning becomes less direct. If a prompt contains dense jargon, inconsistent spelling, or pasted material with strange formatting, the model may spend more of its context on structure rather than substance.
For ordinary use, you do not need to manage tokens by hand. It helps to know that they exist. When a prompt is too long, shorten repeated instructions, remove irrelevant background, and keep the most important context close to the request. That gives the model cleaner bricks to build with.