How do large language models actually work?

A language model lives in two separate phases, and almost every quirk people notice about tools like ChatGPT, Gemini or Claude traces back to that split. First comes training: a one-off, extremely expensive process where the model reads enormous amounts of text and adjusts billions of internal numbers until it gets good at predicting what comes next. Then comes inference: the fast, repeated process that runs every time you send a message, using the model exactly as training left it, without changing it further. Training happens once and stops; inference happens millions of times a day. Keep that boundary in view and most of what feels strange about these systems — the outdated facts, the forgetfulness, the fact that correcting it once doesn’t stick — stops being strange.

Phase one: training

During training, a model is shown vast quantities of text and repeatedly asked, in effect, to guess the next piece of that text. Each guess is checked against what actually came next, and the model’s billions of internal numbers, usually called parameters or weights, are nudged very slightly to make similar guesses more accurate in future. Repeated across huge volumes of text and huge amounts of computing power, this process is how the model comes to produce fluent, often useful text without anyone writing explicit rules for grammar or facts.

This is the framing researchers commonly use to describe how these systems are built, and organisations such as NIST and Stanford’s Institute for Human-Centered AI (HAI) frame it the same way in their public work on foundation models: a distinct, resource-intensive learning stage that produces a fixed model, followed by a separate stage where that model is used.

Training is not continuous. It runs for a period, then stops, and the resulting model is fixed until someone builds and trains a new one. That is why every model has a knowledge cutoff: a point after which it simply was not shown any more text, so it cannot know about events, releases or developments that happened afterwards. A cutoff is not a bug or an oversight — it is the direct, unavoidable consequence of training being a one-off phase rather than something that happens continuously in the background.

Phase two: inference

Inference is what happens every time you type a prompt and press send. The trained model — its parameters now fixed — runs forward over your text and produces a reply one small piece at a time, each piece chosen based on everything written so far, including its own reply as it goes. Nothing about the model itself changes during this process. It is the same fixed set of parameters being run again, on new input, over and over.

This is also the phase you actually experience and, in commercial products, the phase that is metered and paid for. The wait while a reply appears, the sense of the answer being “typed out” gradually, and any usage limits or rate limits you encounter all belong to inference, not training. Training already happened, long before you opened the chat window; what you are interacting with live is a fixed model being run, not a model being taught.

Tokens, briefly

Neither phase works directly on letters or whole words. Models read and generate text as tokens — chunks that might be a whole word, part of a word, or a piece of punctuation. That choice affects things like how a model counts length limits and why it sometimes struggles with tasks that depend on exact spelling. It is a big enough topic to deserve its own explanation; treat this as a pointer to that glossary entry rather than the full story.

Why it forgets

A model’s only memory of your conversation is the conversation itself, fed back in as text each time — what is usually called its context window. Everything you and the model have said sits inside that window and is available when it generates its next reply. Once a conversation grows past that window’s limit, or once you start a new conversation entirely, earlier material is no longer there for the model to draw on. It is not that the model chooses to forget; it genuinely cannot see text that has fallen outside what is being fed to it. This is also why a model has no memory of a different conversation with a different person: each session’s context window is its own, self-contained world.

Why it can’t learn from your chat

Correcting a model mid-conversation — telling it it made an error, giving it a fact it lacked — clearly changes what it says next in that same conversation, because the correction is now sitting in its context window. What it does not do is change the model’s underlying parameters. Ordinary use of a chat product is inference, not training, and inference by definition runs the model without altering it. Your correction helps the reply you get next; it leaves nothing behind for the next person who opens a new chat, and it leaves nothing behind for you either, once the conversation ends. Any lasting improvement to the model itself would require a further, separate training phase, built deliberately by whoever develops the model — not something that happens as a side effect of people talking to it.

Researchers describe concepts like whether a model “understands” or “reasons” about what it produces as genuinely contested, rather than settled, and the terms in this guide are used in that same careful, descriptive sense.