Training vs inference

Training and inference are two separate jobs, done at two different times, at two wildly different costs. Training happens once: a model works through vast amounts of data and adjusts itself until it gets good at predicting text. Inference happens every time afterwards: the trained model reads a prompt and generates a reply. One is rare and expensive; the other is routine and comparatively cheap. Confusing the two is why people expect a model to “remember” a conversation permanently, or wonder why a tool that costs so little to use apparently cost so much to build.

Training

Training is how a model learns in the first place. It is fed enormous quantities of text (and increasingly other data types) and repeatedly adjusted so its internal predictions get closer to correct. This process runs on large clusters of specialised computing hardware over an extended stretch of time, and it is, by a wide margin, the most expensive part of a model’s existence.

Crucially, training has an end point. Whatever the model absorbed up to that point is what it knows — and nothing after. This is where a model’s knowledge cutoff comes from: a fixed date beyond which it simply has no information, because training had already finished before those events occurred. Once training ends, the model itself is frozen. Its internal settings do not change again unless someone deliberately retrains or fine-tunes it, which is itself a new, separate training run.

Organisations such as NIST and Stanford HAI, which track the resources behind AI development, treat training as the phase where the bulk of computing effort and cost is concentrated — a one-off outlay that produces a reusable, static artefact: the finished model.

Inference

Inference is everything that happens after training, every single time someone uses the model. You send a prompt, the model runs it forward through its already-learned internal settings, and it produces a reply. Nothing about the model changes as a result. It does not update itself, remember you specifically, or fold your conversation back into its knowledge — it simply computes an answer and forgets the input once the session ends (unless a separate product feature stores that history for you).

This is the phase people actually experience, whether they are typing into ChatGPT, Claude, Gemini, Copilot or any other assistant. It is also what determines the practical things users notice day to day: how fast a reply appears, how much computing a provider must run to serve millions of requests, and what running the service actually costs its operator per use. Inference is repeated an enormous number of times for every single training run, which is exactly why it is engineered to be as fast and lightweight as possible compared with training.

Why the split matters

Separating training from inference explains several things that otherwise seem strange about AI assistants.

It explains why a model does not learn from your chats the way a person would from a conversation. Learning, in the technical sense, only happens during training. Inference is pure application of what was already learned — however useful or humanlike the exchange feels, the model itself leaves it unchanged.

It explains the knowledge cutoff: any assistant’s grasp of the world stops wherever its training data stopped, regardless of how recently you happen to be chatting with it.

It also explains a pattern people notice about pricing structures generally: a model can be relatively cheap to run per use while having been extraordinarily expensive to build. That is not a contradiction. Training cost is paid once, upfront, by whoever builds the model. Inference cost is paid, in some form, per use, at a far smaller scale each time. The two are accounted for completely differently, which is why researchers and industry bodies analysing AI’s resource use, such as Stanford HAI, generally report training compute and inference compute as separate figures rather than combining them.

An analogy

Training is like writing a textbook: a long, effortful, one-off project that draws on a huge range of source material before a single copy reaches a reader. Inference is like a tutor answering questions out of that finished textbook — quick, repeatable, and done fresh each time someone asks, without the tutor rewriting the book itself. The book only gets rewritten if someone commits to a whole new edition, which is what retraining or fine-tuning a model amounts to.

FAQs

Does a model keep learning while I talk to it?

No. What researchers commonly call “learning” happens during training, not during a live conversation. During inference the model applies its already-fixed internal settings and does not permanently update them from your input.

Why do AI models have a knowledge cutoff?

Because training uses a fixed batch of data collected up to a certain point, and the model’s knowledge is frozen at whatever that data contained. Anything that happened after training finished is simply outside what the model was ever shown.

Is inference always cheaper than training?

Per use, yes — inference for a single reply uses far less computing than training the model did. But because inference happens an enormous number of times across all users, the combined computing cost of inference over a model’s lifetime can still be substantial; NIST and Stanford HAI generally report the two as separate categories rather than comparing them directly.