What is natural language processing?
Natural language processing (NLP) is the branch of artificial intelligence that gives computers the ability to understand, interpret, and generate human language — turning messy sentences and paragraphs into something a machine can work with. From the translation tool that turns a menu into English to the spam filter that catches a dodgy email, NLP is the technology that lets software handle language the way we speak and write it, rather than forcing us into rigid commands or forms.
The core tasks — what NLP actually does
NLP breaks language into a set of reusable jobs. Translation converts text from one language to another by learning patterns across vast collections of parallel examples. Sentiment analysis reads a sentence and decides whether the tone is positive, negative, or neutral — used everywhere from brand monitoring to customer support triage. Summarisation condenses a long article or meeting transcript into a few readable sentences, keeping the key facts. Named entity recognition picks out people, places, dates, and organisations from raw text. Question answering reads a document and returns a direct answer to a natural question. All of these tasks share the same foundation: the computer needs to turn words into numbers, recognise patterns in those numbers, and then turn the result back into language.
How language models handle text — tokens as the basic unit
Modern NLP is driven by large language models, and they process text as tokens. A token is a chunk of text — typically a word or part of a word — that the model treats as a single unit. “Unbelievable” might be split into two tokens: “un” and “believable”. The model reads a sequence of tokens, builds a mathematical representation of the relationships between them, and predicts what token comes next. This next-token prediction, trained on enormous datasets, is what lets a model generate fluent sentences, answer questions, or translate between languages. Every word you type into a chatbot gets broken into tokens before the model does anything with it.
The shift brought by large language models
The field changed dramatically when models grew large enough to capture subtle patterns across whole languages. Earlier NLP systems relied heavily on handcrafted rules or small machine-learning models trained separately for each task (one for translation, another for sentiment, another for summarisation). Large language models blurred those boundaries. A single model, trained once on a vast corpus of text, can handle translation, summarisation, sentiment, and a broad range of other tasks without retraining. The same model that writes a poem can also rewrite it in a formal tone, extract the key dates from a contract, or explain a physics concept to a teenager. This generality is the core shift: NLP moved from many narrow tools to one flexible engine that understands language broadly enough to adapt on the fly.
That engine is far from perfect — models still hallucinate facts, struggle with nuance, and reflect the biases in their training data. But the principle is now established. Language models are the current engine of the field, and NLP today is largely the art of steering that engine toward a specific task, checking its output, and knowing where it still falls short.