Natural language processing (NLP)
Definition
Natural language processing is the AI discipline that enables computers to understand, interpret, and generate human language. It spans tasks such as translation, summarization, sentiment analysis, entity recognition, and question answering. Once driven by hand-built rules and statistical models, NLP is now dominated by large language models built on the transformer architecture.
How it works
NLP turns unstructured human language into something a machine can work with. Text is first tokenized into units, then represented numerically, historically with counts and statistics, and today with learned embeddings that capture meaning. Models then perform tasks on those representations.
Core NLP tasks include classification such as sentiment analysis, extraction such as named entity recognition, sequence tasks such as translation and summarization, and generation such as answering questions. Information retrieval techniques like keyword and vector search also sit within the broader NLP toolkit.
The field has shifted dramatically. Early systems relied on hand-crafted rules, later ones on statistical methods, and modern NLP is dominated by large language models that handle many tasks with a single general model rather than a separate pipeline per task.
Why it matters
NLP is the bridge between human language and machines, and it underpins the interfaces people now use to find information. Search engines, chat assistants, voice assistants, and answer engines all depend on NLP to interpret a query and produce a useful response.
For search and content strategy, NLP determines how systems understand intent and meaning rather than just matching keywords. Algorithms that parse the language of a query and the language of your content decide what surfaces, so understanding NLP clarifies how modern, meaning-based search actually works.
Frequently asked questions
What is the difference between NLP and a large language model?
NLP is the broad field concerned with computers and human language. A large language model is a particular, very capable technology that now performs many NLP tasks. LLMs are a tool within NLP, not a replacement for the discipline.
What are common NLP tasks?
Common tasks include translation, summarization, sentiment analysis, named entity recognition, question answering, text classification, and search. Modern large language models can handle many of these with a single general-purpose model.
How did large language models change NLP?
They replaced task-specific pipelines with general models that handle many tasks through prompting. This raised performance sharply on language understanding and generation, and made it possible to tackle new tasks with little or no task-specific training data.
Why does NLP matter for search?
Search increasingly understands meaning and intent rather than just matching keywords. NLP techniques let systems interpret a query's language and the language of content, which determines what gets surfaced and cited in modern and AI-powered search.
Machine learning
Machine learning is the subset of AI in which systems learn patterns from data to make predictions or decisions, rather than following explicitly programmed rules. By training on examples, models improve at tasks like ranking, classification, recommendation, and language understanding. It is the foundation beneath modern AI, including the large language models that power AI search.
Large language model (LLM)
A large language model is an AI system trained on vast amounts of text to understand and generate human language. Built on transformer architecture and containing billions of parameters, LLMs predict the next token in a sequence, enabling them to answer questions, write, summarize, and reason. They power modern chat assistants, AI search, and autonomous agents.
Transformer architecture
The transformer is the neural-network architecture behind modern large language models. Introduced in 2017, it uses self-attention to weigh how strongly each token relates to every other token in the context, letting models capture long-range meaning and process sequences in parallel. This design made today's LLMs and multimodal models possible.
Embeddings
Embeddings are numerical vector representations of text, images, or other data that capture semantic meaning. By mapping content into a high- dimensional space where similar items sit close together, embeddings let AI systems compare meaning mathematically — powering similarity search, retrieval, clustering, and recommendation.
Google BERT algorithm
The Google BERT algorithm is a natural-language model — Bidirectional Encoder Representations from Transformers — that Google rolled into Search in October 2019 to better interpret the full context of a query rather than reading it word-by-word. BERT is now part of the foundation that AI Overviews and AI Mode build on, making it the bridge between traditional SEO and 2026's generative search.
Vector search
Vector search is a retrieval method that finds information by comparing numerical meaning representations called embeddings, rather than matching exact keywords. Queries and documents are converted to vectors, and the system returns items whose vectors are closest in space — surfacing semantically relevant results even when the wording differs.