Demystifying Natural-Language Processing
How can computers truly understand—and even generate—human language, given its complexity and nuance?
Imagine asking your phone a question and receiving a thoughtful, relevant answer. Or reading a summary of a lengthy article, generated instantly by AI. Natural-language processing (NLP) is the invisible force powering these advances—bridging the gap between human communication and machine understanding.
Natural-language processing (NLP) is the branch of AI concerned with giving computers the capability to understand human language in written and spoken form. NLP can be further divided into two subsets:
- Natural-language understanding (NLU) involves the analyzing of text and speech to figure out the meaning of sentences.
- Natural-language generation (NLG) is concerned with giving computers the capability to write.
NLP is used for a wide variety of purposes, including the following:
- Machine translation: Translating text from one language to another
- Speech recognition: Understanding spoken language and translating it into text
- Text summarization: Generating summaries of longer texts, such as books, articles, or briefs
- Question answering: Answering questions asked in natural language
- Sentiment analysis: Identifying the sentiment of text, for example, whether it’s positive, negative, or neutral
- Computer programming: Writing or assisting with the writing of computer programs
Human language includes not only the languages we use for communicating with each other (such as English, French, and Greek) but also the languages we use for communicating with computers (such as C++, JavaScript, and Haskell).
The earliest “chatbot”—ELIZA—was created in the 1960s as a joke, yet it inspired decades of research in human-computer conversation.
History of NLP
Natural-language processing has been around since the first experiments in automatic translation in the 1950s. The creators of the first automatic translation machine predicted that the problem of machine translation would be solved within a few years. However, it ended up taking much longer, and we still haven’t perfected it.
Early natural-language processing systems relied on a list of handwritten rules that the computer applied to the data it received. The ELIZA chatbot, created in the 1960s, could have a somewhat convincing conversation with a human by following rules and responding as a non-directional psychotherapist. Although ELIZA was created as a joke, it successfully demonstrated that natural-language conversations between people and computers were possible.
Machine learning algorithms and increased computing power led to advances and new techniques for natural-language processing starting in the late 1980s.
In the 1990s, Richard Wallace created the ALICE chatbot. ALICE is similar to ELIZA in that it uses pattern-matching rules. The operator of an ALICE chatbot could add rules, which made it useful for customer service applications such as answering questions about a company’s products.
The growth of the internet made large collections of text readily available for training models. As a result of all this data, new techniques were created in the 1990s and 2000s that depended on statistical analysis of large datasets.
Computers simply need a dictionary and grammar rules to understand language.
Human language is far more complex, requiring exposure to vast amounts of real-world data to grasp nuance, dialects, and context.
Machine learning models made it possible for natural-language processing systems to be trained without having to hand-code rules. Further advances in machine learning, including neural networks, made virtual assistants such as Apple’s Siri and Amazon’s Alexa possible in the 2010s and eventually enabled the remarkable advances in natural-language processing that made tools such as ChatGPT possible.
- NLP began with rule-based systems but evolved with machine learning and neural networks.
- Early chatbots like ELIZA and ALICE demonstrated the basics of computer conversation.
Overcoming the challenges of NLP
In the past, creators of NLP systems focused on breaking down human languages into rules. The idea was that if you could give a computer access to a language’s dictionary and a grammar book, it would know that language.
In the case of programming languages, this approach was somewhat successful. If you know the rules of a programming language and the right keywords to use, you can at least write statements.
But human language is more complex. Here are just a few of the things that a computer can’t understand by memorizing a dictionary and grammar:
- Dialects
- Accents
- Sarcasm
- Metaphors
- Humor
- Grammar and usage exceptions
- Homophones
The fact that a box of chips and wires can understand any of what people say is incredible. Until recently, it seemed that computers were doomed to understand only a formal and limited subset of human languages and that chatting with an AI assistant would always be a frustrating and disappointing experience.
So what changed? The biggest breakthrough in NLP was when researchers decided to throw out the rules and start teaching computers to talk the way humans teach babies to talk, namely, by exposing them to language and letting them figure things out.
What aspects of human language do you think are hardest for computers to understand, and why?
Understanding supervised and unsupervised learning
Supervised learning relies on labeled data, which is data annotated with tags that describe what the data is. For example, if you want to train a model to recognize spam email messages, you could create a dataset containing millions of email messages, each labeled either “spam” or “not spam.” However, the process of labeling data can be costly and time consuming.
Unsupervised learning is the finding of patterns in unlabeled data. With unsupervised learning, algorithms sort through unlabeled data looking for patterns. Unsupervised language-learning models can be trained on very large datasets to create large language models (LLMs). Unlike supervised learning, unsupervised learning can be done inexpensively and quickly.
Virtual assistants like Siri and Alexa use NLP techniques to recognize speech, interpret meaning, and generate natural responses—making everyday human-computer interaction possible.
Want to go deeper? The science behind supervised vs unsupervised learning in NLP
Supervised learning depends on curated, labeled datasets—such as spam/non-spam emails or positive/negative movie reviews—making it powerful but expensive. Unsupervised learning, by contrast, enables models to “learn” from vast unlabeled data, finding structure in language and enabling the creation of LLMs like GPT. Both methods underpin modern NLP but have different strengths and limitations.
Practitioners in NLP must often balance the desire for high performance with the cost and effort involved in labeling huge datasets for supervised learning.
The branch of artificial intelligence focused on enabling computers to understand and generate human language.
A machine learning model trained on huge amounts of text data to understand, generate, and analyze natural language.
How has the availability of large datasets impacted the development of NLP technologies?
Compare the conversation styles of ELIZA and ALICE. What differences do you notice?
- Review the sample dialogues with ELIZA and ALICE above.
- Make a list of the conversation patterns and responses each chatbot uses.
- What advances do you observe between the two generations of chatbots?
Why do you think NLP for programming languages is easier than for human languages?
Consider a time when you interacted with a chatbot or virtual assistant. Did it understand you accurately? Reflect on what worked well and what could have been improved.
What is Natural-Language Processing (NLP)?
Tap to revealNLP is the branch of artificial intelligence concerned with giving computers the ability to understand and generate human language.
What are the two main subsets of NLP?
Tap to revealNatural-language understanding (NLU) and natural-language generation (NLG).
What is supervised learning in NLP?
Tap to revealSupervised learning uses labeled data to train models, such as classifying emails as “spam” or “not spam”.
Define unsupervised learning in the context of NLP.
Tap to revealUnsupervised learning involves finding patterns in unlabeled data, allowing models to learn structure from large datasets.
Which of the following best describes unsupervised learning in NLP?
NLP has evolved from rule-based systems to advanced machine learning models, allowing computers to understand and generate complex human language.
The transition from supervised to unsupervised learning has enabled the rapid development of large language models, transforming the capabilities of AI in language tasks.
The fact that a box of chips and wires can understand any of what people say is incredible.
The Shift
- NLP allows computers to interpret, analyze, and generate human language—making communication with machines increasingly natural.
- Modern NLP relies on both supervised and unsupervised learning, with large language models unlocking new potential.
- The complexity of human language, including dialects, humor, and context, continues to drive innovation in AI-powered understanding.