Understanding Transformers
How do transformer models revolutionize the way machines understand language compared to earlier neural network approaches?
Imagine reading a sentence and instantly knowing which words matter most. Transformer models bring this human-like “attention” to AI, enabling smarter, faster language understanding. Ready to discover how?
Learning to Pay Attention
Transformer models use a self-attention mechanism to find dependencies between inputs and outputs. To understand what that means, you first need to know what attention and self-attention are in machine learning. Read on!
The mathematical technique that transformer models use is called attention. The goal of attention is to allow the model to focus on important parts of the input while generating its output. As people, we do this naturally. When you read a sentence or look at an image, you can easily see which parts of the sentence or image are the most important in terms of understanding it.
When the idea of attention first became popular, it was combined with recurrent neural networks (RNNs). But RNN models have to consider words in sequence one at a time, which is a slow process. Even worse, RNNs tend to assign more importance to recent words and the ends of sentences.
Another side effect of considering words one at a time is that you lose important context. Consider this sentence:
The player swung his bat, and he ran to first base.
Human readers naturally link pronouns to their referents, but traditional language models had to work word-by-word, often missing these connections.
Anyone with a passing familiarity with baseball will know that bat refers to a baseball bat and that he refers to the player. Considered one word at a time, however, a language model can’t make the same connections.
This is where the idea of self-attention comes in. Self-attention allows a model to learn information about an input sequence from the input sequence itself. With self-attention, a transformer model finds relationships between the words player and bat as well as between player and he.
The visualization in Figure 5-8 was created using a tool called BertViz. You can learn more about how BertViz works and try it out yourself at https://github.com/jessevig/bertviz.
Self-attention was first proposed as a solution for improving large language models in the paper “Attention Is All You Need.” This paper, published in 2017, led to the giant leaps forward in transformer models that we’ve seen in recent years.
Why do you think it’s important for language models to consider relationships between words, rather than just processing them in order?
AI models read and understand language exactly like humans, word by word.
Traditional models process text sequentially, but transformer models use self-attention to find relationships between words across the entire input.
Getting Tokens
You may be surprised to learn that language models can understand only numbers; they don’t read or understand words as we do. When you ask an NLP system a question, your input must first be converted into a sequence of numbers called tokens. You can think of tokens as the language equivalent of pixels in an image.
These tokens are read by a model’s input layer and then processed through the hidden layers to predict and output responses to your prompts.
Hidden layers are the layers between the input and output layers that process data and learn features of it.
Text can be converted to tokens in a variety of ways. One common method is word tokenization, which simply creates a token for each word in the text. Transformer models use sub-word tokenization, which converts text into common sequences of characters, such as token and ize. Using smaller units improves the model’s performance.
Sub-word tokenization allows AI models to handle new words and spelling variations, making them more robust in real-world language applications.
To see how OpenAI converts text to tokens, go to OpenAI’s Tokenizer at https://platform.openai.com/tokenizer. Figure 5-9 shows the list of tokens that were generated when I entered the sentence Transformer models use sub-word tokenization.
After the text is tokenized, it’s encoded into token IDs. The token IDs created from the text in Figure 5-9 are shown in Figure 5-10.
![A screenshot displaying the token IDs generated by the OpenAI tokenizer for a given text. It shows that the input text was broken down into 8 tokens, represented by the numerical sequence: [47458, 4211, 1005, 1207, 38428, 4037, 2065, 13].](/wp-content/uploads/course-media/course_02_Book_1_Understanding_AI_Foundations/media/9781394341726-fg010510.png)
Token IDs are input into a neural network as a sequence of numbers called a numerical vector.
Being aware of tokens and the tokenization of your input is important. If you’re using a model through OpenAI’s API, the combined number of tokens in your input and the model’s response determines how much you pay for the service.
How does sub-word tokenization help transformer models handle language more flexibly than word-level tokenization?
Want to go deeper? The science behind self-attention
Self-attention works by assigning weights to every word in the input sequence, allowing the model to consider all relationships simultaneously. This enables transformers to capture long-range dependencies and context, which traditional models struggled with. The technique is fundamental in modern NLP, powering models like BERT and GPT.
A mechanism that enables AI models to focus on the most relevant parts of the input when generating output.
A numerical representation of a segment of text, such as a word or sub-word, used by language models for processing.
Practitioners often monitor token usage closely in production systems, since exceeding token limits can impact both performance and cost.
See how a simple sentence is broken down into tokens and token IDs.
- Go to OpenAI’s Tokenizer.
- Enter the sentence: Transformer models use sub-word tokenization.
- Observe how the tool divides the sentence into tokens and then token IDs.
- Try a phrase with an uncommon word or spelling variation. What changes?
Think about how token limits could affect a conversation with an AI model. What might happen if your prompt or the conversation gets too long?
What is self-attention?
Tap to revealA mechanism where a model learns relationships within an input sequence by considering all parts at once.
Define tokenization.
Tap to revealThe process of converting text into numerical tokens for AI models to process.
What is a token limit?
Tap to revealThe maximum number of tokens a language model will consider during a conversation or prompt.
- Transformer models use self-attention to understand relationships between words.
- Tokenization converts text into numbers for AI models to process.
Transformer models revolutionize language processing by using self-attention to capture relationships across entire input sequences, enabling much more advanced and nuanced understanding than previous models.
Tokenization is the essential step that converts human language into numerical data for AI models, making modern NLP possible.
Self-attention allows a model to learn information about an input sequence from the input sequence itself.
How might self-attention improve the accuracy and usefulness of AI-generated responses in real-world applications?
Reflect on a time when you misunderstood a sentence because you didn’t have enough context. How do you think AI models overcome this challenge with self-attention?
What is the primary purpose of self-attention in transformer models?
The Shift
- Transformer models use self-attention to understand relationships across an entire input, not just word by word.
- Tokenization is the key process that enables AI models to convert text into numerical data for processing.
- Understanding token limits and sub-word tokenization is crucial for effective use of modern NLP systems.