What makes generative AI models capable of creating new content, and how do their scale and training data impact their abilities?
Imagine a computer program that can write poetry, compose music, or generate realistic images—without ever having experienced the world as humans do. These feats are possible thanks to generative AI models, which use vast amounts of data to learn patterns and create entirely new outputs.
A generative AI model is a model that is trained on content (such as images or text) and can use that content to make predictions to generate new content. OpenAI’s GPT-3, the model behind the first public version of ChatGPT, is a generative model for natural-language processing. By leveraging what it’s learned about how people talk and write, it can accurately predict what the next word or sentence should be in response to input.
The world’s largest generative AI models contain more parameters than the number of stars in our galaxy!
For example, if you provide the words peanut butter and to GPT-3, it will most likely predict that the next words should be jelly sandwich. GPT-3 doesn’t have any experience with eating or making peanut butter and jelly sandwiches, but it has analyzed a lot of text, and the most common way to finish the phrase peanut butter and is with jelly sandwich.
Generative AI models understand and experience the world just like humans do.
Generative AI models learn from vast datasets; they predict patterns in data but do not have real-world experiences or consciousness.
Like the hot dog photo classifier (which you may recall from the module on output layers), a generative AI model is only as good as its training data. GPT-3 was trained with over 45 terabytes of text data, so it has amazing capabilities, such as taking context into consideration when responding to input.
Generative AI is used in chatbots, creative writing tools, image synthesis, and even drug discovery, revolutionizing industries by automating content creation and accelerating innovation.
Generative AI models have been steadily getting better as a result of being trained on more and higher-quality data. Table 5-2 compares the size of some recent generative models.
TABLE 5-2 Parameters in Generative AI Models
Model |
Developer |
Parameters |
|---|---|---|
Gato |
DeepMind |
1.18 billion |
ESMFold |
Meta AI |
15 billion |
LaMDA |
137 billion |
|
GPT-3 |
OpenAI |
175 billion |
Bloom |
Hugging Face and BigScience |
176 billion |
MT-NLG |
Nvidia and Microsoft |
530 billion |
WuDao 2.0 |
Beijing Academy of Artificial Intelligence |
1.75 trillion |
GPT-4 |
OpenAI |
1.76 trillion |
As you may recall, parameters are the values that the model has learned from its training.
Want to go deeper? The science behind generative AI models
Generative AI models use complex neural network architectures, such as transformers, to identify and learn patterns in their training data. The more parameters a model has, the more nuanced its predictions can be. However, bigger models also require exponentially more data and computing power, making their development both exciting and challenging.
See for yourself how generative AI models use learned patterns to predict outputs.
- Write a phrase such as “peanut butter and” on paper.
- Ask friends to complete the phrase. Compare their answers to the output of an AI chatbot like ChatGPT.
- Reflect: Did the AI’s answer match the most common human response?
How does the size of a generative AI model impact its ability to generate accurate or creative responses?
Why is training data quality just as important as quantity for generative AI models?
Practitioners often focus on curating high-quality datasets, knowing that even the largest generative models can produce biased or nonsensical results if their training data is flawed.
A type of artificial intelligence trained to generate new content based on patterns it has learned from existing data.
A learned value within an AI model that helps it make predictions or generate outputs, based on its training data.
- Learned what generative AI models are and how they generate new content
- Explored how model size and training data influence performance
Generative AI models have been steadily getting better as a result of being trained on more and higher-quality data.
What is a generative AI model?
Tap to revealA model trained to generate new content, such as text or images, based on patterns learned from large datasets.
What are parameters in AI models?
Tap to revealParameters are the values that an AI model learns during training, enabling it to make predictions or generate outputs.
What does training data refer to?
Tap to revealTraining data is the collection of information (text, images, etc.) used to teach an AI model to recognize patterns and make predictions.
Reflect on how generative AI models might impact industries you care about. How could their ability to generate new content transform creative work, communication, or problem-solving?
Generative AI models create new content by learning patterns from vast amounts of data, and their capabilities are determined by the quality and quantity of their training data and the number of parameters they contain.
The largest generative AI models—like GPT-4—have trillions of parameters, enabling them to generate highly sophisticated and context-aware outputs.
Which of the following best describes what “parameters” are in the context of generative AI models?
What ethical considerations might arise from generative AI models creating content that mimics human communication?
The Shift
- Generative AI models use learned patterns to create new content, not human-like understanding.
- The capabilities of generative AI depend on both the model’s size (number of parameters) and the quality of its training data.
- Understanding how these models work is key to responsibly using and advancing AI technology.