How can artificial intelligence interact with the world in ways that are both useful and understandable to humans?
Imagine an AI that never interacts with the world—it wouldn’t be able to help, learn, or even be noticed. To be truly useful, AI must bridge the gap between computation and real-world experience, engaging with humans and environments in compelling ways.
Some modern AIs can “smell” chemicals using sensors—helping detect explosives and hazardous substances more quickly than traditional methods.
An AI that is self-contained and never interacts with the environment is useless. Of course, that interaction takes the form of inputs and outputs. The traditional method of providing inputs and outputs is directly through data streams that the computer can understand, such as datasets, text queries, and the like. However, these approaches are hardly human-friendly, and they require special skills to use.
Interacting with an AI is increasingly occurring in ways that humans understand better than they understand direct computer contact. For example, input occurs via a series of microphones when you ask Alexa a question. The AI turns the keywords in the question into tokens it can understand. These tokens then initiate computations that form an output. The AI tokenizes the output into a human-understandable form: a spoken sentence. You then hear the sentence as Alexa speaks to you through a speaker. In short, to provide useful functionality, Alexa must interact with the environment in two different ways that appeal to humans, but that Alexa doesn’t actually understand.
Voice assistants like Alexa and Google Home interpret spoken language, transforming what you say into data that AI can process—and then responding in ways humans intuitively understand.
What are some everyday examples where AI interacts with humans in ways that feel natural?
Interactions can take many forms. In fact, the number and forms of interaction are increasing continually. For example, an AI can smell (see “Artificial intelligence grows a nose” at www.science.org/content/article/artificial-intelligence-grows-nose). However, the computer doesn’t actually smell anything. Sensors provide a means to turn chemical detection into data that the AI can then use in the same way it uses all other data. The capability to detect chemicals isn’t new; the ability to analyze those chemicals isn’t new; nor are the algorithms used to interact with the resulting data new. What is new is the datasets used to interpret the incoming data as a smell, and those datasets come from human studies. An AI’s nose has all sorts of possible uses. For example, think about the AI’s capability to use a nose when working in some dangerous environments, such as to smell explosives at an airport before being able to see them by using other sensors.
Practitioners are finding new ways for AI to sense the environment, such as using chemical sensors in factories or airports to detect hazardous substances—even before humans can see or smell them.
Want to go deeper? The science behind AI “sensing” abilities
AI doesn’t literally “smell” or “hear”—it relies on sensors to convert physical phenomena (like sounds, smells, or movements) into data. Advanced machine learning models then interpret these signals, using large datasets built from human studies to create associations (such as mapping chemical data to specific odors). These systems can quickly analyze environmental data, offering fast responses in situations where human senses or traditional tools may be too slow or limited.
How might AI’s ability to detect chemicals or interpret environmental signals change safety and security in public spaces?
Physical interactions are also on the rise. Robots that work on assembly lines are old hat, but consider the effects of robots that can drive. These are larger uses of physical interaction. Consider also that an AI can react in smaller ways. Hugh Herr, for example, uses an AI to provide interaction with an intelligent foot, as described in “Is This the Future of Robotic Legs?” at www.smithsonianmag.com/innovation/future-robotic-legs-180953040/ and “New surgery may enable better control of prosthetic limbs” at MITNews.edu. This dynamic foot provides a superior replacement for people who have lost their real foot. Instead of the static sort of feedback that a human receives from a standard prosthetic, this dynamic foot provides the sort of active feedback that humans are used to obtaining from a real foot. For example, the amount of pushback from the foot differs when walking uphill than walking downhill. Likewise, navigating a curb requires a different amount of pushback than navigating a step.
A device that detects physical phenomena (such as sound, chemicals, or movement) and converts them into data that AI can analyze.
The process of converting raw input (like spoken language) into structured data “tokens” that AI systems can process and understand.
- AI systems can interact with the environment through various sensors and interfaces.
- Human-friendly modes of interaction, such as voice and touch, are becoming more common.
What challenges might arise when AI interacts with the environment in ways that humans do not fully understand?
The point is that as AI becomes more able to perform complex calculations in smaller packages with ever-larger datasets, the capability of an AI to perform interesting tasks increases. However, the tasks that the AI performs may not currently have a human equivalent. You may not ever truly interact with an AI that understands your speech, but you may come to rely on an AI that helps you maintain life — or at least make it more livable.
AI systems directly experience the world just like humans do.
AI relies on sensors and data streams to interpret the environment; it does not have human-like perception, but can analyze signals and perform useful tasks based on them.
AI systems use sensors, tokenization, and large datasets to interact with the environment, enabling them to perform tasks and deliver outputs in ways that are increasingly understandable and useful to humans.
As AI evolves, its ability to perform complex tasks based on environmental interactions is expanding, often in ways that do not currently have direct human equivalents.
“You may not ever truly interact with an AI that understands your speech, but you may come to rely on an AI that helps you maintain life — or at least make it more livable.”
Identify ways AI interacts with the environment in your daily life.
- List three different devices or services you use that involve AI interacting with the world (e.g., voice assistants, smart appliances, self-checkout systems).
- For each, note what types of inputs (e.g., touch, speech, sensors) and outputs (e.g., voice, screen responses, actions) are involved.
How would you design an AI system to interact with a new type of environment—such as underwater or in space? What sensors and outputs would be most effective, and how would you ensure the AI’s responses are meaningful for human users?
What is tokenization in AI interaction?
Tap to revealThe process of converting raw input, such as speech or text, into structured data tokens that AI can understand and process.
How do AIs “smell” chemicals?
Tap to revealAIs use sensors to detect chemicals and turn those signals into data, which are then analyzed using human-built datasets to interpret them as specific smells.
What is a dynamic prosthetic foot?
Tap to revealA prosthetic foot that uses AI to provide active feedback, adjusting pushback depending on conditions like walking uphill or downhill, much like a real human foot.
Which method enables AI systems to interpret human speech for interaction purposes?
How confident are you that you can explain how AI uses sensors and tokenization to interact with the environment?
The Shift
- AI systems are increasingly interacting with the environment in human-friendly ways, such as speech and sensory input.
- Sensors and tokenization allow AI to interpret signals and respond in ways humans understand—even if the AI itself doesn’t perceive like a human.
- As AI advances, its ability to perform complex environmental tasks is expanding beyond solely human capabilities.