ChatGPT, the advanced AI language model developed by OpenAI, has been trained on a vast corpus of text to understand and generate human-like responses. However, its ability to comprehend color metaphors, such as “feeling blue” or “seeing red,” has been a subject of recent academic scrutiny. A study published in Cognitive Science by Professor Lisa Aziz-Zadeh and her team reveals intriguing insights into whether AI can truly grasp these colorful expressions beyond textual analysis.
While ChatGPT has processed billions of words about emotions and colors, it lacks the direct sensory experiences humans have, such as seeing a blue sky or a red apple. This raises critical questions: Do human experiences with color enhance understanding of color metaphors more than language alone, or is textual knowledge sufficient for both AI and humans? The study, partially funded by a Google Faculty Gift, delves into these questions, offering new perspectives and raising further inquiries.
Exploring the Limits of AI Understanding
Professor Aziz-Zadeh, the senior author of the study, explains, “ChatGPT uses an enormous amount of linguistic data to calculate probabilities and generate very human-like responses. But what we are interested in exploring is whether or not that’s still a form of secondhand knowledge, in comparison to human knowledge grounded in firsthand experiences.”
As the director of the USC Center for the Neuroscience of Embodied Cognition, Aziz-Zadeh’s research focuses on how neuroanatomy and neurocognition contribute to complex skills like language, thought, and social communication. Her study’s interdisciplinary team comprised experts from UC San Diego, Stanford, Université de Montréal, the University of the West of England, and Google DeepMind, Google’s AI research company in London.
Comparative Analysis: Humans vs. AI
The research involved large-scale online surveys with four participant groups: color-seeing adults, colorblind adults, painters, and ChatGPT. Participants were asked to assign colors to abstract words like “physics” and interpret both familiar and novel color metaphors. The results were revealing.
Color-seeing and colorblind humans showed similar color associations, challenging the hypothesis that visual perception is essential for understanding metaphors. Painters, however, excelled at interpreting novel color metaphors, indicating that hands-on experience with color deepens conceptual understanding.
“ChatGPT also generated highly consistent color associations, and when asked to explain its reasoning, often referenced emotional and cultural associations with various colors,” the study noted.
For instance, ChatGPT explained the metaphor “a very pink party” by associating pink with happiness and love, suggesting a positive atmosphere. However, it struggled with novel metaphors like “the meeting made him burgundy” and inversions like “the opposite of green.”
Implications for AI Development
This research highlights the limitations of language-only models in capturing the full spectrum of human understanding. As AI continues to evolve, integrating sensory inputs, such as visual or tactile data, may help models like ChatGPT better approximate human cognition.
“This project shows that there’s still a difference between mimicking semantic patterns and the spectrum of human capacity for drawing upon embodied, hands-on experiences in our reasoning,” Aziz-Zadeh remarked.
Future research may focus on enhancing AI models by incorporating sensory data, potentially bridging the gap between textual understanding and human-like comprehension. Such advancements could revolutionize AI’s ability to interpret and interact with the world more holistically.
Study Support and Acknowledgments
In addition to the Google Faculty Gift, the study received support from the Barbara and Gerson Bakar Faculty Fellowship and the Haas School of Business at the University of California, Berkeley. Notably, Google had no role in the study’s design, data collection, analysis, or publication decisions, ensuring the research’s independence and integrity.
