Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster B in Poster Session B: Tuesday, August 4, 2:00 – 3:45 pm, Kimmel Center, Shorin & Rosenthal Rooms

Semantic efficiency and human-alignment in multilingual multimodal large language models

Nathaniel Imel¹, Sofie Chung², Selena Zheng¹, Dorothy Peters³, Thomas A Langlois⁴, Noga Zaslavsky¹; ¹New York University, ²Boston Children's Hospital, ³Wellesley College, ⁴University of Texas at Austin

Presenter: Nathaniel Imel

Large language models (LLMs) have achieved staggering linguistic competence, but it remains unclear to what extent they can acquire human-aligned semantic representations, especially in perceptually-grounded domains. It is believed that human semantic systems are shaped by a drive toward communicative efficiency. LLMs, however, are not trained for this objective, raising the question: Can these models recover efficient, human-aligned semantic systems? We address this question by studying the naming patterns of multilingual multimodal LLMs (MLLMs) across two domains, color and visual household objects, and comparing them with the naming patterns of native English and French speakers. In both domains, we find that: (1) models tend to be efficient but vary substantially in their complexity and human-alignment, with larger models generally achieving better alignment; and (2) models exhibit language dominance, aligning better to English than French. Our work lays the groundwork for a cognitively-motivated framework for studying semantic alignment and efficiency in human-AI interaction.

Topic Area: Auditory, Speech & Language Processing