Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster B in Poster Session B: Tuesday, August 4, 2:00 – 3:45 pm, Kimmel Center, Shorin & Rosenthal Rooms

Do Social Micro-processes Emerge in Multimodal Language Models? A Case Study of Egocentric Referential Communication

Dota Tianai Dong¹, Julian Jara-Ettinger², Jennifer Hu³, Paula Rubio-Fernandez¹; ¹Max Planck Institute for Psycholinguistics, ²Yale University, ³Johns Hopkins University

Presenter: Dota Tianai Dong

Have we built MLMs that truly communicate like people? We investigate whether MLMs exhibit a behavioral signature of human social micro-processes—temporally bound mentalistic representations that support real-time communication by allowing interlocutors to reason about shared context without fully modeling one another’s minds. We treat demonstrative use (e.g., THIS and THAT in English) as such a signature, since it requires reasoning about the physical distance between a speaker’s position and the intended referent. We evaluate four MLMs in egocentric communication. Our preliminary results show that only advanced models produce demonstratives consistent with the expected near–far distinction, providing behavioral evidence of an emerging signature of social micro-processes. However, these models remain insensitive to the speaker’s viewpoint, struggle to correctly interpret egocentric descriptions of speaker location, and fail to generalize to third-person perspective-taking.

Topic Area: Auditory, Speech & Language Processing