Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster D in Poster Session D: Wednesday, August 5, 2:00 – 3:45 pm, Kimmel Center, Shorin & Rosenthal Rooms

Comparing Human and Machine Communication Patterns Through a Tangram Game

Haoran Zhao¹, Colin Conwell²; ¹University of Washington, ²After Thought

Presenter: Haoran Zhao

When humans communicate about visual objects, they develop shared linguistic conventions that progressively reduce referential ambiguity through collaborative dialogue. To test whether vision-capable large language models (VLLMs) exhibit similar communicative behaviors, we compare human-human and agent-agent interactions in the tangram communication game, where two players establish shared references for abstract shapes across six repeated rounds. We analyzed existing human-human data and conducted agent-agent experiments with five VLLMs, measuring performance and using representational probes to explore the structure underlying performance. Humans demonstrate clear convention formation, with representations becoming increasingly distinguishable as accuracy improves from 78% to 96%. In contrast, AI agents showed consistently low performance, typically around 10–30%, with little evidence of convention development, despite access to conversation history, interim accuracy reports, and what appear to be largely accurate initial descriptions by “director” agents. These results suggest that current VLLMs may still struggle with grounded, evolving, coreferential communication in this collaborative setting.

Topic Area: Auditory, Speech & Language Processing