Keynotes | K&Ts | GACs | Talks | Posters | Search

Contributed Talk Session: Thursday, August 6, 11:15 am – 12:15 pm, Skirball Theater
Poster E in Poster Session E: Thursday, August 6, 10:30 am – 12:15 pm, Kimmel Center, Shorin & Rosenthal Rooms

Data Diversity Drives the Emergence of Symbolic Mechanisms in LLMs

Melody Zixuan Li¹, Taylor Whittington Webb²; ¹McGill University, ²Université de Montréal

Presenter: Melody Zixuan Li

Recent work has identified an emergent three-stage symbolic architecture in large language models (LLMs) supporting abstract reasoning. What training conditions drive its emergence? We train transformers from scratch on abstract sequence tasks and test how data diversity shapes symbolic mechanisms. Using a combination of mechanistic interventions, and behavioral assessments of systematic (i.e., out-of-distribution) generalization, we find that both mechanistic and behavioral signatures of symbol processing scale with data diversity, suggesting that this may be a key factor driving the emergence of symbolic computation in LLMs.

Topic Area: Methods, Tools, Theory & Neural Coding