Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster D in Poster Session D: Wednesday, August 5, 2:00 – 3:45 pm, Kimmel Center, Shorin & Rosenthal Rooms

Transformers capture individual differences in reinforcement learning via in-context learning

Jacob Russin¹, Krishn Bera¹, Ellie Pavlick¹, Michael Frank¹; ¹Brown University

Presenter: Jacob Russin

Recent work has shown that neural networks trained to directly predict behavior can discover patterns not captured by cognitive models. This conclusion, however, is based on comparison to models fit at the population rather than the individual level. Population-level parameters may not capture behavior of any individual, even if the underlying model is correct. In contrast, neural networks such as transformers condition their predictions on previous observations "in context" and are thus capable of adjusting to individual differences on the fly via "in-context learning" (ICL). We reasoned that ICL may endow neural networks with the flexibility to adapt to individual differences, improving fits even when the network merely rediscovers the same cognitive model. Here, we train transformers on synthetic data generated by reinforcement learning (RL) agents and demonstrate that they can adapt to individual differences via ICL. Moreover, we show that ICL in this setting resembles hierarchical Bayesian inference, where learned weights determine a population-level prior, and ICL tunes predictions toward the likelihood as evidence is accumulated over the context. Finally, we evaluate Centaur, a large language model finetuned on human behavior, and find that it slightly outperforms individual-fit RL models, suggesting it may have learned regularities not captured by these models.

Topic Area: Memory, Learning & Knowledge Structures