Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster F in Poster Session F: Thursday, August 6, 1:45 – 3:30 pm, Kimmel Center, Shorin & Rosenthal Rooms

An Algorithmic Account of Few-Shot Concept Induction in Language Models

Michael A. Lepori¹, Aalok Sathe¹, Zhuonan Yang¹, Ellie Pavlick¹; ¹Brown University

Presenter: Zhuonan Yang

Both models and minds not only encode features that describe an entity, but can specifically represent relevant features for a given context. This core capability underlies in-context learning in language models and a variety of cognitive phenomena in minds, yet it is unclear how intelligent systems identify and promote relevant features. In this work, we propose Amortized Direction Accentuation (ADA), an algorithmic hypothesis that provides an explanation for this mystery in the context of few-shot concept induction---the ability to infer a latent concept from a small number of positive examples. Under ADA, concept induction proceeds by first forming a summary representation of the examples, then reweighting the features of that summary according to amortized knowledge on useful features. We investigate and validate ADA in language models using a few-shot concept induction task. This work provides a novel perspective on the mechanisms supporting contextualization in language models and a promising hypothesis for a core aspect of cognition.

Topic Area: Auditory, Speech & Language Processing