Poster Presentation

Poster D in Poster Session D: Wednesday, August 5, 2:00 – 3:45 pm, Kimmel Center, Shorin & Rosenthal Rooms

The adaptive relational bottleneck: regulation of attention in the service of abstraction

Keith Thomas Murray¹, Rodrigo Carrasco-Davis¹, Jonathan D. Cohen¹; ¹Princeton University

Presenter: Keith Thomas Murray

Abstraction is a fundamental capability required to solve a wide range of problems. While large language models (LLMs) are capable of abstraction to solve many such problems (McCoy et al., 2024; Yang et al., 2025), they require as much as three orders of magnitude more data to accomplish the same levels of abstraction as humans (McCoy & Griffiths, 2025). Prior work, inspired by the architecture of the brain (Bouchacourt & Buschman, 2019; McClelland et al., 1995), has identified structural inductive biases that may help explain this difference through promoting more efficient learning of abstract representations. This is achieved by constraining the flow of information within the network to emphasize relational structure between input items while attenuating information about their specific features, a design principle referred to as the Relational Bottleneck (Webb et al., 2024; Webb et al., 2021). This can be implemented by a simple modification of the attentional heads, referred to as relational cross attention (RCA), which has been shown to achieve faster and more effective solutions on tasks that require abstraction (Altabaa & Lafferty, 2025; Altabaa et al., 2024). However, it remains unclear under which conditions a given task benefits from the inclusion of a relational bottleneck, and how a system may detect and deploy this capability. In this work, we introduce an adaptive version of RCA, termed ARCA, that regulates its relational bottleneck through learning. This mechanism is implemented as an interpolation between the standard cross-attention mechanism and RCA, and is parameterized by a scalar that can be optimized during training. We demonstrate that ARCA can learn both abstract and feature-based tasks and outperforms other attention mechanisms equipped with a positional code.

Topic Area: Methods, Tools, Theory & Neural Coding