Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster E in Poster Session E: Thursday, August 6, 10:30 am – 12:15 pm, Kimmel Center, Shorin & Rosenthal Rooms

Brain-Tuning Language Models on Task fMRI

Yujin Yun¹, Jiook Cha¹; ¹Seoul National University

Presenter: Yujin Yun

Fine-tuning language models on neural recordings has been shown to improve brain alignment in naturalistic settings, but whether this approach extends to task-based fMRI remains unclear. We apply LoRA brain-tuning to LLaMA-3.1-8B and Centaur-8B, a cognitive foundation model (Binz et al., 2025), using fMRI from the HCP Gambling Task (N = 1,057). Brain-tuning changes the nature rather than the overall magnitude of alignment—prediction accuracy decreases in visual cortex while increasing in reward- and decision-related areas such as amygdala, nucleus accumbens, and orbitofrontal cortex, where frozen models showed near-zero or negative correlations. These results indicate that brain-tuning on task fMRI reduces reliance on low-level visual features and gains sensitivity to reward- and decision-related neural activity absent from pretrained representations.

Topic Area: Methods, Tools, Theory & Neural Coding