Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster B in Poster Session B: Tuesday, August 4, 2:00 – 3:45 pm, Kimmel Center, Shorin & Rosenthal Rooms

Optimizing Language Model Embeddings to Voxel Activity Improves Brain Activity Predictions

Anuja Negi¹, Christine Tseng¹, Anwar O Nunez-Elizalde², Xue Lily Gong³, Fatma Deniz¹; ¹Technische Universität Berlin, ²Independent, ³University of California, San Francisco

Presenter: Anuja Negi

Recent studies have shown that encoding models with contextual semantic embeddings from transformer-based large language models (LLMs) can accurately predict brain responses during naturalistic language processing. Most of these studies construct encoding models using contextual embeddings from a single LLM model layer and with a fixed context length across all brain regions. However, this approach overlooks meaningful variations across the brain. Different brain regions are known to support different levels of the language processing hierarchy and represent different timescales of information. Thus, contextual embeddings from a fixed model layer and context length may not accurately capture brain responses in all regions. In this study, we investigate whether optimizing the contextual embeddings for each voxel improves their ability to predict brain activity. We optimize contextual embeddings for each voxel by selecting the best predicting context length and model layer. We use an existing fMRI dataset in which four participants read narrative stories and isolated sentences. We find that voxel-specific optimization significantly improves the prediction accuracy of contextual semantic embeddings. In addition, a preliminary analysis suggests that voxel-specific optimization may change the estimated semantic tuning in many language-selective voxels. These findings show that voxel-specific optimization of contextual embeddings provides a more accurate account of how contextual semantic information is represented in the brain.

Topic Area: Methods, Tools, Theory & Neural Coding