Keynotes | K&Ts | GACs | Talks | Posters | Search
Poster B in Poster Session B: Tuesday, August 4, 2:00 – 3:45 pm, Kimmel Center, Shorin & Rosenthal Rooms
Brain Foundation Model Representations of Naturalistic Emotional Dynamics
Elena Skoullou1,2, Tatsuya Daikoku1, Masaki Tanaka1,2, Toyotaro Suzumura1, Yasuo Kuniyoshi1; 1The University of Tokyo, 2Tokyo Institute of Technology
Presenter: Elena Skoullou
Emotional experience is thought to arise from distributed and temporally evolving brain activity. Recent advances in brain foundation modeling have led to models that can support multiple downstream prediction tasks, but it remains unclear whether these representations capture affect-related information during naturalistic emotional stimulation, such as movie viewing. We evaluated BrainLM, a masked-autoencoder transformer pretrained on large-scale fMRI data from the UK Biobank. Finetuning and evaluation were performed using a dataset of fifteen participants who viewed the full two-hour audio-visual film Forrest Gump along with movie-aligned character-level portrayed-emotion annotations. Parcel-wise activity from 424 AAL parcels was segmented into 200-TR windows and used as input to BrainLM. We compared normalized parcel-level activity, frozen BrainLM embeddings, and LoRA-finetuned BrainLM CLS-token embeddings for leave-one-subject-out prediction of valence, arousal, and categorical emotion labels. BrainLM embeddings supported higher affective prediction performance than normalized parcel time series using simple linear Ridge regression, and LoRA fine-tuning further improved performance across targets. To examine how affect-related information was organized in the embedding space, we implemented a perturbation-based interpretability analysis estimating which parcels most strongly shifted CLS embeddings along valence and arousal dimensions. High-sensitivity parcels were summarized using sparse weighted graphs and graph metrics across intrinsic brain networks. Valence-related embedding shifts were associated with visual, salience, and executive networks, whereas arousal-related shifts involved broader contributions across sensorimotor, executive, default mode, and subcortical networks. These findings suggest that pretrained brain foundation model embeddings contain decodable affect-related information and that perturbation analysis can provide a model-based summary of embedding sensitivity on a brain network level.
Topic Area: Methods, Tools, Theory & Neural Coding