Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster F in Poster Session F: Thursday, August 6, 1:45 – 3:30 pm, Kimmel Center, Shorin & Rosenthal Rooms

Two Cortical Mechanisms for Natural Audiovisual Processing

Subha Nawer Pushpita¹, Leila Wehbe¹; ¹Carnegie Mellon University

Presenter: Subha Nawer Pushpita

Understanding how the brain processes natural audiovisual information remains a central challenge in cognitive neuroscience. Progress has been limited by reliance on short, controlled, or unimodal stimuli, leaving cortical mechanisms supporting real-world comprehension poorly characterized. While AI now enables extraction of high-dimensional, time-resolved features from naturalistic stimuli, how cortical regions dynamically process or prioritize auditory and visual information over time remains largely unexplored. Using large-scale fMRI data from participants watching movies, we developed two complementary computational approaches to map moment-by-moment sensory processing dynamics across cortical regions: one detecting sustained periods when one modality predicts a region substantially better than the other, identifying regions that switch the modality they encode; the other identifying periods when both modalities predict the region well, revealing regions maintaining balanced representation of both auditory and visual information. These analyses reveal two types of audiovisual processing across the cortex: ``bows" that switch modalities (one posterior bow encircling category-selective visual cortex and another anterior bow spanning dorso-lateral frontal areas) and an arrow-like axis of regions that jointly represents both modalities (extending from lateral occipital cortex into the temporal cortex).

Topic Area: Methods, Tools, Theory & Neural Coding