Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster A in Poster Session A: Tuesday, August 4, 9:30 – 11:15 am, Kimmel Center, Shorin & Rosenthal Rooms

Better than reality: implied auditory information across visual and multimodal regions revealed by generative AI

Ryuto Yashiro¹, Adrien Doerig¹, Tim C Kietzmann²; ¹Freie Universität Berlin, ²Universität Osnabrück

Presenter: Ryuto Yashiro

Cognitive computational neuroscience typically models neural representations using features derived from the sensory stimuli presented during an experiment. However, everyday perception is inherently multimodal, and the brain learns statistical correspondences across sensory modalities. Consequently, sensory information implied but not physically present in the stimuli may also shape neural responses evoked by those stimuli. Here we show that neural representations elicited by silent videos can in part be explained by features derived from AI-generated sounds implied by those videos, over and above the sensory visual responses. This predictive contribution of implied auditory information is strongest in the posterior superior temporal sulcus (STS), a region implicated in multisensory integration. Importantly, AI-generated sounds better explained the data than the actual sounds present in the original videos. These results demonstrate that implied sensory information may need to be considered for modelling and understanding neural representations of unimodal sensory stimuli.

Topic Area: Methods, Tools, Theory & Neural Coding