Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster E in Poster Session E: Thursday, August 6, 10:30 am – 12:15 pm, Kimmel Center, Shorin & Rosenthal Rooms

Neural Encoding and Reinstatement of Multimodal Information during Naturalistic Viewing

Chengcheng Wang¹, Jixing Li¹; ¹City University of Hong Kong

Presenter: Jixing Li

Video comprehension requires the integration of visual, auditory, and linguistic information, yet how these multimodal streams are jointly represented and later reconstructed from memory remains unclear. Here, we analyzed functional magnetic resonance imaging (fMRI) data from 30 participants during naturalistic viewing and subsequent mental replay. We extracted text, audio, and video embeddings from a multimodal large language model and used them to build encoding models that predict cortical activity during viewing. We observed modality-specific spatial patterns, with each modality correlating with activity in its corresponding cortical regions. We then assessed the reinstatement of the modality-specific representations by comparing predicted brain activity with neural responses during the replay period. We observed a systematic reverse temporal structure across modalities during replay, with early recall aligning more strongly with later encoding moments. These findings suggest that multimodal information may be supported by closely coupled mechanisms during both encoding and retrieval, reflecting a common representational framework across perception and memory.

Topic Area: Memory, Learning & Knowledge Structures