Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster C in Poster Session C: Wednesday, August 5, 9:30 – 11:15 am, Kimmel Center, Shorin & Rosenthal Rooms

Beyond snapshot vision: decoding dynamic videos from temporally evolving neural activity

Anne W. Zonneveld¹, Louise van der Eijk¹, Pascal Mettes¹, Iris Groen¹; ¹University of Amsterdam

Presenter: Iris Groen

Visual perception requires processing of dynamic, continuously changing inputs, which elicit neural responses that unfold dynamically over time. Temporal dynamics of visual representations are extensively studied in the context of static vision, e.g. by testing how well visual images can be decoded from time-resolved human brain signals. Less is known about the temporal characteristics of neural responses to dynamic stimuli. Here, we directly compare the neural processing of static and dynamic inputs by computing temporal decoding profiles from electro-encephalography (EEG) activity measured in human subjects viewing short naturalistic videos and matched single-frame images. We ask: 1) Are dynamic videos and static images similarly decodable from EEG responses? 2) How well do neural representations generalize between images and videos? and 3) Is this generalization between formats modulated by video dynamics? We find that images and videos yield comparable within- format decoding accuracies at early stages of the EEG response time course, but videos are better decodable later in time (>400 ms). Neural decoding does substantially but not fully generalize across formats, with later convergence seeming to depend on the temporality of the video content. Specifically, image decoding generalizes least to highly temporal videos, suggesting that additional, video-specific representations are involved. These findings reveal unique neural dynamics for video over images.

Topic Area: Computational Models of Vision & Visual Cortex