Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster A in Poster Session A: Tuesday, August 4, 9:30 – 11:15 am, Kimmel Center, Shorin & Rosenthal Rooms

How Does the Brain Dynamically Combine Visual and Conceptual Features to Support Goal-Oriented Behavior?

Bruce Hansen¹, Michelle R. Greene², Audrey Kris¹; ¹Colgate University, ²Barnard College

Presenter: Bruce Hansen

Scene perception is thought to rely on a dynamic, goal-dependent neural code that flexibly prioritizes and combines visual and conceptual features over time. However, the direct evidence for this has been limited by methodological challenges. To address this, we built a novel brain-guided convolutional neural network (CNN) with dedicated channels that were guided by the neural variance explained by each of eight features. Neural data (via 128-channel EEG) were collected from participants who viewed 78 scenes while performing either a path navigation task or a seating location task. The output layer was designed to classify each input image according to the observer’s task. To successfully predict an observer’s task, the CNN combined image features with task-specific neural data. In each convolutional layer, eight independent sets of nodes were added for dedicated processing of features ranging from low-level wavelet and texture models to high-level task-specific sentence embeddings. The results revealed a late emergence (450ms; convolutional layer 4) of task-dependent differences in feature combination. These findings indicate that task goals shape feature combination late in processing.

Topic Area: Methods, Tools, Theory & Neural Coding