Community Event

Wednesday, August 5, 4:00 - 5:00 pm

Toward a unified science of perception and memory: empirical foundations and computational frontiers

     

Organizers: Stephanie Fu1, Chris Iyer2, Serra Favila3,  Tyler Bonnen4, 1UC Berkeley, 2Columbia University, 3Brown University, 4University of Pennsylvania

 

Additional Speakers: Anna Schapiro1, Akshay Jagadeesh2, 1University of Pennsylvania, 2Harvard Medical School

Abstract

Adaptive behavior depends on the integration of perception and memory, yet these abilities have historically been studied in isolation. In this community event, we will discuss how to integrate the science of vision and memory, leveraging experimental and computational methods from neuroscience, cognitive science, and computer science. Concretely, we focus on the functional relationship between ventral temporal cortex (VTC) and the medial temporal lobe (MTL), highlighting how distinct neurocomputational motifs interact to support human behavior. These behaviors pose difficult computational problems: integrating information across long temporal windows, generalizing from sparse naturalistic experience, and retrieving context-appropriate representations in the face of interference. The VTC-MTL system resolves these challenges by operating over the high-dimensional sensory data of natural experience. Yet most existing models of human memory do not process such data, relying instead on idealized, low-dimensional inputs hand-tailored for a given task. We argue for the necessity of developing mechanistic models of memory that operate over natural sensory inputs, and highlight modeling strategies from the computer sciences that may help us build unified models of perception and memory. Our goal is not to survey a field as much as it is to help build one: connecting researchers who are working on related problems, identifying bottlenecks, and articulating a research agenda for the VTC-MTL system that is both experimentally grounded and computationally tractable.

Session Plan

Our event is organized in three stages:

  1. Empirical foundations: We begin with a series of brief talks that provide an integrative account of VTC and the MTL, each centering on a distinct component of the circuit and describing its architecture, the algorithm it implements, and the behaviors it supports. 
  2. Computational frontiers: We survey state-of-the-art approaches to modeling vision and memory, highlighting the shared computational challenges between biological and artificial systems. We also review several influential models of memory in the cognitive and neurosciences, and how their core theoretical contributions may be expressed in an integrated perceptual-mnemonic framework. 
  3. Community discussion: We transition to small group discussions, where participants discuss points of alignment between biological and artificial memory systems, experimental puzzles well-suited to recent modeling innovations, and promising paradigms to coordinate the field’s progress.

By the end of the event, we hope participants will better understand how this VTC-MTL system interacts to support human behavior, identify the key computational bottlenecks preventing progress modeling this system, and leave with concrete ideas about datasets, tasks, and models needed to move the field forward.  We hope this structured exchange of perspectives will help clarify key challenges in studying visual and mnemonic behaviors, and stimulate a more unified research agenda.