Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster B in Poster Session B: Tuesday, August 4, 2:00 – 3:45 pm, Kimmel Center, Shorin & Rosenthal Rooms

Inferring Scene Segmentations from fMRI Responses in the Natural Scenes Dataset

Tiasha Saha Roy¹, Kayla Hartman¹, Mario Serrafero¹, Kendrick Kay¹, Thomas Naselaris¹; ¹University of Minnesota

Presenter: Tiasha Saha Roy

Scene segmentation - the partitioning of a visual scene into meaningfully distinct segments - emerges rapidly and automatically when viewing complex natural scenes. Segmentation is thought to depend upon representations across multiple visual areas, from local edge detection in V1, border ownership in V2, to object recognition in the ventral stream. The way that these representations across multiple areas in the visual cortex interact and merge to form full scene segmentation is not currently understood. To increase our understanding, we explored segmentations of natural scenes inferred from human brain activity. We developed a decoding pipeline based on the Segment Anything Model (SAM), which outputs segment masks in the neighborhood of each pixel, or “prompt”, in an input image. We mapped brain activity in human visual cortex to the high-dimensional image embedding utilized by SAM, decoded segment masks for a dense grid of prompts, and then consolidated these masks into a single coherent scene segmentation that assigns each image pixel to one of a fixed number of segments. We report that scene segmentations inferred from brain activity at test time can be highly accurate, even for complex test images with many distinct objects, and that pixel assignments are most accurate away from the borders of the segments. This decoding approach lays the groundwork for future investigations of the unique and complementary contributions of visual cortical areas to scene segmentation.

Topic Area: Methods, Tools, Theory & Neural Coding