Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster B in Poster Session B: Tuesday, August 4, 2:00 – 3:45 pm, Kimmel Center, Shorin & Rosenthal Rooms

Distribution shift predicts the dynamics of 3D perception

Václav Knapp¹, tyler bonnen¹; ¹University of California, Berkeley

Presenter: Václav Knapp

Here we draw on an established phenomenon in machine learning---distribution shift---to understand the temporal dynamics of human visual perception. We begin by modeling a tractable setting where both train and test distributions are known: we procedurally generate objects, fine-tune DINOv2-L to distinguish between them, then evaluate on an out-of-distribution 3D perception benchmark. We find that trial-level distribution shift predicts perceptual accuracy on held-out data: fine-tuning improves performance on the test set, but only on trials close to the train set. Critically, we identify a model-based proxy for this distance that does not require direct access to the training data. This allows us to extend our analysis to large pretrained models, where we estimate the distance between images in this 3D benchmark and a more naturalistic distribution of visual data. Humans are far more robust to distribution shift than large vision encoders (e.g., DINOv2-G, CLIP-G), and human reaction times scale with distributional distance: humans are fast on 3D inferences "close" to the distribution of natural images, but take more time for trials "further" away. Our results demonstrate that 3D shape perception can be understood as out-of-distribution generalization, where humans compensate for distribution shift by flexibly allocating additional visual processing at test time.

Topic Area: Computational Models of Vision & Visual Cortex