Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster A in Poster Session A: Tuesday, August 4, 9:30 – 11:15 am, Kimmel Center, Shorin & Rosenthal Rooms

NeuroTIMM: Harmonizing the TIMM Model Zoo with Human Visual Strategies

Akash Nagaraj¹, Drew Linsley¹, Jay R Gopal¹, Peisen Zhou¹, Thomas Serre¹; ¹Brown University

Presenter: Akash Nagaraj

In previous work by Fel2022, the authors showed that "harmonizing" deep neural networks with human psychophysics to force them to rely on the same visual features as human observers aligned models with human perception while maintaining classification accuracy. Here, we scale up this approach with ClickMe 2.0, a larger and more robust behavioral dataset, and more efficient harmonization methods that make it feasible to align models across the full PyTorch Image Models (TIMM) library. Harmonized models exhibit substantially improved alignment with human visual strategies on ImageNet, which also yields higher correspondence with electrophysiological recordings from macaque visual cortex, and improved robustness against visual perturbations. We release all the harmonized weights as an open resource for the community.

Topic Area: Computational Models of Vision & Visual Cortex