Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster A in Poster Session A: Tuesday, August 4, 9:30 – 11:15 am, Kimmel Center, Shorin & Rosenthal Rooms

Directional Confusions Reveal Inductive Bias Through Rate–Distortion Geometry in Human and Machine Vision

Leyla Roksan Caglar¹, Pedro A. M. Mediano², Baihan Lin³; ¹Icahn School of Medicine at Mount Sinai, ²Imperial College London, ³Columbia University

Presenter: Leyla Roksan Caglar

To humans, a robin seems more like a bird than a bird seems like a robin, but does this asymmetry also hold for machine vision? Humans and modern vision models can match each other in accuracy while making systematically different kinds of errors, differing not in how often they fail, but in who gets mistaken for whom. We show that these directional confusions reveal distinct inductive biases invisible to accuracy alone. Using matched human and deep vision model responses on a natural-image categorization task under 12 perturbation types, we quantify asymmetry in confusion matrices and link it to generalization geometry through a rate–distortion (RD) framework, characterized by slope (β), curvature (κ), and efficiency (AUC). Humans exhibit broad but weak asymmetries, whereas deep vision models show sparser, stronger directional collapses — a dissociation confirmed to be independent of accuracy. Robustness training reduces global asymmetry but fails to recover the human-like breadth–strength profile. Mechanistic simulations show that broad–weak and sink-like organizations produce opposite shifts in RD frontier geometry even when matched for performance, suggesting that human-aligned robustness requires redistributing error structure, not just reducing it.

Topic Area: Computational Models of Vision & Visual Cortex