Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster B in Poster Session B: Tuesday, August 4, 2:00 – 3:45 pm, Kimmel Center, Shorin & Rosenthal Rooms

A systematic analysis of representations in artificial neural networks across hyperparameters

Kirsten Devolder¹, Thomas R. Colin², Sven Wientjes¹, Clay Holroyd¹; ¹Universiteit Gent, ²Ghent University

Presenter: Kirsten Devolder

Artificial neural networks (ANNs) are indispensable tools for modeling human cognition, yet their internal representational geometry remains poorly understood. Researchers often select model hyperparameters based on task performance, but this practice assumes that high accuracy implies biological relevance. However, if networks with comparable performance develop different internal structures, accuracy-based selection may lead to misleading comparisons with brain data. Building on prior studies, which have usually focused on single hyperparameters, we provide a systematic multi-factor analysis of representational variability using a fully crossed hyperparameter design. This study explored the influence of eight hyperparameters – batch size, hidden layer size, learning rate, nonlinearity, optimizer, L1 and L2 regularization, and weight initialization – on the representations learned by 1504 feedforward networks trained on a dual-task MNIST dataset. Using representational similarity analysis, we quantified the consistency of the representations of the networks. Spearman rank correlations between representational dissimilarity matrices revealed substantial representational variability (ρ = .55), which decreased when averaging across networks with identical hyperparameter configurations (ρ = .70), indicating that representational differences arose from both hyperparameter choices and training stochasticity (random initialization and batch sampling). In addition, we ran a principal component analysis, which suggests that activation function, hidden layer size, optimizer choice and learning rate had the largest influence on network representations. Together, these findings show that networks with comparable behavioral performance can develop substantially different internal representations as a function of hyperparameter choice. More broadly, the results demonstrate that hyperparameters matter for representational geometry in complex and interacting ways that are not fully captured by task performance alone. This suggests that implementation details can impact inferences on model-brain comparisons, where representational conclusions may vary across hyperparameter settings.

Topic Area: Methods, Tools, Theory & Neural Coding