Theory & Methods

Contributed Talk Session: Wednesday, August 5, 10:15 – 11:15 am, Skirball Theater

A Nonlocal Variational Framework for Optimal Neural Representations

Talk 1, 10:15 am

Gengshuo John Tian¹, Brent Doiron²; ¹Flatiron Institute, ²University of Pittsburgh

Presenter: Gengshuo John Tian

Despite decades of study it still remains unclear how neural populations should organize their representation of sensory, motor, and cognitive variables so as to optimize discriminability. There are two important limitations in previous works. First, Fisher information (FI) is commonly used to measure the quality of population codes (Seung and Sompolinsky, 1993), yet FI is fundamentally a local measure that is incapable of capturing global structures of the representation, as is needed in coarse discrimination (Berens et al., 2011). Second, simplified forms of tuning functions (such as Gaussian bumps) are usually imposed for analytic tractability, but in real neural data, tuning curves are diverse and in most cases defy being described by an overly simplistic structure. Here, we tackle these issues by focusing on the representation of a one-dimensional periodic variable θ and analytically minimizing the average binary classification error between all θ pairs without any restriction on the shape of the tuning curves for various noise models. Our cost function accounts for both fine and coarse discrimination and the feasible set is an infinite-dimensional function space, making our formulation a nonlocal variational problem. We obtained the solution by viewing the space of neural response distributions as a Riemannian manifold in the sense of information geometry (Amari, 2016) and using a result from knot energy theory (Abrams et al., 2003). The optimality of the derived representation is demonstrated with simulations. We deduce two predictions from the model, one concerning the range of tuning curve shapes and the other relating neurons' variability to their tuning sharpness. Both of these predictions are verified in a head-direction cell dataset (Duszkiewicz et al., 2024). Our results point to a new framework for studying the global structure of neural representations.

Hyperalignment Reveals Shared Information in Monkeys’ Idiosyncratic Fine-Grained Movie fMRI Patterns

Talk 2, 10:25 am

Yuqi Zhang¹, Haiyan Wang², Qi Zhu³, Xiaolian Li⁴, Jane Han¹, Guo Jiahui⁵, Won Mok Shim⁶, Royoung Kim⁶, Maria Gobbini⁷, James Haxby¹, Ma Feilong⁸, Wim Vanduffel⁴; ¹Dartmouth College, ²Institute of Automation, Chinese Academy of Sciences, ³INSERM, CEA, Université Paris-Saclay, NeuroSpin Center, ⁴KU Leuven, ⁵University of Texas at Dallas, ⁶Sungkyunkwan University, ⁷University of Bologna, ⁸University of South Carolina

Presenter: Yuqi Zhang

Hyperalignment (HA) maps individual brain activity into a shared high-dimensional representational space, allowing finer cross-subject comparison than anatomical alignment. Although HA has been widely used in humans, its application in non-human primates remains limited. Here, we applied HA to fMRI data from 12 rhesus monkeys watching Monkey Kingdom. We used the first half of the movie to build a common template and to estimate individual transformation matrices and evaluated performance on the second half. HA improved inter-subject correlation, especially in occipital, temporal, and lateral prefrontal cortex, and increased between-subject movie timepoint classification accuracy by about 80% relative to anatomical alignment. HA also enabled prediction of idiosyncratic functional topographies from other monkeys’ movie-watching and localizer data. Applying the same framework to 24 humans watching the same movie showed similar results. Together, these findings show that HA supports robust cross-subject alignment of fine-grained functional organization in both monkeys and humans.

Task-guided cross-subject latent alignment: a multi-encoder-decoder VAE

Talk 3, 10:35 am

Angeliki Papathanasiou¹, Jascha Achterberg¹, Thomas E. Nichols¹, Rui Ponte Costa¹; ¹University of Oxford

Presenter: Angeliki Papathanasiou

Aligning neural activity across subjects offers the promise of discovering shared computational principles and generalizable decoders. However, traditional alignment methods require shared stimuli across subjects, a constraint that limits applicability to naturalistic paradigms with limited or non-overlapping data. We introduce a Multi-Encoder-Decoder Variational Autoencoder (MED-VAE) that achieves cross-subject alignment without shared stimuli by anchoring representations to a common scaffold provided by a pretrained ANN. Using the Natural Scenes Dataset, we show that MED-VAE creates common latent spaces with superior semantic organisation, achieving higher cross-subject alignment than common methods while maintaining robust generalisation to held-out stimuli where traditional methods degrade. Reconstructing from these common spaces back to each subject's original neural space, MED-VAE preserves equal stimulus-driven signal in its cross-subject latent space. Finally, we show that this superior alignment directly enables cross-subject neural prediction, as demonstrated via cross-subject image decoding. In summary, we introduce a framework to identify generalisable common subspaces for cross-subject predictions and downstream tasks, demonstrated here for visual cortex responses to static images.

The Wisdom of a Crowd of Brains: A Universal Brain Encoder

Talk 4, 10:45 am

Roman Beliy¹, navve wasserman¹, Amit Zalcher¹, michal Irani¹; ¹Weizmann Institute of Science

Presenter: navve wasserman

Image-to-fMRI encoding is important for both neuroscience research and practical applications. However, such "Brain-Encoders" have been typically trained per-subject and per fMRI-dataset, thus restricted to very limited training data. In this paper we propose a Universal Brain-Encoder, which can be trained jointly on data from many different subjects/datasets/machines. What makes this possible is our new voxel-centric Encoder architecture, which learns a unique "voxel-embedding" per brain-voxel. Our Encoder trains to predict the response of each brain-voxel on every image, by directly computing the cross-attention between the brain-voxel embedding and multi-level deep image features. This voxel-centric architecture allows the functional role of each brain-voxel to naturally emerge from the voxel-image cross-attention. We show the power of this approach to (i) combine data from multiple different subjects (a "Crowd of Brains") to improve each individual brain-encoding, (ii) quick & effective Transfer-Learning across subjects, datasets, and machines (e.g., 3-Tesla, 7-Tesla), with few training examples, and (iii) use the learned voxel-embeddings as a powerful tool to explore brain functionality (e.g., what is encoded where in the brain).

Neural Footprints: Variance-Based Methods Systematically Miss High-Level Representations

Talk 5, 10:55 am

Jon Gauthier¹, Tyler BrookeWilson²; ¹University of California, San Francisco, ²Yale University

Presenter: Jon Gauthier

The brain computes at vastly different levels of representation, encoding both abstract knowledge and fine-grained sensory information about the world. These two ends of the spectrum have distinct statistical properties: while abstract knowledge about the world is compact and low-dimensional, sensory information is highly variable and difficult to compress. This statistical asymmetry creates fundamental issues for several neuroscience analysis techniques, biasing these methods to favor high-dimensional sensory features over low-dimensional abstract representations. In a simulation experiment, we demonstrate that variance-based techniques fail to distinguish between obviously correct and incorrect models of visual perception due to this statistical asymmetry. Reliable neural analysis therefore requires models that are not merely fit to explain variance in neural activations, but that more deeply link representations to the computations they support.

Implications of hierarchical Markov models of behavior: on irreversibility, predictability, and dimensionality

Talk 6, 11:05 am

John J. Vastola¹, Kanaka Rajan¹; ¹Harvard University

Presenter: John J. Vastola

The maturation of quantitative tools for studying the high-level structure of animal behavior, and especially tools which represent spontaneous behavior as a sequence of stereotyped and neurally well-defined 'syllables', demands that the field revisit a fundamental theoretical question: if the coarse structure of behavior can be accurately described by Markov models, what do these models really tell us about behavior? In this work, we explore the theoretical implications of these models and discuss how they allow us to quantitatively formulate questions about the sequence-like nature and effective dimensionality of behavior. One important insight is that the eigenvalues and eigenvectors of various model-associated matrices furnish interpretable time scales and modifications of behavior that occur on those time scales. We illustrate our points using both toy examples and Markov models fit to real data. By analyzing the consequences of Markov representations, we clarify the theoretical meaning of progress in quantifying behavior.