Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster E in Poster Session E: Thursday, August 6, 10:30 am – 12:15 pm, Kimmel Center, Shorin & Rosenthal Rooms

A Reproducibility Framework for Subject-Level Alzheimer’s Decoding

Adel Sahuc¹; ¹New York University

Presenter: Adel Sahuc

Many EEG papers warn about leakage, unstable validation, and weak reporting, yet these issues still appear frequently in current disease-classification studies (Brookshire et al., 2024; Kapoor & Narayanan, 2023; Kunjan et al., 2021). We make these risks concrete in one exemplar problem: Alzheimer's disease (AD) vs control (CN) decoding on OpenNeuro ds004504 (Miltiadous et al., 2023; Markiewicz et al., 2021). We evaluate three traps together, in one auditable workflow. Trap 1 (identity leakage): when epochs from the same subject appear in both train and test sets (subject-overlap), accuracy is inflated. Trap 2 (cohort sensitivity): even when no subject appears in both sets (subject-disjoint), results can still vary strongly depending on which subjects are held out. Trap 3 (objective mismatch): epoch-level metrics (an epoch is a short EEG segment treated as one sample) and subject-level metrics can rank models differently. To empirically demonstrate these traps, we compare PCA against a univariate T-test feature selector across KNN, SVM, XGBoost, and MLP. Overlap versus subject-disjoint splits test Trap 1, varying held-out cohort size and composition across repeated cohorts tests Trap 2, and epoch-versus-subject ranking comparisons test Trap 3. This work jointly tests three EEG evaluation traps and quantifies a previously unquantified Trap 3 effect: the best subject-level model ranks tenth by epoch accuracy. We recommend the use of subject-disjoint LOSO/LPSO (Leave-P-Subjects-Out) splits, reporting fold/cohort variability (SD), and reporting both epoch and subject accuracy.

Topic Area: Methods, Tools, Theory & Neural Coding