Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster F in Poster Session F: Thursday, August 6, 1:45 – 3:30 pm, Kimmel Center, Shorin & Rosenthal Rooms

Processing the Unexpected: Contributions of Multiple Semantic Dimensions

Harshada Vinaya¹, Seana Coulson¹; ¹University of California, San Diego

Presenter: Harshada Vinaya

How our semantic memory and the underlying representational structure support online language comprehension remains an open question. Events when our semantic expectations are violated provide an opportunity to dissociate different dimensions of similarity between the expected and the observed continuations. Here, we ask whether neural responses to three different kinds of semantic violations are processed in a singular way or show differential sensitivity to distinct dimensions of semantic knowledge. We quantified semantic deviation using GPT2 surprisal together with representational distances derived from BERT and ImageBind embeddings, and assessed their ability to predict EEG responses elicited by violation words. Neural responses were not fully explained by surprisals, and language-derived and multimodal representational distances provided selective improvements over lexical predictability depending on how the observed continuation deviated from the expected one.

Topic Area: Auditory, Speech & Language Processing