Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster D in Poster Session D: Wednesday, August 5, 2:00 – 3:45 pm, Kimmel Center, Shorin & Rosenthal Rooms

Perceptual Distance from a Deep Neural Network (DNN) Predicts Human Brain Response to Foreign Accented Speech

Shang-En Huang¹, Sophie Neale¹, Seana Coulson¹; ¹University of California, San Diego

Presenter: Shang-En Huang

Recent research has used embeddings from a popular DNN for speech representation to derive measures of the perceptual distance between different utterances of the same sentence, showing that these DNN-derived distance measures of foreign-accented speech are better predictors of their intelligibility than are those derived from acoustic-phonetic cues. Here we examine whether DNN-derived distance measures can model differences in event-related potentials (ERP) elicited by speech produced by talkers whose accent diverged to varying degrees from that common to the language community of the participants. EEG was collected as 40 healthy native English speakers listened to sentences read by four different talkers. Item-based ERPs were derived by averaging (across participants) the EEG that was time-locked to the final word in each sentence, and as voiced by each of the talkers. A talker-specific modulation analysis used measurements of difference waves formed by subtracting ERPs to speech voiced by a fourth, reference talker (with a canonical American accent) from those elicited by each of the three other talkers. Analyses here used the DNN-derived perceptual distance measure as one of five continuous predictors (cloze probability, perceptual distance, mean f0, number of pauses, articulation rate) of the amplitude and latency of item-level ERPs to gauge neural sensitivity to each talker’s distance from the reference talker. Talker-specific modulation analysis revealed an effect where perceptual distance of accents scaled positively with P2 amplitude. Perceptual distance measures from the DNN were thus associated with neural responses suggesting foreign accented speech incurred changes in sound extraction processes (auditory P2). This study extends prior behavioral results that suggest the plausibility of DNN-derived perceptual distance measures, revealing their relationship to the brain’s real time response to words in accented speech. The contextually informed DNN representations of speech present an interesting alternative to cue-based attempts to quantify accents.

Topic Area: Auditory, Speech & Language Processing