Authors: Jonas Auernheimer¹, Tobias Reichenbach¹
¹FAU Erlangen-Nürnberg
Introduction: Speech comprehension improves when visual cues are added to auditory input, especially under challenging listening conditions—a phenomenon known as inverse effectiveness. While previous studies have linked audiovisual (AV) integration to cortical tracking of speech and lip movements, it remains unclear whether early subcortical auditory processing, such as at the fundamental frequency (f₀), is also modulated by AV input and whether this relates to behavioural performance.
Methods: We analyzed EEG recordings from 25 participants listening to audiovisual speech passages with varying facial degradation. Two background maskers—native English (informational) and non-native Dutch (energetic)—were used. From the speech, we extracted two features: a waveform oscillating around f₀ and higher-frequency envelope modulations (200–4000 Hz). Neural encoding was based on accuracies of the reconstructed EEG yielded by multivariate linear models. Comprehension was measured via post-trial questions. Finally, linear mixed-effects models were used to relate the neural encoding to behavioural scores, accounting for within-subject and AV condition effects.
Results: Comprehension scores were significantly higher when visual input was present, confirming the AV benefit. However, no significant AV effect was found at the neural encoding level. Encoding accuracy was higher with the Dutch background speaker, suggesting early auditory processing is more sensitive to acoustic masking. A significant correlation emerged between neural encoding strength and behavioural performance, influenced by both subject and condition.
Conclusions: Early neural encoding of AV speech-in-noise is modulated more by acoustic interference than visual input. While comprehension benefits from AV integration, early subcortical processing may not reflect this advantage. Still, the link between neural encoding and behavioural performance suggests individual and condition-specific influences at early stages of AV processing.
