Brain-controlled hearing technology: Setting targets for real-world applications

Authors: Kiki van der Heijden1, Vishal Choudhari2, Xiaomin He2, Nima Mesgarani2

1Donders Institute
2Columbia University

Decoding the focus of a listener’s attention from brain activity with auditory attention decoding (AAD) algorithms is a promising route towards intelligent, brain-controlled assistive hearing technology. However, as academia and industry move towards real-world applications of AAD algorithms, fundamental questions regarding the requirements of such AAD technology have not been addressed. Specifically, it is unclear how objective metrics of AAD performance such as decoding accuracy and latency relate to the subjective experience of a user. How requirements vary across different contexts and user groups is poorly understood as well. We addressed this gap in a series of three online psychoacoustic experiments in which we simulated AAD outcomes while normal hearing (NH) and hearing impaired (HI) participants listened to complex auditory scenes consisting of two simultaneous conversations. We assessed the impact of AAD parameters on user experience by quantifying how they affect objective experience (repeated word detection and speech intelligibility) as well as subjective experience (subjective ratings). In Experiment 1, we investigated what degree of suppression of an unattended talker is optimal for intelligibility of the attended talker, while ensuring sufficient audibility of the unattended talker to facilitate attention shifts. In Experiment 2, we examined the impact of AAD decoding accuracy on user experience. Finally, in Experiment 3 we assessed the interaction between AAD decoding accuracy and AAD latency in static and in dynamic listening scenarios. The results characterize the effects of AAD parameters on user experience for NH and HI listeners and illustrate that AAD user experience is dependent on the properties of the listening scene as well as the listening context. Our findings highlight the importance of identifying the requirements of AAD technology for different user groups and use cases in order to develop effective real-world AAD applications.