Authors: Joanna Luberadzka¹; Umut Sayin¹
¹Eurecat, Centre Tecnològic de Catalunya
Background: Although hearing aids (HAs) aim to improve communication, their benefit is usually assessed with non-interactive listening tests. These tests measure hearing thresholds and speech intelligibility, but not communication ability. The need for an assistance-free, scalable, and reproducible speech communication test has been recognized in the literature, but practical challenges have prevented its realization.
Method: Advances in speech technologies enable the use of AI voice agents as partners in communication testing. We present a prototype of such a system: a web-based voice agent built using a native streaming LLM API that allows bidirectional, low-latency audio interaction. The acoustic scene is managed via the Web Audio API, allowing difficulty to be scaled by adjusting SNR, noise type, and target spatialization. We designed a café role-play task in which participants interact with the agent and collect requested information. Agent behavior and task goals are controlled through prompt engineering. Two normal-hearing groups were tested at SNRs of +5 dB and −5 dB.
Results: Participants rated the interaction as sufficiently natural. At both SNRs, speech latency was judged acceptable and conversational flow was maintained. Perceived difficulty differed between groups, but most participants successfully collected the target details. While this confirms effective communication with the agent, it indicates limited sensitivity of the paradigm to quantify communication difficulties.
Conclusion: Our prototype shows the potential of an AI voice agent for speech communication research. In the future, we plan to use HA and hearing loss simulations and develop a more sensitive paradigm combining an information-gap task with a secondary task. Planned metrics include task scores, clarification requests, repair sentences, and turn-taking analysis.
