Authors: Jack Collins¹, Simon Carlile¹, Julian Maclaren¹, Dick Lyon¹, Chris Collier¹, Alejandro Ballesta Rosen¹, Adrian Buzea¹
¹Google Research
Background: Individuals regularly experience hearing difficulty moments in everyday conversation. Identifying these moments of hearing difficulty has particular significance in the field of hearing aid technology where timely interventions are key for realtime hearing assistance. In this paper, we proposed and compared machine learning solutions for continuously detecting utterances that identify these specific moments in conversational audio.
Method: We evaluated Audio Language Models (prompting and fine-tuning) as classifiers, alongside a more traditional wav2vec classifier, and a baseline ASR heuristic solution. The evaluation dataset was a multi-gender, multi-speaker audio dataset which made use of labeled ground truth “signal-no-understanding” act tags available in the Switchboard Dialog Act Corpus and the Meeting Recorder Dialog Act Corpus.
Results: We observed that Audio Language Models are a good fit for modelling this hearing difficulty detection task, with both few-shot prompting and fine-tuning approaches converging to a similar performance with increasing number of examples (F-score 0-shot: 0.82, 200-shot: 0.94), showing clear uplift against the baseline heuristic approach (F-score 0.45).
Conclusion: Leveraging Audio Language Models for nuanced audio classification tasks involves a level of audio reasoning that goes well beyond ASR. We have established a Hearing Difficulty Detection task and shown that it is possible to achieve excellent results for this task with Audio Language Models using both few-shot prompting and fine-tuning approaches.
