Authors: Sarthak Mangla¹, Afagh Farhadi¹, Michael Heinz¹
¹Purdue University
Background: Self-supervised speech representations are widely used in hearing-aid enhancement, but are usually learned as listener-independent features, limiting personalization. This study investigates whether these representations can be made listener-aware by incorporating audiogram-derived phoneme confusability without requiring direct behavioral responses.
Method: Using data from 31 Clarity Prediction Challenge listeners, we synthesized isolated words and degraded them using MSBG simulation and noise. Transcriptions from a wav2vec 2.0 ASR model were aligned to estimate listener-specific phoneme substitutions, generating 39 × 39 confusion matrices. These matrices were used to adapt frozen wav2vec 2.0 embeddings through auxiliary tasks: predicting confusion patterns, estimating pairwise phoneme confusability, or a combination of both. Performance was evaluated on a listener-aware benchmark using hearing-aid-processed speech.
Results: Listener-aware objectives significantly improved prediction performance compared to generic self-supervised adaptation. The phonetic objective (confusion-matrix row prediction) performed best, reducing RMSE from 27.38 to 26.80 and increasing Pearson correlation from 0.674 to 0.700. All listener-aware objectives showed significant RMSE reductions in paired tests (p < 0.05), with the combined objective achieving ΔRMSE = -0.56 (p < 0.001).
Conclusion: Audiogram-derived phoneme confusability is a viable supervisor for learning listener-aware speech representations. The success of confusion-matrix row prediction suggests that hearing-aid personalization can be effectively integrated at the representation layer using simulated perceptual structures.
