Authors: Annika Magaro¹; Gasser Elbanna¹; Josh McDermott¹
¹MIT
Background: Models capable of predicting human speech recognition have diverse applications. While recent machine speech recognition systems explain certain neural responses to speech, it remains unclear how well they account for human recognition behavior across various auditory distortions. This study compared human speech recognition performance against five recent industry systems.
Method: Human participants and five speech recognition models (Parakeet-TDT-0.6B-v2, SeamlessM4T-v2, Whisper-large-v3, Whisper-large-v1, Whisper-tiny-v1) transcribed words from two-second speech excerpts. To ensure fair comparison, scoring was limited to 230 common words. Performance was evaluated across 37 different types of distortions, most of which were varied parametrically.
Results: Models showed high correlations with human performance patterns across distortions. However, several industry models demonstrated superhuman performance, achieving higher intelligibility than humans for most distortion types. Whisper-tiny-v1, which shared training data with Whisper-large-v1 but utilized a smaller architecture, underperformed humans, suggesting that superhuman capabilities may depend on large model architectures.
Conclusion: Current industry speech recognition models exhibit human-like performance patterns across speech distortions, yet some systems have surpassed human accuracy. These findings encourage further research into developing models with architectures and training datasets specifically designed to match, rather than exceed, human performance levels for scientific modeling.
