Authors: Amin Pourhoseingholi¹, Paul Bateman¹, Catherine Killan¹, Nicola Wray¹, Sara Rafiee², Derek Hoare³
¹National Institute for Health and Care Research Nottingham Biomedical Research Centre, Nottingham
²School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran
³Hearing Sciences, Mental Health and Clinical Neurosciences, School of Medicine, University of Nottingham, Nottingham
Background: AI chatbots are increasingly used to disseminate health information, yet their accuracy, consistency, and readability in hearing health remain underexplored. This study evaluates the validity, reliability, and readability of four AI chatbots—ChatGPT (GPT-3.5), Bing AI, Gemini, and Perplexity—across five key audiology domains: adult hearing loss, paediatric hearing, hearing aids, tinnitus, and cochlear implants.
Methods: One hundred expert-designed questions (20 per domain) were submitted twice to each chatbot, generating 800 responses. Responses were scored by audiology experts using a 5-point Likert scale adapted from the Global Quality Score. Validity was assessed via low (score ≥4) and high (score =5) thresholds. Cronbach’s alpha measured reliability. Readability was analyzed using Flesch Reading Ease and Flesch-Kincaid Grade Level metrics.
Results: Perplexity achieved the highest overall performance (score = 0.829), followed by Bing (0.796), Gemini (0.769), and ChatGPT (0.716). Under low-threshold validity, GPT-3.5 scored highly in all domains (70–90%), but high-threshold validity dropped significantly across all models. Readability was poor overall: 84.3% of responses were “Difficult” or “Very Difficult”, with 67.7% written at a college reading level. Reliability was acceptable to high across all platforms (Cronbach’s α ≥ 0.70).
Conclusions: AI chatbots can provide reasonably accurate hearing-related information, but high-threshold precision and public readability are limited. These tools should complement, not replace, clinical guidance. Continued refinement is needed to enhance accuracy, readability, and usability in hearing care applications.
