Authors: Hector Gabriel Corrale de Matos¹, Jan-Willem Wasmann², Lilian Cássia Bornia Jacob¹
¹University of São Paulo
²Radboud University Medical Center
Background: Chatbots offer highly accessible hearing-health advice, however their readability in different languages remains unclear. This study examines whether incorporating Wikipedia-derived audiology excerpts alters the textual complexity of AI-generated responses in English and Brazilian Portuguese, evaluated with standard readability indices.
Methods: ChatGPT-4o, DeepSeekR1, Claude3.7, Gemini-Flash, LeChat, SábiaAI, and Microsoft-Copilot produces 112 responses to eight hearing-related questions generated intrinsically and Wikipedia-orientated (Retrieval-Augmented Generation approach). Readability was assessed with metrics in English and Brazilian Portuguese: Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), Coleman-Liau Index (CLI), Automated Readability Index (ARI), Gunning Fog Index (GFI), Word Count (WC) and Average Words per Sentence (AWPS). Two language-specific indices were used: SMOG and Gulpease Index (GI). FRE and GI scores from 0 to 100 whereas FKGL, SMOG, ARI, CLI and GFI approximate grade-level. Descriptive statistics and Student’s T-tests (α = 0.05) were computed for each metric within each language-condition.
Results: English texts (n=56) demonstrate variation in readability metrics with Wikipedia orientation: FRE 30.29 → 29.34, FKGL 12.92 → 13.32, SMOG 14.38 → 14.93, ARI 14.37 → 14.73, CLI 15.24 → 15.17, GFI 13.96 → 14.99, WC 281.68 → 164.21, AWPS 15.63 → 16.73. GFI showed significance alteration and SMOG marginal increase (p ≈ 0.05) in the Wikipedia condition. WC differed significantly without Wikipedia orientation. Portuguese texts (n=56) averaged FRE 20.05 → 21.63, GI 56.00 → 51.20, FKGL 14.58 → 15.26, GFI 12.27 → 14.78, ARI 12.78 → 13.95, CLI 15.74 → 15.34, WC 268.86 → 166.09, AWPS 13.49 → 17.81. GFI, ARI, WC, AWPS varied significantly and FKGL showed a marginal increase (p ≈ 0.09) with Wikipedia. GI and WC differed significantly without Wikipedia direction.
Conclusion: Portuguese texts remained more complex. Wikipedia nominally affected readability, warranting more rigorous analyses.
