Authors: Loonan Chauvette¹; Anne Sophie Grenier¹; Emily Coffey²; Robert Zatorre³; Philippe Albouy¹; Jérémie Voix⁴; Andréanne Sharp¹
¹CERVO Brain Research Center; ²Concordia University; ³Montreal Neurological Institute; ⁴École de technologie supérieure
Background: Auditory–tactile assistive technologies can complement conventional hearing devices, notably for music perception and noisy listening environments. However, designing effective devices requires knowledge of how acoustic cues are encoded and integrated across sensory modalities. Timbre perception, emerging from complex spectral and temporal acoustic cues, provides a framework to examine the limits and benefits of multisensory augmentation for complex sound perception.
Method: 30 adults with normal hearing and screened for vibrotactile acuity completed a timbre discrimination task in auditory, tactile, and auditory–tactile conditions. Stimuli were presented via insert earphones and the **Multichannel Vibrotactile Glove**, an assistive device we developed to convey acoustic signals as vibrations with 5 independent actuators per hand. Discrimination thresholds were estimated using an adaptive staircase method for 6 timbre-related acoustic parameters: 3 spectral (number of harmonics, harmonic roll-off, even-harmonic attenuation) and 3 temporal (attack time, amplitude modulation depth, modulation frequency).
Results: All timbre-related parameters could be discriminated using vibrotactile stimulation alone, indicating that complex acoustic cues can be conveyed through touch. Tactile thresholds were worse than auditory ones for spectral parameters but were comparable across modalities for temporal parameters. Auditory-tactile performance was statistically better for two out of three temporal features, suggesting cross-modal integration when unimodal cues share similar baseline sensitivity.
Conclusion: The study reveals constraints and opportunities for multisensory technologies. Vibrotactile augmentation seems best suited for temporal acoustic features, and multisensory gains depend on cross-modal threshold alignment. The Multichannel Vibrotactile Glove provides a flexible platform to explore signal processing strategies for multisensory hearing rehabilitation.
