Device Converts Images Into Music to Help the Visually Impaired "See" With Sound

A study published this month in the journal  Restorative Neurology and Neursocience  demonstrates the potential of a new device to help the blind reach for objects in space by converting images into sounds.Gadgets like these are called sensory substitution devices (SSDs); they use another sense, usually sound or touch, to help the visually impaired perceive the world around them.  Ideally, an SSD would not only assist in sensing the environment but also in interacting with it. Such a device would allow blind people to perform daily tasks like reaching out to shake someone's hand or grab a cup of coffee.The SSD described in this paper comes from the lab of Amir Amedi at Hebrew University of Jerusalem.  Amedi and his team call their device EyeMusic. It's a non-invasive SSD that converts images into a combination of pleasing musical tones, or soundscapes.EyeMusic works by scanning an image and then representing pixels of it with different aspects of sound. High vertical locations are represented as high-pitched musical notes and low vertical locations as low-pitched notes. To represent horizontal location, the image is scanned from left to right, with an auditory cue marking the start of the scan. The horizontal location of a pixel is indicated by the timing of the musical note relative to the start cue (with later sounds indicating pixels farther to the right). Brightness is encoded by the loudness of the sound.Different musical instruments are used for each of five basic colors: white (vocals), blue (trumpet), red (reggae organ), green (synthesized reed), and yellow (violin). Black is represented by silence. Amedi and his team collaborated with musicians to choose the instruments and the musical notes to ensure their SSD created musical sounds and not unpleasant cacophonies for the user. You can listen to sample sound recordings here.Amedi and co-authors trained blindfolded sighted test subjects to use EyeMusic. Some of the participants picked it up in as little as half an hour, suggesting it is not that difficult for people to create a representation of space in their brains without visual input. The participants trained on EyeMusic performed fast and accurate movements, and the authors are hopeful about the SSD's potential. Eventually, EyeMusic may be used by the visually impaired in rehabilitation or in everyday activities.