HEEYA MODY

THE PROBLEM

1. People with hearing disabilities have to rely on sight to engage. When in conversation, they have to be facing the person to decipher lip movement or interpret sign language, and cannot multitask.


2. People with hearing disabilities can only interpret direct conversations– they do not have the autonomy to passively understand conversations around them.


3. Although they may have sight as a sense, they have more tunnel vision than others because sounds can be indicators as to where to look and what to expect.

How might we use OpenAI’s Whisper and GPT-3 to unite people?

 
a hardware product that assists people with hearing disabilities in navigating daily life with tactile sensory feedback

SOLUTION

1. People with hearing disabilities no longer have to rely solely on sight. They can feel the voices around them as physical sensations.


2. Their tunnel vision is now expanded due to haptic feedback as a substitute for sound indicators. This helps direct their gaze and know what to expect.


3. People with hearing impairments now have a sense of the kinds of conversations occurring around them due to the sentiment analysis feature. This helps them further investigate and potentially avoid dangerous situations, for example someone having an angry and potentially violent conversation around them.


4. People with hearing disabilities can know when their name is being mentioned or called without having to be near them or facing them, with the user’s name detection feature.

︎︎︎ Haptic technology overcomes tunnel vision that our users experience when relying on sight by sending tactile signals; touch does not have a limited field of view like sight.

︎︎︎ Haptic technology allows for a wide range of forms that signals can take: simple vibrations, vibrations in the form of letters, morse code vibrations, etc.

case study 1

Conversation Interpretation with Haptic Belt

case study 2

Interpretation with Haptic Ring & Bracelet

︎︎︎ When walking on the street

︎︎︎ In social gatherings or events

︎︎︎ When navigating through general daily life

1. Understanding how people with hearing disabilities currently communicate: sign language, lip movement


2. Understanding how people who are both blind and deaf engage in conversation: tactile sign language drawn on palm


3. How do we apply the haptic solution to multiple conversations? How do we figure out which person’s voice to transcribe at a particular time?


4. What if multiple haptic feedback sensors are used on the body? 3D spatial awareness can be achieved, and the user can know where the sound is coming from.


5. Understanding Whisper and the limitations of the model

1. Does Whisper detect non-speech related sounds? e.g. police sirens, car sounds and crashes (Application: user knowing more about dangers around them)
2. Whisper has multilingual training, but can it transcribe in braille?


1. How can words be turned into haptics? What patterns can be deployed?
1. Morse code
2. Mimicking speech intonation
3. Research on positive vs. negative patterns

Haptic Belt and Ring, modeled in Blender

Using the arduino microphone we are able to send the information from the microphone to a microchip storing the logic code we made before

After reading in the binary signal sent from the microchip storing the code logic sets off the vibration or keeps off

WordSense Haptic Controller Circuit Design

1. Applying more sensors and haptics on the belt, allowing for more accurate 360° inputs and corresponding haptics


2. Testing haptic patterns and their interpretations to more accurately represent certain sentiments and emotions


3. Expanding sentiment analysis to cover a broader range of emotions and events, which each sending a unique signal


4. Exploring how ASL can be translated into haptic feedback for more autonomy (being able to multitask and have conversations without having to face a person)