"Soundbyte"
Microsoft Inclusive UX design challenge - 2018.5
Design a product, service, or solution to solve for exclusion in a deskless workplace.
Think about software, hardware, varying interaction modalities, AI & Cognitive augmentation, and/or, a physical workspace. Your design must meet a clear need and be extensible to wider applications. Your project may be near-term practical or blue sky, but the idea must be innovative, technically feasible, and have a realistic chance of adoption if instantiated. Imagine a future that puts human empathy at the design core. Your approach must use the Inclusive Design principles.
overview
This project is aiming to design a wearable product to help people under variable situations. Wearable devices are widely popular over recent years, in order to expand functionalities, companies and designers are pushing frequent and innovative updates to both the devices and the partner applications. However, wearable devices existing in the market are not as accessible as we think. How to make wearable devices more accessible to all? With inclusive UX design challenge for wearable devices, we are able to have a deeper look in this problem.
Working as part of a six-people-group, we spent eight weeks on researching and designing for people hard of hearing. Instructor: Dana Karwas. T.A.: Will Hsu
In this project, I was responsible for UX research, information architecture, feasibility testing and visual design.
Guest client: Nick Katsivelos, Microsoft Inclusive Design Team
During the process, we designed a mobile application with a wearable device. With each prototype, we Iterated several times based on user testing. For deliverable, we made a 140-page booklet to document our process and introduce our products.
Design
Process
Problem
statement
Over 460 million people worldwide have disabling hearing loss. People are constantly exposed both to industrial and recreational activities that result in noise-induced hearing loss, potentially severely reducing the quality of life for those affected. Once lost, hearing cannot be restored in humans. How can we help prevent hearing loss and also improve the hearing of those who have lost it?
Hearing loss is a problem that has existed for more than thousands of years, tracing back to Ancient Egyptian society. Historical records have cataloged many strange remedies for hearing loss: the Ancient Egyptians recommended ingesting a brew of goat urine and insect eggs. Romans simply cupped their hand around the back of the affected ear. The hearing trumpet was a monumental achievement during the Renaissance.
As technology continues to advance, hearing loss treatment has improved by leaps and bounds in the past two centuries.
potential scenarios
Our product focus on users who experience loud noise on a regular basis, like construction workers, airport service agents, and bartenders. The current solutions to hazardous noise in construction site are big, bulky earmuffs or small, foam earplugs. These two choices make it more difficult to properly communicate and may even interfere with real-time information.
Storyboard
Storyboard-1 Construction Site worker, need to filter out noise and keep communication with their colleague.
Storyboard-2 Outdoor Worker need to talk with their co-workers in a noisy environment.
Storyboard-3 People want to get rid of noise.
user research
Hearing and the Brain Event
Hearing and the Brain Event
Kelly Tremblay, Ph.D., an auditory neuroscientist, professor in the Department of Speech & Hearing Sciences at the University of Washington, and HLAA Board Member, discussed the role of the brain in hearing, and how auditory rehabilitation strengthens that brain function.
Interview with people with hearing-loss
Interview with people with hearing-loss
Many participants feel embarrassed to wear hearing aids. Though helpful under general circumstances, hearing aids can be more annoying when there are echoes or loud surrounding noise.
Interview outdoor workers: walkie talkie and earplug
Interview outdoor workers
Construction workers have commented walkie talkies/noise-cancelling headphones/earbuds are most effective in their daily routines. But these headphones/earbuds make it difficult to notice alarms, potentially leading to dangerous accidents.
Microsoft Seeing AI Team and Soundscape Team
Skype with Microsoft teams
We interviewed Microsoft Seeing AI team and Soundscape team through Skype, asked them how to do user testing with people with disabilities and some tips about designing a product for people with disabilities.
Microsoft Hearing AI Team
Hearing AI answers the question: why sound must be heard? It introduced a visualized way to feel sound not only by your ear. This meeting inspired us of sensory substitution and how to design product for people with hearing problems.
Discussions of potential scenarios, persona spectrum and contextual inquiries allowed us to narrow down our target users. Through these observations, we were able to have a much better grasp on the design of our product.Based on our initial idea of smart noise-cancelling earbuds, we then expended our ideas and focused on specific functions that our product should feature.
Ideation
Brainstorming 1.0
competitive analysis
While focusing versatility and portability, current smart earbuds are not concentrating on sound and hearing - the two fundamental functions of earbuds. From this angle, it could be said that many companies are heading towards the wrong direction.
Existing apps lack diverse functionalities. Many focus on health tracking, AI, and general settings. We can aim to build additional functions based on different scenarios our users face in their daily lives.
The second problem for existing apps is making a clear distinction between the wearable device and its partner software. Applications for earbuds should have a different focus than wrist bands or head bands. Earbuds should filter out background noise and focus on useful sounds.
Analytic on existing earbuds
Analytic on application
Application made for wearable devices
After research of existing smart earphones and application for wearable devices, we decided that our product should focus on how to generate sound and make AI assistant as our add-on function. One of the main functions of our earbuds is to filter out noises and contact with people you want to hear. Another function is to transform undetectable frequencies to hearable sound frequencies.
Ideation
Brainstorming 2.0
Sound Filtering
Sound “quality” or “timbre” describes those characteristics of sound which allow the ear to distinguish sounds which have the same pitch and loudness. For example, a note on the piano does not sound like the same note on the clarinet. Timbre is then a general term for the distinguishable characteristics of a tone. Timbre is very important for the voice recognition and differentiation. The timbre depends on the shape of the sound wave, determined by the superposition of the sinusoidal waves characterized by the fundamental sounds and their harmonics. From the point of view of the production of sound, the timbre is determined by the nature (form and composition) of the source of the sound and by the manner in which it is placed in oscillation.
Speaker Diarisation
Speaker diarisation answers the question “who spoke when?” by utilizing speaker segmentation and speaker clustering. By automatically partitioning an input audio stream into homogeneous segments (segmentation), changes in an audio stream could be recognized. Assigning these segments to the same source based on speaker characteristics is speaker clustering. This process can allow to enhance readability by structuring an audio document, or provide the speaker’s true identity when it’s used in conjunction with speaker recognition system.
Sound Detect, Recognition and Distinguish
Having a database of preset sounds (like alarms, voices etc…) stored on our phone. When the software detects sounds, it matches them to the database and identify them. When a particular sound is detected with a high frequency, the software matches this audio input with information in its databases and smooths the high frequency/potentially dangerous sound.
The second way is to associate frequencies to sources of sounds. For example, most alarms have the frequency range 1-3 kHz where the human hearing is the most sensitive. Our device can detect these sounds by associating the features of the sound (frequency, timbre, amplitude) to “alarms”. The same can be done with airplane noise pollution and etc. For example, the airplane engine range is 1,000 to 4,000 Hz. Our device would detect this range and associate it with the airplane. There are health consequences when exposed to this noise -- it can cause annoyance, sleep disturbance, decreased school performance, hearing impairment, hypertension, and ischemic heart disease. By reducing the noise, we can help prevent such dire health consequences.
Noise Cancelling
Noise-cancelling headphones reduce unwanted ambient sounds using active noise control - a method for reducing unwanted sound by the addition of a second sound specifically designed to cancel the first.
Position Detection
There are many ways to locate a sound in a space. This is called acoustic location and was developed in the 40s. There are various methods for positioning and identifying sounds around us. Perceptual-based 3D sound localization works only if the listener is in a narrow listening area (“sweet spot”). The Doppler effect is a phenomenon that consists in the change, with respect to the original value, of the frequency or wavelength perceived by an observer reached by a wave emitted by a source that is moving with respect to the observer. A lot of devices use this phenomenon to detect the positions of sounds, like the echocardiogram.
Bone Conduction
We usually hear sounds through tiny ear bones (bone-conducted or bone-transmitted) and our eardrums (air-conducted or air-transmitted). Most sounds are heard by our eardrums, the membrane of the middle ear, which vibrates in response to sound waves. Its function consists in transmitting the vibrations generated by the sound waves to the ossicles, constituted in a latero-medial sense (from the outside towards the inside) by hammer, incus and stirrup. The tympanic membrane (eardrum) moves the handle of the hammer, to which it is firmly bound. The head of the hammer consequently vibrates in turn, transmitting the sound to the other two ossicles. The vibration of the base of the bracket through the oval window moves the perilinfa of the vestibule and therefore of the cochlea, causing the movement of the cilia of the sensory cells present here. The cochlea in turn is responsible for the transduction of the kinetic impulse into electrochemistry.
Some types of hearing loss, such as tympanosclerosis or Otosclerosis, are due to a malfunction of the eardrum or the ossicles, respectively. Yet, even if the external system (eardrum) is compromised, the internal ones (cochlea, vestibule) are perfectly functional. For this reason, there is heavy research on bone-conduction technology. Bone conduction devices (such as headphones) can imitate the role of your eardrums. These devices decode incoming sound waves and convert them into vibrations that can be received directly by the Cochlea so the eardrum is never directly involved. The “sound” reaches the ears as vibrations through the bones (or skull) and skin.
Research
Proposal
What is Soundbyte?
Google Glasses and Apple Watches spearheading wearable technology focused on the eyes and the wrists. Our product focuses on the ears - a “hearable” device to allow users complete control over real-world sounds. Built with smart noise-cancellation and automatic frequency transformation at its core, Soundbyte is our vision of an inclusive product that can parallel as a speech enhancement device for those hard-of-hearing.
Soundbyte is a wearable device meant for preventing users from the real-world sound environment and helping user with hearing loss by customizing sound into ears.
Besides, it can also interpreting Microsoft Cortana as your artificial inteligence assistant to make life easier. With filter selection, users can focus on what they want to hear in reality. Transform mode help user to adjust frequency based on the hearing preferences.
Who are the users?
Our users include outdoor workers who need to prevent themselves from hearing-loss and keep communicating with their co-workers at the same time, people with hearing problems and want to get rid of embarrassing situation and people live their daily life wanting to get rid of noises.
Sitemap of our Application
Prototyping
Paper prototype for application
Paper prototype 1.0. This prototype includes basic functions like pairing earbuds, AI assistant and our three main function - filter out noise, contact friends and adjust sound frequencies based on users hearing range.
Paper prototype 2.0. We iterated our very-first paper prototype based on testing with our friends and group members. After a lot of discussion on how to make every step more logical, we came up with a new version paper prototype.In this version, a new function allow users to mute a specific direction on the map; equalizers in filter page enable users to directly adjust different sound source around them.
Our first user testing went out not that well. We noticed that without our description about this whole process, most of our users felt confused about how to navigate the application and what would happen after clicking a button.
User Testing
v 1.0
Prototyping
Low-fidelity digital version for application
Based on feedback we received from first user testing, we improved our interface and adjusted some of the buttons. We added instructor before login page and detailed other pages.
In this prototype, we focused on not only functions of our application but also user interface. We did user test with different type of target users, and the results of user testing surprised us.
Lo-fi prototype. Try on by click this.
User testing
v 2.0
Prototyping
High-fidelity digital version for application
Product Prototyping
Product design sketch. Different color stands for different functional part.
Prototype v 1.0. Made from super light clay.
CAD modeling pic for v 2.0
Prototype v 2.0, made from 3D printing.
Based on feedback on the second prototype, we changed the bone conduction part to back of the ear.
CAD modeling pic for v 3.0
Prototype v 3.0, made from 3D printing.
Solution Deliver
Booklet for this project. Check this link for the whole book.
Assessment
User Needs
Starting with contextual inquiries and observation, we focused on the needs of those who have hearing disabilities and also those hard of hearing.
Test
We have researched through video observation, brainstorming sessions, and flow analysis to identify the “pain points” that could discourage potential users during the prototype experience. By faking the environment by using physical sounds, we made it easier and clearer for users to understand our products.
Post-Test
We surveyed more than 20 workers and people with hearing problems with a questionnaire on current hearing products. The questions included ones such as “how to communicate with people in a working environment”, “what are the pros and cons when switching among various technologies”, “which tools they were most comfortable with”, ”whether the app’s UI is easy to understand”, and “what is your understanding of the goals of Soundbyte”
© 2018 TIANYUE WU ALL RIGHTS RESERVED