VisionTranslate helps hard-of-hearing users talk to anyone by translating sign language on the Apple Vision Pro
UI/UX Design Lead | Shipped September, 2024 for visionOS
My Contribution
Spatial UX and mixed reality (MR) design
UX Research
Low and High fidelity prototyping
Co-project management
Outcomes
Launched an MVP on the App store exclusively for visionOS attracting 100+ users in the first week
Won the Strategy & Design Sprint at Michigan Ross in 2024
Development and design speed went up 83%
In a room full of people, the hardest part of a conversation shouldn’t be the setup
For many hard-of-hearing people, the friction exist before the conversation even starts: open an app, prop up a phone, or wait for an interpreter. By the time they’re ready, the moment has passed. Conversations live in fluidity, one participant put it simply: “Even when I’m physically present, it feels like I’m not really there.”
VisionTranslate puts the conversation back in the room. It’s a native visionOS app that turns sign language, speech, and text into live captions across 40+ languages, right in your line of sight. No phone to look down at. No one to wait for. While the VisionPro focused on entertainment, we found an opportunity to leverage both: mixed-reality technology and the new VisionPro, to solve this challenge for an overlooked community.
I led design and co-led the project with Kunal. No one in our team had built for visionOS before. We shipped to the App Store, reached 100+ users in the first week, and won the Ross Strategy & Design Sprint.
Things to keep in mind
Reasoning
Hard-of-hearing users juggle apps, interpreters, and tools just to start a conversation.
Each tool takes a different setup, so the environment ends up deciding how people communicate.
Goal
Help hard-of-hearing users feel in control of any conversation in flexible spaces.
Personal goal: pushing my limits in spatial computing, visionOS, and design for 3D spaces
Dynamics
Our limited experience and expertise with visionOS, XR, and spatial design across the team
We had restricted access to hard-of-hearing users for continuous testing
Time and resource limits with student responsibility
Hardware we didn’t own: only at an Apple Store or our technology library
Context
Users need to talk directly, without an interpreter, without asking permission, and without fighting an app first.
Hearing loss is on a spectrum. Each individual has a different set of criteria to meet.
Formation
What actually breaks a conversation?
I chose to focus on one thing: helping users feel present in a conversation. Every existing tool pulled people’s eyes and attention away from the person they were talking to. Spatial UI could keep both in the same view. I set out to design three tight flows (sign-to-text, speech-to-text, and personalizations) to remove friction.
Form 02
People felt invisible, isolated, and loss of control
In interviewing three hard-of-hearing users, what they did explained a lot more than what they said. For example, each participant spent extra time preparing their setup before talking. At the end, all three participants raised the need for autonomy. They didn’t want more help, they wanted to need less of it.
Attention perception can be fake
Since users depend on technology and external tools, it can falsify perceived attention by the person(s)
Slow and Cumbersome Apps
Whether it was an app, translator or tool, we saw how people needed time to set up before conversation
Difficulty in initiating conversation
Starting a conversation is an area people often feel the most friction. Sometimes between conversations
Reduced confidence due to dependency
Over time, users became dependent on their tools and constant users hampered their confidence
Form 03
Testing Apple's Vision Pro with a beginner’s mindset
My teammate and I would go to the Apple store to try the Vision Pro and observe design as new users. I studied the overlap of spatial design, accessibility, and human factors contributing to it, giving me three concrete principles that shaped every screen while we designed.
Clear sight lines
Use zones and fewer occlusions so users can lip-read and pick up visual cues on the user interfaces
Adjustable by default
Fun fact: hearing loss is on a spectrum, so users need to tune the capabilities to their needs
Signals that scale
Hard-of-hearing users lean on sight, so environmental cues should adapt to how they perceive
Form 04
Scoping features to protect the core capabilties
We ran cross functional sprints often where every feature had to pass three core questions:
Can we build this with visionOS and MR limits?
Would our users actually find relief in this and why?
Does it remove friction from a conversation, and at which point exactly?
Sharing conversations, playback controls, conversation history, and save/favourite were great but had to be cut. Mapping feature interactions in a 3x3 matrix, ChatGPT as in-house critic helped find edge cases and solve logic complexities. The team made decisions quicker. For example, Text-to-voice sounded promising, however no user raised through our interviews- so out of scope.
Form 05
Get engineering, design, and business to work from one picture
Pushing for visual sprints early helped us clarify and test usability in a low-risk environment before any code. Apple’s SwiftUI design system and spatial patterns like, multi-sensory feedback, spatial text placement, captions, and minimal motion helped prevent situations like motion sickness. I ran informal tests through sprints and looped results straight back into design.
Form 05
Relaxing the interface by using pop-on captions
During active translation the UI moved too much. The screens were scalining and captions appeared word by word, which stacked two layers of motion on top of each other. I switched us to the highlighted pop-on captions for both sign-language and speech-to-text. They’re easier to read and focus on, especially for users with dyslexia or other neurodivergence which are commonly paired with hard-of-hearing. The change also saved a week of dev time!
Form 05
Put language switching one tap away
Initially, changing the system language took about 4.6 seconds and around 2 wrong clicks. The translation control was buried. Since it supports both the core features, I moved it into the tab bar ornament so it stays in the same place on every screen. They cut wrong clicks to 1, and nobody had to pause a conversation to switch. It was a much better fit for multilingual users like P3 from research.
Form 05
Carry accessibility consideration through to the build
Accessible wireframes don’t count for much if the build loses it. I set up dedicated meetings to check build against WCAG AA, then adjusted contrast, spacing, and layer blur for readability. I also used the serial position effect to decide where simplicity mattered most. Onboarding and home page came first, settings came last.
Screens from the first iterations
Results
Available on the visionOS App Store with 100+ new users the first week.
Won the Strategy & Design sprint at Michigan Ross Business 2024
Conversations on their own terms without interpreters or tools
Now, people who are hard-of-hearing can just start talking. People sign and VisionTranslate turns their signs into text and a spoken voice. No matter the language spoken people can understand and adjust settings as well. No setup, no interpreter, no looking away- the conversation runs on their terms, not their tools’.
Next
Over 300+ kinds of sign languages
Senior citizen & Dyslexia friendly
