Enabling Inclusive Communication on the Apple Vision Pro
AR/VR Design
Accessibility
project management
My Role
My Role
Project Manager
Lead UI/UX Designer
Project Manager
Lead UI/UX Designer
Area
Area
Product Strategy
User Experience
Product Strategy
User Experience
My Contribution
My Contribution
Product Concept
UX Research
Rapid Wireframes
High-fidelity Prototyping
Product Concept
UX Research
Rapid Wireframes
High-fidelity Prototyping
Tools
Tools
Figma
Swift UI
Figma
Swift UI
Teamwork
Teamwork
Software Developers*4
Business Analysts*2
UI/UX designers*2
Software Developers*4
Business Analysts*2
UI/UX designers*2
The Challenge
Simple conversations can be a daily struggle when one person can hear and the other cannot.
Simple conversations can be a daily struggle when one person can hear and the other cannot.
What are some paths we can take to explore bridging this gap and increasing quality of communication for each human. The team had no experience developing or designing for accessibility or mixed-reality.
What are some paths we can take to explore bridging this gap and increasing quality of communication for each human. The team had no experience developing or designing for accessibility or mixed-reality.
The SOLUTION
Breaking into both virtual and physical spaces
Breaking into both virtual and physical spaces
Mixed-reality gives us a window of opportunity and we began bridging worlds to give everyone a voice. Whether it is sign language to text or speech to text, there were many task flows to consider and we had a lot of decisions to make.
Mixed-reality gives us a window of opportunity and we began bridging worlds to give everyone a voice. Whether it is sign language to text or speech to text, there were many task flows to consider and we had a lot of decisions to make.
Timeline
A linear timeline that was not so linear.
Exploration & Research
Exploration & Research
USER RESEARCH
Individuals hard of hearing feel invisible,
isolated and dependent.
Individuals hard of hearing feel invisible,
isolated and dependent.
With the help of the amazing MCommunity, my teammate and I conducted observational studies with three individuals with hearing impairment. Through user observation and discourse we brought clarity to understanding the problems users currently had.
Looking back, I would have urged the team to spend more time recruiting more participants for deeper insights.
With the help of the amazing MCommunity, my teammate and I conducted observational studies with three individuals with hearing impairment. Through user observation and discourse we brought clarity to understanding the problems users currently had.
Looking back, I would have urged the team to spend more time for deeper insights.
Users want to :
Users want to :
Be aware of their surroundings
Be aware of their surroundings
Have autonomy and control
Have autonomy and control
Experience Frictionless Applications
Experience Frictionless Applications
Translating insights to actionable-findings
Translating insights to actionable-findings
Reducing the activation energy to start the communication process
Reducing the activation energy to start the communication process
Effective translation in flexible environments
Effective translation in flexible environments
Empowerment in communicating with any individual
Empowerment in communicating with any individual
Exploring the Connection Formula
Exploring
Exploring
Exploring the Connection Formula
Exploring the Connection Formula
Technology research
VisionPro Is Here. Is Mixed Reality the Right Move?
VisionPro Is Here. Is Mixed Reality the Right Move?
Seeing an opportunity in the new technology seemed to be incredibly exciting, but to intentionally see if it was viable we needed to take some steps to ensure and explore other solutions. Experiencing the VisionPro in-person for the first enabled us to utilize a beginner's mindset. We began thinking about spatial design, placement of the navigation bar. Many of my design iterations were derived from experience on the headset.
Seeing an opportunity in the new technology seemed to be incredibly exciting, but to intentionally see if it was viable we needed to take some steps to ensure and explore other solutions. Experiencing the VisionPro in-person for the first enabled us to utilize a beginner's mindset. We began thinking about spatial design, placement of the navigation bar. Many of my design iterations were derived from experience on the headset.
Intersectional research
Navigating intersections between disciplines
Navigating intersections between disciplines
By exploring spatial design, accessibility, and AR/VR technology, a three-way Venn diagram highlighted intersecting areas to focus on:
Clear visuals and lighting- using sight lines, zoning, and fewer occlusions so people can rely more on visual cues and lip-reading
Personalized settings- letting users adjust features to match different signal processing needs
Adaptive, immersive cues- building in environmental signals that scale with perception and spatial awareness.
By exploring spatial design, accessibility, and AR/VR technology, a three-way Venn diagram highlighted intersecting areas to focus on:
Clear visuals and lighting- using sight lines, zoning, and fewer occlusions so people can rely more on visual cues and lip-reading
Personalized settings- letting users adjust features to match different signal processing needs
Adaptive, immersive cues- building in environmental signals that scale with perception and spatial awareness.
Putting the pieces together
Putting the pieces together
User A, B and C indicated feeling excluded in conversations and lack of physical presence. Leveraging Mixed Reality (XR),we can tap into spatial awareness and UI for multimodal screen integration to facilitate more inclusive interactions- recognizing both sign language and speech- while enabling more accessible task flows
User A, B and C indicated feeling excluded in conversations and lack of physical presence. Leveraging Mixed Reality (XR),we can tap into spatial awareness and UI for multimodal screen integration to facilitate more inclusive interactions- recognizing both sign language and speech- while enabling more accessible task flows
PROBLEM STATEMENT
How do we facilitate communication between people with different abilities to increase the quality of communication?
How do we facilitate communication between people with different abilities to increase the quality of communication?
Feature Implementations
Feature Implementations
Scoping & Converging
Narrowing down the features
Narrowing down the features
As Co-Project Manager, the team and I ran
cross-functional sprints between engineering, design and business to ask three questions for each feature:
Does this meet our user’s needs?
How feasible would implementation be?
How viable it seems?
As Co-Project Manager, the team and I ran
cross-functional sprints between engineering, design and business to ask three questions for each feature:
Does this meet our user’s needs?
How feasible would implementation be?
How viable it seems?
Feature will be implemented?
Conceptualizing & Visualizing
Implementing Concept into AR/VR Spaces
Implementing Concept into AR/VR Spaces
The low-fidelity enabled me to quickly validate usability and feature placement early on. Due to constraints for technical implementation we narrowed it down to three main features- Sign language to Text, Speech-to-Text and Translation.
The low-fidelity enabled me to quickly validate usability and feature placement early on. Due to constraints for technical implementation we narrowed it down to three main features- Sign language to Text, Speech-to-Text and Translation.
Conceptualizing and decision making through low-fidelity wireframes Enables quickly validate usability and feature placement early on.
Mapped out how users would move through key actions, ensuring clarity and efficiency before adding complexity
Balance accessibility & feasibility post-development
Balance accessibility & feasibility post-development
I began by creating accessibility-focused wireframes, but once the first feature was developed, I noticed major gaps. The implementation didn’t align with basic WCAG guidelines, and intent behind my designs had been lost in translation. I revisited my files and set up focused meetings with the team, to ensure that accessibility was also carried through into development.
I began by creating accessibility-focused wireframes, but once the first feature was developed, I noticed major gaps. The implementation didn’t align with basic WCAG guidelines, and intent behind my designs had been lost in translation. I revisited my files and set up focused meetings with the team, to ensure that accessibility was also carried through into development.
Dive into Key Design Decisions
Dive into Key Design Decisions
High-fidelity wireframes
Feature highlight website
Built a Product Feature website highlighting Features and basic marketing efforts
Apple VisionOS App
VisionTranslate
VisionTranslate is an AR/VR-based sign language tool that provides real-time speech-to-text, multilingual support, and customizable settings for users with hearing impairments.
Translate sign into text in real-time
Translate sign into text in real-time
Live captions appear instantly when someone else signs
Live captions appear instantly when someone else signs
Speech-to-text converts spoken words into on-screen captions.
Speech-to-text converts spoken words into on-screen captions.
Seamless multilingual translation (40+ languages) across task flows
Seamless multilingual translation (40+ languages) across task flows
Expanding Impact
Expanding Impact
Learnings
It takes a village to raise a product
It takes a village to raise a product
There were moments of disconnect between the engineering and design team, especially after implementation. Since it was my first time taking on a leadership role for an end-to-end process, I learnt how important it was to mend to ensure alignment between teams. It was the answer to making or breaking the quality in the final product. Another very important thought I intentionally carry with me today, is bringing everyone's strengths to the table. Each team member displayed a different strength, regardless of what role was on the paper. Sometimes, the engineers had better design solutions and other times designers had better engineering solutions- and thats great!
While we shipped out this application on the app store, due to time and technical constraints it wasn’t the highest quality of work I would have liked it to be (a classic cross-functional team set back)- even today I remember things I can do to make it better!
There were moments of disconnect between the engineering and design team, especially after implementation. Since it was my first time taking on a leadership role for an end-to-end process, I learnt how important it was to mend to ensure alignment between teams. It was the answer to making or breaking the quality in the final product. Another very important thought I intentionally carry with me today, is bringing everyone's strengths to the table. Each team member displayed a different strength, regardless of what role was on the paper. Sometimes, the engineers had better design solutions and other times designers had better engineering solutions- and thats great!
While we shipped out this application on the app store, due to time and technical constraints it wasn’t the highest quality of work I would have liked it to be (a classic cross-functional team set back)- even today I remember things I can do to make it better!
Next Project
University of Michigan, Improvised Route prediction Model
University of Michigan, Improvised Route prediction Model
Developed and Designed enhanced Bus transportation system
Oversaw end-to-end Process for Development
and design
/Development
/UX Design