Enabling Inclusive Communication on the Apple Vision Pro
Enabling Inclusive Communication on the Apple Vision Pro
AR/VR Design
AR/VR Design
Accessibility
Accessibility
Project Management
Project Management
My Role
My Role
Project Manager
Lead UI/UX Designer
Project Manager
Lead UI/UX Designer
Project Manager
Lead UI/UX Designer
My Contribution
My Contribution
Product Concept
Research
Wireframes
MockUps
Product Concept
Wireframes
Prototyping
Product Concept
Research
Wireframes
MockUps
Area
Area
Product Strategy
User Experience
Product Strategy
User Experience
Product Strategy
User Experience
Teamwork
Teamwork
Software Developers*4
Business Analysts*2
UI/UX designers*2
Software Developers*4
Business Analysts*2
UI/UX designers*2
Software Developers*4
Business Analysts*2
UI/UX designers*2
The Project
During my first semester as a sophomore, I joined a product development club that matched students with engineers, designers, and business majors across campus, that’s how I met my wonderful teammates: Kunal Mansukhani, Tim John, Aarush Khanna, Selina Sun, Will Draper, and Sherrie Li. With only a vague sense of timeline and potential, this became our chance to create a project with real value. During this time, we also won the Ross + Strategy Design Sprint 2024 at the University of Michigan.
During my first semester as a sophomore, I joined a product development club that matched students with engineers, designers, and business majors across campus, that’s how I met my wonderful teammates: Kunal Mansukhani, Tim John, Aarush Khanna, Selina Sun, Will Draper, and Sherrie Li. With only a vague sense of timeline and potential, this became our chance to create a project with real value. During this time, we also won the Ross + Strategy Design Sprint 2024 at the University of Michigan.
My Role
I joined the team as a UX Designer in a cross-functional group of software engineers, designers, and business majors responsible for taking a product from zero to one. As my involvement and impact grew, so did my responsibilities. I oversaw the design direction, helped in feature prioritization, and led project meetings. As the design lead, I handled the user interface, high-fidelity wireframes, and mockups that informed accessibility, development, and marketing. In parallel, I co-led the project with Kunal, focusing on team alignment and managing project timelines.
I joined the team as a UX Designer in a cross-functional group of software engineers, designers, and business majors responsible for taking a product from zero to one. As my involvement and impact grew, so did my responsibilities. I oversaw the design direction, helped in feature prioritization, and led project meetings. As the design lead, I handled the user interface, high-fidelity wireframes, and mockups that informed accessibility, development, and marketing. In parallel, I co-led the project with Kunal, focusing on team alignment and managing project timelines.
Task overview
When Apple released the Vision Pro, we saw an opportunity to dive into the spatial computing space and push our technical skills. As we brainstormed ways to meaningfully use this medium, we recognized how underrepresented the disabled community is, and how communication remains one of the hardest challenges for people who are hard-of-hearing.
Another honest motivator? We were tired of seeing the usual dining hall apps, bus apps, and housing apps that students kept building year after year. This was our moment to use our 18-year-old beginner’s mindset to create something new, something we weren’t sure would work, but knew we could learn from.
Over six months, our Design, Engineering, and Business/Marketing teams built an accessibility-focused communication platform for the Vision Pro. We created an experience that empowers users who are hard-of-hearing to communicate with others, regardless of their ability.
When Apple released the Vision Pro, we saw an opportunity to dive into the spatial computing space and push our technical skills. As we brainstormed ways to meaningfully use this medium, we recognized how underrepresented the disabled community is, and how communication remains one of the hardest challenges for people who are hard-of-hearing.
Another honest motivator? We were tired of seeing the usual dining hall apps, bus apps, and housing apps that students kept building year after year. This was our moment to use our 18-year-old beginner’s mindset to create something new, something we weren’t sure would work, but knew we could learn from.
Over six months, our Design, Engineering, and Business/Marketing teams built an accessibility-focused communication platform for the Vision Pro. We created an experience that empowers users who are hard-of-hearing to communicate with others, regardless of their ability.
Goals
Improve communication abilities for people with hard-of-hearing
Leverage the Apple Vision Pro to build skills in spatial computing
Pushing focused skills, even without certainty about feasibility or the development process
Improve communication abilities for people with hard-of-hearing
Leverage the Apple Vision Pro to build skills in spatial computing
Pushing focused skills, even without certainty about feasibility or the development process
Limitations
Limited experience with VisionOS and spatial design
Restricted access to hard-of-hearing users for continuous testing and feedback
Time and resource constraints as a student-led project
Building rapport with the team
Limited experience with VisionOS and spatial design
Restricted access to hard-of-hearing users for continuous testing and feedback
Time and resource constraints as a student-led project
Building rapport with the team
No team member had previous experience with spatial or SwiftUI
Limited personal access to community with disabilities
No established rapport with teammates
Timeline
A linear timeline that was not so linear.
Research Process
The Challenge
Once we began research, the data was undeniable. The challenge wasn't just about the technology, it was about learning how to design and develop for a type of user that was completely new. We had to make sure we were targeting a problem that is accurate and relevant.
Once we began research, the data was undeniable. The challenge wasn't just about the technology, it was about learning how to design and develop for a type of user that was completely new. We had to make sure we were targeting a problem that is accurate and relevant.
Simple conversations is a daily struggle when one person can hear and the other cannot.
Simple conversations is a daily struggle when one person can hear and the other cannot.
critical thinking
Here were two questions I got from my 'sketchbook' unfiltered. There were two spaces we needed to really think about in terms of what exactly were going to solve and how were we going to do that.
Here were two questions I got from my 'sketchbook' unfiltered. There were two spaces we needed to really think about in terms of what exactly were going to solve and how were we going to do that.
What exactly are points of friction for users that are hard-of-hearing ?
Explore the value technology can provide to improve the experience?
Interviewing and reviewing
Tp understand the problem space better, we conducted three interviews with participants that were hard-of-hearing. Interviews and contextual inquiry revealed that they felt invisible, isolated and dependent. During the interview, the key to derive the most productive insights was to observe, what they have said, but most importantly how. Here are three key insights we derived:
Tp understand the problem space better, we conducted three interviews with participants that were hard-of-hearing. Interviews and contextual inquiry revealed that they felt invisible, isolated and dependent. During the interview, the key to derive the most productive insights was to observe, what they have said, but most importantly how. Here are three key insights we derived:
Technology dependence can falsify perceived attention between two people communicating
Users need to have autonomy given any moment in an conversation
All existing technology applications carry friction in set up
Technology dependence can falsify perceived attention between two people communicating
Users need to have autonomy given any moment in an conversation
All existing technology applications carry friction in set up
VisionPro Is Here. Is Mixed Reality the Right Move?
Technology research
Seeing an opportunity in the new technology was exciting, but we had to verify to make disciplined choices and explore other solutions. I began thinking about spatial design, placement of the navigation bar. Many of my design iterations were derived from experience on the headset.
Seeing an opportunity in the new technology was exciting, but we had to verify to make disciplined choices and explore other solutions. I began thinking about spatial design, placement of the navigation bar. Many of my design iterations were derived from experience on the headset.
Experiencing the VisionPro for the first time
Cross-domain Knowledge
I had to understand what it meant to converge between three non-technical and technical disciplines in particular to Spatial Design, Accessibility and AR/VR Technology. I used a mix of YouTube videos, interviews and development observations
I had to understand what it meant to converge between three non-technical and technical disciplines in particular to Spatial Design, Accessibility and AR/VR Technology. I used a mix of YouTube videos, interviews and development observations
Clear visuals and lighting:
Using zone and fewer occlusions so people rely on visual cues and lip-reading
Clear visuals and lighting:
Using zone and fewer occlusions so people rely on visual cues and lip-reading
Clear visuals and lighting: Using zone and fewer occlusions so people rely on visual cues and lip-reading
Personalized settings:
Letting users adjust features to match different signal processing needs
Personalized settings:
Letting users adjust features to match different signal processing needs
Personalized settings: Letting users adjust features to match different signal processing needs
Adaptive, immersive cues:
Building in environmental signals that scale with perception and awareness
Adaptive, immersive cues:
Building in environmental signals that scale with perception and awareness
Adaptive, immersive cues: Building in environmental signals that scale with perception and awareness
Three domains; spatial design, accessibility and XR Technology
What we found, need to solve and how?
ReseArch summary
User A, B and C indicated feeling excluded in conversations and lack of physical presence. Leveraging Mixed Reality (XR),we can tap into spatial awareness and UI for multimodal screen integration to facilitate more inclusive interactions- recognizing both sign language and speech- while enabling more accessible task flows
HOW MIGHT WE…
Improve communication for people with hard-of-hearing in order to empower them with autonomy and a physical space resulting in confident conversations?
cONCEPT PROPOSAL
A mixed-reality (XR ) communication tool for sign language that provides live speech-to-text, multilingual support, and customizable settings for users with hearing impairments.
What we found, need to solve and how?
ReseArch summary
User A, B and C indicated feeling excluded in conversations and lack of physical presence. Leveraging Mixed Reality (XR),we can tap into spatial awareness and UI for multimodal screen integration to facilitate more inclusive interactions- recognizing both sign language and speech- while enabling more accessible task flows
HOW MIGHT WE…
Improve communication for people with hard-of-hearing in order to empower them with autonomy and a physical space resulting in confident conversations?
cONCEPT PROPOSAL
A mixed-reality (XR ) communication tool for sign language that provides live speech-to-text, multilingual support, and customizable settings for users with hearing impairments.
Design Process
Transitioning from Research to Action-oriented Insights
The results from our research gave us concrete insights on what users who are hard-of-hearing needed from an accessibility forward mindset. We understood that the environment was complex and by breaking the physical and virtual space we could offer solution to these pain points.
The results from our research gave us concrete insights on what users who are hard-of-hearing needed from an accessibility forward mindset. We understood that the environment was complex and by breaking the physical and virtual space we could offer solution to these pain points.
Reducing the activation energy to start the communication process
Reducing the activation energy to start the communication process
Provide effective translation in flexible environments
Provide effective translation in flexible environments
Empower users to communicating with any individual with any ability
Empower users to communicating with any individual with any ability
Scoping and features decisions
The team and I ran cross-functional sprints between engineering, design and business to ask three questions for each feature: 'Does this meet our user’s needs? 'How feasible would implementation be? 'What would the timeline look like? '
The team and I ran cross-functional sprints between engineering, design and business to ask three questions for each feature: 'Does this meet our user’s needs? 'How feasible would implementation be? 'What would the timeline look like? '
Icons next to the features indicated whether it would be implemented at this stage
Implementing Concept into AR/VR Spaces
As we began looking at the variety of possibilities after narrowing the features, the team needed something to visualize with. As I began sketching concepts it enabled us to quickly validate usability and feature placement early on. We narrowed it down to three main features- Sign language to Text, Speech-to-Text and Translation
As we began looking at the variety of possibilities after narrowing the features, the team needed something to visualize with. As I began sketching concepts it enabled us to quickly validate usability and feature placement early on. We narrowed it down to three main features- Sign language to Text, Speech-to-Text and Translation
Low-fidelity concept and wireframes helped team communication and decision making
Mapping out interactions through key actions and flow before adding complexity
Over 70+ wireframes to refine ideas
Needed more focus between accessibility & feasibility post-development
I began by creating accessibility-focused wireframes, but once the first feature was developed, I noticed major gaps. The implementation needed improvement to align with WCAG AA guidelines. We set up focused meetings, to ensure that accessibility was also carried through into development.
I began by creating accessibility-focused wireframes, but once the first feature was developed, I noticed major gaps. The implementation needed improvement to align with WCAG AA guidelines. We set up focused meetings, to ensure that accessibility was also carried through into development.
Post-developement Session A
Diving into key design decisions
One of the coolest parts of doing research before hand, is that high-fidelity designing was so much easier now that we had clarity on the user and purpose. I began designing wireframes key flows and screens as we had weekly team syncs.
One of the coolest parts of doing research before hand, is that high-fidelity designing was so much easier now that we had clarity on the user and purpose. I began designing wireframes key flows and screens as we had weekly team syncs.
I also created a website for our marketing goals
Bonus: website for our marketing goals
Built a website highlighting our product's capabilities and building history
Built a website highlighting our product's capabilities and building history
Apple VisionOS App
VisionTranslate
VisionTranslate is an AR/VR-based sign language tool that provides real-time speech-to-text, multilingual support, and customizable settings for users with hearing impairments.
VisionTranslate is an AR/VR-based sign language tool that provides real-time speech-to-text, multilingual support, and settings for users with hearing impairments
Real-time translation of sign language to text & audio
When you sign to someone else, your signs will display as a text and can be voiced out aloud for another person to hear live.
When you sign to someone else, your signs will display as a text and can be voiced out aloud for another person to hear live.
Understanding someone else's sign language
There are 300+ sign languages, so if you don't know one- don't worry! VisionTranslate instantly converts sign-language to text
There are 300+ sign languages, so if you don't know one- don't worry! VisionTranslate instantly converts sign-language to text
Speech to Text mode: View live captions instantly when someone else is speaking
Be physically present while you sit in a room with multiple people, you are able to view live captions when they talk or choose not to
Be physically present while you sit in a room with multiple people, you are able to view live captions when they talk or choose not to
Customizable Settings & Personalization
Change your settings based on your comfort. Whether you want a bigger font, slower transcription or higher contrast do it all here
Change your settings based on your comfort. Whether you want a bigger font, slower transcription or higher contrast do it all here
Expanding Impact
Learnings
This was not a feasible product.
This project is a good example to show that the application's success metric was also defined by the success of over-heading product: VisionPro. The physical product served as the primary influencer before our application made any contact. While the VisionPro served as a fantastic medium to get our idea implemented, this was not a great example of a feasible product in the market.
This project is a good example to show that the application's success metric was also defined by the success of over-heading product: VisionPro. The physical product served as the primary influencer before our application made any contact. While the VisionPro served as a fantastic medium to get our idea implemented, this was not a great example of a feasible product in the market.
Scoping is important first
As I took a leadership role for an end-to-end process, I learnt how important alignment between teams was- it was the answer to making or breaking the product quality. There were moments of disconnect between the engineering and design team, especially after implementation. With a little more scoping and planning, we could have shipped the MVP at a higher quality of work that I would have liked it to be (a classic cross-functional team set back)- and today I think of certain ideas to make the next project better.
As I took a leadership role for an end-to-end process, I learnt how important alignment between teams was- it was the answer to making or breaking the product quality. There were moments of disconnect between the engineering and design team, especially after implementation. With a little more scoping and planning, we could have shipped the MVP at a higher quality of work that I would have liked it to be (a classic cross-functional team set back)- and today I think of certain ideas to make the next project better.
The purpose of a MVP is to collect data
Looking back, the purpose of a product on the market is to validate certain hypothesis, assess the product and derive insights that shape the next iteration. As a team we did pretty well only until the MVP launch. I could have said, "Beta on the way" or something, but considering school timelines and shifting priorities, we did not move forward. I would like to see how this would play out if given a shot to improve.
Looking back, the purpose of a product on the market is to validate certain hypothesis, assess the product and derive insights that shape the next iteration. As a team we did pretty well only until the MVP launch. I could have said, "Beta on the way" or something, but considering school timelines and shifting priorities, we did not move forward. I would like to see how this would play out if given a shot to improve.
Meanwhile, we won the Michigan Ross + Tech design Sprint 2024! 🏆
Devyani Jain
devyanij@umich.edu
Devyani Jain
devyanij@umich.edu
Devyani Jain
devyanij@umich.edu