John Tang

ICS Colloquium (1/31): John Tang

ICS Colloquium
Thursday, Jan. 31, 11:30-1:00
POST 302
Pizza at 11:30, Talk 12-1

John Tang
Senior Researcher
Microsoft Research

Accessible Video Calling:
Using AI to Convey Conversational Cues for Diverse Abilities

Video calling has become a popular conversation medium that conveys emotions and visual cues in ways that importantly augment what’s being said. Expressing disagreement, confusion, surprise, etc. are often done visually, but how can those vital conversational cues be shared with people who cannot perceive them? Our research on making video calling accessible explores using AI to detect these cues and find ways to convey them to people who cannot perceive them. Last year, we explored Video Calling for the Visually Impaired (VC4VI), which uses an AI agent to detect conversational cues, such as nodding head in agreement and shaking head in disagreement, and conveys them to the visually impaired through movie soundtrack-like audio cues so they can react to them. This year, we are exploring Video Calling for the Autistic (VC4A) that detects emotional expression in a conversation and conveys them to people with Autism Spectrum Disorder so they can perceive and respond to them. I will discuss our experiences designing user experiences for these user populations and invite input on some of the challenges we are facing.