THE CHALLENGE
Well-intentioned AR users often unknowingly violate bystander privacy by collecting and sharing sensitive data. Some apps use machine learning to make inferences without user knowledge, further compromising privacy. The current permissions within the state of the art are flawed as colluding apps deceive users to access and share data covertly. Bystander privacy concerns have risen with modern AR devices continuously capturing the physical world. Surveys show public worry about data collection even in medical or assistive use.
There is an undeniable need to find a solution that effectively protects bystander privacy while still offering an immersive AR experience.
OUR SOLUTION
This new technology invented by Bo Ji and his team builds on a key insight that the device user’s eye gaze and voice is a highly effective indicator for subject/bystander detection in an interpersonal interaction, and leverages novel AR capabilities such as eye gaze tracking, near-field microphone, and spatial awareness to achieve a frame rate of 52 frames per second for a head-mounted AR device while detecting and obscuring bystanders captured within sensor data in real-time. BystandAR correctly identifies and protects over 98% of bystanders within the data stream while allowing access to over 96% of the subject data. This solution does not require offloading unprotected bystander data to another device for analysis - a potential avenue of privacy leakage in prior art.
A malicious AR application is designed to request bystander visual data from a device, infer sensitive information from this data, and offload the inference results to another location for exploitation. BystandAR, in the form of a privacy-preserving API, shown between the device sensors and the malicious applications, is designed to prevent this.
