Android users will soon have access to a revolutionarily accessible form of device control: Completely hands-free navigation, using the power of Google AI and facial tracking.
Part of the company’s long list of updates and announcements rolling out at its developer keynote event today, the new feature is a mobile version of Google’s desktop offering known as Project Gameface. The new virtual cursor uses Android accessibility services and a database of facial expressions from MediaPipe’s Face Landmarks Detection API to allow for broader customization and manipulation of hands-free tech for both users and developers.
“Through the device’s camera, it seamlessly tracks facial expressions and head movements, translating them into intuitive and personalized control. Developers can now build applications where their users can configure their experience by customizing facial expressions, gesture sizes, cursor speed, and more,” Google explained in its announcement.
Google partnered with international accessibility solutions group Incluzza to test the Gameface expansion in broader, non-gaming contexts, including work and social tasks.
Mashable Light Speed
Project Gameface first launched in 2023 as an open-source, hands-free gaming mouse allowing users to operate computer cursors with just head and facial movements. The technology was designed in collaboration with viral video game streamer Lance Carr, who is quadriplegic, as a more accessible option to expensive head-tracking systems. It also introduced the option of gesture size, allowing more customizability for users of different mobility.
“We’ve been delighted to see companies like playAbility utilize Project Gameface building blocks, like MediaPipe Blendshapes, in their inclusive software,” Google wrote. “Now, we’re open sourcing more code for Project Gameface to help developers build Android applications to make every Android device more accessible.”
Google also announced new AI features for its screenreader technology, TalkBack, which will provide more detailed descriptions and fill in information for unlabeled images on the web to help users who are blind or have low vision.
Project Gameface is now available for developers on GitHub.