Watch scientists control a robot with their hands while wearing the Apple Vision Pro

Scientists have developed an app for the Apple Vision Pro mixed reality headset that lets users control a robot using head and hand gestures alone. It could be used to control machines remotely in different scenarios — from playing pranks to navigating a disaster zone.

The app, called "Tracking Streamer," tracks human movements — in particular how the head, wrist and fingers move — and streams this data over a Wi-Fi connection to a robot on the same network. The connected robot then translates the data into corresponding movements. 

The researchers published their work in a short paper published March 9 on Github — an open-source code repository. Their system tracks 26 points in the hands and wrists, as well as separate points in the head. It also records spatial data — such as how high up from the ground you are.

Related: Please don't wear the Apple Vision Pro while driving, study urges, but buying coffee with VR is an 'exciting novelty'

Developer of the app Younghyo Park, a doctoral student at MIT published a video to the website X, formerly known as Twitter, showing the system in action.

In a short video clip, study co-author Gabe Margolis, a grad student at MIT, controls a four-legged robot using his hands and body movements. 

Margolis performs gestures to command a robot to approach a closed door, turn the handle with its gripper and enter. In another sequence, he gestures for the robot to pick up a plastic trash can lid and throw it into the garbage. At one point, he also bends down — which triggers the robot to mimic him.

Apple launched its Vision Pro virtual reality (VR) headset in Feb. 2024, with footage since emerging of people wearing the device in their day-to-day lives. Shortly before its launch, scientists published research that investigated how well VR headsets performed if you were to use them in daily life. 

They found the devices had a lag time, cut off peripheral vision and caused fun-house-mirror-like distortions. They warned against wearing VR headsets while driving or walking downstairs.

Park and Margolis speculated in their paper that some users may wear the Apple Vision Pro all the time, like glasses, and that continued use will provide more data that can be fed into teaching robots how people move.