Reported on the Google Research blog, a team looking at better immersive experiences with virtual reality and mixed reality have come up with technology. The technology shows faces behind the VR headsets. A mixed reality experiment puts a fake face on top of your VR headset, said The Verge.
Chaim Gartenberg reported Wednesday on what’s up.
“Mixed reality videos—where a video camera and a green screen are used to show what a person using VR is experiencing—aren’t particularly new. But a recent collaboration between Google’s Daydream and Research teams has come up with a new spin on mixed reality by making the VR headsets transparent so you can actually ‘see’ the user’s face.”
The authors of the Google Research blog post reported that “We have used our headset removal technology to enhance Mixed Reality, allowing the medium to not only convey a VR user’s interaction with the virtual environment but also show their face in a natural and convincing fashion.”
Gartenberg said use of “a combination of a 3D scanned model of a person’s face, along with an HTC Vive headset modified with eye tracking technology allows for a digital recreation of the person’s face to be substituted in place.”
As for the scan process: “Each 3D scan typically takes less than a minute and creates a database of eye gazes and blinks, which the software then maps to the data from the eye tracking system in the headset,” said The Verge.
(Gartenberg also said “the software color corrects the digitally created face portion to match the lighting of the rest of the user’s face.”)
The authors of the blog went into detail. They talked about the capture a personalized 3D face model for the user with “gaze-dependent dynamic appearance.” There is the initial calibration step, a one-time calibration procedure, which typically takes less than a minute, they said, for getting a 3D face model of the user, and learn a database that maps appearance images (or textures) to different eye-gaze directions and blinks.
The blog authors said their approach “uses a combination of 3D vision, machine learning and graphics techniques.”
The team working on this are enthusiastic about future possibilities. “Going from an utterly blank headset to being able to see, with photographic realism, the faces of fellow VR users promises to be a significant transition in the VR world, and we are excited to be a part of it.”
Vivek Kwatra, research scientist and Christian Frueh and Avneesh Sud, software engineers, authored the post Tuesday on the Google Research blog.
They wrote about the limitations of virtual reality: sharing these experiences with others. Sharing can be difficult, they said. VR headsets pose a challenge to create a complete picture of the people participating in the experience.”
Over to Mixed Reality, which alleviates the disconnect. They defined it as “a related medium that shares the virtual context of a VR user in a two dimensional video format allowing other viewers to get a feel for the user’s virtual experience.”
However, they added, the headset continues to block facial expressions and eye gaze so you still get that “hurdle to a fully engaging experience and complete view of the person in VR.”
That is where the team working on the tech aims to make a difference. They show a picture of what they have done, with the following caption. “A VR user captured in front of a green-screen is blended with the virtual environment to generate the MR output: Traditional MR output has the user face occluded, while our result reveals the face.”
So in short you get a see-through headset, with the user’s face revealed. Their true eye-gaze has been recreated, subsequently merged with the virtual environment to create the final MR video. The synthesized face looks natural and alive.
More information: TechXplore
