A Holodeck to Train Robots
I’m a Trekkie (not a “Trekker,” thank you), so I’m tickled every time I see one of these “Star Trek inspires tech” stories.
If you’ve watched TNG (The Next Generation) with the venerable Sir Patrick Stewart serving as captain, the crew of the USS Enterprise used the Holodeck—a “virtual reality” chamber that created 3D environments for training and recreation. (Fun fact: Gene Roddenberry, the creator of the Star Trek universe, came up with the idea for the Holodeck in 1968. The tech debuted in a Star Trek: The Animated Series episode called “The Practical Joker” in 1974.)
Today, robots use these interactive virtual environments to train robots with a process called “Sim2Real.” Artists used to spend weeks making these virtual environments, but there were more robots needing training than there were virtual spaces, so researchers looked to AI.
“Generative AI systems like ChatGPT are trained on trillions of words, and image generators like Midjourney and DALLE are trained on billions of images,” says Callison-Burch, associate professor of Computer and Information Science at the University of Pennsylvania.
“We only have a fraction of that amount of 3D environments for training so-called ‘embodied AI.’ If we want to use generative AI techniques to develop robots that can safely navigate in real-world environments, then we will need to create millions or billions of simulated environments.”
Enter Holodeck, a system Callison-Burch, and others. Named for the Trek virtual environment, Holodeck can create a virtually limitless (yes, pun intended) range of environments, using AI to interpret user requests. Holodeck leverages knowledge embedded in large language models (LLMs), like the one that ChatGPT and other chatbots use. It “has a converation” with the LLM to understand user request parameters. Holodeck can change “a 1b1b apartment of a reasercher who has a cat” into a space matching that description.
It starts with the floor and walls, then adds windows and doorways. Then, Holodeck searches Objaverse, a library of pre-made digital objects, to furnish the space. Finally, it queries a researcher-designed layout module created to constrain object placement (that way, a toilet doesn’t get placed on a wall).
The research team generated 120 scenes one generally sees in today’s spaces (offices, apartments, etc.), then generated atypical scenes, like stores and public spaces. Throughout these tests, human evaluators chose the Holodeck spaces over those using an older technology called ProcTHOR.
In June, the researchers will present Holodeck at the 2024 Institute of Electrical and Electronics Engineers (IEEE) and Computer Vision Foundation (CVF) Computer Vision and Pattern Recognition (CVPR) Conference in Seattle, Washington.
