The Nuts and Bolts of Spatial Computing
Spatial computing combines the physical and digital worlds, enabling users to interact with digital objects as if they were in the real world; it is a fusion of digital content with the physical environment, enabling new ways to interact with the digital world. Transcending traditional interfaces, it allows users to engage with 3D digital elements through gestures, voice, and movement, perhaps changing forever how we play, learn, work, and create.
Spatial computing is the convergence of virtual and physical worlds, integrating augmented reality (AR), virtual reality (VR), and mixed reality (MR). It changes how individuals experience, interact with, and understand their surroundings and allows machines to navigate and understand their physical environment.
Spatial Computing Basics
Spatial computing is yet another catchphrase encompassing technologies that can integrate our physical world with virtual experiences. It adds a level of intuitive interactions not possible before advances in augmented reality (AR), virtual reality (VR), and mixed reality (MR).
These hardware and software advances include spatial sound, which can simulate real-world sounds while the user is at the center of the 3-D space. Apple’s Spatial Audio feature in its AirPods is a good example of its current use. Depth sensing technology now provides an accurate, faster, and more detailed understanding of the physical environment. The technology allows devices to create a 3D map of the space for a realistic and interactive AR experience. Simultaneous localization and mapping (SLAM) allows devices to map the environment while simultaneously tracking their own movements within it. It is necessary for navigation, object placement, and interaction, providing an AR/VR application foundation. LiDAR SLAM makes environmental reconstruction more accurate for direct robotic positioning and navigation.
Augmented reality provides the ability to view the overlay of digital content and information onto our physical world through a device. Virtual reality adds sound and 360° views that simulate the physical experience, providing user immersion into the physical environments.
Mixed reality comes into play by combining AR and VR, enabling digital objects to interact with the physical world in real time. Users can interact and even manipulate digital content as if it were physically available.
The main technologies at the heart of spatial computing include sensors, cameras, and displays. Sensors such as LiDAR, motion-sensing inertial measurement units (IMUs), and depth-sensing cameras gather information from the environment, drawing a live-action map of every move and moment in the physical environment in real time so that devices understand and interact in three dimensions. Display technologies, including head-mounted displays (HMDs) and AR glasses, combine the digital and physical worlds by overlaying digital content onto the real world or creating entirely virtual environments, and projection systems map digital content onto physical surfaces.
These technologies deliver spatial mapping, gesture recognition, and immersive user interfaces. Spatial mapping is scanning the physical environment to render a 3D digital map. This includes a user’s position, orientation, context, and the physical objects and surfaces within the environment. Then, it enables the placement of virtual objects and interaction within the natural world that blends the two. Incorporating human gestures via sensors and cameras allows users to manipulate digital content naturally so that they seem to touch or move physical objects. Spatial computing interfaces, including headsets, are intuitive and give users a natural way to navigate and interact with digital content. It leverages hand gestures, finger motions, gaze tracking, and voice commands through wearables equipped with high-resolution cameras, scanners, microphones, and sensors.
Where Are We Today?
Spatial computing applications include gaming, entertainment, healthcare simulations, and interactive educational tools. That’s just the beginning. Apple, for example, launched its first spatial computing device, the Vision Pro headset, featuring the processing power of a MacBook Pro. Apple’s ARKit development tool allows developers to create AR experiences. It provides tools for motion tracking, environmental understanding, and light estimation to create interactive AR applications. Microsoft’s HoloLens standalone holographic computer is a hardware solution enabling high-definition holograms to integrate with the physical world using its sensors, cameras, and a custom holographic processing unit (HPU) for spatial computing. ARCore by Google is a software development kit for AR that supports motion tracking, environmental understanding, and light estimation. It provides APIs to build experiences across Android, iOS, Unity, and Web.
Although Apple’s Vision Pro kicked off greater interest in spatial computing this year, many companies have been working on the development of hardware and software solutions in this space. Stay tuned. In November, we’ll deliver Part 2—Spatial Computing—when Virtual and Physical Worlds Collide, a deeper dive into the technology, especially its future.
