AC thVRsday
In his weekly column, Android Central Senior Content Producer Nick Sutrich delves into all things VR, from new hardware to new games, upcoming technologies, and so much more.
The first two-hour session of Starship Home that I finished was met with a surreal sense of spatial juxtaposition. One moment ago, my living room was a starship complete with alien plants and an infinite storage system that tied in neatly with my basement. The next moment, I was back to my normal old living room with nothing nearly as exciting in it.
Another recent mixed reality game, The Last Stand, turned my living room into a store in the mall. This sounds rather strange until the zombies start breaking in the boarded-up windows, and you suddenly realize you’ve been placed in a movie frame of the classic movie Dawn of the Dead. Even after taking the headset off, I expected a zombie to break through my sliding glass door at any moment.
It’s this sense of “making new memories in people’s physical spaces” that gets me so excited about mixed reality games and why Meta’s latest mixed reality capabilities update is so important. Meta Quest developers can now use your Meta Quest 3’s cameras to create even more immersive content thanks to the updated camera passthrough API.
So what does this mean and how is Meta protecting your privacy? Between Meta’s blog post, the company’s presentation at GDC, and my own discussions with Meta representatives and Meta Quest developers, I’ve found some rather interesting new concepts developers are working with to create the future of immersive mixed reality gaming.
Camera data is the Pandora’s Box of mixed reality
For a long time, Meta has restricted Meta Quest camera access to developers. Games like Starship Home, The Last Stand, The Infinite Inside, and all others rely entirely on what the Meta Horizon OS tells them your room looks like. These games only know your walls exist because the OS tells them. Likewise, your dining room table might be considered a desk because you defined that during the headset’s setup process.
Everything was pre-baked, including the size and shape, which means developers could only work with whatever data the headset fed them. Now, thanks to a new camera passthrough API, Meta Quest apps are able to actually see your room — assuming you permit them — and understand the objects within.
Let’s look at a side-by-side of what I’m talking about:
The current mixed reality kit is very boxy. Future apps and games will finally understand room geometry and nuance that makes mixed reality games more convincing.
While the headset is mapping your room during the initial setup process, you’ll see it building a complex mesh of polygons so it can recreate your physical space in a virtual environment. This data was previously only accessible by the headset itself and never shared with an app.
Instead, the app would only be given the data on the right. Notice how the box that defines my couches doesn’t line up with the actual height of the couch since the couch isn’t a box? The same problem occurs with my dining room table, which is oval in shape, yet the older room setup only allows for square or rectangular tables.
This new solution allows apps and games to better identify the space around you, resulting in more realistic (and less hokey) mixed reality experiences.
Now that Meta is expanding the use of camera data to apps, developers may soon be able to program their apps and games to more intelligently understand the geometry of objects. That means no more weirdly floating characters in a mixed-reality space.
That also means when a virtual object hides behind a physical one, it’ll look much more convincing since it won’t be hiding behind a square box but, instead, your rather curvy living room chair. This can extend to zombies banging on your windows instead of virtual recreations of a window, goop oozing out of sinks, spiders convincingly crawling out of vents, etc.
The future is in flux
After Apple Vision Pro launched, lots of people came up with interesting ideas of what mixed reality concepts could look like. Some used mixed reality to gamify chores so you can vacuum and never miss a spot again. Apps like this will work even better now that they have access to camera data, as they’ll be able to properly identify areas rather than just using prebuilt data from the headset.
Others used the idea to completely transform their rooms into completely different decor using generative AI. While this video is a mockup and not a real app you can use today, it might give you an idea of what to expect from mixed reality concepts in the future:
The new passthrough camera API is built upon Android’s Camera2 API, according to Meta, helping it tie deeply in with the Quest’s hardware. This means apps can utilize data from the camera and adjust it as needed, helping virtual objects to better match the lighting and color of your room.
Things look different during sunlit days than they do in artificial nighttime light, and this new API can help play a part in making those virtual objects look more convincing in your actual space.
In addition to that, the new API lets apps use the Quest’s camera to capture imagery of your room and add it to virtual objects. This can be as simple as taking a picture of a person’s face and adding it to a rudimentary character model — as you see above — or something more realistic like your actual living room reflecting in a pool of virtual water in a game.
The new passthrough API allows the Quest’s cameras to capture your room and put it on virtual objects, or just more seamlessly blend the virtual and real worlds.
Better spatial awareness will enable your headset to further transform the room around you into something both familiar yet unique and always memorable. Spatial awareness holds a special meaning for our brains and even bespoke experiences like this masterful airplane cockpit someone built.
From what I’ve seen, this is one of the most convincing uses of mixed reality to date. Most people don’t have access to an empty airplane cockpit in their garage, but being able to turn your favorite chair into a convincing airplane cockpit — complete with its texture accurately positioned inside the virtual cockpit — would make an incredibly convincing game of Flight Simulator without all the expensive add-ons and accessories some people spend a lot of money on.
As with anything XR, this is just the beginning of something new and interesting that’ll play a part in what’s coming down the line. Google’s upcoming Android XR operating system also allows passthrough access for apps, meaning developers can get a jump start on making concepts now so that games and apps are more mature when that OS — and future AR glasses hardware — begins shipping to customers, content will be there waiting for them.
For the price of a Nintendo Switch, you can grab the most affordable VR headset to date, featuring cutting-edge mixed reality capabilities and a powerful Snapdragon XR2 Gen 2 processor inside. No need for a PC or console, just strap it on and you’ll be transported anywhere you want.
