Valve’s Lighthouse as USB: Anything More than a Bunch of Spin?
This is the third article in a series on the Valve/HTC Vive Ecosystem. If you you need additional context, please begin with the first article in the series.
A famous quote from Gabe Newell is about a lesson that Valve learned early-on when dealing with the Internet. You can find it in Episode 306 of the Nerdist Podcast at 00:12:14.
Don’t ever, ever try to lie to the Internet because they will catch you. They will deconstruct your spin. The will remember everything you ever say for eternity. -Gabe Newell
At this year’s Game Developers Conference where Valve announced their Virtual Reality partnership with HTC, and at that time, Gabe made an incredible claim about the Lighthouse tracking technology:
So we’re gonna just give that away. What we want is for that to be like USB. It’s not some special secret sauce. It’s like everybody in the PC community will benefit if there’s this useful technology out there. -Gabe Newell (Valve)
The story which accompanies the interview describes Lighthouse as a way of providing infinite input solutions into Virtual Reality. “As long as tracking is there, anything can be brought into VR, like how USB ports enable you to plug (virtually) anything into your computer.”
What the Technology Brings
In the previous two articles, we’ve dug into the technology itself, and it supports what we’ve been told. Spend perhaps $100-150 for two of Valve’s Lighthouse units and mount them in opposite corners of the room. At that point, you can almost forget about them. But any enabled device that you bring into the room can take advantage of:
- Rock-solid positional data with high precision and resolution
- Rock-solid orientation data with high precision and resolution
- Very low additional power use (passive sensors, undemanding electronics)
This support would be available for an arbitrary number of devices, and “at a low enough cost to be incorporated into consumer electronics items such as televisions, headsets, input devices, or mobile devices.”
Given Valve’s ambitions for the technology, it is expected that they will create a complete solution that will feed fully resolved positional and orientation data to an electronic device without the need for additional processing.
That last bit of functionality has yet to be confirmed. If not the case, the processing power required to compute the position and orientation is extremely lightweight. Valve may also have an additional solution for wireless connectivity back to a PC.
It is unclear if the default Lighthouse mode will support any identity features, but our review seems to suggest that it would be easy for Valve to enable the following functionality with a user-installed firmware update:
- Ability to instantly identify a room and to distinguish it from others
- Ability to give the room a unique identity to be used as a database key
More on the significance of these later in this article.
It is important to note that while this technology seems quite promising, it is still being developed. An early developer release is expected in the spring, and consumer release is slated for November of this year.
Commonly Suggested Uses
To be honest, the apparent uses (provided by Valve and speculated by third parties) are quite plausible, but by themselves don’t seem especially compelling:
- New virtual reality controls and 3rd party input devices
- Tracking technology for 3rd party virtual reality head-mounted displays
- Ability to find real-world objects in the room while you are still in VR
- Solving robotic navigational issues
Now that we have finished our technical review in the previous two articles and have a better idea about the system and its capabilities, why don’t we try our own hand at developing some new features which can take advantage of it?
If this isn’t going to be an upcoming feature for the HTC Vive, even for novelty’s sake, then the obvious has been missed. The concept of creating a depth map just from two images is very well known.
What would make the process even more robust is combining a camera of well known characteristics with the precision of Lighthouse tracking (providing known position and aim at all times). If not with a unique device built especially for that purpose, then we’re talking about the HTC Vive itself with built-in camera and tracking.
How might it work? It couldn’t be simpler: walk around the room and look at everything. The software will merge image stills or video with high resolution position and orientation data for camera. Once completed, it would process the images, determine the depth of elements which have been seen from multiple angles, and reconstruct the entire scene in three dimensions and display it in virtual reality.
Worth noting, the internal development version of the HTC Vive appears to have two cameras in front. One cannot help but wonder if they contemplated yet another method of 3D image acquisition, perhaps more appropriate for real-time processing?
Room scanning is something that might play well with Valve’s announced room-scaled VR, where you actually move around the physical room in tandem with your character moving in virtual reality. If you’re going to move around your living room, why not use it as the location for a virtual world at the same time? (Give some thought to how that might work. We’ll circle back around to it later in this article.)
What else might room scanning open the door for? Social engagements and playing games with friends and family in a familiar environment. It could serve as a wonderful bridge between virtual reality and augmented reality.
This is similar to room scanning, but you would indicate to the software a specific item in the room. You would get up close to the item and slowly look all around it while the software reconstructs it before you in real-time. The software could automatically determine any holes in the model and prompt you as needed to inspect specific areas in more detail (or from other angles) to get a more complete picture.
Yet another version might take advantage of a special mode which could be made available in the Lighthouse system. While the first Lighthouse unit provides high resolution tracking information for your head mounted display or camera, the second Lighthouse could temporarily enter a second mode where a carefully strobed and swept infrared laser light assists the camera in constructing a high-definition model of your object.
Once created, your object could be imported into your virtual library which you could shared with others.
We touched on this briefly when covering room scanning, but this topic deserves serious consideration by itself. What if it was as simple as walking into a room with a Lighthouse enabled webcam, putting on your Lighthouse-enabled Augmented Reality glasses, and having a conversation with your aunt who is sitting on both your couch and her couch from 200 miles away?
Maybe you are like me and you never liked what you saw with augmented reality. So many startups are quick to promise yet unable to deliver these pie-in-the-sky aspirational tech demos which are little more than ridiculous techno-fantasies.
There is no way these things could even do the required computer-vision based processing to constantly track the images with the user’s changing head movement, not to mention have any idea where to place objects in the room or how to share the same content with others in the room.
Or is there?
The curious thing is that the Valve Lighthouse solves quite a number of augmented reality problems. Tracking directly solves the viewpoint problem, but what about places to project content or knowing who to share data with? That would be tied to the room identity features mentioned earlier.
Lighthouse-enabled AR glasses could be able to instantly identify the room they are in and distinguish it from others. The next time you or someone else walks into the room, any special information (such as pre-defined areas to project images onto) are referenced and download based on the Lighthouse ID number. When Lighthouse-assisted, your AR device can focus more of its limited resources on communications, content, and graphics.
Take another look at one of those aspirational augmented-reality videos from earlier this year and imagine a Lighthouse in every room. Now that you know more about Lighthouse, doesn’t this look less aspirational and more like a blueprint for something that could be available next year?
Here’s the funny thing: CastAR was founded by two ex-Valve employees that did not want to make the transition from Augmented Reality to Virtual Reality. Valve let them go, but they also let them take their AR technology with them. It might be a good time for someone to ask Jeri Ellsworth and Rick Johnson about Lighthouse.
Commercial Lighthouse Units and Augmented Reality
After making the connection between Lighthouse technology and Augmented Reality, I started to wonder how it would work in the commercial space. I’m not much of a creative type, so I’m going to play this one straight.
As you enter the front door, your pair of Lighthouse-enabled glasses automatically picked up the ID beacon off of an in-store Lighthouse unit. You have AR Beacon Roaming enabled, so your glasses looked up the beacon’s unique ID in an online database, and determined that the available scene is compatible with your hardware and is consistent with your filter settings. The scene is tied to a specific location in the store.
Curious, you walk over to the indicated area, and give your glasses permission to download and execute the scene over your wireless connection. Within moments, a lifelike, distinguished, tall man with white hair in a gray suit appears in your field of view. He addresses you from the speakers built in the end pieces of your glasses.
Okay, let’s stop there. I’m not going to blow any more of this article’s word budget on this particular scenario, and I think you might have some idea where it can go from there. Yes, such an experience could not only be interactive, but it could also independently complete a transaction with the user.
Lighthouse can mean the ability to authoritatively signal the availability of pre-defined content that is tied to location, and to enable augmented-reality glasses to better take advantage of it (by providing stable tracking that would far exceed what smart glasses might be able to do on their own).
Can you imagine some other uses? Museums, bakeries, real estate, self-service kiosks? Creative technical types might operate a public sandbox for like-minded individuals to come and show off their latest efforts in front of a live audience.
Perhaps this is a world that Valve explored and decided that it was best to leave this to others?
Other Potential Uses
Home automation (visualizing the state of your home and making changes) could benefit greatly from Lighthouse-enabled room scanning or augmented reality.
Devices could be created for the blind which allow them to see objects in a room using depth scanning (and if combined with Lighthouse identity, features and functions of the room could be indexed and tagged in remote databases).
Small sets of freestanding Lighthouse-enabled cameras with network connections could become popular. Two or more in the same room could be used to create movies where the scene can be reconstructed from many different arbitrary angles. With the right processing, an entire room or stage could be broadcast in virtual reality in real-time. Streaming performances.
What about using an enthusiast level PC to deliver next-generation augmented reality features in the home or office, with today’s technology? This might deserve an article in its own right, so the description here is going to be brief.
Combine the augmented reality features made available with Lighthouse (such as room identity and presets), PC-based room scanning and depth-mapping, PC-based processing and graphics power, the Vive head-mounted display, and the idea behind one pre-existing Jeri Ellsworth patent assigned to Valve which includes re-rendering a live camera feed with the same perspective as the human eye would see.
What do we have? Just as mobile augmented reality and Lighthouse made the CastAR video look possible, a PC-driven augmented reality system and Lighthouse could make last week’s fantasy “Just another day in the office” high concept demo look like a blueprint for next year’s technology.
I have to wonder, is this part of why Valve is dismissive of mobile? (See 00:46:20 of the Geoff Keighley interview with Gabe Newell.)
Automatic check-in and directions for augmented reality clients.
Automatic logging of physical patrols by security guards.
Assisted vehicle parking in a garage with sub-millimeter resolution.
The number of different things, both big and small, which Lighthouse enables is staggering. What are some of the uses that you can think of for Lighthouse?
So you run into a case where there something we think is really important, it is an abstract, but something we think is really important and we want to push in that direction. The reason why fans haven’t arrived at the same conclusions is because they don’t have the same data as us. –Erik Johnson (Valve) [55:50]
When Gabe Newell looks at virtual reality, he asks how long will it be stable? How long until a VR display is replaced by direct neural stimulation? “You just want to test to make sure that you’re not investing in something that’s fragile.” –Geoff Keighley interview with Gabe Newell (00:47:48).
When I look at Lighthouse, it is anything but fragile. It solves core issues in Virtual Reality with inputs and tracking and does not seem easily replaced. What I find surprising is that it seems to have solid practical applications that match with Valve’s core mission as much as it has additional applications that go well beyond anything that Valve seems to be interested in.
Is this another USB, a common standard that is picked up and used across the industry? It sure is starting to look that way. If Valve is offering to license the technology for free, there is a lot of promise in this new enabling technology.
Development on this product still needs to continue (as planned), but from all appearances, Lighthouse’s potential as a common technology is a claim that passes the spin test.
March 28th, 2015 – For the next few weeks, Alan Yates of Valve is taking questions on Lighthouse technology.