Apple has started delivering its amazing mixed reality headset, the Apple Vision Pro, and I am confident it will wow consumers and invigorate the industry. You might think my opening sentence would please Apple, but if I was a developer with an app for the Vision Pro, three words would violate Apple’s recent language guidelines — mixed reality headset.
Instead, Apple calls its headset a spatial computer, a thoughtful term justified by the product’s advanced features. That said, Apple has taken its branding one step further by imposing language restrictions on app developers that have raised eyebrows: ’Refer to your app as a spatial computing app. Don’t describe your app experience as augmented reality (AR), virtual reality (VR), extended reality (XR), or mixed reality (MR)’.
I appreciate the need for disciplined branding, but I worry Apple is going too far by actively working to suppress language that has a long and rich history. I say this as someone who began working in this field when the phrase virtual reality had just emerged and long before augmented reality, mixed reality, or extended reality had been coined. This means I’ve lived through the full evolution of this language over the last 30+ years, with various twists and turns.
Maybe I’m being nostalgic, but when I started out, virtual reality was new and it quickly became the most exciting concept in the world of technology. I was a young researcher conducting VR depth-perception experiments at NASA and the photo below was a large poster in the lab where I was working. To me, it was a deeply inspiring image, capturing both the present and future of the field.
The fact is, the human experience depicted in the photo above has been called ‘virtual reality‘ for almost 40 years. If you are a developer for the Vision Pro and create a fully simulated immersive experience, is it really such a problem to describe it as virtual reality? After all, the VR headset (above) is now in the Smithsonian. This is our history and culture, and it should not be branded away by any corporation.
Of course, the Vision Pro is orders of magnitude more sophisticated than the NASA headset shown above, not just because it’s higher fidelity but because it adds entirely new capabilities. The most significant capability is the power of the Vision Pro to seamlessly combine the real world with spatially projected virtual content to create a single unified experience — a single perceptual reality. This is called augmented reality or mixed reality depending on the capabilities and both phrases have a long history in academia, government research labs, and industry.
So What’s the Difference Between AR and MR?
This is probably the most misunderstood schism in the world of immersive technologies so it’s worth taking a quick trip back in time to explain how today’s divide came to be.
For most of my career only one term was needed, augmented reality, but its definition has been diluted over the years as marketeers pushed simpler and simpler systems to fall under the banner, confusing the public. I suspect the pendulum will swing back in the future, but for the next five to ten years, both the mixed and augmented reality phrases will be helpful.
As background, I began working on merging real and virtual environments in 1991 before there was language to describe such a combined experience. My focus back then was to explore the basic requirements needed to create a unified perceptual reality of the physical and virtual. I called this pursuit ‘design for perception‘ (admittedly, not very catchy).
Still, I learned a lot, determining that the real and virtual realms needed to be spatially aligned in full 3D space with sufficient precision that the flaws are beyond the limits of human perception (called the ‘Just Noticeable Difference’ or JND in the field of psychophysics). Even subtle flaws destroy the illusion and your brain perceives the real and virtual as separate, not one reality.
In addition, both realms needed to be simultaneously interactive. For example, the user needs to be able to reach out and engage naturally with both real and virtual at the same time, cementing the illusion that the virtual content is an authentic part of the physical surroundings.
And finally, the real and virtual need to engage each other (bi-directionally interactive), because without that consistency the illusion is lost. If you grab a virtual book and place it on a real table and it falls through — it’s not perceived as a unified reality and suspension of disbelief is gone.
Because no language existed back then, I referred to merging the real and the virtual as creating ‘spatially registered perceptual overlays‘. Also, not very catchy.
The Evolution of a Name
Fortunately, the phrase ‘augmented reality’ was coined at Boeing soon after and quickly took off. I liked this language a lot. After all, AR clearly describes the objective of the technology — to add virtual content to a real environment that is so naturally integrated, the two worlds merge together in your mind, becoming a single reality. And for almost 20 years, that’s what the phrase augmented reality meant (while simpler devices that merely embellished or annotated your field of view were called head-up displays).
And then in 2013 Google Glass appeared. I respect that product and believe it was ahead of its time. Unfortunately, the media incorrectly referred to it as augmented reality. It was not. It didn’t enable virtual content to be placed into the real world in a way that was immersive, spatially registered, or interactive. Instead, it was what we now call smart glasses, which are deeply useful and will become even more useful as AI gets integrated into these product, but it wasn’t AR.
Still, the phrase augmented reality got watered down during the 2010s, not only because of Google Glass but because the makers of smartphones were pushing simple visual overlays as ‘augmented reality’, even though they were not immersive and lacked 3D registration with the real world. They also lacked user interactivity and bi-directional interactivity between the real and virtual. This was before LiDAR and other scanning technologies were added to phones, enabling spatial registration and interactivity. Today’s phones are much better, but AR got tainted.
I’m sure I wasn’t the only one frustrated by AR language being watered down. I suspect that the team at Microsoft working on the first commercial product (the HoloLens) that enabled a true augmented reality were equally annoyed. In fact, I speculate that this is why Microsoft, upon launching their innovative HoloLens product focused their language on the phrase mixed reality. The phrase had been around since 1993, but it was with the HoloLens launch, which was also ahead of its time, that MR really took off. It basically came to mean — genuine augmented reality.
And so, we now have two terms that describe different levels of augmenting a user’s surroundings with spatially registered virtual content. To help clarify the difference between AR, MR, and VR, we can look at definitions that were published in 2022 by the U.S. Government Accountability Office (GAO). I have to assume the GAO cares about the differences between these phrases to clarify if government contracts are paying for virtual reality, augmented reality, or mixed reality devices. To address this, the GAO put out a public document that featured this simple image to summarize the differences:
Hardware vs. Experience
It’s worth noting that the difference between AR and MR has nothing to do with the hardware and everything to do with the experience. I bring this up because many people incorrectly believe that AR hardware refers to glasses with transparent screens you can peer through, and MR hardware refers to headsets that use ‘passthrough cameras’ to capture the real world and display it to the user on internal screens. This is not true.
I say that as someone who used passthrough cameras in the first system I built for the U.S. Air Force back in 1992 (called the Virtual Fixtures platform). I made that design choice because it allowed me to register the 3D coordinate systems for the real and the virtual with higher precision, not because it changed the user experience. And besides, simple phone-based AR also uses cameras, so that is not the differentiator.
Back to Apple’s Vision Pro
This leads me back to the Apple Vision Pro — it’s a mixed reality headset, not because it uses passthrough cameras, but because it enables users to experience the real world merged with interactive virtual content that is spatially registered with a user’s natural surroundings with precision, creating one unified reality.
And because mixed reality is the superset technology, the Vision Pro can also provide simpler augmented reality experiences and fully simulated virtual reality experiences. And for all three (VR, AR, and MR), I expect the Vision Pro to amaze consumers with experiences of far exceed any device ever been built at any price. It’s a true achievement for the Apple and the team of engineers who made it happen.
The Vision Pro also enables other capabilities that are entirely unique, including a spatial operating system (visionOS) that breaks exciting new ground by using a user’s gaze direction for input. In other words, I agree that the Vision Pro is not only a mixed reality headset, but also a spatial computer and frankly, a work of art. I also believe that spatial computing is a great overarching term for AR, MR, and VR experiences and adds some new twists that relate to spatial operating systems — all good.
My only recommendation is that Apple not be too heavy handed in suppressing the historic and accepted language of the field. After all, I am old enough to remember Apple’s biggest product launch, the famous “1984” super bowl ad that unveiled the Mac. It featured a runner throwing a massive hammer to shatter an Orwellian future where Big Brother controls society by replacing accepted language with ‘newspeak’ and enforcing it with ‘thought police’.
From that perspective, I hope Apple will soon allow developers to reference VR, AR, and MR experiences on the Vision Pro by name. After all, we want an immersive future where 2+2 still equals 4.
This post was written by FMI Advisory Board Member Louis Rosenberg.
Louis Rosenberg, PhD is a pioneer in the fields of virtual and augmented reality and a longtime AI researcher. He is known for founding Immersion Corporation (IMMR: Nasdaq) in 1993 and Unanimous AI in 2014, and for developing the first mixed reality system at Air Force Research Laboratory. His new book, Our Next Reality, is available for preorder from Hachette.
Connect with FMI
Want to stay ahead of the marketing wave and prepare yourself for the future? Connect with us on our LinkedIn, Instagram or Twitter/X.
If you enjoyed this newsletter, please share it and subscribe to receive it directly in your inbox.