Immersive experiences are changing how we view and interact with the world around us. Extended reality (XR) technologies like virtual reality (VR), augmented reality (AR), and mixed reality (MR) have all claimed a space in our data-driven world, by helping us work better and understand abstract concepts in an intuitive way.
Each of these technologies offers distinct values, making them suited to a different range of applications. VR, for example, is entirely immersive – it places the user in a virtual world, often enabling them to interact with it. Augmented reality, on the other hand, blends virtual elements with the physical world by superimposing them onto the ‘real’ visual field.
Perhaps one of the most intriguing of these technologies, however, is mixed reality. This concept has already gained popularity as a powerful tool for business, and in this article we’ll see why.
What is mixed reality?
Mixed reality can be defined as a technology platform that allows users to experience and interact with virtual 3D content, merged with the real world. In many ways, it can be thought of as a form of AR, but with the added ability to manipulate virtual objects, and the capacity of mapping and including the real world environment in the virtual one too. As a result, they become thoroughly mixed – hence the term ‘mixed reality’.
It strives to create the most natural-feeling and seamless experience, using gesture and motion sensors to connect user actions with the virtual objects they manipulate. Some mixed reality technologies include haptic feedback and interfaces that enable users to ‘feel’ the virtual objects as well as see them. These ‘visuo-haptic mixed reality’ or VHMR platforms are still under active development, with several technologies being used to create a realistic sense of touch and feel for mixed reality experiences. These haptic technologies include vibration, forced air and ultrasonic pulses.
Benefits of mixed reality
There are numerous benefits of mixed reality, and many ways it can add value and efficiency to businesses. It’s worth noting that mixed reality requires the use of special MR glasses, whereas most AR experiences can be viewed with a smartphone or regular AR glasses. The need for specialized equipment makes MR more similar to VR, and it raises the barrier for adoption. However, we are now beginning to see that the latest VR headsets offer the option of pass-through cameras (Quest 2 and PRO, Pico 4). As a result, VR headsets are moving closer towards enabling people to experience MR as well.
Despite the historical (and current) barriers to entry, MR has already shown incredible value for many situations. A big part of this value comes from being able to interact with virtual objects, but more significantly, there are numerous possibilities opened up by linking virtual and physical objects to a rich data layer that is also accessible to users. This ability makes MR one of the best technologies for using digital twins in a practical way.
According to Microsoft, companies using the HoloLens2 MR headset have been able to massively decrease costs and improve outcomes across multiple sectors. Lockheed Martin has used mixed reality to deliver a 90% increase of efficiency on assembly processes, and Mercedes Benz has lowered its carbon footprint by reducing technician travel by 40%.
Other benefits include:
- 14% reduction in construction costs
- 30% decrease in time spent by doctors on ward visits
- 83% reduction in university learning time
- 5% increase in manufacturer revenue
(Source: Microsoft, https://www.microsoft.com/en-us/hololens)
As mixed reality solutions become more widely available, the applications and value of this technology will only become more widespread. While it may not yet have application for every sector, there are some where it has already demonstrated a compelling business case.
Real-world examples of mixed reality
There are several industries where mixed reality is already being used for practical purposes. In some cases, MR is replacing more cumbersome technologies, and in others it is creating an entirely new area of application.
Prototyping
Designers can rapidly refine prototypes and accelerate development time with mixed reality. Collaborating with stakeholders and demonstrating a proof-of-concept is easier when you know you’re all working from the same concept to begin with. By being able to ‘handle’ different iterations of a prototype in mixed reality, it’s possible to eliminate a ‘physical’ mockup stage until the design basics have already been agreed.
This saves time and materials, and enables a designer to get customer feedback at a really early design stage, which ensures the shortest path to a successful design. Using mixed reality, it is also possible to implement and test new software for the purpose of controlling hardware – even when it hasn’t been created yet.
For example, testing software for a new robotic arm (3D model) in a real auto factory setting, could help identify the real-world ‘fit’, and identify safety issues or potential collisions. Using mixed reality for virtual prototyping in this way can not only save money, but potentially lives as well. If this technology had been used for Spain’s new fleet of trains, they could have saved €258 million, by building trains that actually fit in their tunnels.
Process optimization
Mathematical models can determine optimal process configurations, but these are hard to understand without knowing what they look like. By seeing and modifying processes virtually with MR, it’s easier to understand physical implications to each decision. Also it can help understand the safety side of process optimization: what happens if a machine runs at 105% of its capacity for a longer period of time? With MR, the effects can be modeled and shown directly in the real-world setting.
Urban planning
Using digital twins, urban planners can run realistic simulations of new developments. Will a new housing project negatively affect traffic or harm wildlife? With MR, it’s possible to see these effects in action. Users can also make changes to the model and see the results instantly. New mathematical models can show how urban planning creates micro-, meso-, and macro climates in a city.
High-rise buildings can create microclimates with high temperature extremes, and the wider landscape can affect the flow of air and heat throughout the year. Using MR, planners can explore multiple variations and see how, for example, windmills sited on a hill near to a city could reduce the required air streams for cooling the city in the summer. Visualizing these models makes a conversation between multidisciplinary teams easier too, because it takes these models out of the abstract realm and projects them – tangibly – into the real world.
Architecture
Considering a total renovation of a large building? Maybe you have a few ideas in mind, and perhaps you’ve asked for a few architectural options to choose from? With mixed reality, you can explore a potential new configuration as you walk through the current building, and easily switch between different designs. By placing your ‘virtual house’ in the intended location, you can already see the view from the kitchen to the street where your kids are playing, and the view from your living room into the garden.
This gives you a taste of the real experience of living in the space, before it is finalized. You can now truly experience your new home and change the floor plan before it’s too late! It’s also easier to communicate with the architects and builders, who can see what you mean when you give feedback on the design.
Training
With mixed reality, engineers and technicians can be trained on a wide variety of different equipment without needing to use physical examples. This can be especially valuable when the equipment is unusual, expensive, or hard to transport. In addition, the mixed reality environment allows for more data to be available – a real step-up from traditional workshop training.
Penske Truck leasing, for example, has started using mixed reality training for its truck technicians, helping to accelerate their training program for new recruits. VR offers a safe and easy access training environment, especially for dangerous environments or hard to reach places. You can do a VR demo with a VR headset and others can join with different devices. This also works very well for remote assistance.
Education
In addition to highly practical learning applications, such as pilots, technicians, and engineers, mixed reality is also useful for other areas of education. One such example is training of medical staff, including surgeons and doctors. Mixed reality virtual cadavers are already available, and there are several Human Anatomy experiences already on the Fectar app, including Human Anatomy 101, Human Heart, and Human Eye. Doctors’ diagnostic skills can be tested with realistic holopatients, who experience little suffering while trainee doctors hesitate over the diagnosis for defined patient scenarios. The XR initiative at the University of Michigan is actively identifying opportunities for mixed reality learning and working to deploy them in new areas.
Maintenance and Support
When manufacturers operate globally, it is more common to rely on strategic partnerships to help distribute products and solutions. The ability to have a representative in every major geographical area is a massive benefit for companies, especially those in the technology and industrial sectors. However, innovation must happen fast, and it isn’t easy to train partners around the globe with physical equipment.
In these cases, mixed reality can be used to demonstrate and teach fixes, maintenance, and installation of complex equipment. With VR and AR you are able to train more people in less time, for a greater variety of tasks. And, with remote assistance, you can help everyone everywhere with any device.
An important application of AR is for First Time Right Diagnosis (FRD). This helps a third-party to make the right diagnosis, using an error code on the display of a machine. The advantage of AR for FRD is that the user can either fix the issue themself, or a maintenance employee will be properly prepared the first time they drive out – because they already know what the problem is and have the right parts to fix it.
Extended reality, mixed reality, virtual and augmented reality – what’s the difference?
There are now so many kinds of virtual and blended experiences and technologies, it’s hard for people to discern what is what. With so many different terms, it’s hard to know what they all mean – and there seems to be some overlap. What’s the difference between these technologies?
Firstly, extended reality, or XR, is the blanket term for all these virtual experience technologies; it includes AR, VR, and MR.
That’s clearer, at least. While some people claim that mixed reality is a combination of AR and VR, this is an oversimplification – but also not far off the mark.
Ultimately, all of these are about the 3D content, and about using it to communicate complex ideas. Each of these technologies is all about sharing and collaboration in our shared 3D world, and people shouldn’t be limited by a specific technology, device or software. This is the vision behind Fectar.
Mixed reality vs augmented reality
Mixed reality is best described as a subtype of augmented reality: you can still see the real world around you – and that’s the defining feature of AR. The difference between MR and ‘regular’ AR, however, is that MR enables you to interact with the virtual objects as if they were really there, using gesture control or specialized gloves. Check out Ultraleap and their amazing hand tracking (this is integrated as standard in the Fectar platform).
Another big difference between MR and AR is that people can experience augmented reality with a wider range of devices, including smartphones and tablets. More than 2.4 billion devices are able to experience AR, making it much more accessible. This is because the technical requirements are not as demanding; AR doesn’t need to know what you’re doing. Mixed reality, on the other hand, requires MR glasses or headsets, and some kind of gesture control or other sensory input to enable interaction. And real-time connections with several data sources, for showing IoT data with support screens
Mixed reality vs virtual reality
Both mixed reality and virtual reality enable users to interact with virtual objects. They also both require specialized equipment to facilitate this, such as headsets, gloves, or gesture control.
The distinction is that, with VR, those virtual objects exist entirely within a virtual world. Unlike AR, which allows the user to see virtual objects blended with the real world, VR – by definition – excludes the real world entirely.
In simple terms: placing virtual objects in reality is AR. Viewing entire virtual worlds; that’s VR.
What technologies are used for mixed reality?
Many of the supporting technologies for mixed reality are the same as for other extended reality experiences, however the technical requirements for mixed reality are a lot more complex. Designing and building an MR environment or experience is a difficult task that must cover concepts like user point of view, 3D content creation, safety, and spatial awareness. Incorporating haptic elements is also difficult.
However, there are some companies that have already created proven platforms for mixed reality.
Perhaps the most notable is Microsoft’s HoloLens. Currently on its second iteration, the HoloLens 2 is becoming a leader in the mixed reality space. Microsoft is devoting considerable effort into developing a complete platform and pre-packaged solutions for major industries.
Another major player in mixed reality technologies is MagicLeap, whose MagicLeap 2 glasses are remarkably lightweight and capable. In recent years, MagicLeap is moving to the B2B market with their glasses.
There are many other manufacturers of AR glasses out there, including the somewhat underappreciated Lenovo ThinkReality A3, however most of these lack the sensor inputs needed to turn augmented reality into a fully interactive mixed reality experience.
There are now, however, new VR headsets becoming available with the pass-through option.
How to get started using mixed reality
Mixed reality can offer some huge benefits, and this is amply demonstrated by some notable examples from industry, including Japan Airlines’ HoloLens-powered hands-on training for pilots and mechanics, and NASA’s OnSight platform for remote virtual exploration.
Collaboration between market leaders is also making MR technology more accessible and this is increasing the possibilities for application in more businesses. For example, the collaboration with Fectar and Ultraleap for hand-tracking: https://youtu.be/3m4EjH3PVYo
According to research the global market for AR and MR will reach nearly US$607 billion by 2029, with a CAGR of 50.5% during this period. However, the share of this taken by mixed reality alone will only represent a small fraction of the total, due to the wider applicability of AR technologies. That’s simply because of the fact that, for the next 3 to 5 years, the smartphone will be the main device to access 3D content.
For businesses today, the question is how they can start using mixed reality to improve efficiency and effectiveness. To date, most firms using MR are still experimenting to find the ideal applications for this technology. The cost and time used to be a significant hindrance, and companies are under no illusions about this. However, they have calculated the potential advantages to be worth the costs because of the transformative potential.
Today, it makes sense to put yourself in the position to easily ‘option’ new MR solutions as they become available. This means experimenting and adopting ‘lower threshold’ augmented reality and related technologies today, so you’ve already established a baseline. As AR assets and spaces form the foundation for more complex mixed-reality experiences, companies that get the edge with AR will be in a naturally better position to adopt MR. They will already be familiar with many of the development processes involved, and may have already generated a substantial portfolio of virtual 3D assets.
It’s worth making a comparison to the period after the internet bubble popped, between 2000 and 2005. This was the most important phase for companies to prepare for a new, mature, and potentially disruptive market. Companies and schools that waited and tried to adopt the internet after 2005 found it much harder, and some of these companies had already lost too much ground, and didn’t make it at all.
Over the next 3 years you will see the developmental phase with VR and AR technologies. We already see the development of AI at a very fast pace; VR and AR are a very important interface for all the potential that AI can offer for businesses and education. Simply ‘doing nothing’ is not an option.
Perhaps one of the easiest ways to get started with AR is with the Fectar app. This AR app gives users free access to the extensive Fectar platform, featuring inspirational AR content created with the Fectar Studio.
