Human understanding and interactions lean heavily on our experiences with the real world that we are most used to and perfectly suited for. However, skill levels in interacting with digital information are a different issue, and various segments of the population feature widely different degrees of knowledge of how to use digital devices and content.
Comprehensively multisensory engagements marry extended reality (XR) and in real life (IRL) to create genuinely immersive experiences that allow us to perceive them as a true amalgamation of virtual and real worlds. They can also find use to create inclusive interactions for users whose disabilities can make it difficult to access commonplace computing technologies.
Truly multisensory environments require comprehensive technology approaches to address a variety of senses. This requirement creates new opportunities for developers of devices and applications, but also faces substantial hurdles for technological reasons, cost considerations and consumers’ readiness to engage with such environments. Many researchers and developers are taking up the challenge, and are working on a wide range of offerings to address human senses in more comprehensive ways than current applications allow for.
Multisensory technologies are under investigation to expand the ways individuals can interact with digital and virtual applications. Multisensory interfaces and environments have been research and development topics for a long time. Olfactory, haptics and tactile interfaces are available, and even wind- and temperature-interface efforts exist.
Currently, these types of interfaces are relegated to niche applications or small market pockets. But virtual environments and novel technology approaches could result in diffusion of such applications to a wider range of users. Technological, cost and adoption issues exist, but first steps in creating multisensory engagements are under way.
Multisensory environments serve users by connecting in natural ways with virtual information and elements. But companies also can leverage multisensory approaches to create more meaningful – and impactful – connections with consumers and customers.
Paul Silcox, executive creative director at FutureBrand, a brand strategy and design company, believes that from gesture control to in-store design, and mixed, virtual and extended reality, “multisensory marketing is here to stay”. And a crucial aspect of multisensory engagement is the opportunity to make brands and products stand out in a world of visual overload.
Sound and scents for brands
Sonic branding has been around for some time. After more than a quarter-century in use, Intel’s sound logo – the “Intel Inside” musical notes – is perhaps the example that comes to mind most readily. The logo manages to reach consumers if their eyes are focused somewhere else, or even if they are in a different room when watching TV, for example.
Sonic engagement is not new, but there are many more approaches emerging. Recently, a number of experimental sound applications have been launched, and there are many more senses brands – and applications – can make use of.
Smell is another sense that brands frequently leverage. Hotels, shops and entire franchises use scents to evoke a branded experience. Mood Media, an experiential media company, for example, helps clients to create emotional connections with scent marketing. The company is also working with immersive audiovisual solutions, “connecting physical and digital with integrated media for a seamless customer journey”.
For some showings of the movie Heretic, entertainment company A24 partnered with Joya Studio, which researches, develops and produces fragrances and scented objects. During a pivotal scene in the film, selected screenings featured scents that were pumped into the auditorium.
Silcox highlights super-additivity as an important aspect of multisensory branding. Engaging multiple senses simultaneously “is exponentially more powerful than the sum of their individual effects”.
Despite clear benefits, he also points to challenges – challenges that will apply to the entire category of XR-enhanced environments as they become more common. Just randomly embedding sensory effects will not result in desired outcomes – instead, developers will need to focus on “defining individual sensorial assets and bringing them together as a powerful suite for a clear purpose”.
Excitement about multisensory engagements will inevitably lead to designers bundling a smorgasbord of technologies simply because they can. But “it’s important to show restraint and use these tools in deliberate ways in order to avoid an empty, gimmick effect”.
Venues as experiential landscapes
Perhaps expectedly, music venues and events are exploring the use of multisensory sensations to increase the entertainment value. Since the autumn of 2023, the Sphere close to Las Vegas, Nevada, has established a showcase in modern entertainment.
The giant spherical venue features advanced sound systems such as directional sound and virtual acoustic environments, as well as many motion and environmental technologies. The costs were also tremendous, with a price sticker of more than $2bn, providing an understanding of how high the hurdles for wide diffusion are.
However, according to Brian Mirakian, senior principal at Populous, a design firm that focuses on the creation of experiential environments: “Tomorrow’s concerts are more than just performances; they are moments that immerse audiences in environments that engage all five senses, transforming live shows into unforgettable, multi-sensory journeys.”
He adds that “advancements in technology are enabling venues to integrate sensory elements” that require a design and planning process that can be challenging to translate to the many types of venues artists perform in.
Mirakian also cautions that the introduction of advanced technologies, including scents, motion and wind, comes with additional considerations. Creating immersive experiences “necessitates fine-tuning, a process that requires the expertise of those who know the venue to meld the artist’s vision with the venue’s specifications”.
Touch and go on haptics
There exist a wide range of interface technologies, such haptics and selected interfaces, and an obvious market exists for gaming applications, but there are challenges to expanding the use of haptic sensations to create immersive environments in real-world locations that add digital experiences. Marrying haptic sensations of digital interfaces meaningfully with real-world situations and activities is not a trivial task. Nevertheless, new applications are slowly emerging.
“As digital devices evolve, we’re at an exciting inflection point, with the likes of gaming consoles, headphones, smartwatches, fitness trackers and headsets incorporating more features, which will allow brands to develop truly immersive experiences,” says FutureBrand’s Silcox.
FutureBrand created a haptic logo for Mastercard, which uses distinctive haptic vibrations combined with a sonic logo to let customers feel their smartphones when shopping online or paying at shops’ payment terminals with the firm’s credit cards.
Meanwhile, researchers at Northwestern University developed a wearable device to create a “sophisticated variety of haptic sensations”. The device connects wirelessly to VR headsets or smartphones and offers the sensations of “vibrations, stretching, pressure, sliding and twisting”.
The device has a small form factor, attaches to the skin, and can easily be worn on the move. The researchers “envision their device eventually could enhance virtual experiences, help individuals with visual impairments navigate their surroundings, reproduce the feeling of different textures on flat screens for online shopping, provide tactile feedback for remote health care visits, and even enable people with hearing impairments to ‘feel’ music”.
The researchers also mention the use in applications where touch supports users with visual or hearing impairments, and other companies are focusing on related applications. OneCourt has developed a device that enables tactile sports broadcasts. The device resembles a tablet that outlines the game courts, and is described as “transforming gameplay into trackable vibrations”. The entrepreneurs created the offering to help visually impaired sports fans experience games.
Jerred Mace, the CEO and founder of OneCourt, says: “We’ve essentially developed a laptop-sized haptic display that’s capable of communicating dynamic information like sporting events through touch.”
The device is focused on sporting events, but similar services may find use in many more commercial applications to enhance immersive environments – and could support individuals with and without visual impairments.
The long way for puzzle pieces to fall into place
New interface technologies can enable multisensory sensations that will elevate metaverse environments. Initial use cases exist in industrial, healthcare and entertainment markets, for instance. However, truly immersive environments will remain elusive in consumer markets for quite some time.
The long-term prospects look better. Touchscreens, app-supported stores and public venues also required time to diffuse, and are now almost ubiquitous.
For companies trying to leverage these new opportunities, the question remains what sensory technologies to bundle in what form factor. What is the right breadth and depth of multisensory sensations for what kind of applications, and for which consumer segments? Possible combinations are virtually limitless.
A better understanding of human interactions with virtual environments and digital objects will be crucial to drive commercial applications. As Silcox advises, “we need to ask what our end desired purpose or reaction is that we are looking to provoke”.
Martin Schwirn is a strategy and innovation consultant for Global 2000 companies, and the author of Small data, big disruptions: How to spot signals of change and manage uncertainty (ISBN 9781632651921).
