Imagine you have a cold or flu and are resting at home. You ask your Alexa digital assistant to play some soft music to help you get ready for a much-needed nap. As you speak, Alexa detects a sniffle or cough in your voice. Before playing the song, Alexa suggests medication, such as cough drops and asks if you want one-hour delivery. You say ‘Yes’ and the cough drops are waiting at your door after you wake up from the nap.
This may seem like something from the distant future, but the reality is that this technology is very close at hand. The ‘voice and sound intelligence’ category has been an emerging trend over the past few years. In the near future, voice and sound technology combined with artificial intelligence will have a significant impact on the future of marketing.
What is Voice Intelligence?
If you have ever used Amazon’s Alexa, Google Home, or Apple’s Siri, then you have used voice intelligence technology. Voice technology allows devices to understand human speech. Not only have companies integrated voice technology with smart devices and phones, but it is also used within call centers and through Internet chatbots. Voice intelligence refers to the use of Artificial intelligence through Natural Language Processing and Machine Learning to analyze speech and sounds, mostly in real-time. This means that the analysis is being done during a conversation or while a sound is being made.
Different Forms of Voice and Sound Analysis
1. Anonymous Voice Analysis
Think about a situation where you recently purchased a coffee machine, only to find out that it does not work properly after you unbox it. You dial the service number listed on the back of the machine and are greeted by a chatbot. After interacting with the chatbot for a short period of time you’re redirected to a live customer service agent who talks you through your problem. This may seem like a routine interaction. However, there can be several interesting aspects of Voice and Sound Intelligence that can occur within that call.
First, when the chatbot hears your voice, even though it does not identify the individual at the other end, the bot can sense a feeling of “stress” or “anger” in the voice of the customer. The analysis could help the chatbot pair the annoyed customer with a specific representative best suited to handle such an interaction. That is, the customer can be automatically routed to an agent that is the most skilled, from a personality standpoint, to work well with an angry or stressed customer. This would be a use case of ‘Anonymous Voice Intelligence’ where the caller is not specifically identified through their voice.
2. Voice Recognition
We have all come across numerous smartphones that use fingerprints or facial recognition to unlock the device. However, voice is as unique as the user’s fingerprint and can also identify the user. Indeed, financial organizations like banks are increasingly using voice print analysis to verify the user’s identity. In the future, we may see our phones using our voices to unlock our devices instead of facial recognition. All of this falls into the second category of voice intelligence, which is voice recognition. With voice recognition, the device can identify a specific person, and then we can use data about that person to personalize the interaction.
The Future of Voice Intelligence in Marketing
The measurement of voice has increased exponentially in recent years. Almost 90 million people use smart speakers in the U.S. alone, which amounts to more than one-third of consumers. People are also talking more and more to all types of devices including watches, laptops, automotive interfaces, shopping mall kiosks, thermostats, TVs, refrigerators, fitness bands, children’s toys, and wearables to name a few. This leads to a goldmine of voice data that can be collected and used by marketers. But, what personality, behavioral, or physical characteristics of a person can voice intelligence identify?
Sentiment / Emotion Analysis
Smart speakers and other devices can analyze tone, vocabulary and other characteristics of voice to quantify how someone is feeling. This quantification of feelings and emotions is known as Sentiment Analysis. For example, Amazon’s fitness band, Halo, has a built-in microphone that can determine the emotional state. Amazon reports that the device is capable of inferring whether the user is happy, sad, or tired and determines their level of stress throughout the day based on their voice.
This data is valuable to marketers, who can use the information to target customers with personalized ads when they are in certain emotional states. For example, if someone is stressed then advertising for a yoga class, or alcohol may be better received by the consumer. If Halo is connected to Alexa, then this information about emotions can be used to suggest various products and services based on mood. While Amazon states that the data Halo collected is only stored internally, given past privacy breaches by many large companies, one is left to wonder how this data might be used.
This is only scratching the surface, beneath which lies a plethora of possibilities. What if the data accumulated by these devices could be purchased by other organizations? This data could help narrow down aspects such as the best time to call users based on their emotions. The user may be annoyed receiving a call about a new credit card promotion at 1 pm, but might be more receptive to the conversation at 6 pm if the device, such as the Halo, indicates that stress levels are down. Moreover, the calls can be made by representatives that best match the conversation style of the user. This methodology is already being used at numerous call centers.
Voice Intelligence and Identification of Physical and Behavioral Characteristics
Emerging research suggests that not only can voice identify the emotional state of the speaker, but it can also identify physical attributes. For example, voice intelligence software company Nuance has created technology that purports to determine a speakers’ age based on their voice. With knowledge about the speaker’s age, marketers will be able to target advertising more effectively. National Geographic reports that voice can be used to identify an individual’s height, based on pitch and how deep someone’s voice is.
However, determining age and height by voice is just the tip of the iceberg. In a fascinating book, voice research expert Rita Singh reviews hundreds of peer-reviewed studies and identifies that the following characteristics of a person are potentially identifiable by voice: health conditions (Alzheimer’s, autoimmune disorders, Parkinson’s, Dementia, endocrine system disorders, epilepsy, arthritis, lung cancer, sinusitis, anemia, Lyme disease, chronic fatigue syndrome, acid reflux, dehydration), drug usage (alcohol, opioids, cocaine, birth control pills), psychological conditions (anxiety, depression, schizophrenia, suicidal risk) to name a few.
Research reported in this book also indicates that voice intelligence has the potential to identify pregnancy, genetic disorders (Down’s Syndrome), and even sexual orientation.
If voice intelligence can reliably identify these emotional, physical, and behavioral characteristics then marketers would have powerful profiling information on consumers. That is, if an individual’s smart speakers or other devices can identify these human conditions, then marketers can target people very effectively. If someone is identified with arthritis based on their voice, for example, then advertisements for arthritis drugs could be sent to that individual’s media feeds. If someone is detected with an opioid condition, then marketing could be sent/ suggested various rehabilitation centers.
Voice Intelligence: Beyond Words
In the opening paragraph of this article, we looked at the case where a smart speaker detects sniffling and coughing and suggests the listener purchase cough drops. This scenario is not science fiction but is based on a recent Amazon Patent. A graphic developed by Amazon for the patent is shown below.
This patent suggests that Amazon is in the process of investigating the identification of ‘non-word’ verbal sounds for marketing and selling purposes. This type of sound analysis opens up many marketing possibilities for the future of marketing. For example, technology may be able to pick up sound expressions such as a deep *sigh*. If a person sighs in a certain way that represents disappointment, perhaps Alexa can suggest a pint of ice cream or maybe a bottle of whiskey to cope with the emotional state. Further, based on past purchase data, marketers may be able to personalize the specific advertisement to a specific individual based on the intensity of the *sigh*.
Many other sounds in the environment may also be used for marketing purposes. What if Alexa or another device detects a baby crying?
In that case, the smart speaker can suggest certain products or services that may be useful in that situation. Indeed, advertising can then be sent to all the devices owned by that individual. The list of sounds that can be identified by smart speakers, and used for marketing, is endless. If you chuckle at how crazy that thought is, Alexa will likely be able to detect that also!
The Dark Side of Voice and Sound Intelligence
It may seem like a cool idea to be able to speak to your microwave, fridge, or other devices. However, this technology raises privacy and ethical concerns. If a voice can be used to identify the person, their demographics, underlying medical conditions, and drug usage amongst other things, then how do we protect this information?
Companies like Google, Amazon, Microsoft and Apple, collect and store voice data collected from smart speakers and other devices. This data storage is then analyzed to help improve the performance of the devices. Even though these companies state that a “small subset of data is analyzed and saved”, it is not entirely clear from their Privacy Policies what data is specifically saved and how it is used.
This raises further questions like:
Is my voice data safe?
Will my voice data be shared with other organizations?
Will my voice data ever be sold to other enterprises such as a data broker?
These are all valid questions, and the most prominent one may be the security of the data. We have witnessed cases such as the Facebook scandal where 87 million profiles were harvested illegally or the Microsoft data breach where 38 million records were exposed. With precedents such as these, it would be more than justified to have concerns over the safety of voice and sound data. Another potential safety concern is selling this data to other enterprises. It can easily be argued that this information is a potential goldmine for organizations, considering the personal information that can be derived for voice and sound intelligence.
Conclusion
Voice and Sound intelligence is increasingly being collected, used and analyzed by technology companies. Most consumers are likely unaware of the amount of voice data that is being collected. The fact that Amazon has applied for a patent that allows Alexa to use different emotional cues from the voice, suggests that this data is going to be used for marketing purposes. It is reasonable to think that Apple’s Siri and Google Assistant are not far behind with the integration of this technology in their voice-enabled devices. The future of Voice Intelligence and marketing is filled with exciting new possibilities.
However, the field is equally challenged by safety and privacy concerns around this emerging technology.
Written By:
Ahmad Noshad
Assistant Marketing Manager, Pepsico Inc.
Professor David Rice
Schulich School of Business, York University
Executive Director, Future of Marketing Institute