11 Dr. Jonas Heller, Postdoctoral Researcher, about AR as a decision making & visualization tool
Dr. Jonas Heller does research on Augmented Reality in the field of consumer psychology and the optimal usage of technology in retail for almost 5 years now. Through his work, he has been able to identify many factors which help retailers to use AR as an advantage. Nevertheless, he still sees great potential in research for topics related to the technology, as there is little scientific evidence available so far. He is convinced that the new form of information presentation will profoundly change our society and would like to see a stronger exchange between research groups, as well as between science and practice.
Postdoctoral Researcher at Brightland Institute for a Smart Society (BISS), Maastricht University
Points of contact with AR/VR:
AR for shopping and for optical measurement
no usage of VR
since 5 years due to his PhD about AR and the influence on Consumer Behavior (https://www.augmented-research.com/)
started only recently due to his research about VR
"I would really like to see AR/VR/XR research groups working more closely together - to find out more about the technologies faster, and to benefit from the other's experiences."
During your PhD, you have successfully researched about AR in the field of Consumer Psychology, have published all your articles, were a finalist in this year's SERVSIG Awards for the best dissertation and you are still researching on this topic.
1) What exactly fascinates you personally about the technology and which advantages and potentials from a scientific point of view does AR offer to the retail industry?
I am personally fascinated by the fact that it is something completely new that our society has never experienced before. We no longer have text on paper and we no longer have a picture printed in a newspaper. Information is suddenly presented three-dimensionally and is detached from the medium on which it is projected. This has never existed in this form before. I think this way of presenting information will change society very much. Accordingly, it will also change the retail sector, because it will make it easier for customers to make decisions from home. Decisions could be made without AR before, but the decision-making process in the customer's mind was much more complicated. In addition, the devices, the smartphone or tablet, for the application of AR have already reached 99% of the customers. This again speaks for the potential in retailing, because no major technological revolution is needed to roll out the application
In your opinion, are there differences in the fields of application for AR and VR?
AR Geräte sind wie gesagt schon beim Kunden angekommen. Bei VR-Brillen sehe ich das etwas anders. Ich sehe großes Potenzial für VR im Gaming-Bereich und auch im Personalwesen und Training, sprich bei Unternehmen, die die noch teure Geräte für bestimmte Situation stellen können. Da ist das Potenzial für VR eventuell auch größer als für AR.
AR devices have already reached the customer, as mentioned above. With VR glasses I have a different opinion. I see great potential for VR in the gaming sector and also in human resources and training, i.e. in companies that can provide the still expensive devices for certain situations. Here, the potential for VR is probably also greater than for AR.
On the other hand, AR has great potential in the scientific area, because we still know very little about it. We do not know how AR improves the customer's decision and in which context AR is better than the classical media such as print or traditional websites. We also don't know exactly which people respond better to AR than others - which is of course also relevant for the retailing industry once you segment customer groups. Our research group, initiated by Dr. Tim Hilken (www.augmented-research.com), has been doing intensive research on this topic for more than 4 years. Nevertheless it is still a very young and unexplored field. For example, there is currently no article on AR in the Top 4 Marketing Journals . There is a lot to discover: we want to create new insights, we want to do research and therefore it is of course very interesting for us. Due to the fast moving innovations in technology, both in hardware and software, the topic is also incredibly relevant.
2) Which of your results impressed you most from a personal point of view? Has there been any result that you would not have expected?
My first project was about the imaginative power of customers (Journal of Retailing 2019). We gave restaurant visitors a traditional menu card and an interactive AR menu card. With the AR menu card, they could display and interact with holograms of the food, such as zoom in, zoom out, move and rotate them. We found out that customers who receive an interactive augmented reality menu in the restaurant instead of a traditional menu, speak more positively about the restaurant to others. The explanation for this is that the AR menu relieves the customer of some of the effort required to imagine the food and drinks, thus improving the customer's imagination. As a result, customers feel more comfortable with their purchase decision. They may not know whether they have made the right decision when ordering, but they are more confident in their decision.
example of an AR menue card
We then tested for which customers this works better or worse. We distinguished between customers who are better able to think spatially (spatial visualizers) and customers who are mentally better able to produce clear images, such as clear colours and outlines (object visualizers). In this case we were not 100% sure in the first place for which customer type AR would work better in this situation. For people who already have a clear visual idea anyway, or for people who can think more spatially. We found out that for people who have a strong imagination but are worse at spatial thinking, AR is more useful. For consumers it is important to have a clear idea of all sides of the product as well as the position of the product and this leads to a higher benefit for the so-called object-visualizers (i.e. customers who have worse spatial thinking).
The other results were in line with our expectations. We tested how the type of interaction with the AR holograms affects the willingness to pay. For the experiment, each subject had the Microsoft HoloLens on and was asked to place a chair in a room. They could move the hologram of the chair as they wanted or zoom in and out. One group of participants controlled the holograms using hand gestures (which are recognized by the HoloLens and converted into movement of the holographic chair), while the other group used voice commands. The results show that control by means of gestures has a much better effect and thus leads to a higher willingness to pay. This was not unexpected, because it was a chair and for consumers speaking with a chair is a rather unnatural action.
Nevertheless, it was important for us to prove this hypothesis because of its far-reaching implications for retailers. Regardless of AR, we are seeing a strong increase in voice control in the consumer sector right now - see Siri, Alexa and Co. Retailers are currently trying to develop apps or build their own holographic nervous system to make shopping easier for consumers. Through the study, we discovered that they should invest in hand gesture control, AR controllers and connecting controllers in one app rather than expanding voice control.
Have you always used the HoloLens for your experiments or have you also used the participants‘ personal smartphones? Was there an influence of the used device on the consumers' behaviour?
During my PhD, we provided the test persons with a tablet or the HoloLens. In current studies at the University of Maastricht, participants also use their own smartphone, whereby of course we control possible influencing factors, such as operating system, screen size, etc. In fact, we have never observed an influence of certain smartphone characteristics.
My two PhD supervisors, Prof. de Ruyter and Prof. Chylinski, conducted a study in Sydney in 2014/15 in which they gave a tablet or a smartphone to test persons in a supermarket. The supermarket visitors were able to scan a shelf with both devices to obtain additional information. Here it was found that consumers prefer the smartphone over the tablet. However, this was mainly due to the size and slower processing of data with the used tablet.
However, it probably depends on the area of application which device has the greatest benefit. In a restaurant, a tablet is supposedly better because you can look at the food more easily. For other tasks, a smartphone is probably sufficient.
Small supermarket to test AR in an offline retail setting.
Could there also exist an interaction effect between HoloLens, smartphone and voice and gesture control?
Interesting point. With the handheld device (smartphone, tablet) you do not have the possibility to work with natural gestures. Instead, you can only perform touch screen movements. Here, a voice control would possibly have a greater advantage. With a HoloLens natural movements are applicable, which makes the control by gestures more attractive.
3) Now when you look at the currently available consumer AR applications on the market: Do the companies fully exploit the potential of AR?
In my opinion, many companies have so far hardly exploited the potential added value of AR. In general, companies can currently be divided into one of three categories:
Companies that are not yet involved in AR at all.
Companies that are already doing something with AR but in my opinion do not generate any added value for the customer.
Companies that successfully use AR to provide added value to the customer.
Companies of the second category have jumped on the AR hype, but have not yet managed to use the technology efficiently. ING Bank, for example, has developed an AR filter where a virtual man jumps across the screen after each transfer and tells you that the transfer is now completed. Since it does not improve the bank's service, I believe that this application in this form has only limited benefits for the customer. You could definitely get even more out of it.
Ikea and Amazon are example of companies, that already use AR in a useful way.
Ikea AR Hey Place
But there are also many useful applications in the cosmetics sector, for example from Douglas or L'Oréal. These companies use AR to improve the process of "Product Trial and Preview".
L’Oréal Virtually try on
The insurance and pension industry can also benefit greatly from the use of AR. Pension fund providers often find it difficult to reach the young target group. For young people, the subject of pensions is boring, too far away, and they cannot imagine what life is like at the age of 70. Nevertheless, it would be important for them to deal with it at an early stage and make appropriate arrangements. One of our current research projects is devoted to this problem. We are conducting studies in which we show students an AR-Face Aging Filter for their own face. This enables them to see themselves aging. We assume that by visualizing their own aging, the participants can better put themselves in the situation of old age and therefore think more about their own retirement afterwards.
Example of an Face Aging Filter AR APP
I therefore recommend companies to first put themselves in their customers' shoes and consider what problems their customers have and whether they can solve these problems through visualization. If the problem cannot be solved by visualization, I do not think AR/VR is the right solution to invest in.
What can companies do if they find it difficult to develop suitable use cases for AR or are unsure whether their AR application will add value?
I can only recommend to these companies to work with universities. Of course, there is often a trade-off between a strong theoretical focus and controlled laboratory conditions of the universities vs. very case-specific and results-oriented studies of the industry. Nevertheless, I am convinced that both parties can benefit greatly from the combination of research and practice if everyone agrees to compromise.
4) Concerning the Corona crisis: Have you personally noticed changes in your professional or private environment regarding the relevance of AR/VR, for example with regard to requests for research cooperation from companies?
Yes and no. We did not directly gain easier access to companies due to the distance rules in the wake of the Corona crisis.
However, by shifting the physical to Digital, we have received significantly more requests to give webinars via AR/VR. In the last few months I have therefore given several webinars on the following topics: AR & Customer Interaction, AR & Asset Management, Logistics, Supply Chain Management, Industrial Machinery Repair and Remote Service Systems.
From the research side, such as the European Commission, I have also seen that the call for research on AR/VR has increased. We have also initiated several projects here during this time, such as the CoronAR project, which is intended to contribute to the clarification of the disease through visualisation. In order to avoid the initial hamster purchases, we also had the idea to reproduce products in supermarkets with holograms. With this we wanted to show that the products will be available again next week and that the consumers do not have to hoard.
We have submitted applications for research funding for some of these ideas, but we are not yet working on any of them.
There was also great interest from the university in the area of digital teaching. Here, too, I have received enquiries asking whether and how they could integrate AR/VR in order to make digital teaching more sustainable and exciting.
Have you already implemented something yourself?
Unfortunately not yet. We discuss it a lot, because we assume that it is more comfortable for students and increases their attention if they sit in a VR lecture hall instead of hours of lectures via Zoom, MS Teams, etc. First study results also speak for this. However, the practical implementation is still a problem here, as we currently do not have the financial means to provide each student with VR glasses.
More hints from Jonas Heller on how AR can help in times of Covid-19 can be read on Medium.
5) "In 5 years, almost every household will own XR glasses like e.g nowadays mobile phones and laptops and use them for different applications like holiday simulation, shopping, meetings, for teaching/learning”.
What do you think about this statement?
I am often asked this question. In my opinion it depends more or less on when Apple will release the first version of such XR glasses. There are indications that this will happen within the next 1-3 years. Here I hope that it will be glasses that are customer-friendly and well thought-out and that really add value for the end customer. Counter examples are Google Glasses or HoloLens, which are mainly intended for the B2B sector.
For this reason, I think it depends on when one of the big companies launches a customer-friendly version of such glasses. 5 to 10 years sounds like a realistic time frame to me.
This is why it is so important to increase research in the AR/VR/XR area. There is still a lot that we need to find out and learn about XR glasses and their useful applications in order to know which glasses will ultimately be accepted by customers and which areas of application will provide a benefit.
What do you think about the applications listed?
Holiday simulations already exist to some extent in the form of 360° tours through hotel rooms and holiday resorts. Airline companies such as Emirates now also offer the option of selecting the appropriate seat in VR or 360° before booking a ticket. I therefore see great potential in the tourism sector.
3D Cabin Emirates Experience
I think the way to and from the shops will also be strongly influenced by AR, so we'll get a lot of information on the windshield in the car. In the store itself, I expect a kind of "pick by vision" in the future, like you already know it from warehouses. The consumer inserts his shopping list in advance, so that the way to the individual shelves and products is automatically displayed in the store itself.
For meetings and teaching, it depends on what the glasses look like at the end of the day and what turns out to be the more suitable option. Is it therefore better to hold meetings in your own four walls and see the other participants as holograms or if everyone meets in a virtual room. For business meetings or lectures I find a professional virtual meeting room without distraction from the outside world more suitable. For a consultation with a psychologist, on the other hand, it might be more relaxed if you don't have to leave your own safe four walls.
I see potential for both technologies. Which technology is more suitable for which area of application will then become apparent. As far as I know, there are still no reasonable research results that compare AR and VR and investigate when which technology is more beneficial for the customer. There are still a number of unanswered questions that can be investigated. For this reason, I am very curious to see what will happen in this area in the future.
Final Statement of Jonas Heller:
For research, I feel it is important that there is even more collaboration between the various research groups of AR/VR, so that we can learn even more about the technologies and benefit from experiences and learnings. While our research group has published extensively in the areas of AR, consumer psychology and marketing, we have learned most about the technologies from studies that have not worked from a scientific perspective. For this reason, I would like to see more exchange, because only through a transparent exchange of experiences we can advance the research field AR/VR/XR more strongly.
In our interview on next weeks #digitalthursday, we will talk with Arnd Hutmacher, Head of Digital & Brand PR, from Nestlé Wagner about the company's latest AR promotion and highlight the benefits of the application for the consumer sector.