CTO André Neubauer shows advantages of AR for retailing & gives insights about process decisions
André Neubauer, former CTO of Mister Spex, accompanied the development of the AR application and knows the difficulties until the optimal solution is found. Mister Spex was one of the first online stores in Europe that enabled a virtual try-on of glasses, and is still causing great enthusiasm among consumers today. In our interview, he explains the questions that every company should ask itself when introducing a new application.
touch point to AR/VR:
already worked on AR projects during his studies
implemented 10 years ago an application for ImmobilienScout in AR
uses apps for true-to-scale visualization (Mister Spex, IKEA APP, measuring in general etc.)
misses "killer" app with unique value for using AR
no use of VR applications due to mental anchoring in the gaming area
since 5 years AR at Mister Spex
"Augmented Reality is one of the most disruptive technologies we will experience in the next 10 years."
Mister Spex was one of the first companies integrating an AR application as a central feature in an online store and has since been constantly expanding the range of products available for the application.
1) What was the need and the idea behind the AR application and how has it developed since its introduction at Mister Spex?
The virtual try-on is one of the features that most people at Mister Spex are excited about. When I talk to someone about the fact that I worked at Mister Spex, I am very often asked about this feature. Especially for people who are not technically skilled, such features - besides the actual added value - trigger great enthusiasm. At Mister Spex, for example, the virtual fitting supports the selection process in a playful way that is really fun.
The technology at Mister Spex has changed a lot over the years. It was recognized early on that customers need to be offered a way to simply try on glasses. For the beginning, the 3D view was considered too complex and the company started with a self-developed 2D view. Customers uploaded a picture of themselves. The glasses were placed over the image and aligned using pupil recognition. With this the first view was ready. The customer could then adjust the position of the glasses and got a first feeling whether the glasses fit him.
In order to provide greater added value for the customer, Mister Spex then developed a 3D live view based on the TryLive technology from AcepTryLive.
For this purpose, the live image from the camera is supplemented by a virtual 3D model of a pair of glasses. As the customer moves, the model is adjusted so that he or she can see what the glasses look like on his or her head from different perspectives. However, the problem was that this solution did not work on mobile devices.
Live-Try-On of Glasses at Mister Spex
Why did you stop developing this application yourself?
This is a decision that companies should consider carefully before implementing. With every technological application the question "Make or Buy? As long as the technology is not yet mature, it may make sense to buy it externally. The development in the AR environment in recent years shows how much the applications for use on the web have changed. While Flash was used in the past, all modern solutions today are based on HTML5. This entails large investments. The question to ask yourself is whether you want to do it yourself as a company or whether you want to buy a ready-made solution, which in case of doubt is easier and cheaper.
How did the application been refined afterwards?
Since the technology was still relatively in its beginnings at the time, it took us a long time before we were able to offer a good try-on on mobile devices as well. Therefore, 1.5 years ago we decided to work with ditto. Based on their technology, we developed a video AR instead of a live AR application as before.
During the first fitting, a video of the customer's face is recorded and used to create a 3D model. The customer only has to look left and right once. Then ALL glasses can be tried on. The calculation takes place centrally, so that the customer experience is independent of the computer capacity of the device.
A further advantage compared to the live try-on is that the initial image can be taken without the customer's own glasses, but the try-on can be done with glasses again afterwards. This way, the view is not distorted and the selection process is easier.
Virtual Try-On with video at Mister Spex
Which factors play a role in a smooth presentation?
There are two factors:
1) The Rendering
This only applies to the live view and refers to how many polygons (i.e. how many polygons are used to display the image) are taken into account. A higher number of polygons allows a higher level of detail, but also leads to a higher processing power on the customer's end device.
2) The Asset Quality
The quality of the 3D model of the glasses to be projected onto the image or video. In the case of Mister Spex this is reflected in the details of the glasses.
What other differences result from using live or video model?
Limitations exist for both approaches and result primarily from mobile uses. As mobile usage is constantly increasing in all industries, it is particularly important to ensure a good customer experience.
For the live view, only the 3D model of the glasses needs to be downloaded. The positioning in the live image is done on the device, so that the data consumption is low. However, the required computing capacity is comparatively high, which has an impact on battery consumption when used on the mobile device.
In the video view, the centrally calculated representation is downloaded. The data consumption is high depending on the display quality, but the required computing capacity is lower due to the display.
So there is no such thing as a perfect solution.
2) How often is the AR feature actually used and do you think that this feature is the reason why many of your customers shop at your store and not at another eyewear provider?
At Mister Spex, customers can choose from more than 10,000 frames in the online store. There is also the option of having up to four glasses delivered to their home for a free fitting. The 3D fitting helps enormously in the decision-making process and is indispensable for reducing the complexity of the product range.
Our analyses also show that prospective customers who have undergone a 3D fitting are much more likely to have glasses sent to them for a fitting and end up buying one.
Do customers also have difficulties in using the 3D view?
This varies from customer to customer. Experienced online shoppers are often familiar with it. But of course we want to introduce all customers to this innovation. That's why we do a lot of research on both a quantitative and qualitative level and continually revise interaction concepts to improve the customer experience and reduce potential skepticism. We have achieved a lot in this area in recent years.
An existing problem is the true-to-scale representation of glasses. Since neither video nor live images contain reference information on the size of objects, it is still necessary to resort to alternative solutions. These can be estimates or the use of a standard chip card as a reference object. Stereo cameras, as they are increasingly installed in smartphones, will solve this problem in the long run.
However, the technology should never be in the focus of attention, but should only serve to improve the customer experience. Together with other tools such as an online vision test and pupil distance recognition, we are getting closer to our vision of redefining the way people buy glasses. Customers can simply order their glasses - even on a Sunday - and a few days later hold the finished product in their hands.
3) Due to your long experience as CTO and software developer:
Which hurdles did you have to overcome on the way to implementation and which learnings can you give to entrepreneurs, educational institutions and interested parties who also want to develop a project with AR?
The first question is "Do I have to do it all myself?". Due to rapid technological development, there is always the risk that a technology that we use today will be obsolete tomorrow. An advantage then turns into a burden. Nevertheless, this approach can make sense, for example, if it is of high strategic relevance for the company.
Another obstacle that regularly drives me to "frustration" in my job is the different development of platforms in which AR is used. Browsers, for example, are becoming increasingly restrictive. For example, users need explicitly give permission to use cameras and data. Depending on the customer, this can affect the user experience and buying behavior, especially if customers are more skeptical about digital offerings. Access to other hardware (e.g. stereo cameras in modern smartphones) is also virtually impossible in a web context.
Mobile apps, on the other hand, have an easy solution: If the user agrees to all access rights once after installation, the customer experience is no longer restricted afterwards. In addition, native apps can directly access the hardware of the device and provide more functionality or a better display.
If companies decide to develop their own mobile apps, however, they must be aware beforehand that the development effort is higher and, in case of doubt, will have to be repeated if several platforms are to be covered. In addition, the app download represents a hurdle for customers, depending on how often the app is used afterwards.
Are there other use cases in your area with AR?
Yes, for example, we are looking at solutions for our stores to make the entire online product range available
4) Do you see VR use cases in your field in the future? (e.g. a walk-in virtual Mister Spex Store)?
Not really. We sell a physical product and not a virtual one. That means we connect physical and virtual objects and display them in augmented reality.
5) Keyword Corona crisis: Have you personally noticed changes in your professional or private environment regarding the relevance of AR/VR? In general, do you estimate that Corona has an influence on the relevance of AR&VR and do you expect industry-/ application-specific differences?
The Covid-19 situation has led to a situation where society has to deal much more with technologies, for example, to be able to work from home or hold virtual meetings. What used to be the exception is now "the new normal". Through playful approaches to AR, such as virtual backgrounds in video phone calls, many hurdles are removed. For me Corona is an enormous digital boost. Studies show that in the last 3 to 4 months, online penetration has developed in some areas in a way that would otherwise have been predicted for a period of 10 years.
In the past, customers were often skeptical about e.g. buying furniture on the Internet. Due to Corona and the taken opportunity to shop in the store, they are forced to choose another way and shop online. Since this is the only option left and the experiences with it are hopefully positive, inhibitions are automatically reduced.
Overall I see Covid-19 as a strong driver for technologies like AR. With VR I can't judge it, because I have too little contact with the subject. But in general I am convinced that Corona is advancing these topics.
6) "In 5 years almost every household will own XR glasses like e.g. today's smart phones and laptops and use them for different applications like vacation simulation, shopping, meetings, for teaching/ as learning rooms".
What do you think about this statement?
I think this is a totally good vision. Whether this will already be fully achieved in 5 years or maybe only in 10 years is another question. In my opinion, it depends on how long the current situation will last and how digitalization will progress as a result. Moreover, the trend will first push through to the younger generations.
The purposes will surely also change. There are a lot of things we cannot yet imagine today, and we overestimate others, for example vacation simulation: I'm not sure whether this is the right way to recharge our batteries. But I do believe that XR glasses will be integrated into everyday life.
Surely the first versions will not be as powerful as any other disruptive technology. Google already offered a niche product a few years ago with Google Glasses. Now design and other aspects such as good battery performance have to be solved to enter the mass market.
According to this, maybe not in 5 years, maybe not all households, but basically this is exactly what will happen. As new technologies evolve, there will be a corresponding stagnation in the mobile sector.
Finale statement of André Neubauer:
I believe that AR has been a niche application so far, mainly used for special cases.
However, this was important in order to create many basics. In a few years we will see completely different use cases that will make our lives in general easier or even safer. Surely AR will also be used interactively and not just to display information. I am looking forward to that.
All in all it is one of the most disruptive technologies we will experience in the next 10 years!