The critical role of humans in AI - AI Change Makers Podcast

Episode transcript: #2 eBay – From helicopter parts to office utensils: improving the ecommerce experience using data and AI

Brought to you by GAIn, this is the AI Change Makers Podcast. My name is Wouter Huygen and on this show, I talk to business leaders about how they create industry breakthroughs with AI.

In this episode I talk to Allard de Boer who is head of global analytics at Ebay Classifieds Group. He talks about the many areas where eBay uses Data and AI to improve their platform. In particular also, how the role of human creativity and problem solving remains critical in the process, for instance for devising relevant experiments.

 

Allard welcome to the show!

Allard: Hi Wouter, good to see you!

 

You are head of global analytics at Ebay Classifieds Group. I am curious what path led you to this role.

Allard: I started my career at eBay at Markplaats actually, almost 10 years ago. I worked in the area of products analytics and web analytics and I have been involved with product analytics and experimentation for the first half of my career. Then I moved to the central team about 4 or 5 years ago to see if we can scale that experience on product analytics and experimentation to the rest of the portfolio. So I have been doing that for the last years and currently I am head of the global analytics team overseeing multiple topics. Next to product analytics, we also look at marketing, commercial analytics, and the verticalization of our platform.

 

Obviously, eBay is one of the oldies in terms of internet companies; digital natives. When was eBay founded?

Allard: I think in 1995.

 

I believe you are right, I looked it up.

Allard: Oh good! I passed this one.

 

In general you see that big internet companies that are still around from that time are some of the early adaptors of Data and AI. What are some of the most prominent examples for eBay that spurred the growth in its early days?

Allard: A company like eBay is a really a technology driven company and what you see is that technology and the data they are so much combined. So actually all of the products that we built since the start have a very strong data component in there. For eBay it is very important and also one of our biggest challenge to make sense of the huge amount of inventory we have to our customers. The matching of the inventory to the customer needs that has been the priority from the start, using search engines and the ranking of the content and matching this in a way that people can navigate through this enormous amount of inventory. That has been the backbone of what we do from the start.

 

You could say that those are some of the critical capabilities that the successful ecommerce players adopt. There were some retailers back then that moved the line, but they didn’t make it. They were first movers, but in the end they did not make it. I think this is not because they were too late, but I think they didn’t realize this thing that you’re mentioning; how to manage your digital shelve space is different online from offline.

Allard: Exactly. What you also see is that everybody jumped on the digital shelve space, the longtail. For ecommerce players that is a bit closer to what they know, but when you think about customer journeys and customer experience you get a much better view on what customers are experiencing throughout their journey on the website and you will also understand that you really need to scale the experience instead of trying to match your offline experience to the online experience. The shelve space that you just mentioned sounds to logical, but at the beginning this was not logical at the beginning at all for many retailers.

 

How come that eBay got it right?

Allard: I think that the difference for eBay was just the abundance, that is always the power of the eBay brand. The abundance of products exceeds any part of imagination that you can bring to it. If you go to eBay, you can find anything, from spare parts for your helicopter to new utensils for your office. Everything can be found in every way on eBay. I think that has been the power of eBay since the beginning and there is no offline retailer that can match this in any way. Especially not in the early days.

 

We learned from some of the classic war stories of AI and product recommendations in retail that diapers go with beer.

Allard: You do see typical combinations of products that you might not think of. Especially on a platform like Marktplaats, you see combinations of moments in a customer’s life. You can see the combination of baby chairs and new cars which is a combination that you see on our platform quite often. And this is possible because we sell all of these different things.

 

I suppose that those algorithms are pretty mature at the moment, so your recommendation and personalization are the algorithms that run at the front line. I can imagine that eBay is pretty advanced in this because of their long history. What is in your view the next evolution in applying data and machine learning?

Allard: You would be amazed. A platform like eBay still spend a lot of time understanding what the inventory on the platform is. An example is that if someone sells a couch, we have thousands examples of varieties of couches, matching and grouping these is still very important for us. The better we understand what kind of ad the customer posts, the better we will be able to match it to the needs of the customer. This shows that we are still on a journey and that there are so many topics that we still need to handle. The other thing you see is that much more business development moving to data. The data is picked up at the front end, the platforms with all of the recommenders and notifications. This is native to a company like eBay. However, using this data at the back end and really understanding the topics that we should be focusing on as a business, this is where we are currently going through a data transformation. Even though we already are a digital native company, there is still much more maturity to be gained in this area.

 

So internally there is a difference between the front and back end in terms of sophistication in terms of using data and AI technology?

Allard: To clarify, we distinguish between data used for platforms and data used for people. In both the decision making is different. With data used for platforms, the recommenders etc., we invest a lot of time in it and we have a lot of experience. Using that data towards the people and in the decision making process, this is the area I spend a lot of time and energy myself to help evolve that and to get to that level of sophistication in these teams as well.

 

How do you do that? How do you raise this level of knowledge and competencies?

Allard: One thing to really understand is that nowadays a company like ours has more data than anybody can make sense of. For any topic we can find data points that are relevant, but not all data is relevant or a good indicator or predictor for future success. People become overwhelmed with all the data that is available and this causes a grid lock to move to the next phase; How to transform that data into actions and into the organization. Structuring the data in a way that makes sense for the business is a big chunk of what analytics teams do and also one of our biggest assignments.

 

The challenge is to come up with the relevant insights. There is no lack of data and therefore you could produce a firehose of insights, but which of those really matter and what do you want to do with it?

Allard: Yes, and it sounds really straight forward, but really it is not. It is very easy to understand what is happening, we can find several metrics and data points that show us what is happening. We can also have a bit of a brainstorm about why things are happening. But really understanding the causational effects and the drivers of those, is where the analytics team in our organization nowadays really makes a big impact.

 

So that is using the data analytics for operational improvements and also strategic choices. Can you give an example of such operational improvements that the teams are working on and they recently came up with?

Allard: One of the tools that we use for example is experimentation. This is a tool that is used a lot in the product teams for actually two typed of use cases. One is a discovery use case: I have an idea, lets quickly test this and make it very simple and objectively involve the customers in this decision making process. The other use case is about understanding the incremental value that we deliver from a product improvement that we are doing. Experimentation has unlocked much more insights towards the company for us to really understand how the things I am doing are impacting the customer and the business. This can be all kinds of different metrics; conversion, retention, acquisition. All of these play a role in this. If you look at a mechanism this is where you really saw an acceleration in the past 5-10 years.

 

So it is coming up with a hypothesis on how to improve your main business drivers like acquisition and retention but then translating those to concrete actions to improve your product or platform, and then testing it?

Allard: Yes, exactly.

 

Experimentation is for some executive still a bit of a magic word. It is seen as a capability that most big digital natives have that bring them a lot of value. But there is also a bit of a mystification around it. Is that the case if you look at experimentation within eBay, how does it work?

Allard: I have been running the experimentation at our business for at least 8 years already. Before that we were already doing this at a technical level. Because our platforms generates so many visits and activity, it is pretty easy for us to say, lets pick a small part and lets try something different with this group. Switching away from 1/5/10 percent of those visitors and showing them something completely different and then just measuring what they do. That is the way experimentation is done most often, and of course there are more complex methodologies around it. However, this is basically what it boils down to. We split people in groups and we ask them ‘’which part of our solution do you like better – version A or version B? Then we just measure what people do. It starts with the technology. You need to be able to build the technology and have it available to make this split testing possible.

 

So you basically have hundreds or thousands of versions of your products live at any moment. And if I go online I get a different version than you?

Allard: Yes. In the early days we ran one or two experiments at the same time and we tried to make sure that they did not collide. But what you see is that the experimentation methodology has improved enormously over the years and that we are able to run several experiments at the same time all the time. This means that you and I could be in several experiments at the same time, but not in the same groups at the same time. So we could be in the same group for one experiments but in opposite groups for another experiment. That scalability really helps to increase the velocity of experimentation.

 

The magic maybe in the technology and not so much in the analytics. I suppose the analytics is an AB test where you compare two groups for a given experiment and you look at the effect. The fact that you can do this at scale with many experiments at the same time, that requires technology to do that.

Allard: Without the technology, this is not possible. I think storing the data with the different flags is pretty straight forward. But then you also need to make sure that the interpretation of those results make sense. Your testing a subset of your whole population, so making sure that any conclusion that you derive from that, there’s methodologies that you need to apply to that. Coming to statistical significance sounds very simple, but there are so many different flavors that you can put towards that. The analytics teams have spent a lot of time in optimizing and perfecting that.

 

So it is actually both, the technology and statistics.

Allard: You do see third party AB testing tools nowadays; they have much more out of the box statistic significant testing. In all honesty those are pretty good. For a company like eBay, a first party testing solution fits better with our business model so we need to do a few of these steps ourselves.

 

If you have an experimentation capability like you described, it still requires someone to formulate the experiments. Coming back to the human part of the equation and given the fact that there is more data then all the people can probably chew on, how do you get your organization to move in that direction?

Allard: That’s a great question. The great thing is that if you think about experimentation, the technology is becoming much more accessible. Setting up an experiment is really becoming much easier, but still it takes time. What you see happening, because it is so accessible to many teams, is that there are several approaches towards experimentation. One of the approaches we have seen a lot in the past is that teams just start testing. Anything that they can think of they start testing to see what the results bring them. Those do give you good results from time to time, but the way I picture it is like throwing spaghetti at the wall and just seeing what sticks; you have all of these ideas and you just throw them at the wall and hopefully there is a winner that drives the business forward. Building a bit more sophistication in this process by doing your homework and really understand what the drivers and scenarios are that most likely will make an impact and which ones probably won’t. Good hypothesis creation and really understanding the results and thinking through your problem. That is an evolution that we have been going through in the last couple of years. Even though experimentation is cheap, the engineering capacity is one of our scarcest goods. So any waste created in this process is suboptimal.

 

And how do you build these competencies?

Allard: At Marktplaats we started with the tooling first to make sure that the technology was available. Throughout the first stages we learned that third party tools only could bring us to a certain point. Our ambitions on experimentation exceeded the capacity of these third party tools. So that’s when we started building our own AB testing framework. But this framework started running into limitations around the data scientists and analysts not being able to evaluate the experiments. You cannot scale these capabilities endlessly so we started looking for a way to automate this. This is where another evolution on this AB testing tool was needed and then giving it to the team so that they can operate it themselves. But the whole thinking around it was that we would give this capability to teams but without the proper training, education and understanding of all of the different moving parts there, you cannot expect high quality to come out. This is when we started thinking that we also need to educate people more.

 

I remember when I was 8 years old, my father got his first personal computer at home but he basically never touched it. It gradually moved in to my room – he never used it because it was a new tool and he didn’t know what to do with it. So you can give people a tool but they still need to be trained on how to use it and what the potential is.

Allard: Yes exactly. And the questions we wanted to answer 10 years ago with experimentation were much more basic than the questions we are answering today. Really understanding what statistical significance means and what kind of metrics should we be looking at and which ones should we not be looking at. Pitfalls that we saw were that people would put in 10 different metrics, just look at the results – and declare whatever goes up the winner. They did not think about the possibility that there could be a false positive. Really helping the teams to do that better also improved our AB testing framework along the way because they started asking better questions and that also helped us to evolve our thinking around this as well.

 

Technological tooling is relatively easy to scale especially in a global firm like eBay, but how do you do that with training and people competencies?

Allard: In the beginning we started by giving people instructions on how to use the tool and that started evolving a bit more. People were now able to access the tool and they were able to use it, but in the end they were not fully confident that what they were looking at was actually something that makes sense from a business perspective. In the end they came back to the analysts again, so this is when we took a step back and started to think about how we can ensure that the people in our company fully understand how to make decisions based on data and which questions you can answer with it, which ones you cannot and which methodologies you want to use. That sparked an inhouse training program which we call the analytics university. This is a business program that really takes people who are not day to day working with an analytical background and raises the bar across the whole organization on using data in operations. We kicked it off around three years ago and we run it almost every quarter with a big group of participants, and we continue to drive it further and have been scaling it ever since.

 

And that helps you to have the demand side better talk to the supply side when you talk about data and analytics? So the demand side and the business side, are also better equipped to ask the right questions?

Allard: There are a lot of knock-on effects here. First, people feel much more empower to use the data and the tooling that we have so you see that they are using it more and more, we really see an increase in usage and testing. Next to that, the questions they start asking to the data and analytics team become a lot more sophisticated. Instead of asking the relatively straight forward question ‘’what happened’’, they really started thinking about the why and brainstormed together with the analytics teams on what answers we should be looking for. The third one that we see happening is that you get a much broader data stewardship throughout the company. The insights are just as good as the quality of the data. Where in the past years data quality was owned by data or analytics teams, nowadays you see that this is much wider owned by the whole company because people really see the value it brings.

 

What you described still means that all the improvements from data and analysis still is a very close collaboration between humans, problem solving, thinking about where we could have improvements, formulating analysis and experiments, and doing these cycles, right? If you look or dream ahead, do you see a future where AI takes over part of this? Where you also automate some of these improvement cycles.

Allard: This is a very interesting area of development. We are becoming really good at answering questions which we know we need to ask. Do we see certain segments of customers in the journey? Do we see certain drop off point? But still, the questions that are being asked a limited by the people asking those questions. Currently we are going through an evolution where we really want to augment these teams with more forward looking insights so that we alert certain opportunities without us already going through the path of understanding what is going on. This is currently the evolution we are going through. I am hoping that in the next years this will evolve more and that we can really empower the business, marketing, product and customer service teams with forward-looking insights that they might not have foreseen and making the impact they can make much higher compared to what we can deliver today. eBay is already going through this. They call parts of this AI analytics and in the next few years I really see this as a growth opportunity.

 

So if I understand correctly, where you now would have your analysts doing analysis on a customer journey to validate a certain hypothesis, you would have an algorithm running on your customer journey, all the data that the platform generates, and come up with certain hypotheses, flagging certain irregularities, classify, categorize or cluster certain behavioral patterns, and basically come up with a suggestions to your analyst for areas to look at?

Allard: We already see this in different areas and on a specific level, within marketing there are already certain segments of customers or customer behaviors that are being identified automatically. But also the product improvements, there is so much more opportunity there and this is where I think the data we have, especially on the scale of what this platform generates, we can really come to new insights that without AI support we probably wouldn’t have uncovered, unless maybe by accident.

 

To close off, what has been your biggest personal learning over the past couple of years being part of this journey.

Allard: A couple of things. I think that for myself that the data capability is really part of a change management for an organization. I think that everybody in our organization understands the value and how it should be used, but actually making it work for your organization has been very interesting. Analytics can push out much more BI dashboards or these kinds of things, but that is just generating much more noise and really making that impact and changing the internal behavior of our employees, and that is a big chunk of what we do. Over the years that has become much more apparent for me.

 

This is one, are there any others?

Allard: It is really around the people, the process, and the tools. In a company like ours we tend to look at the tools first, but the business and the people should really dictate what kind of tools or solutions we need to drive the business further. That has been a big change for us.

 

Thank you Allard, it was exciting to hear all of these inside stories from a well-known brand, so thanks for that!

Allard: Thanks for having me!