Data & Algorithms in Retail: How to Introduce AI in Operations and Create a Winning Strategy

Why retailers should re-evaluate their focus when it comes to investing in Advanced Analytics & AI

Retailers have in the past couple of years especially been experimenting with AI in typical marketing use cases. They have however foregone the domains where most substantial gains can be won, such as replenishment, pricing strategies and high-level design of promotional activities.

Indeed, the archetypical destination for investments in advanced analytics for a retailer is the marketing department. Innovating with CRM tools such as Salesforce Marketing Cloud, Selligent, Zaius or one of the many other alternatives has a proven return on investment.

Especially given that many innovative cases depend on tailored communication to customers, this is an excellent place to start exploring the capabilities of data science and advanced analytics.

Once successful initial investments have been made in this area it can be tempting to attempt to try to repeat the successes by investing in more of the same; adding bells and whistles on top of these systems, with ever decreasing marginal returns.

Much like the salesperson who is reluctant to switch to a novel product because his current portfolio has served him successfully in the past - people tend to stick to things which are familiar. However, limiting your focus on solely this area can leave objectively more interesting opportunities unexplored.

The million-dollar question to be answered is then : “What should I be investing in next; as I continue to transfer my retail organization into an organization driven by data & algorithms?" This is a question we get asked a lot while visiting these retail organizations looking to find a new path for sustainable growth. The following therefore stems from our experience in the field.

We have come to the conclusion that replenishment, pricing and promotions are domains in which substantial opportunities can be found. The fact that there is value in this area is evident, since it touches upon the very essence of what it means to be a retailer; and the essence of the value a good retailer brings to its customers.

"The essence of retail is getting the right product at the right price in front of the right customer; at exactly the right time."

Some bravery is required, since this implies venturing into domains which have been paining retailers since times immemorial. The greatest complexity comes forth from the complex interactions which take place between these domains.

These complex interactions are the reason why the majority of retailers have resorted to a "satisficing" approach - pursuing a course of action which satisfies the minimum requirements necessary to achieve a goal. This satisficing approach is often based on experience and intuition, making these processes hard to scale and heavily dependent on individuals; something which can pose a substantial risk to an organization.

The aim of this long-read article is to provide insight into these processes, showing how these depend on each other; as well as providing a clear vision on how traditional and novel technology can be combined to make a company better at solving this puzzle.

Retail is a market in duress, and those who will survive in the long run will be the players who are able to implement a system that can outperform traditional satisficing approaches.

This article is the first in a four part series. Part one gives a general overview of the dynamic we want to study, and the overall philosophy we employ in doing so. This also includes the demand function (i.e. demand prediction) as a core component of most of these processes. The second part dives into depth on the replenishment task; making sure that the right balance is found between the cost of lost demand and the inventory holding cost.

Part three in turn focuses on the automated pricing, assessing the right price given the current market conditions in order to optimize gross margin. The fourth and last part includes feedback of the former to domains into the domain of marketing and promotions, which can actively leverage insights of the latter two domains to make highly personalized offers and communication more effective and better integrated.

A more interesting playing field for your data & algorithms

This first part gives a high-level overview of the playing field in which retailers can reap massive benefits from investments in analytics: distribution and pricing.

"Too much of today’s marketing is 1P marketing. Companies mainly concentrate on promotion and sales and disregard product, price and place (distribution). This results in ineffective marketing. - Philip Kotler"

Three key areas of excellence

No two retailers are identical, and being a retailer has always meant being capable of a great diversity of tasks. However, when it comes to analytics investments we are going to single out three key domains of interest: pricing, replenishment and marketing and promotions.

These three domains are of course interrelated; the right price can be heavily dependent on the current inventory position or the lead times observed at suppliers. Aspects of replenishment such as safety stock levels are in turn influenced by the gross margin which is attained for each specific product.

Marketing and promotion in turn can also be a bidirectional influence on both replenishment and pricing. Examples of this can be promoting specific products because there is excessive inventory present, or increasing the inventory position in anticipation of a planned promotion.

All this talk of innovation tends to remain rather abstract, so the main goal of this whitepaper is to provide some very concrete examples of approaches which can be taken to innovate in these domains. This is done in parts 2, 3 and 4 of this series; after the general market forces have been sketched in this first part.

First key area of excellence: Replenishment

Replenishment is used here as an umbrella term for anything related to the ordering and logistics decisions taken by a retailer. Within this domain the key objective for a retailer is to optimize the balance between the cost of inventory, and the opportunity cost of lost sales.

Both of these concepts are seemingly simple; but the exact balance between these terms as well as the correct numbers and figures can be profoundly hard to come by.

Various economic trends have caused supply chains to be become shorter. One of these is the need for much shorter response times which are required to capture more volatile demand. Another is the new global trend towards protectionism, which is causing companies to re-think their supply chains (Rose and Reeves, 2017).

Where it used to be sufficient to have cost-optimal supply chains, having supply chains which are resilient is fast becoming an equally important consideration. Decreasing the number of links in the chain, as well as the number of legislative domains through which the chain travels is the most straightforward route to this result.

A shorter supply chain does not necessarily mean that things are becoming simpler! As a matter of fact, new technologies are popping up which completely change the way in which companies are handling their supply chains.

This has lead to some people declaring the death of supply chain management (Lyall, A., Mercier and Gstettner, 2018), but in all earnestness this should be seen as more of a rebirth of the supply chain than an actual demise.

In the vanguard of this new wave of technology are novel applications of artificial intelligence in the domain of the supply chain. These applications typically serve to remove repetitive and transactional tasks from the human to-do list.

Automating these tasks opens up brainspace for people to focus on finetuning how algorithms work, and thinking about how the systematic process can be managed better. Overall resulting in better performance, and more interesting job content.

For more in-depth examples for what this shift means for retail companies, the reader is referred to part 2 of this series.

Second key area of excellence: Pricing

Only very few retailers have the luxury of serving markets with price insensitive customers, who don't really change their purchasing habits based on the prices which are asked. And whereas these markets exist, they come with a completely different set of problems; ask anyone who has recently been running a drug cartel.

“The result is that incumbents stay in business, facing only limited competition, even as they charge high prices for poor service.” ― Tom Wainwright, Narconomics: How To Run a Drug Cartel

In spite of the fact that price is one of the key dimensions customers take into consideration when making retail purchases, many retailers are stuck with highly archaic processes to define prices.

Effectively relying on gut feel, simple anecdotal comparisons with competitors or simple markup rules (cost plus pricing) to define prices. Moreover, prices are then often no longer reconsidered for change until it is time for clearance sales on these products.

This is wasting enormous amounts of potential, especially for online players whose menu cost (= the cost of changing prices) is practically zero. The flip side of this low cost being that it has become increasingly easier for customers to compare prices of different retailers online.

The fact that comparing is so simple makes it ever more important to have a clear system in place to respond to imbalances in this area.

With the advent of the internet, customers are able to compare prices easily and this has raised the importance of pricing among the 4Ps. [...] they are more likely to buy their preferred brand from the retailer offering the lowest price. - Philip Kotler

Naturally the quote above has to be nuanced, and price is not limited to the dollar amount put on the article. Substantial value can be derived from convenience, or a trusted name - value for which a customer is willing to pay a premium.

All this has been known and researched extensively in economic literature. However, recent advances in data science in combination with the data collection apparatus that many retailers have in place today have made it possible to move many of these insights from the domain of academia into economic reality. This makes ‘pricing’ a second domain of interest, a second key area of excellence.

For more concrete examples of how a typical retailer can implement a structural solution to optimizing pricing decisions the reader is referred to the third part in this series.

Third key area of excellence: Promotions

Another consequence of the pressure being placed on the retail sector is that many companies have been reverting to discounting strategies to keep revenues up.

This traditional control lever is both easy to pull and fairly easy to measure, and therefore provides a short term solution to a problem. One severely impacted branch of retail is fashion retail, with year-round heavy discounters finding more and more traction (https://www.economist.com/business/2016/01/09/to-the-maxx).

However, in mindlessly pulling this lever the discounter trap becomes a very real thing (https://hbr.org/2013/09/escaping-the-discount-trap)! Falling into this trap means that the only thing which is still drawing customers are heavy discounts, heavily taxing retailers' margins.

Needless to say that such a situation is not viable in the long term, and retailers who want to continue being relevant have to change things up.

One qualitative thing a retailer has to ask is what key service am I providing for my customers, and how can I focus more on the value that I provide to them rather than just providing them with ever cheaper products. The answer to this question will vary, but focusing on providing the right experience to clients who are passionate about your product in a holistic sense will be paramount.

Besides these qualitative and strategic decisions, intelligent automation can also help on these fronts. In both cases the essence being to stop carpet bombing discounts to a wide audience, but rather to strike with surgical precision when discounting.

Offering discounts only to the people who are expected to require them for convincing, adjusting the depth of the discount to minimize money left on the table, and to only discount products which needs to be discounted to move.

A first variation of this approach is intelligent reactive discounting. When analytics show that the inventory of a specific product exceeds the acceptable levels, expiry dates for products are creeping up scarily close or new product introductions are around the corner the point in time might have come to launch a promotional discount on a product.

But the way in which this is done leverages an intricate demand function, which can make predictions on the level on the individual customer.

The second variation of this pops up during the end-of-season sales, which occur bi-annually for most retail sellers. At this point in time even big retailers often rely on judgement calls by individual category managers to set the appropriate discount levels for the product portfolio.

Suboptimal discounts leave money on the table in the form of lost sales (when discounts are too shallow), and in the form of lost revenue (when discounts are too steep and product flies off the shelves too quickly).

For this second case, predictive algorithms based on historic consumer behavior can be used to predict the optimal discounting level for various products. Specifically, for retailers with negligible menu costs, even live learning algorithms can be an option.

These algorithms can actively experiment with prices to determine the price sensitivity of various offerings to clients. These algorithms continually strike a balance between doing experiments to gather more knowledge about customers and using what they have already learned about customers to optimize the revenue generated from these customers.

Of course, a system with this amount of complexity cannot simply be put in place at once - but should be seen as a long term goal.

This is why promotions should be considered to be the third key area of interest. For a more intricate insight into how these systems can be put into practice, the reader is referred to the fourth part of this series.

Why the demand function is your starting point

The motor of the recent advances in artificial intelligence is the ability to create prediction machines. These do what it says on the cover: they predict a wide array of things. When sufficient quality data is available, these techniques allow for making much better predictions than were traditionally possible.

The second part of this long read delves into more detail into the practical implications of building such demand models for various applications.

Outstanding performance in each of the three domains which have just been covered relies on the quality of the demand function. That is why demand prediction should be considered as your starting point for intensified investments in data science & advanced analytics.

Ordering the right amount depends on being able to estimate what is going to sell, setting the right price requires getting an accurate reading on price elasticity, and providing the right promotions require the correct estimate of the promotional sensitivity of various products and customer segments.

Again, it is key to note that a base price and a promotion are structurally different things in the mind of consumers.

A prediction machine is inherently a very simple thing; it takes various inputs (independent variables) and based on these values estimates the value for the output variable (dependent variables); in this case the customer demand.

Complete books have been written on the creation of the demand function, which are completely outside of the scope of this whitepaper. However, it can be very enlightening to touch briefly upon the various inputs which can be used to feed such a demand function.

These inputs give a clear view on the kind of parameters which can be taken into account when investing in data science and advanced analytics. Building effective prediction machines can move these areas away from typical gut-feeling into domains where things are truly measurable.

A conceptual representation of this is given in the image below, where simpler and more traditional variables are placed close to the center and vice versa.

Historical sales as input for demand prediction

Observing historic sales and trying to extrapolate from these figures to predict future demand is where most retailers start off. The simplest and most well known (and still widely used) technique is to simply use the averaged past sales as a prediction for future sales.

More advanced techniques will try to extract trend and seasonality information to further improve the accuracy of a forecast. This might also be augmented with additional information such as the fact that a new product catalog is being launched at certain set points in time.

There are however severe limitations to models which only rely on historical sales to make predictions. A first problem which pops up is what to do with newly introduced products. By definition these do not have a sales history to fall back on.

For fashion retailers this question is compounded by the fact that the majority of the orders have to be placed at once at the beginning of the selling season. This leads us to the second most frequently used type of information used to create demand prediction models.

Other products which pose challenges are products with sparse sales. This might have a wide range of causes, such as inventory ruptures, demand driven by very specific conditions or simply the fact that it is a niche product for a very select group of customers.

Product properties as input for demand prediction

To overcome the limitations of models based solely on historical demand product information is often introduced. This not only means including more data, but also means that the kind of model being built is changing.

Rather than building a model for each product, a model is now used to make prediction for groups of products, or even for all products in some cases.

Typical product properties which are included are the categories of the products as defined by the retailer, as well as other properties such as price, performance, colors,... and basically all other things which might be relevant in the perception of consumers.

An interesting side note in this regard is that the data consistency for the product properties can sometimes be heavily neglected by retailers. Having many empty fields, or big odd-come-short categories where very different products are bunched together. Situations such as these should be corrected sooner rather than later.

A last-ditch resort in situations such as these might be to look for more complete data online (which is feasible for products such as electronics), or to engineer features using image recognition algorithms; another core staple of machine learning. Naturally, neither of these two should be viewed as a valid alternative for good data.

By including this information predictions can be made even for products which do not have a full history. Another benefit will be the lessening of the impact of outliers in sales, which will be moderated by a larger number of observations.

Promotional impact as input for demand prediction

Whereas price can be viewed as a basic property of a product, promotions are completely different beast altogether. Promotions can take a whole plethora of different shapes and forms such as percentage discounts, product bundles, rebates, and many many more.

Moreover promotions are often combined with some form of communication which informs customers of the promotion. The type of this communication (email, banners, in-store advertising,...) should also be incorporated in the set of independent variables to get an accurate reading of the expected promotional impact.

Whereas most retailers have good information on past sales, and reasonable information on product properties, few have good structured information on past and future promotional activities.

The reason for this being that database design for such information can be very challenging. Nevertheless, getting this right should be a top priority for retailers since the lack of doing so is wasting some of the most valuable information which can be collected in order to make the demand function suitable for more advanced use cases.

The exact manner in which to design such a database is sufficient material to fill at least another complete white paper, and is not something which we are going to cover in depth in this series. However, if this is something you are currently struggling with we are always keen to help.

Price elasticity as input for demand prediction

The inclusion of price elasticity is not so much including more data than it is looking for the right patterns and setting up experiments to validate assumptions. More than the other aspects previously discussed this boils down to something which can be considered to be a classification problem: which of your products can be considered to be price elastic and which can be considered to be price inelastic?

Oftentimes historical data will be sufficient to make a rough-cut classification for a subset of the products. Other products will tend to remain in the gray area; where it is not possible to make a very explicit judgement as to whether or not these products are (in-)elastic.

To collect this data proactive experiments will have to be conducted; varying price levels in such a way as to collect the right pieces of information.

As mentioned earlier this information is of paramount importance for strategic pricing decisions. Being able to estimate how customers will respond to price changes is the key piece of information required in order to set the price which is going to maximize overall gross margin.

Competitor prices as input for demand prediction

In spite of valiant attempts; most retailers are unable to differentiate themselves to such an extent that they have no competition in the market. This inevitably means that customers will be influenced by prices at competitors.

Fortunately, acquiring this information has become easier than ever in online B2C markets. Automated scraping technology allows for systematically collecting information from websites of competitors.

Structuring and analyzing this information can already yield valuable insights about average price levels, price evolutions, discounting behavior as well as the product portfolio of competitors.

As such this is an exercise which yields immediate dividends even without the full-on integration into a predictive demand function.

Going beyond the demand function itself, this information can also be directly used in the strategic decisions for automated pricing. This will be explored in depth in part 3 of this series.

And much, much more!

The above of course is only a simplified and uniform view on the demand function. It is there to help you understand certain inputs that can be used to predict customer demand; without getting bogged down in the intricate inner workings of these models.

In reality many other factors might enter into the equation. For someone who is selling barbecues the weather will be a major factor in trying to predict short term demand patterns. For fashion the color schemes being used by great designers might be essential input to make the right predictions.

A single best demand prediction model does not exist, since too much depends on the specifics of a retailer. Hence, the need for experimentation and correctly designed experiments to validate what works and what doesn't work for a specific scenario.

Our hope would be that the preceding can serve as kindling, sparking some ideas on how demand prediction could be improved for your specific case.

Takeaways

This article made the case for (re-)allocating investment budgets for advanced analytics to domains where they can be expected to have a greater impact than the more peripheral domains where they are most frequently bestowed.

Our aim was to help you understand the importance of spending resources where they can enable or improve demand prediction, as we demonstrated how it functions as the motor to excel in the aforementioned three key areas; replenishment, pricing and promotions.

This is no easy feat since this means changing the core processes currently used within a retailing organization - rather than adding some new processes supported by artificial intelligence.

The cornerstone of innovation in these domains is the demand function. A key thing to ask yourself now would be: am I already collecting the right information to allow me to build such an advanced demand function?

Every retailer probably has access to the sales history (or else the tax man might get somewhat nervous), but few companies probably have an accurate reading on the price and product portfolio of competitors.

This series continuous with three more parts which give an in-depth view on replenishment, pricing and promotions from a data-driven perspective. If you want to know more about this don't hesitate to contact us!

Bio

I am someone who is passionately curious - a scientist and sceptic at heart - always discovering new things that pique my interest. Not solely a theorist but pragmatic when it comes to problem solving.

My main role is that of CTO & co-founder at Crunch Analytics, where it is my privilege to lead and mentor a team of passionate and talented people. Akin to my own broad ranging interests, this team consists of people with diverse backgrounds - including data scientists, business strategists, data engineers, statisticians, and more! This range enables Crunch to effectively tackle difficult business challenges and deliver scientifically sound and at times unconventional solutions.

Louis-Philippe Kerkhove - Co-founder & CTO Crunch Analytics

References

Phillips, Robert Lewis. Pricing and revenue optimization. Stanford University Press, 2005.

Agrawal, Ajay, Joshua Gans, and Avi Goldfarb. Prediction machines: the simple economics of artificial intelligence. Harvard Business Press, 2018.

Economist special report on Global Supply Chains, July 2019 (https://www.economist.com/spec...)

https://medium.com/swlh/how-zara-spent-0-in-advertising-to-disrupt-the-fashion-industry-59526b5000af

Rose, Justin, and Martin Reeves. "Rethinking your supply chain in an era of protectionism." Harvard Business Review, published on 22 (2017).

Lyall, A., Mercier, P., & Gstettner, S. (2018). The death of supply chain management. Harvard Business Review, 15, 2-4.

Wainwright, T. (2016). Narconomics: How to run a drug cartel. PublicAffairs.