Use of AI in forecasting leaves a lot to be desired

In recent years, people have had high expectations of the use of artificial intelligence and external data in Sales & Operations Planning (S&OP). At S&OP Flavour Day, it turned out that those expectations are far from being met. Tineke Kok of coffee producer Jacobs Douwe Egberts (JDE): ‘We are still stuck in the trough of disillusionment.’

By Marcel te Lindert

Jonathon Karelse (pictured) recalls training the demand planners at a solar panel manufacturer. When the discussion turned to the possibilities of artificial intelligence (AI) and the use of external data, the demand planners became hugely enthusiastic. They wanted to get started with machine learning and weather forecasts sooner rather than later. ‘I really had to temper their enthusiasm,’ says the forecasting expert. ‘They were not yet using statistical models and had no idea how accurate their forecasts were. In such situations, using AI is like fixing the hand for the seconds on your watch when the hour hand is broken.’

Use models sparingly

Karelse denounces the huge focus many companies place on the accuracy of their forecast. Measuring and continuously improving it has become almost a dogma. This leads to all kinds of efforts to adjust the forecast, not all of which are effective. ‘The question is whether all these efforts ultimately add value. It’s easy to spend millions on solutions that may improve the forecast, but don’t actually do anything for the bottom line. The costs often exceed the benefits.’

British statistician George Box once said that all models are wrong, but some are useful. ‘What he actually meant is that you should use models sparingly. Even the most complex and expensive model produces an inaccurate forecast. Most companies don’t even come close to a perfect forecast. So don’t spend hours arguing about whether the forecast for a particular product should be 3% higher or lower if you are always 40% wide of the mark anyway.’

Trough of disillusionment

One company that has gained experience in using AI to improve forecasting is Jacobs Douwe Egberts (JDE). ‘Everyone knows Gartner’s hype cycle. After the peak of inflated expectations comes the trough of disillusionment. After that, the path of enlightenment leads to the plateau of productivity,’ says Tineke Kok (pictured), Programme Manager at the coffee and tea producer. ‘We also embarked on that hype cycle. Our expectations were extremely high, but we are actually still stuck in the trough of disillusionment.’

JDE started working with AI believing that the basics were right. ‘We had set up a solid process for forecasting, but the adoption of the standard way of working left a lot to be desired in practice. This was due to the lack of proper support for the people who have to make the forecast. So we went in search of a tool to automate that process. We found a start-up that could help us with that.’

Peaks and troughs

Forecasting at JDE is complex because most people only buy coffee when it is on special offer. That leads to big peaks and troughs in the demand pattern. When the start-up managed to predict those peaks and troughs pretty successfully, Kok and her colleagues got excited. The company decided to invest in the tool and implement it in seven markets. ‘Then we suddenly saw a significant difference emerging between the forecast and the realized sales. This turned out to be caused by new situations that had not arisen before. And no model can accurately predict things that haven’t ever happened in the past.’

Stopping the implementation was not an option. ‘Instead, we started feeding the model with more data. External data about retailers’ other promotions, for example. But none of that helped. The only external data that seemed to influence demand was data about trends in consumers’ purchasing power. But the model did not know what to do with that, because it had been developed primarily for predicting events rather than long-term trends.’

Training the model

Another reason why the AI model has not yet lived up to expectations is due to “noise”. When a retailer runs a promotion, JDE has to start delivering the extra stock to stores two to three weeks beforehand. ‘That’s what we actually want to know. But there is quite some time and therefore noise between the promotion and the operational peak – noise in the shape of changing stock levels and changing prices,’ says Kok. ‘In addition, we hugely underestimated how much time and energy it takes to train the model. You have to keep telling the model whether a peak is due to a promotion or a price change. In short, it’s very difficult to set up a model that can make a good forecast. Consumers remain unpredictable.’

Estée Lauder uses several key performance indicators (KPIs) to monitor the quality of the S&OP process. One is the forecast value-add, a measure of the added value of all actions to improve the statistical forecast. ‘When we do a certain promotion for the first time, it is difficult to forecast it. That’s why we send our statistical forecast to the marketing and sales departments so they can add information about promotions,’ says Amber Roxas (pictured), IBP Lead at the cosmetics brand.

Forecast value-add

Roxas stresses how important it is to handle these KPIs carefully. If people are judged on the forecast value-add, it could affect their input. ‘The forecast value-add is calculated as the difference between the accuracy of the enriched forecast and the accuracy of the statistical forecast. People who know that could negatively influence the statistical forecast to increase the forecast value-add. But that would not help to improve the business performance, of course.’

Want to know more? Download the Best of S&OP Flavour Day 2024 – November 20, 2024