The quality of a forecast depends largely on the reliability of the data on which it is based. But we still need to know how to measure that reliability.
Assess the accuracy of demand forecasts is actually less straightforward than it seems.
Should the variance be measured in units, as a percentage, or relative to a baseline forecast? Over what time frame should the comparison be made? And how should the results be interpreted when some products sell every day while others regularly go through periods with no sales at all?
A small mean error can also mask a persistent bias. Conversely, a statistically less accurate forecast can sometimes lead to better decisions if it allows for better anticipation of uncertainty and the associated risks.
Measuring forecast accuracy therefore involves not only choosing the right indicators, but also analyzing them at the appropriate level of detail, over the appropriate time horizon, and in accordance with the demand profile of each product.
Above all, this measure serves as the starting point for a process of continuous improvement in Demand Planning. It helps determine whether discrepancies stem from the data, the model, an unaccounted-for event, or a human adjustment, and then take action on the appropriate factor.
The accuracy of forecasts thus becomes a key indicator for ensuring the reliability of demand planning and to better guide planning and procurement decisions.
Let’s take a look at how to properly measure the accuracy of forecasts and improve their performance over the long term.
Demand Planning: How Can We Measure and Improve Forecast Accuracy?
Measuring forecast accuracy helps identify discrepancies between anticipated demand and actual demand. However, this analysis must take into account the main Factors Affecting the Quality of Forecasts, such as data reliability, seasonality, promotions, stockouts, price fluctuations, and exceptional events. In demand planning, understanding these factors allows you to better interpret forecasting errors and take action on the most relevant levers to improve forecast accuracy.
So, how can we measure and improve the accuracy of forecasts?
1. Define what you are measuring before assessing accuracy
Before measuring the accuracy of a forecast, you must first clearly define what you are trying to forecast: orders, sales, warehouse shipments, or actual demand.
This distinction is essential because observed sales figures do not always reflect demand. In the event of a stockout, for example, low sales may simply reflect a lack of availability.
The same bias may occur in the case of one-time orders, delivery delays, or unusual fluctuations in inventory.
As part of the demand planning process, it is therefore necessary to specify the forecast data, the level of detail, the scope, the frequency, and the forecast horizon.
A precision measurement is only meaningful if it measures the right variable, at the right level, and at the right time.
2. Improve the reliability of the data used to measure error
Data quality naturally affects the quality of forecasts. But it also affects how their performance is measured.
A consistent forecast may appear inaccurate if the actual data it is compared to contains anomalies. Conversely, certain errors in the data can artificially improve an indicator.
Before evaluating performance, it is therefore necessary to identify events that could alter the observed signal: stockouts and partial availability, promotions, returns, cancellations, data entry errors, changes in product codes, temporary closures, or transfers between stores and channels.
However, the goal is not to systematically remove outliers.
A spike in sales may result from a data entry error, but it can also stem from a particularly successful promotion or a local event. In the first case, the data must be corrected. In the second, it represents actual information that should be retained and explained.
Corrections must also remain traceable. Retaining the raw data, the corrected data, and the reason for the change makes it possible to audit the process and prevent excessive data cleaning from artificially inflating performance metrics.
To accurately measure the accuracy of forecasts, the key principle remains simple: the forecast and the actual results must be comparable and represent the same business reality.
3. Take into account the predictability of products
Not all products are equally predictable.
A product sold in large quantities every day provides much more information than a replacement part ordered only a few times a year. Similarly, a mature, stable product does not have the same characteristics as a new product, a highly seasonal product, or an item with highly volatile sales.
The difficulty of forecasting depends, in particular, on:
- sales volume and frequency;
- variability in demand;
- whether it is regular or intermittent;
- whether there is a trend or seasonality;
- of the product life cycle;
- its sensitivity to prices and promotions;
- the frequency of exceptional events;
- possible substitutes from other product lines.
Directly comparing the accuracy of products with very different profiles can therefore lead to misleading conclusions.
A volume-variability segmentation—similar to an ABC-XYZ approach—allows, for example, the distinction between SKUs with a high economic impact and products for which demand is particularly erratic. Other dimensions can complement this segmentation, such as margin, criticality, life cycle, supplier lead time, or substitutability.
This classification then makes it possible to adapt the models, indicators, and level of human intervention.
Stable, high-volume SKUs can be largely automated. Intermittent demand requires methods and metrics tailored to periods with no sales. High-stakes, highly volatile SKUs, on the other hand, may warrant greater oversight.
The goal, therefore, is not to achieve the same level of accuracy across the board, but to evaluate each forecast in light of the product’s actual predictability.
4. Measure accuracy at the appropriate level of detail
The more aggregated a forecast is, the more stable demand tends to be.
Fluctuations across multiple SKUs, stores, or channels can offset each other when aggregated. It is therefore often easier to forecast national sales for a category than to forecast sales for each SKU at each retail location.
But the statistically most accurate forecast is not necessarily the most useful one.
A national forecast may be excellent but still insufficient for determining how many units should be sent to each warehouse. Conversely, measuring performance solely at the SKU-store level can paint an overly negative picture if many SKUs have low volumes or intermittent demand.
The assessment level must therefore balance three constraints:
- have a sufficiently extensive history;
- correspond to the level at which decisions are actually made;
- maintain consistency across the various forecasting levels.
Performance can thus be tracked simultaneously by reference, category, store, region, or channel.
This hierarchical analysis makes it possible, in particular, to identify a common situation: an overall accurate forecast can mask significant allocation errors. The company then forecasts the correct total volume, but not necessarily in the right place.
Hierarchical reconciliation methods can then help maintain consistency across the various levels of demand planning.
5. Measure accuracy at each decision point
A forecast does not have a single level of accuracy.
The degree of uncertainty varies depending on the time horizon. Forecasting demand for next week and for the next three months are two different exercises.
In the short term, more information is already available: recent orders, availability, confirmed sales transactions, or recent sales indicators. As the time horizon extends, more factors become uncertain.
We must therefore view accuracy as a performance metric that varies depending on the time horizon, rather than a single figure.
Above all, the measurement must correspond to the time when the company was actually supposed to make its decision.
Let’s imagine that a supplier requires an order to be placed three months in advance. Relying solely on the forecast revised fifteen days before the sale does not allow us to determine whether the information available at the time the commitment to the supplier was made was sufficiently reliable.
A more robust approach is to keep different versions of the forecast and then measure, for example:
- the forecast available 7 days in advance;
- the forecast available 30 days in advance;
- the forecast available 90 days in advance.
Each one can then be compared to the actual result.
This approach avoids having to measure, after the fact, a forecast that has been updated with information that was not available when the operational decision had to be made.
So the right question isn’t just “What did we do wrong?”, but also “What was our mistake at the moment we had to make a decision?”
6. What metrics should be used to measure the accuracy of forecasts?
There is no single universal indicator that can, on its own, summarize the quality of a forecast.
Each metric highlights a different aspect of the error. The choice should depend on the demand profile, the level of analysis, and the operational use.
MAE: Measuring the Mean Absolute Error in Units
The MAE, or Mean Absolute Error, measures the average of the absolute differences between the forecast and the actual value.
It has one major advantage: the result is expressed in the unit of the product.
An MAE of 50 therefore means that the forecast deviates by an average of 50 units from the actual result.
This metric is easy to interpret, but it makes it difficult to compare products with very different sales volumes. An error of 50 units may be negligible for a product that sells several thousand units, but significant for a product that typically sells 70 units.
RMSE: giving greater weight to large errors
RMSE, or Root Mean Squared Error, places greater weight on large deviations.
It can be particularly useful when major errors result in disproportionately high operational costs.
On the other hand, this sensitivity also means that a few extreme events can significantly influence the result.
MAE and RMSE therefore do not answer exactly the same question. The former measures an easily interpretable average error; the latter focuses more on errors with large magnitudes.
MAPE: Express the error as a percentage
MAPE, or Mean Absolute Percentage Error, expresses the error as a percentage of the actual volume observed.
The fact that it is expressed as a percentage makes it intuitive and largely explains its popularity.
However, it has a significant limitation: when actual sales become very low, the margin of error can become extremely high. When actual sales are zero, the calculation becomes problematic.
MAPE is therefore ill-suited for applications characterized by low or intermittent demand.
MASE: Comparing Performance Across Different Series
MASE, or Mean Absolute Scaled Error, compares the model’s error to that of a reference forecast.
This standardization makes it easier to compare data sets of different sizes and, above all, helps answer a key question:
Does the model really perform better than a simple method?
This is an important consideration when evaluating a demand planning process. A sophisticated method may appear to perform well in absolute terms without actually offering any real improvement over a naive forecast.
Bias: Detecting Persistent Underestimates and Overestimates
The average error isn’t enough.
A forecast may appear to be accurate but still systematically overestimate or underestimate demand.
This is what makes it possible to identify forecast bias.
This information is essential because the operational consequences vary. Persistent underestimation increases the risk of stockouts. Repeated overestimation can lead to excess inventory, tied-up capital, or markdowns.
Since false positives and false negatives may offset each other in certain aggregate measures, bias must be tracked separately.
Also measure the uncertainty
A one-time forecast of 1,000 units gives the impression that only one possible future is being considered.
However, two products with the same average forecast may have very different levels of uncertainty.
Linking the forecast to a prediction interval makes it possible to estimate a plausible range of demand. This information is particularly useful for determining safety stock levels, developing multiple scenarios, and balancing the risk of stockouts against the risk of excess inventory.
A robust system, therefore, does not rely on a single accuracy percentage. It generally combines several factors: error, bias, performance relative to a benchmark, changes over time, and level of uncertainty.
7. Compare the models to a baseline forecast
A complex model is not necessarily a better model.
A moving average, a simple seasonal method, or a relatively simple statistical model can produce good results when demand has a suitable structure.
Before seeking greater sophistication, it is therefore helpful to establish a baseline—that is, a reference forecast simple enough to serve as a point of comparison.
For example, it might involve using:
- the last observed value;
- the average for the most recent periods;
- sales for the same period last year;
- a simple seasonal forecast.
The candidate model must then demonstrate that it represents an improvement over this baseline.
However, this comparison should not be based solely on the data used to build the model.
A highly flexible model can reproduce historical data remarkably well while performing poorly on new data. This is known as the phenomenon of overfitting.
To more accurately simulate real-world demand planning conditions, validation can be performed at various points in the history.
The model first learns from the data available up to a given date, generates a forecast for the desired time horizons, and then repeats the process as it moves forward in time.
This backtesting allows us to compare the methods across different time periods, different demand patterns, and, most importantly, the time horizons actually used by the teams.
Machine learning can add value when a company has a large number of data series, a sufficiently rich historical record, and relevant explanatory variables. However, its performance must be demonstrated under the same conditions as that of simpler models.
The goal is not to select the most sophisticated technology, but the model that actually improves predictions on data it has never seen before.
8. Identify the source of forecasting errors
Measuring an error is only the first step. To improve forecasts, you must then understand the source of the discrepancies.
A decline in performance can stem from several sources:
- incorrect or incomplete data;
- a disruption that masked part of the demand;
- a change in behavior;
- a promotion or a price change that was incorrectly listed;
- an exceptional event not included in the model;
- a break in the trend or seasonality;
- a model that is no longer as suitable;
- an inappropriate human modification.
Not all of these errors call for the same response.
Changing the model will not solve a problem caused by incorrect inventory data. Adding more historical data will not necessarily correct a recent structural break. And increasing the number of external variables will not automatically improve results if those variables are not sufficiently reliable or available at the time the forecast needs to be produced.
Error analysis must therefore precede the selection of a solution.
It is particularly useful to break down the variances by product, category, location, channel, and time frame in order to identify areas where performance is actually declining.
This line of reasoning allows us to move from a general observation—“our forecasts lack precision”—to a actionable diagnosis: where do we lose accuracy, since when, over what time frame, and for what type of demand?
9. Measuring the Added Value of Human Adjustments
A model does not necessarily contain all the information known to the teams.
A sales representative may know that a new customer is about to sign up. The marketing department may have approved a campaign that doesn’t yet exist in the data. The purchasing department may be aware of an upcoming change to the product lineup.
Human intervention can therefore improve forecasting.
But it can also damage it.
Increasing a forecast because it “seems too low,” making repeated undocumented adjustments, or confusing a forecast with a sales target can gradually introduce bias into the process.
The question, then, is not whether to choose between automation and human expertise, but rather to measuring the actual value added by each intervention.
This is, in particular, the objective of Forecast Value Added, or FVA.
The company can successively compare the performance of a naive forecast, the statistical model, the forecast after adjustment by the demand planner, and, if applicable, the sales consensus.
At each step, it assesses whether the intervention actually improves or worsens the forecast.
If a step consistently undermines results, its effectiveness must be reevaluated, even if it has historically been part of the process.
It is also a good idea to document adjustments: author, date, reason, initial value, new value, and validity period.
This traceability transforms human intervention into a measurable process and allows teams to focus their expertise on situations where it truly adds value: launches, exceptional events, high-priority cases, or structural changes.
10. Link the accuracy of forecasts to operational results
A statistical improvement is only meaningful if it contributes to better decisions.
Anticipating demand for 1,000 units over the next four weeks does not necessarily mean that you have to order 1,000 units right away.
The decision also depends on available inventory, orders already placed, expected sales before the next delivery, supplier lead times, minimum order quantities, logistics capacity, and the desired service level.
We must therefore distinguish between three dimensions.
The quality of the demand signal : Does the forecast accurately represent the expected demand and its uncertainty?
Inventory Policy : Are the reorder parameters, safety stock levels, ordering frequencies, and service levels consistent with this uncertainty?
Logistics Execution : Are orders and deliveries actually carried out as planned?
This distinction is fundamental.
A shortage can be caused by an underestimation, but also by an order placed too late or a delayed delivery from a supplier.
Similarly, excess inventory does not necessarily prove that the forecast was too high: it may result from a minimum order quantity, a change in the product mix, or an outdated inventory rule.
The accuracy of forecasts must therefore be viewed in the context of operational KPIs: service levels, out-of-stock incidents, availability, inventory levels, markdowns, and related costs.
The goal of demand planning is not to achieve the best possible statistical score. It is to produce information that is reliable enough to improve operational decisions.
11. Develop a process for continuously improving accuracy
Improving forecast accuracy in a sustainable way does not necessarily mean immediately replacing the model or adding more complexity.
A structured approach begins by pinpointing exactly where performance can be improved.
Define the target precisely
First, we need to determine what is planned, at what level of detail, how often, and to inform which decision.
Establish a baseline
Existing models must be compared to a simple, reproducible method in order to assess their true added value.
Segment demand profiles
Stable, seasonal, intermittent, new, or highly volatile benchmarks do not necessarily have to be evaluated and treated in the same way.
Measure Based on the Horizons Actually Used
Forecasts should be evaluated based on the version available at the time decisions are made, not solely on the most recent version prior to the actual outcome.
Combining Multiple Indicators
The error must be supplemented by an analysis of bias, performance relative to a baseline, and, where relevant, the uncertainty associated with the forecast.
Test the models on data sets not used for training
Backtesting that accurately reflects real-world conditions helps prevent the selection of models that perform well only on the historical data used to build them.
Analyze the causes of discrepancies
Errors should be attributed as much as possible to their source: data, model, unforeseen event, structural change, or human intervention.
Measuring Forecast Value Added
Every step in the process—whether automated or performed by humans—must be able to demonstrate that it effectively contributes to improving the quality of the forecast.
Linking Precision to Operational Results
Improved forecasting must be viewed in the context of operational objectives: availability, service level, inventory control, and the ability to anticipate demand.
Monitoring Performance Over Time
A model that performs well today will not necessarily continue to do so.
Product assortments evolve, consumer behavior changes, and insights derived from historical data may gradually become irrelevant.
Monitoring bias, data drift, declining performance, or an increase in the number of adjustments makes it possible to detect these changes and reevaluate the models before the discrepancies become structural.
Making Accuracy a Valuable Tool for Demand Planning Management
Measuring the accuracy of forecasts is not about finding a single percentage that is supposed to summarize the entire performance of demand planning.
A meaningful assessment must take into account what is actually planned, the demand profile, the level of detail, and, above all, the time horizon within which the decision must be made.
It must also combine multiple readings. Error measures the magnitude of deviations. Bias reveals persistent tendencies to overpredict or underpredict. A comparison with a baseline indicates whether the model actually adds value. Prediction intervals, meanwhile, help highlight the uncertainty that a point forecast does not reveal.
But a measure is only useful if it leads to action.
By pinpointing exactly where, when, and why forecasts deteriorate, companies can take action on the right areas: data quality, segmentation, model selection, explanatory variables, human adjustments, or operational processes.
The challenge, therefore, is not to eliminate all uncertainty. Some demand will always be difficult to predict.
Rather, the goal is to reduce avoidable errors, quantify the remaining uncertainty, and incorporate it into inventory and procurement decisions.
Ultimately, a good forecast isn’t one that claims to be right all the time. It’s one that the company knows is reliable enough to be able to Where to secure inventory, where to limit commitment, when to respond, and which SKUs to focus the teams’ attention on.
It is this ability to measure, understand, and leverage the quality of forecasts that transforms demand planning into a true decision-making tool.
