Pitfalls of Using MAPE as a Forecast Accuracy Metric

The mean absolute percentage error (MAPE) is one of the most popular forecast accuracy metrics. It is intuitive, but it has limitations that anyone evaluating forecasts for heat demand, load, solar or wind power should know. This article looks at how MAPE behaves in typical energy situations and which metrics are better suited where it fails.

How MAPE is calculated

MAPE = (100 % / N) · Σ |Aₜ − Fₜ| / |Aₜ|

Aₜ is the actual (measured) value at time t, Fₜ the forecast value and N the number of time steps. In words: the average of the absolute errors, each expressed as a percentage of the actual value.

Strengths

• Easy to interpret: a MAPE of 10 % means the forecast was off by 10 % on average.
• Scale-independent: it allows comparisons between series of different size, for example two utilities' load portfolios.

Limitations

1. Values close to zero. Because every error is divided by the actual value, small actual values inflate the metric. Example: a 15 MWp PV plant at dawn produces 0.2 MW, the forecast says 1.0 MW. The error of 0.8 MW is small relative to the plant, but its percentage error is 400 %. At night, with actual values of exactly zero, MAPE is undefined, and many tools then silently drop these time steps or return infinity. Hyndman and Koehler (2006) point out exactly this: MAPE is infinite or undefined when actual values are zero and extremely skewed when they are close to zero.
2. Asymmetry. For non-negative values, an under-forecast can produce at most 100 % error, while an over-forecast has no upper limit. A model optimised for MAPE therefore learns to forecast systematically too low.
3. No direction. MAPE doesn't show whether the forecast is too high or too low. For balancing group management that is exactly the decisive question, so MAPE should always be reported together with the bias.
4. Intermittent series. Series with many zero or near-zero periods, such as solar output or demand of single customers, cannot be meaningfully evaluated with MAPE.

What is a good MAPE?

An often-quoted scale goes back to Lewis (1982): below 10 % is highly accurate, 10–20 % good, 20–50 % reasonable and above 50 % inaccurate. It was developed for business and sales forecasting, though. For solar or wind power, with many values near zero, these thresholds are not meaningful. A comparison against a simple reference forecast says more than any fixed threshold.

Better alternatives

• MAE and RMSE: the mean absolute error and the root mean square error, both in the unit of the forecast (for example MW). The RMSE weights large errors more heavily, which often matches their cost.
• nMAE and nRMSE: MAE or RMSE normalised by the installed capacity. This is the standard for solar and wind because it makes plants of different sizes comparable and is well-defined at night. We report our solar and wind benchmarks this way.
• WAPE: the sum of absolute errors divided by the sum of actual values. Like MAPE it is expressed in percent, but single small values don't make it explode. Useful for load and heat.
• MdAPE: the median instead of the mean of the percentage errors. More robust against outliers, but still undefined for zero values.
• MASE: proposed by Hyndman and Koehler (2006). It scales the MAE by the in-sample MAE of a naive forecast; values below 1 mean the forecast beats the naive one. It handles zero values as long as the series isn't constant.
• Skill score against a reference: how much better is the forecast than persistence or climatology? In our wind benchmark, for example, the nRMSE of 9.3 % compares with 18.9 % for a persistence forecast.

What this means in practice

• Choose the metric by use case: capacity-normalised metrics for solar and wind, MAE or WAPE in MW for load and heat.
• Always report the bias as well, and compare against a reference forecast over the same period.
• Optimising blindly for MAPE leads to systematically low forecasts, and these turn directly into balancing energy.

How to evaluate forecasts fairly overall is described in our guide to evaluating forecasts for the energy sector.

References

1. Armstrong, J. S., Collopy, F. (1992): Error measures for generalizing about forecasting methods: Empirical comparisons. International Journal of Forecasting 8(1), 69–80. doi:10.1016/0169-2070(92)90008-W
2. Goodwin, P., Lawton, R. (1999): On the asymmetry of the symmetric MAPE. International Journal of Forecasting 15(4), 405–408. doi:10.1016/S0169-2070(99)00007-2
3. Hyndman, R. J., Koehler, A. B. (2006): Another look at measures of forecast accuracy. International Journal of Forecasting 22(4), 679–688. doi:10.1016/j.ijforecast.2006.03.001
4. Lewis, C. D. (1982): Industrial and Business Forecasting Methods. London: Butterworths.