We hit 92.9% accuracy across nearly 15,000 commodities with a forecast that simply predicts next month’s price will be the same as last month’s. Here’s what headline accuracy figures really measure.
There’s no shortage of commodity price forecast providers who can make their accuracy measures look impressive: 95% at a one-month horizon, and still close to 90% six months out. The pitch is always the same – near-perfect visibility into where prices are headed, now and further out. What’s rarely explained is how those numbers are actually measured, or what “accurate” even means in context. Look closer and these figures are often built on metrics that say little about real forecasting skill, and even less about whether following the guidance would actually have saved you money.
Does 95% accuracy mean that 95 out of 100 buying decisions taken on those forecasts would have been the right call? It does not. You can actually work out a great deal about what these metrics measure from the numbers alone. Accuracy that is high in the near term and lower further out is the first sign you are being shown a measure you cannot trade on. So why are these inflated metrics used so often, and what do they actually measure? The answer to the first question is simple: metrics like this are used because they sound good and give an impression of perfection. We answer the second question in this article, and show how easy it is to produce these high-accuracy numbers without any forecasting skill at all. We will also show why you should look instead for transparency and a proven track record of helping buyers save money, not flawed headline metrics.
The most quoted measure proves the least
There are many ways to measure forecast accuracy. The one you are almost always shown asks how close the forecast got to the actual price. Say the forecast predicted $1,050 and the price came in at $1,000. It was $50 off, so it missed by 5%, and would therefore be quoted as being 95% accurate. We’ve previously written an article on the problems with reporting accuracy like this. Deviation-based metrics are fine to report in combination with other things, but they don’t say anything about skill.
What no skill at all looks like
To prove this, we tested what a number like that is worth. We built two very simple forecasting strategies that require no skill at all and tested how they would perform. One repeats last month’s price, forecasting no change at all. The other adds a very small price increase each month. Neither has any real predictive power. We ran both over the last two years on roughly 15,000 commodity prices, a broad cross-section of the more than 38,000 series we cover.
The repeat-the-price forecasts came out 92.9% accurate on average one month ahead. It is a real but worthless figure, because the measure behind it asks how close a forecast landed to the price, not which way it said prices would go or whether it could have helped you save costs. A measure like that gets worse the further ahead you look, for reasons unrelated to skill: prices drift further from where they started over six months than over one.
Deviation-based accuracy is not forecasting skill. It is how much the price moved.
Repeat this month’s price, call it a forecast for the next eighteen months, and a deviation-based metric gives you 92.9% accuracy one month out, 85.1% at six months and 80.9% at eighteen. No model, no data, nothing behind it.
Because the forecast only repeats the last price, its “error” is simply how much the price moved. Saying it is 92.9% accurate one month out and saying prices moved 7.1% in an average month are the same statement written two ways. This shows what the measure rewards. The figure is 100% minus the average error, so improving the accuracy means shrinking the error, and the surest way to do that is to forecast as little movement as possible. Nobody should pay for that.
Measured on direction, a forecast should improve with time
In the second test, we focus on directional metrics. This measures which way the forecast said prices would go, up or down, and checks whether they went that way. This kind of metric is quoted far less, mainly because the numbers look less impressive. It also tells you much more about genuine skill. On direction, anything consistently above 50% is a genuine edge, and a real one is worth serious money against a procurement budget. A tradeable directional accuracy of 90% does not exist anywhere. Call direction correctly 60% of the time at a one-month horizon and you have a good career ahead of you in trading. Any number far above that is not measuring direction.
In this test we added one assumption: prices rise a steady 0.2% each month, about 2.4% over a year, close to what general commodity inflation does. This means the forecast predicts rising prices at every horizon, so a directional measure simply checks how often prices actually rose. It is still a forecast that takes no risk: it stays close to the stale price and adds only a small inflationary adjustment. Because it barely moves away from where prices already are, the deviation-based metrics still score it highly – around 92.9% accurate at a one-month horizon – so on that measure it could be presented as a near-perfect forecast. Measured on direction, the picture is very different: accuracy is much lower, but it rises with the horizon, as the chart below shows.
That rise is not skill either. Inflation pushes prices up over long periods, so assuming prices go “up” is right more often the further ahead you look. Therefore, directional accuracy is constant or rising the further out you look – but unlike deviation-based accuracy, it does not fall as the horizon lengthens.
If the accuracy figure falls steadily the further ahead the forecast looks, it is almost certainly measuring price deviation, not direction.
Directional accuracy is a better measure, but it is not everything either. Where the peak lands, how long a trend holds, what happens when a market breaks: these separate a useful forecast from a curve trailing the last price, and no deviation metric shows any of them. No single published figure tells you whether a forecast works for your decision. Only the forecasts themselves can.
Why do people keep reporting 90%+ accuracy?
Some of it is deliberate, a way to make forecasting ability look better than it is. Some of it is simply a lack of understanding of how commodity markets work.
MAPE (mean absolute percentage error), used to calculate the deviation-based accuracy measures above, is a standard error metric in forecasting, and it is what you reach for if you are trained in data science. In a price series it quietly measures volatility instead of skill, and unless you know commodity markets well enough to expect that, it looks perfectly reasonable. Flattering, even. Then it goes in the deck. The deeper problem is optimising the wrong thing. Deep data science capability with shallow market knowledge produces models tuned to score well on a measure nobody can trade on.
Tomorrow we could publish forecasts for all 15,000 commodities used in this test and report an accuracy of 92.9%. Or we could publish only the 5,000 least volatile and report 97% accuracy, still just repeating stale prices. We won’t, because none of these would be good forecasts. Their directional accuracy would be nowhere near the 90s. Nobody’s is. And they would not help a single buyer decide whether to buy now or wait.
Instead of releasing thousands of forecasts optimised for misleading accuracy measures, we assess several performance metrics together and check what actually matters: whether a forecast produces insights that help save money. We only release a forecast once we are confident it has real predictive power, meaning it can help customers decide when to buy and save money as a result. We don’t spend our time chasing a headline accuracy figure. We offer transparency, access to our historical forecasts so you can check them yourself, and a proven track record of guiding decisions that reduce costs. To see how our forecasts have helped clients save money, read more here, or request a demo.
This commentary is prepared by Expana and its group of companies, neither of which is an investment firm. We have no positions in the commodities or derivatives referenced. The views expressed are for information only. See our disclaimer for more information: https://www.expanamarkets.com/disclaimer/
Written by Frederik Kildedal
