In our previous post, we examined the naive forecast, and in particular its performance for forecasting market outruns (such as market prices). We also had a look at a simple but “typical” machine learning model (in this case a feed forward neural network) compared to the naive forecast - and that often such models struggle to perform better than the naive. If you haven’t read it, check it out here.
The obvious follow-up question for anyone interested in generating forecasts that add value is therefore: how do we beat the naive forecast? A very similar, but perhaps even more useful question, is: how to we add value over the naive forecast? Broadly speaking we have four possibilities:
1. Use forward-looking information in the model.
Unsurprisingly, adding future information related to the period you are forecasting can improve forecast accuracy. This can include forecasts of underlying drivers (e.g. weather forecasts), or future values (such as forward curve data) that have already been calculated. Be aware though of the danger here - if such forward information is itself based on (or is no better than) naive forecasts, then your modified forecast will still essentially be performing naive forecasting. As always - be careful of apparent complexity that delivers little value.
As an example of relevant added-value via forward looking data, we test adding a wind forecast feature to our simple NN model from our previous post. This improves both model performance overall, and - in some periods at least - results in forecasts that predict timing in price movements, rather than lagging these changes.
Looking at performance metrics for this new “augmented” model confirms this.
2. Better model architecture
More advanced model architectures than the simple feed forward networks discussed so far may be able to extract additional predictive power out of the data, both historical and forward-looking. However, a common challenge for using these more complex models in power market forecasting is the lack of data.
Taking our example of daily price forecasting, we have only 365 data points per year. Ten years of data gives only 3652 or 3653 data points. Liquid traded power markets have only existed in many countries for a maximum of 20 years or so - giving, at best, around 7000 data points. And that is ignoring the fact that these markets have changed dramatically over this time, driven by such things as changes in market rules, new drivers such as CO2 prices and increases in renewable generation. Patterns in the data that once held true may no longer do so, and features that are important today may not have existed even a few years ago.
Thus in practice we often have very limited data sets with which to build and test models. These are often much too short for advanced models to be trained and used successfully. You can train a complex model architecture on limited data, but you run a real risk of overfitting (among other things). You do not want your model to simply learn the training data set exactly. Such a model may be able to replicate the training data ok, but that’s all - any market situation that it has not seen exactly in the past can result in very poor predictive accuracy.
3. Better feature development from historical data
An alternative to improved architecture is to use domain insight to propose and develop more complex derived features from the data, within a more simple model architecture. However this is often easier said than done, and often requires deep domain knowledge to identify such patterns. This is one reason automatic model fitting methodologies or models built by non-domain experts can perform poorly in practice.
To illustrate this approach, we constructed several data features that are based on the tendency for power prices to “mean revert”. These features were then added to the original simple feed forward neural network model presented in our previous blog post. The idea here is that this enables the model to “learn” the mean-reversion tendency in prices and apply this when constructing forecasts. This simple change provides a small but measurable improvement in model accuracy over the simple network from earlier. In our experience such “expertise” features can often compensate for the lack of training data in many power applications.
4. Go beyond point forecasts
Adding value by forecasting future distributions of outcomes, or providing estimates of forecast confidence can add substantial value, even if the underlying forecast (or “expected value forecast”) performs similarly to the naive forecast. The change in “distribution spread” (as, say prediction intervals or confidence intervals) over time for example provides a clear indication of forecast accuracy as it changes, and aids forecast interpretation. We plan to address this issue in more detail in a later post - “Beyond the point forecast: how to drive value via predicting distributions. ”
Benchmarking
Finally, no matter what model you use you should always benchmark your forecasts against the naive forecast. Ideally you should benchmark also against “modified naive” forecasts such as moving average, exponential smoothing, and simple pattern (e.g. seasonality) models. This may be no more complex that defining a set of standard models and metrics, fitting these to each quantity you will forecast, and comparing to your more advanced model(s).
It is important to define these metrics so they reflect your use case. For example, consider two price forecast models. The first has a lower mean absolute error, but occasionally its forecast error is substantial. The second has a higher mean absolute error, but rarely if ever has very large forecast errors. The second model may be more preferable, as large errors can lead to very poor decisions and substantial losses, whereas small errors may not.
Summary
Performing better than a naive or simple model can often be surprisingly difficult in power market forecasting. This is not a bad thing - it is evidence among other things that the markets are well functioning overall, and that there are few or no obvious patterns that are not already known and exploited. Forecasting with machine learning in particular is also challenged by the limited amount of data available. However it is possible, via the use of good model design and careful feature (data) development and selection, to develop models that perform better and are more robust than naive or simple “modified naive” models. All it takes (!) is the combination knowledgable modellers and analysts, readily available data, a well defined workflow and a modelling system that makes it easy to develop, test and deploy… a subject that we will come back to later.
Comments