Lesson 2: Measuring Forecast Accuracy

Motivation...

Forecasts vary widely in skill: between different systems, different seasons, from day to day, etc. Thus, we need to measure how well competing forecast systems typically do. And, maybe more importantly, how far off they could potentially be. Furthermore, we want to know how well a particular forecast system is meeting our needs. This is key, as needs differ.

For example, one rainfall forecast user, say an emergency manager, may be concerned about a rare extreme event which could lead to flooding, while another rainfall forecast user, say a gardener, might be concerned about the more normal rainfall amounts. They are both concerned about the forecast in rain, but their needs are very different. So, we need to be able to tailor our skill measurements to take into account different user scenarios. Continuing our example, the emergency manager may be concerned only with rainfall amounts over 2 inches in 24 hours and so would need to measure the accuracy of yes/no or a probabilistic forecast that rainfall would exceed this threshold. In contrast, the gardener would want to measure how many inches the forecast was typically off by and whether it was biased wet or dry. Again, same forecast but different need and, thus, different skill assessment. In this lesson, I encourage you to develop the skill of selecting which metric is the most appropriate given the situation. This is not an easy task as there are many fluid parts, but if you carefully consider the goal of the forecast and the intended use, you will succeed.