Wednesday, February 19, 2014

How the National Weather Service Grades Itself For This Winter's Forecast (and a word on forecasting in general)

Via BusinessWeek:
“Not one of our better forecasts,” admits Mike Halpert, the Climate Prediction Center’s acting director. The center grades itself on what it calls the Heidke skill score, which ranges from 100 (perfection) to -50 (monkeys throwing darts would have done better). October’s forecast for the three-month period of November through January came in at -22. 

The NWS' Climate Prediction Center page on scoring one's skills could apply to most market prognosticators as well:
You don't start to count skill (SS) as positive unless and until more than 1/3rd of the forecasts are categorically correct. 1/3rd is the 'zero' skill level. Pointing at a case (here and there) where your method was right is dangerous, particularly when done after the fact and when SS-overall <=0.