Evaluating Probability Assignments -- With Examples from Weather Forecasting

What makes decision making troublesome is that the consequences often will depend on yet unknown factors. When asking experts or forecasters about their opinions concerning the future course of events, it seems reasonable to reward them by a scheme related to the extent to which their assessments become reality. At first sight, it might appear impossible to avoid expert's assessment which at least to a certain degree depend on the particular reward scheme. The concept of probabilistic scores though provides reward systems with an incentive towards giving forecasts in the form of probabilities. These probabilities turn out to be independent of the particular scoring system.

Scores can of course be applied as well in situations where the forecaster is not a human but a machine. In this talk, I will discuss the application of scores to (numerically generated) probability forecasts. Scores can be used either to assess the performance of existing forecasting systems, or to train new machines that issue probabilistic forecasts in a statistical learning context. I will present several examples involving weather forecasting. Finally I would like to discuss a few limitations and problems I encountered.