You are here

Model Evaluation Framework

This page is part of the project: Great Lakes Climate Ensemble

The evaluation of any model data set under consideration for inclusion in the Ensemble will be subjected to several different quality checks.  We build our evaluations off the prototype framework as defined by the National Climate Predictions and Projections Platform:

Group 1: Evaluation of Standard Statistical Measures

Metric abbreviation Metric full name Description Metric group
mean Mean average for a period of time Distributional - Central tendency
median Median median for a period of time  
stdev Standard deviation interannual standard deviation for a period of time Distributional - Variability
max Maximum greatest value for a period of time  
min Minimum smallest value for a period of time  
p5 5th percentile 5% of the data falls below the value of the 5th percentile Distributional - Quantiles
p10 10th percentile 10% of the data falls below the value of the 10th percentile  
p25 25th percentile 25% of the data falls below the value of the 25th percentile  
p75 75th percentile 75% of the data falls below the value of the 75th percentile  
p90 90th percentile 90% of the data falls below the value of the 90th percentile  
p95 95th percentile 95% of the data falls below the value of the 95th percentile

Comparison Metrics:

Bias = difference between the modeled and the observed evaluation metric for a given variable or index.

Group 2: Evaluation of Adaptation Metrics

We are collecting lists of metrics used by adaptation planners.  Initially, we will focus on a small subset that serve a variety of planning topics (i.e., water resources, human health, agriculture, etc.).

Examples include: Hot/Cold Days, Freeze-Thaw Cycles, Growing Season Length, etc.

Group 3: Evaluation of Processes

Protocols for evaluating the following physical processes in the models will be developed and applied to the models under consideration: 

  • Seasonal cycle of lake temperatures and lake ice
  • Other measures of lake/atmosphere interactions (?) 
  • Organized (and extreme) precipitation/convection
  • Seasonal cycle of extreme rainfall
  • Summertime precipitation minimum over Lake Superior
  • Seasonal cycle of net basin supply
  • Lake-effect snowfall
  • Low level jet/moisture entering region
  • others?

and we add a fourth evaluation group to the framework,

Group 4: Evaluation of Model Components

Information about the following components will be collected on each model.  We will use existing research on the performance/representation of each component compared to its peers in providing an analysis of how the quality of the projections are impacted by the select component.

  • Are the Great Lakes represented within the model? (if so, how is the representation carried out?)
  • Does lake ice form within the model simulation? (if so, how is the lake ice simulated?)
  • Dynamical core
  • Shallow and deep convective model schemes