A three-tier benchmark scores physical, data-driven and hybrid climate models on equal terms

Why it matters

Climate projections cannot be checked directly on useful timescales, and new data-driven models make comparison harder. This protocol offers a common, transparent way to compare model types. The authors argue the held-out post-2015 record is a rare out-of-sample test for the current generation of models. The link between these scores and mid-century accuracy is still a working hypothesis.

What they did

The authors define a protocol with three tiers. Tier I is a set of physical sanity checks, such as energy balance, El Niño variability and basic responses to CO2. Tier II scores models against post-2015 observations using fair CRPS, a probabilistic score, relative to the median CMIP6 model. Tier III tests paleoclimate states and perfect-model experiments where data-driven models must predict the future of existing Earth system models. They show example scorecards for CMIP6 models and an idealized demonstration using a simple climate model and one Earth system model.

Key findings

• In the idealized demonstration, adding the 2015–2025 window roughly halves the uncertainty in 2050 warming, and the information sits mostly in the realized warming level rather than the trend within the window.
• In the example scorecard, global-mean 2 m temperature separates models strongly (S from −2.6 to +0.7), with CanESM5, UKESM1-0-LL and KACE-1-0-G warming too fast over the test window.
• Global-mean precipitation and net top-of-atmosphere flux separate models only weakly, and every model under-predicts the observed rise in Earth’s energy imbalance by about half.
• Only 7 of the 60 CMIP6 models evaluated provide every input needed for the required Tier I tests, so results are reported per test.
• The two example scorecards rank models differently: NorESM2-MM and CESM2 lead the spatial-pattern score, while NorESM2-LM leads the trajectory score.

Limitations

• The post-2015 test window is short and dominated by one decade of internal variability, so the diagnostics are necessary but not sufficient proxies for skill at the 2050 target.
• The idealized demonstration uses one Earth system model as truth and simplified uncertainties, and likely overstates the constraint somewhat. Constraining regional changes will be harder.
• The scope is narrow, covering 2050 temperature and precipitation only. Paleoclimate proxy uncertainties remain large, and the example CMIP6 runs omit some forcing changes inside the test window.

Glossary

• Fair CRPS: A probabilistic score of how well an ensemble’s spread matches an observation, adjusted so ensembles of different sizes compare fairly.
• Perfect-model experiment: Training a model on one Earth system model’s historical output and testing it on that model’s future.
• Ensemble member: One simulation among several run from slightly different starting conditions, used to sample internal variability.
• Equilibrium climate sensitivity (ECS): The long-term global warming expected after CO2 concentration doubles and the climate settles.