AI weather models beat WRF on light rain but lose to it at 20 mm per 6 h

Why it matters

Flash-flood warning depends on heavy rain, not light rain. The ranking at a low threshold does not carry over to intense rain. The authors suggest a mixed approach, to be tested prospectively: AI for temperature and rain occurrence, and corrected WRF where the heaviest rates matter.

What they did

They ran a 12 km WRF model, with and without 3D-VAR assimilation of GNSS and conventional observations every 6 h. They ran GraphCast, AIFS and FuXi from the same ERA5 reanalysis. All were verified over flood windows in 2016, 2018 and 2021 in Luxembourg and the Greater Region. Stations were screened to be independent of the assimilated data, giving 92 119 matched rain pairs. For July 2021 they also compared the fields with a radar composite and ran finer WRF nests at 4 and 1.33 km.

Key findings

• At 1 mm per 6 h, GraphCast has the best CSI (0.424), ahead of AIFS (0.415) and assimilated WRF (0.389). AIFS has the highest detection rate (POD 0.835).
• At 20 mm per 6 h the order reverses. WRF reaches CSI 0.160, AIFS 0.141 and GraphCast 0.114. FuXi forecasts no such exceedance at all.
• AIFS and GraphCast cut temperature RMSE by 19.0 % and 17.6 % relative to assimilated WRF. These are the best-resolved differences in the study.
• Assimilation raises WRF’s CSI at 1 mm from 0.364 to 0.389, but the gain is not resolved with 24 h blocks. At 10 mm the assimilated run scores lower (0.189 against 0.212), and the authors give no cause.
• Refining WRF to 1.33 km raises peak rainfall but lowers the neighbourhood score at 10 mm (0.307 to 0.274).

Limitations

• Only three events, and the radar and resolution tests cover a single week of July 2021. The results are 6 h hindcasts re-initialised from reanalysis, not real-time warning tests.
• All three events fall inside AIFS’s fine-tuning period, so its scores may be optimistic. The AI models are also coarser than WRF, so resolution and model type are mixed together.
• The causes of the assimilation effects and of the event-to-event differences are not identified.

Glossary

• 3D-VAR: A method that blends a short forecast with observations to build a better starting state for the next forecast.
• CSI (Critical Success Index): A score for rain-occurrence forecasts: hits divided by hits plus misses plus false alarms. Higher is better.
• GNSS ZTD: Zenith Total Delay, the delay of satellite navigation signals through the atmosphere. It carries information on moisture.
• Fractions Skill Score: A score that compares rain coverage within neighbourhoods of growing size. It credits rain that is nearly in the right place.