Probabilistic LSTMs capture extreme Swiss floods that a deterministic LSTM underestimates

Why it matters

Deterministic deep learning models tend to underestimate the biggest floods, which are the ones that matter most for warnings. Probabilistic output gives forecasters the upper tail of plausible outcomes at little extra cost. The study also shows a first way to learn the step from a forecast distribution to a warning level, though it is a proof of concept.

What they did

The authors trained a standard deterministic LSTM and three probabilistic LSTMs (CMAL, DRN and BQN) on 196 Swiss catchments from the CAMELS-CH dataset. They tested them on 1 October 2000 to 30 September 2020. They checked calibration and sharpness, and looked at three example floods and the top 0.05 % of discharges. They then trained a reinforcement learning agent, FRiCA, to turn CMAL predictions into warnings of six severity classes. They compared it with two fixed rules.

Key findings

• The deterministic LSTM underestimates 97 % of the top 0.05 % discharge values, while 83 % of them fall inside the CMAL 99th percentile range. DRN and BQN capture about 60 % and 57 %.
• CMAL had the best CRPS (mean 0.50, median 0.39), ahead of DRN (0.53, 0.41) and BQN (0.56, 0.43).
• No model was perfectly calibrated. CMAL was overdispersed, DRN mostly too narrow in the upper tail, and BQN underdispersed.
• For floods, FRiCA scored 414 hits against 301 for the deterministic LSTM and 124 for the 99th percentile rule. It had 616 false alarms, against 171 and 1444.
• For extreme floods, FRiCA scored 27 hits and 10 false alarms. The deterministic LSTM scored 18 hits and 8 false alarms. The 99th percentile rule scored 28 hits and 90 false alarms.

Limitations

• None of the probabilistic models is well calibrated, and the three different loss functions make it hard to separate the effect of loss from the effect of distribution.
• FRiCA is an exploratory proof of concept. Results varied a lot across random seeds, no systematic hyperparameter tuning was done, and the reward design risked reward hacking.
• Rewards only represent plausible operational preferences, not real socioeconomic costs.

Glossary

• LSTM: Long Short-Term Memory network, a neural network that learns from time series and is widely used for rainfall–runoff modelling.
• CRPS: Continuous ranked probability score, a score for probabilistic predictions that rewards both accuracy and sharpness, where lower is better.
• PIT histogram: A plot that checks calibration. A flat histogram means the predicted probabilities match how often outcomes occur.
• Reinforcement learning: A way of training an agent to pick actions by rewarding good outcomes and penalising bad ones.