ICON weather model ported to GPUs runs 5.5× faster than CPUs and is now operational

Why it matters

Weather services need fast forecasts to deliver products on time. The work shows a large community Fortran model can be moved to GPUs step by step, without a rewrite. It also shows the same GPUs can serve machine-learning weather models.

What they did

The authors added OpenACC directives to the Fortran code of ICON. They covered the dynamics, tracer transport, physics, pollen, diagnostics and parts of data assimilation. They checked GPU results against a CPU ensemble with perturbed inputs, and against observations over several seasons. They timed a one-hour run of the ICON-CH1-EPS setup on A100 GPUs and compared it with CPUs.

Key findings

• The first GPU version took 125.6 s for the benchmark, about 4.2 times faster than the CPU reference, which is in line with the memory bandwidth ratio of the hardware.
• Tuning steps cut the runtime to 95.80 s. The biggest gains came from CUDA Graphs (5 %) and mixed precision in the dynamical core (8 %). The final socket-to-socket speed-up is 5.5×.
• ICON-CH1-EPS scored better than the older COSMO-1E system for precipitation occurrence and slightly better for 2 m temperature and 10 m wind speed errors. It scored slightly lower for total cloud cover. ICON replaced COSMO at MeteoSwiss on 28 May 2024.
• A 33 h forecast now takes 2642 s on two nodes, within the 3000 s limit. At first deployment it took about 3200 s and needed three nodes.
• Strong scaling stops at about 12 GPUs, when each GPU has fewer than 100000 grid points.

Limitations

• Known model deficiencies remain: a warm bias in Alpine valleys, too much convective rain at maxima, and too much ensemble spread in precipitation.
• The approach works fully only with the nvhpc compiler on NVIDIA hardware. AMD GPUs have unresolved issues for some NWP parts, and there is no solution for Intel GPUs.
• The authors did not measure energy use. The data assimilation code is not in the open-source release, and the OpenACC directives make the code harder to read.

Glossary

• OpenACC: A set of compiler directives, written as comments in the code, that tell the compiler to run loops on a GPU.
• CUDA Graphs: An NVIDIA feature that records a series of GPU tasks once and replays them with less launch overhead.
• Mixed precision: Using lower-precision numbers for some fields and double precision for others, to save memory and time.
• KENDA: The ensemble data assimilation system that combines observations with model output to give the starting state of a forecast.