AI weather model can produce forecasts 5,000 times faster than gold-standard weather model, Microsoft says
According to developers, Aurora outperforms state-of-the-art atmospheric chemistry forecasts on 74% of variables.
Is AI the future of weather forecasting?
One AI researcher from the University of California Los Angeles calls it "the reckoning moment" and explains how AI forecasting differs from forecasting from traditional weather models.
A team of researchers set out to revolutionize weather forecasting using artificial intelligence.
So far, forecasts have come out 5,000 times faster and outperformed the current gold standard of weather forecasting models on 74% of data, such as wind speed, wind direction and humidity, according to the Microsoft Aurora model developers.
"Traditional weather and climate models aim to predict how the atmosphere evolves by formulating a set of governing equations using first physical principles (e.g. conservation of mass, momentum, energy, etc.)," Principal Research Manager of the AI Aurora model Paris Perdikaris said. "Then the solutions to these equations are approximated in space and integrated in time using sophisticated numerical algorithms that run on computer clusters."
AIRLINE HARNESSES ‘NEXT GENERATION OF FORECASTING’ TO CUT DOWN ON WEATHER DELAYS AND CANCELATIONS
File: A supercomputer at the German Climate Computing Center with 100,000 processors.
(Morris MacMatzen / Getty Images)
Euro (ECMWF) model is gold standard of traditional models
According to Perdikaris, the European Center for Medium-Range Weather Forecast's (ECMWF) model called the HRES-IFS, which stands for High Resolution Integrated Forecast System, is the gold standard of traditional forecasts. However, producing a 10-day forecast at a 6.8-mile resolution takes about 65 minutes on large sets of computers with hundreds of processors. That's expensive and slow, he said.
"Traditional numerical weather predictions would require hours to produce the same result using a supercomputer with thousands of CPU cores," Perdikaris said. "The primary advantage here is Aurora circumvents the cost of running a physics-based simulation, and its evaluation is highly parallelized across compute cores and fully leverages the capabilities of modern GPU (graphics processing unit) hardware."
Perdikaris said his team looked at AI models like PanguWeather and Google's GraphCast, which are only trained on a single data set and designed to only create a single specific prediction task like a 10-day global weather forecast with a 6.8-mile resolution. The developer said that these were the first AI models that competed with and sometimes outperformed the gold standard.
However, the team wanted the model to tackle a wide variety of Earth system and science data prediction tasks. Aurora was born.
FORECASTING WITH AI: COMPANY CREATES FIRST WEATHER AND CLIMATE GENERATIVE ARTIFICIAL INTELLIGENCE
Google's AI weather forecasting model gets good grades from researchers
File: A study shows that Google's GraphCast which is an AI based weather forecast model, is more accurate than traditional weather models.
"With Aurora, we asked the question of whether we can build a much more general AI system that can tap into the vast expand of Earth system data and excel at a range of prediction tasks," Perdikaris said.
Building a foundation model to include more weather, climate and atmospheric chemistry solutions
So they developed a "foundation model." It's a machine learning program, "that makes decisions in a manner similar to the human brain, by using processes that mimic the way biological neurons work together to identify phenomena, weigh options and arrive at conclusions," according to IBM.
Aurora was trained in two phases: pretraining and fine-tuning. Pretraining is the most expensive and time-consuming part, Perdikaris said. It took about 32 NVIDIA graphics processing units about 2-and-a-half weeks to ingest and learn from a collection of weather and climate simulation data which included analysis and reanalysis. AI models only need pretraining and fine-tuning once.
HOW AI CAN HELP (AND HURT) CLIMATE CHANGE
File: A single Nvidia graphic processing unit (GPU).
(JOEL SAGET/AFP / Getty Images)
"Aurora’s effectiveness lies in its training on more than a million hours of diverse weather and climate simulations, which enables it to develop a comprehensive understanding of atmospheric dynamics," the developers wrote for a Microsoft blog. "One of the key findings of this study is that pretraining on diverse datasets significantly improves Aurora’s performance compared to training on a single dataset. By incorporating data from climate simulations, reanalysis products, and operational forecasts, Aurora learns a more robust and generalizable representation of atmospheric dynamics."
Then comes fine-tuning, which improves Aurora's accuracy at specific tasks like 10-day global weather forecasting or 5-day global air pollution prediction.
"During ‘finetuning’ Aurora learns from a much smaller corpus (collection) of high-quality simulation data that correspond to the target prediction task," Perdikaris said. "This phase is less computationally demanding, and can be completed in about 5 days using 8 NVIDIA A100 GPUs (graphics processing units)."
Testing Aurora with weather extremes
To test Aurora out, the team forecast northwest Europe's Storm Ciaran. The ferocious bomb cyclone killed 14 and spun up winds equivalent to a strong Category 3 hurricane, along with flooding.
STORM CIARAN SPAWNED RECORD-STRONG TORNADO AS DEADLY STORM TORE THOUGH UK'S JERSEY ISLAND
Aerial view of deadly flash flooding in Italy from Storm Ciaran
At least four people have died in Italy after flash floods swept through parts of Tuscany after nearly 8 inches of rain fell from Storm Ciaran. (Video courtesy: Vigili del Fuoco)
"The low-pressure system associated with Storm Ciarán set new records for England, marking it as an exceptionally rare meteorological event," stated the blog. "The storm’s intensity caught many off guard, exposing the limitations of current weather-prediction models and highlighting the need for more accurate forecasting in the face of climate change."
Other AI weather-prediction models failed to forecast the extreme wind speeds and rapid intensification. Aurora produced results similar to the gold standard traditional weather model at a fraction of the time and cost.
"Aurora’s improved performance on extreme events stems from the fact that it has ‘seen’ a much larger and diverse corpus (body) of data during its pretraining and finetuning phases – this volume of data naturally brings better coverage of extremes," Perdikaris said.
REAL OR FAKE? HOW TO SPOT WEATHER IMAGES GENERATED BY AI
File: A super computer at Meteo France, the French national meteorological service. This new investment was essential in 2021 for traditional numerical weather prediction to process observation data and calculate the forecast.
(FRED SCHEIBER/AFP / Getty Images)
"Aurora’s effectiveness to adapt to new scenarios for which training data may be sparse can be attributed to the fact that the pretraining phase yields a model that has already extracted good features from data, and this allows us to quickly adapt the base pretrained model to new prediction scenarios, especially in case where data scarcity would have otherwise prevented us from training a specialized model from scratch," he continued.
Aurora takes just seconds to form a 10-day global weather forecast, 5-day pollution forecast or concentrations of greenhouse gases using a single GPU.
WE ASKED THE ARTIFICIAL INTELLIGENCE-BASED CHATGPT TO EXPLAIN THE WEATHER. HERE ARE THE RESULTS:
File: The supercomputers develop graphics that look like this from which meteorologists forecast. This is the Meteo France Office.
(AKSARAN/Gamma-Rapho / Getty Images)
Will NOAA ever use Aurora?
The team is currently working on open sourcing Aurora to make it available to the general public. Perdikaris is also in talks with NOAA, ECMWF and the UK Met Office to evaluate the AI model to "asses it's potential in becoming the backbone of future operational systems."
"I think we are still at least a few years from that, but the field is moving really fast," he added.
"The ability of foundation models to excel at downstream tasks with scarce data could democratize access to accurate weather and climate information in data-sparse regions, such as the developing world and polar regions," stated the blog. "This could have far-reaching impacts on sectors like agriculture, transportation, energy harvesting, and disaster preparedness, enabling communities to better adapt to the challenges posed by climate change."