FMI Open Data on Amazon AWS S3

FMI provides two data set in AWS S3: Hirlam numerical weather model and SILAM global weather model. The data are publicly accessible in S3.

SILAM Global Air Quality forecast

SILAM (System for Integrated modelLling of Atmospheric coMposition) is a global-to-meso-scale dispersion model developed for atmospheric composition, air quality, and emergency decision support applications, as well as for inverse dispersion problem solution.

The data is licenced by the Creative Commons Attribution 4.0 International license (CC BY 4.0).

FMI provides operational 20-km global AQ forecast of 7 regulated constituents for free download from AWS S3 bucket.

The 168-hour long forecast is generated once a day every evening with start time 00Z of previous day to accommodate the effect of actually observed emissions of e.g. forest fires. The forecast is available around 06:00 UTC next day with 121 hourly steps form 00Z D0.

Example presentation of SILAM Global Air Quality forecast

Content

The forecast is available on 0.2x0.2 degree global regular lon-lat grid. The values are hourly averages timestamped with end of hour.

The parameters are concentrations of NO2, NO, SO2, O3, CO, PM10 and PM2.5, and air density. All concentrations are given in ug/m3 at ambient conditions, thus in order to concert them to mixing ratios or to EU-regulated concentrations a transformation has to be applied:

Parameter

units

Conversion factor to

vmr (ppb)

EU-concentration (ug/m3)

mass mixing ratio

(ug/kg)

NO2

ug/m3

0.63/airdens

1.225kg/m3 /airdens

1/airdens

NO

ug/m3

0.96/airdens

1.225kg/m3 /airdens

1/airdens

CO

ug/m3

1.04/airdens

1.225kg/m3 /airdens

1/airdens

O3

ug/m3

0.60/airdens

1.225kg/m3 /airdens

1/airdens

SO2

ug/m3

0.45/airdens

1.225kg/m3 /airdens

1/airdens

PM10

ug/m3

-

1

1/airdens

PM2.5

ug/m3

-

1

1/airdens

Note: All PMs are expressed as dry mass.

Disclaimer: Current products are created for scientific use only. Neither quality nor completeness of the presented information is guaranteed and the data producers do not accept any responsibility for its correctness and timeliness.

Accessing the Data on AWS

The data is uploaded into two AWS S3 Buckets:

  • fmi-opendata-silam-surface-netcdf (for surface data)

  • fmi-opendata-silam-surface-zarr (for pressure levels data)

Every model run is stored in separate directories divided into files based on parameters. Every file contains all valid times for one parameter and one model run.

Files are named with a convention:

silam_glob_v5_6_<year><month><day>_<parameter>_<forecast_day>.<format>

For example:

  • silam_glob_v5_6_20190819_CO_d0.zarr

  • silam_glob_v5_6_20190819_CO_d0.netcdf

The data is stored into following directory structure:

/global/<year><month><day>/

Where times are derived from origin time of the model run.

Content of the buckets can be browsed here:

SNS

Public Amazon SNS topics are available for every new object added to the Amazon S3.

  • arn:aws:sns:eu-west-1:916174725480:new-fmi-opendata-rcrhirlam-surface-grib

  • arn:aws:sns:eu-west-1:916174725480:new-fmi-opendata-rcrhirlam-pressure-grib

For more information on subscribing to SNS topics, visit http://docs.aws.amazon.com/sns/latest/dg/SubscribeTopic.html.

For more information on Amazon S3 event message structure, visit http://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html.

More Information

Silam home page: http://silam.fmi.fi

Some usage examples: https://github.com/fmidev/opendata-resources/tree/master/examples/python

Hirlam Weather Model

HIRLAM (High Resolution Limited Area Model) is an operational synoptic and mesoscale weather prediction model. The model is developed by an international program formed by institutes from Sweden, Norway, Denmark, Iceland, the Netherlands, Ireland, Spain, Estonia and Lithuania. There is also close co-operation with France.

FMI provides its operational Hirlam model runs to be downloaded freely from AWS S3 bucket. The data is licenced by the Creative Commons Attribution 4.0 International license (CC BY 4.0).

The data is updated four times a day with analysis hours 00, 06, 12 and 18. Corresponding model runs are available roughly five hours after analysis time (~ after model run has started).

Content

The current horizontal resolution of the HIRLAM RCR model is 7.5 km. At present, the current operational setup produces daily four 54 hour regional forecasts for extended European area (Figure below). The model is initiated by the ECMWF boundary condition files.

The data is divided in two sets: surface data and pressure levels data. The surface level data provide data for surface of the earth and pressure level data for constant air pressure levels in the atmosphere. The following pressure levels are provided: 1000, 925, 850, 700, 500, 400, 300, 250, 200, 100, 50 hPa.

The data is in projection epsg:4326. Parameters

Following parameters are available in the surface data:

Parameter

GRIB ID

GRIB Name

Level Type

Other Information

Pressure

151

msl

meanSea

 

GeopHeight

256

Z

   

Temperature

167

2t

heightAboveGround

 

DewPoint

168

2d

heightAboveGround

 

Humidity

157

r

heightAboveGround

 

WindUMS

131

u

heightAboveGround

 

WindVMS

132

v

heightAboveGround

 

PrecipitationAmount

201113

rain_con

   

TotalCloudCover

164

N

244

 

LowCloudCover

186

Cl

214

 

MediumCloudCover

187

Cm

224

 

HighCloudCover

188

Ch

234

 

Precipitation1h

2059

rr1h

entireAtmosphere

 

MaximumWind

201187

MaximumWind

heightAboveGround

 

WindGust

29

WindGust

105

 

RadiationGlobalAccumulation

169

ssrd

 

"templatenumber": 8 / "aggregatetype" : "accum" / "aggregatelength" : 60

RadiationLWAccumulation

175

strd

 

"templatenumber": 8 / "aggregatetype" : "accum" / "aggregatelength" : 60

RadiationNetSurfaceLWAccumulation

177

str

 

"templatenumber": 8 / "aggregatetype" : "accum" / "aggregatelength" : 60

RadiationNetSurfaceSWAccumulation

228242

fdif

   

 

And following in the pressure level data:

Parameter

GRIB ID

GRIB Name

GeopHeight

256

Z

Temperature

167

2t

DewPoint

168

2d

Humidity

157

r

WindUMS

131

u

WindVMS

132

v

VelocityPotential

135

w

PseudoAdiabaticPotentialTemperature

3014

 

Accessing the Data on AWS

The data is uploaded into two AWS S3 Buckets:

  • fmi-opendata-rcrhirlam-surface-grib (for surface data)

  • fmi-opendata-rcrhirlam-pressure-grib (for pressure levels data)

Every model run is stored in separate directories divided into files based on parameters. Every file contains all valid times for one parameter and one model run.

Files are named with a convention:

numerical-hirlam74-forecast-<parameter>-<year><month><day>T<HOUR>0000Z.grb2

For example:

numerical-hirlam74-forecast-DewPoint-20170322T000000Z.grb2

The data is stored into following directory structure:

<year>/<month>/<day>/<hour>

Where times are derived from origin time of the model run.

Content of the buckets can be browsed here:

SNS

Public Amazon SNS topics are available for every new object added to the Amazon S3.

  • arn:aws:sns:eu-west-1:916174725480:new-fmi-opendata-rcrhirlam-surface-grib

  • arn:aws:sns:eu-west-1:916174725480:new-fmi-opendata-rcrhirlam-pressure-grib

For more information on subscribing to SNS topics, visit http://docs.aws.amazon.com/sns/latest/dg/SubscribeTopic.html.

For more information on Amazon S3 event message structure, visit http://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html.

Tools to handle the data

The data is stored in GRIB2-files. Below is listed few possible tools to handle the data.

PanoPly

PanoPly provides an easy UI to view GRIB, NetCDF and HDF files.

Example image of PanoPly's user interface visualizing total cloud cover in northern hemisphere
Example image of PanoPly's user interface visualizing total cloud cover in northern hemisphere

Integrated Data Viewer (IDV)

Compared to PanoPly Integrated Data Viewer (IDV) provides a slightly more complicated but more versatile user interface for browsing and analysing geographical data.

Example image of Intergrated Data Viewer's user interface visualizing long wave radiation data
Example image of Intergrated Data Viewer's user interface visualizing long wave radiation data

GRIB-tools

Bunch of tools exist for batch processing the data:

ECMWF GRIB tools provides number of convenient command line tools to process the data. For more information, please consult their wiki:

NOAA's WGRIB2 (and WGRIB for GRIB1 files) can for example: inventory and read grib2 files, create subsets and export data to ieee, text, binary, CSV, netcdf and mysql.

http://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/

Geospatial Data Abstraction Library GDAL supports GRIB as well

http://www.gdal.org/frmt_grib.html

SmartMet Server

If you are using the data in an operational manner and live in web environment, SmartMet Server is a great resource to extract the data as JSON or GML and visualize it via WMS.

For more information, please consult: https://github.com/fmidev/smartmet-server

Additional Resources

http://en.ilmatieteenlaitos.fi/numerical-weather-prediction http://en.ilmatieteenlaitos.fi/open-data-manual-forecast-models https://aws.amazon.com/public-datasets/fmi-hirlam/