Jan 7, 2021

Visualizing energy data from ENTSO-E and ENTSO-G

The ENTSO-E provides public data for the electricity load/production while the ENTSO-G provides data of transferred gas amounts in the European union.

Sadly, the data is not nicely visualized at the corresponding website (which is also very slow if a lot of data is requested). So I provided a nice frontend for the data as part of a project for my master’s degree.

The final project can be found at:

or if you are connected to the FH-Aachen university network (by VPN or eduroam) you can visit the page hosted at https://service-fb9.fh-aachen.de/

The service is implemented using python only with Dash from Plotly for the pretty graphs and maps and a Flask Webserver running in a gunicorn docker container. I am currently using SQLite as data storage while I also tried Apache Spark to filter the data on a HDFS, which did not suit quite well for my usecase.

How did you do this?

In the future I will hopefully describe a little further where the challenges were before coming to the result you can see there.

I will just summarize a few:

  • understanding the data format and the energy background of the data (which was far more complicated for the ENTSO-G part)
  • different data results when requesting data from ENTSO-G as JSON vs XML vs CSV
  • map X and Y points from ENTSO-G aren’t usable as latitude and longitude
  • download the whole data (multiple times) and store it in a database (with correct names encoding)

This is a project with a quite cool result which I invested a lot of time into.

Links: