Cegal and Kafka secure great dataflow at Statnett
The system operator of the Norwegian power system, Statnett, ensures that electricity flows smoothly from power producers to consumers. For the power supply to function optimally, data must flow unhindered between systems, departments, power companies and authorities. Kafka’s technology secures this, combined with expertise from Cegal. (Photo: Statnett)
On Statnett’s websites you’ll find an overview of how much power is produced in Norway, almost in real time – and how high the consumption of power is. The data behind this service is made available by Kafka technology, which Cegal assists with.
Statnett owns, operates, and expands the national electricity grid, which means that electricity can be transferred across the country and between Norway and neighboring countries. To put it simply, Statnett ensures that we get electricity when we turn on a switch, and that there is a balance between production and electricity consumption in Norway.
Statnett’s activities include the collection and analysis of enormous amounts of data. A modern energy market and well-functioning power grid will not work without data collection and analysis. Statnett is also required by law to share data with the Norwegian authorities and EU authorities.
Kafka ensures free dataflow
Data collection and data flow can be called the central nervous system in Statnett. However, the data has traditionally been very protected and difficult to access for other business systems. In addition, few people have had access to the data. The challenge has been to achieve a better data flow between IT systems and agencies. To solve this, Statnett has chosen Kafka technology.
So, who and what is this Kafka in a data context? Apache Kafka, which is the full name, is an open-source streaming platform. It makes it possible to handle integration between different systems and process data in real time. Kafka was developed by LinkedIn, who needed a solution that could handle the enormous amounts of data the company eventually had to handle. The project was Apache Kafka, which was open sourced in 2012.
Cegal has assisted Statnett’s highly competent integration team with the setup and development of Kafka-based integration since 2018.
“To begin with the job was to set up Kafka. With the help of Kafka, we retrieve data from databases or systems and make them available to various other systems, quickly and securely”
Dejan Maroc, Senior Sonsultant at Cegal.
From database to data lake
The first phase of the project was about showing that with the help of Kafka it would be possible to distribute data to the various areas and systems in Statnett.
After a successful first phase, Statnett and Cegal have started phase two: moving data from databases to a so-called data lake in a Hadoop environment. Cegal has created a streaming platform that allows data to flow quickly between different systems and makes it possible to use data in new systems.
“This data platform allows an organic growth in the use of data for new systems and new use. It makes it easy to use data in new ways”, explains Maric in Cegal.
Previously, the data was in proprietary databases, and only a few systems and people had access to and insight into the data. With the data platform, the data can be used by several systems and distributed to several, both internally in Statnett and externally to partners, power suppliers and other players in the energy industry, authorities as well as you and me.
Large amounts of data are collected from equipment that is out in the field, such as transformers, turbines and so on. Now this data can be used for smart and predictive maintenance. This way you can fix equipment before it fails, which prevents downtime and saves Statnett large costs.
With the data in a data lake, it is also easier to connect the data with other sources, such as weather data. At the same time, it will be easier to analyze the data and ensure a more efficient power grid and sustainable power supply.
