Speaker
Description
The Russian Scientific Data Lake is a part of Data Lake R&D conducted by the DOMA project. It aims to mitigate the present LHC computing model drawbacks to cope with an unprecedented scientific data volume at the multi-exabyte scale that will be delivered by experiments in the High Luminosity phase of the LHC. The prototype of the Russian Scientific Data Lake is being implemented and it tests different configurations of data caching and buffering mechanisms using real ATLAS and ALICE experiments payloads. In order to compare the efficiency of the resource usage between different configurations and control the state of the deployed infrastructure a unified monitoring system was developed. It aggregates data from various sources into a single ElasticSearch storage. On top of it, a set of dashboards at Kibana and a special web-application based on Django framework were developed for monitoring test jobs and software components of the Russian Scientific Data Lake infrastructure. In this work we present the architecture, components and features of the unified monitoring system.
