Speaker
Mr
Anton Teslyuk
(NRC "Kurchatov Institute")
Description
Modern Free Electron Laser (FEL) facilities generate huge amounts of data and require sophisticated and computationally expensive analysis. For example, recent experiments at European XFEL have generated more than 360 Tb of raw data in five days. Efficient analysis of this data is a challenging task which requires productive use of existing methods and software for data analysis over scalable computing infrastructure. Additional challenge is that different pieces of software are optimized for diverse computing architectures (parallel MPI computing, GPU, SMP and etc) and require various software environments.
In this report we present our experience of setting up experimental data analysis workflow in containerized computing infrastructure. We take individual software packages for certain analysis steps to set up loosely coupled virtualized microservices and use Kubernetes software to orchestrate multiple containers as a scalable data processing workflow. This approach brings us flexibility in setting up software environments inside containers and allows easy parallelization of data processing.
Primary author
Mr
Anton Teslyuk
(NRC "Kurchatov Institute")
Co-authors
Mr
Sergey Bobkov
(National Research Center "Kurchatov Institute")
Mr
Viacheslav Ilyin
(National Research Center "Kurchatov Institute")
