Speaker
Dr
Alexei Klimentov
(Brookhaven National Lab)
Description
The Production and Distributed Analysis (PanDA) system was designed to meet the requirements for a workload management system (WMS) capable to operate at the LHC data processing scale. It has been used in the ATLAS experiment since 2005 and is now being part of the BigPanDA project expanding into a meta application, providing transparency of the data processing and workflow management for High Energy Physics (HEP) and other data-intensive sciences.
BigPanDA was one of the first WMS systems expanded beyond the Grid to support the High Performance Clusters, Supercomputers, Clouds and Leadership Computing Facilities (LFC). This could not fail to attract the attention to the system from other compute intensive sciences such as brain studies. In 2017, the pilot project was started between teams of the BigPanDA and the Blue Brain Project (BBP) of the Ecole Polytechnique Federal de Lausanne (EPFL) located in Lausanne, Switzerland. This proof of concept project is aimed at demonstrating the efficient application of the BigPanDA system to support the complex scientific workflow of the BBP which relies on using a mix of desktop, cluster and supercomputers to reconstruct and simulate accurate models of brain tissue.
During the first phase of the collaboration we demonstrated the execution of the computational jobs on a variety of BBP distributed computing systems powered. The targeted systems for demonstration included: Intel x86-NVIDIA GPU based BBP clusters located in Geneva (47 TFlops) and Lugano (81 TFlops), BBP IBM BlueGene/Q supercomputer ( 0.78 PFLops and 65 TB of DRAM memory) located in Lugano, the Titan Supercomputer with peak theoretical performance 27 PFlops operated by the Oak Ridge Leadership Computing Facility (OLCF), and Cloud based resources such as Amazon Cloud.
To hide execution complexity and simplify manual tasks by end-users, we developed a web interface to submit, control and monitor user tasks and seamlessly integrated it with the PanDA WMS system. The project demonstrating that the software tools and methods for processing large volumes of experimental data, which have been developed initially for experiments at the LHC accelerator, can be successfully applied to other scientific fields.
Primary author
Dr
Ruslan Mashinistov
(NRC KI)
Co-authors
Alexandre Beche
(EPFL)
Dr
Alexei Klimentov
(Brookhaven National Lab)
Dr
Fabien Delalondre
(EPFL)
Prof.
Felix Schuermann
(EPFL)
Mr
Ivan Tertychnyy
(NRC KI)
Prof.
Kaushik De
(UTA)
