Speaker
Dr
Dmitry Golubkov
(Institute for High Energy Physics, Protvino)
Description
The ATLAS Production System (ProdSys) is responsible for processing of petascale data on the Grid. ProdSys – automated system for running jobs in the PanDA. The PanDA Production ANd Distributed Analysis system is a data-driven workload management system for production and distributed analysis. ProdSys performs an extremely important role being an additional level of abstraction for PanDA. The Production System originated in 2006 and now handles O(1M) tasks per year, each task transforms in many jobs. ProdSys evolves to accommodate a growing number of users and new requirements from ATLAS Physics groups. To improve the main characteristics of the system, such as flexibility, maintainability and ease of use, decided to develop a new system to meet current requirements – next generation of Production System, ProdSys2.
ProdSys2 is a multilevel workflow management system for BigData processing that operates with user requests, tasks and jobs. Based on previous experience with managing of tasks the architecture of system is designed so that ProdSys2 consists of two major components representing different levels of data abstraction: DEFT (Database Engine for Tasks) and JEDI (Job Execution and Definition Interface). These components operate together: the first system manages user requests and tasks and the second manages jobs generated from tasks provided. Systems interact with each other according to the developed protocol. As top-level management interface of ProdSys2, DEFT provides unified access to the user to run different types of tasks, such as tasks for Physics (DPD Production and MC Production), data preparation (Data Reprocessing) and Trigger (High-level Trigger Reprocessing). DEFT supports both task workflows (task chains) and single tasks, easily adapts to new types of tasks and delivers enhanced management mechanisms for controlling of tasks.
We present top-level management layer of next generation ATLAS Experiment Production System
Primary author
Dr
Dmitry Golubkov
(Institute for High Energy Physics, Protvino)
Co-authors
Alexandre Vaniachine
(Argonne)
Prof.
Alexei Klimentov
(Brookhaven National Laboratory, Upton)
Dr
Jose Enrique Garcia Navarro
(Instituto de Física Corpuscular, Paterna)
Dr
Kaushik De
(University of Texas at Arlington)
Dr
Mikhail Borodin
(National Research Nuclear University «MEPhI»)
Dr
Tadashi Maeno
(Brookhaven National Laboratory, Upton)
