25–29 Sept 2017
Montenegro, Budva, Becici
Europe/Podgorica timezone

The ATLAS Production System Evolution

25 Sept 2017, 16:20
30m
Conference Hall (Montenegro, Budva, Becici)

Conference Hall

Montenegro, Budva, Becici

Splendid Conference & SPA Resort, 85315 Becici, Montenegro Hotel Splendid
Plenary Computing for Large Scale Facilities (LHC, FAIR, NICA, SKA, PIC, XFEL, ELI, etc.) Plenary

Speaker

Mr Mikhail Borodin (The University of Iowa (US))

Description

The second generation of the ATLAS Production System called ProdSys2 is a distributed workload manager that runs daily hundreds of thousands of jobs, from dozens of different ATLAS-specific workflows, across more than a hundred heterogeneous sites. It achieves high utilization by combining dynamic job definition based upon many criteria, such as input and output size, memory requirements and CPU consumption, with manageable scheduling policies and by supporting different kinds of computational resources, such as GRID, clouds, supercomputers and volunteer computers. The system dynamically assigns a group of jobs (task) to a group of geographically distributed computing resources. Dynamic assignment and resource utilization is one of the major features of the system. The Production System has a sophisticated job fault recovery mechanism, which efficiently allows running multi-terabyte tasks without human intervention. We have implemented new features which allow automatic task submission and chaining of different types of production. We present recent improvements of the ATLAS Production System and its major components: task definition and web user interface. We also report the performance of the designed system and how various workflows, such as data (re)processing, Monte Carlo and physics group production, and user analysis, are scheduled and executed within one production system on heterogeneous computing resources.

Primary author

Mr Mikhail Borodin (The University of Iowa (US))

Co-author

Dr Alexei Klimentov (Brookhaven National Lab)

Presentation materials