Integrating LEAF to data management workflow in LHAASO

Sep 11, 2018, 2:45 PM
15m
406A

406A

Sectional reports 10. Databases, Distributed Storage systems, Datalakes 10. Databases, Distributed Storage systems, Datalakes

Speaker

Mr Haibo Li (Institute of High Energy Physics,Chinese Academy of Sciences)

Description

Nowadays, data storage and management in cloud computing environment has been very important in high energy physics filed. The LHAASO(Large High Altitude Air Shower Observatory) experiment of IHEP will generate 2 PB per year in the future. These massive data processing faces many challenges in the distributed computing environment. For example, some sites may have no local HEP storage which made the distributed computing unavailable. Our goal is to make the data available for LHAASO in any remote sites. In our architecture, we use EOS as our local storage system, and use LEAF as the data federation system. LEAF is a data cache and access system across remote sites proposed by IHEP. LEAF can present one same file system view at local and the remote sites, supporting directly data access on demand. In this paper, we will present the whole data management architecture, data workflow and performance evaluation of LEAF in LHAASO.

Primary author

Mr Haibo Li (Institute of High Energy Physics,Chinese Academy of Sciences)

Co-authors

Mr Qi XU (Institute of High Energy Physics, Chinese Academy of Sciences) Dr Qiulan Huang (Institute of High Energy of Physics(IHEP), Chinese Academy of Science) Dr Yaodong Cheng (Institute of High Energy Physics, Chinese Academy of Sciences)

Presentation materials