Speaker
Description
As next-generation light sources move toward higher brightness and data rates, there is a growing need for software systems that can manage differentiated data processing tasks across various beamlines. Our team has developed a workflow-driven management system to address these requirements. The system emphasizes modular design, allowing for the integration of diverse algorithms and the orchestration of complex analysis sequences. We discuss our approach to building a scalable engine for real-time monitoring and batch processing, as well as our efforts in modeling data elements to support reusable workflows. The system has been tested with typical applications like PDF and protein structure analysis. This report focuses on the practical engineering lessons learned during development and the system's role in supporting cross-disciplinary research at mega-science facilities.