Speaker
Description
The rapid growth of distributed computing systems has necessitated the development of efficient platforms for running optimization workflows. In this talk, we present a comprehensive approach to running optimization workflows using the Everest cloud platform, developed at IITP RAS. Everest enables the publication of code as web services and facilitates the allocation of computing resources, including standalone servers, computing clusters, grids, and clouds. Leveraging a user-friendly web interface and API, users can execute compute jobs for services on available resources.
We have extensively utilized the Everest platform in numerous optimization projects, benefiting from its ability to seamlessly integrate optimization models with external systems through an API. Furthermore, Everest provides a convenient debugging environment, allowing for fine-tuning of optimization models by rerunning them from the intuitive web interface.
Our approach encompasses several essential components. Firstly, an application within Everest is configured following our established guidelines. Secondly, a cloud-based computing resource, such as Google Cloud Platform or Yandex Cloud, is employed within Everest. Additionally, the optimization model's source code is hosted in a Software as a Service (SaaS) git repository, such as GitLab or GitHub, and adheres to our prescribed guidelines in terms of inputs and outputs.
Moreover, Everest offers the flexibility to set up computing resources that dynamically create and destroy virtual machines in the cloud as per demand. We have developed comprehensive guidelines to simplify the utilization of optimization models within Everest, ensuring a streamlined workflow.
To facilitate effective setup, we provide a concise checklist of necessary configurations. In Everest, the application should include essential input parameters, such as the model's version, model parameters in JSON or YAML format, and input data as an archive. The output structure should comprise a version identifier, standard output and error streams, and an archive containing the results. Furthermore, a bootstrap script is set in the application, facilitating the retrieval of the specified model version from the input parameters.
Within the cloud environment, an image is configured with all the requisite Everest agent dependencies, alongside the installation of optimization model dependencies, including tools like git for retrieving code from the SaaS git provider.
Our approach represents a robust and versatile framework for running optimization workflows in the Everest platform. By leveraging its features and adhering to the prescribed guidelines, researchers and practitioners can effectively integrate and execute optimization models, ultimately contributing to the advancement of distributed computing systems.
This work was supported by the Russian Science Foundation under grant no. 22-11-00317, https://rscf.ru/project/22-11-00317/
Keywords: distributed computing systems, optimization workflows, Everest cloud platform, web services, computing resources, debugging, integration, guidelines.
