Data analytics in the cloud

Project goal

This project is evaluating solutions that combine data engineering, machine-learning and deep-learning tools. They are being run using cloud resources — from Oracle Cloud Infrastructure (OCI) — and address a number of use cases of interest to CERN’s community. This activity will enable us to compare performance, maturity, and stability of solutions deployed on CERN’s infrastructure with the ones in OCI.

R&D topic

Machine learning and data analytics

Project coordinator(s)

Eric Grancher and Eva Dafonte Perez

Team members

Luca Canali, Riccardo Castellotti

Collaborator liaison(s)

Vincent Leocorbo, Cristobal Pedregal-Martin, David Ebert, Dmitrij Dolgušin

Collaborators

Project background

Big-data tools — particularly related to data engineering and machine learning — are evolving rapidly. As these tools reach maturity and are adopted more broadly, new opportunities are arising for extracting value out of large data sets.

Recent years have seen growing interest from the physics community in machine learning and deep learning. One important activity in this area has been the development of pipelines for real-time classification of particle-collision events recorded by the detectors of the LHC experiments. Filtering events using so-called “trigger” systems is set to become increasingly complex as upgrades to the LHC increase the rate of particle collisions.

Recent progress

SWAN is a platform for performing interactive data analysis in the cloud. It was developed at CERN and integrates software, compute, and storage resources used by CERN physicists and data scientists. In 2020, we deployed a proof-of-concept version of SWAN on OCI resources.

As part of this work, we developed a custom Kubernetes deployment on OCI resources, in order to take advantage of GPU resources. This proved that it is possible to run interactive analytics workflows and ML in OCI while accessing datasets from CERN’s storage systems.

We also performed a distributed machine-learning training exercise, with a recurrent neural network using over 250 GB of data. For this, we utilised 500 CPU cores and 10 GPUs using Kubernetes on OCI. Using OCI showed us that public clouds are particularly convenient for use cases that need a large number of resources for a short amount of time.

Next steps

The focus of the project in 2021 will include work to integrate CERN’s analytics platform with OCI, enabling users to run their workloads on remote cloud resources using a common interface.

Over the longer term, we are planning to add new features to the analytics platform, with a focus on improving the full lifecycle of the development of machine-learning use cases.

Publications

Presentations