EOS productisation

Project goal

This project is focused on the evolution of CERN’s EOS large-scale storage system. The goal is to simplify the usage, installation, and maintenance of the system. In addition, the project aims to add native support for new client platforms, expand documentation, and implement new features/integration with other software packages.

R&D topic
Data-centre technologies and infrastructures
Project coordinator(s)
Luca Mascetti
Team members
Fabio Luchetti, Elvin Sindrilaru
Collaborator liaison(s)
Gregor Molan, Branko Blagojević, Ivan Arizanović, Svetlana Milenković

Collaborators

Project background

Within the CERN IT department, a dedicated group is responsible for the operation and development of the storage infrastructure. This infrastructure is used to store the physics data generated by the experiments at CERN, as well as the files of all members of personnel.

EOS is a disk-based, low-latency storage service developed at CERN. It is tailored to handle large data rates from the experiments, while also running concurrent complex production workloads. This high- performance system now provides more than 350 petabytes of raw disks.

EOS is also the key storage component behind CERNBox, CERN’s cloud-storage service. This makes it possible to sync and share files on all major mobile and desktop platforms (Linux, Windows, macOS, Android, iOS), with the aim of providing offline availability to any data stored in the EOS infrastructure.

Recent progress

In 2020, Comtrade's team continued to improve and update the EOS technical documentation. Due to the COVID pandemic and the travel restrictions during the year, the hosting of dedicated hardware resources at CERN, to support the prototyping of an EOS-based appliance, was postponed.

The EOS-Comtrade team therefore focused their efforts on the development of a native way to interact from the Windows environment to the distributed storage system. Currently, Windows users access EOS only via dedicated gateways, where the system is exposed with SAMBA shares.

In order to provide more performant and seamless access to EOS for Windows users, Comtrade started developing a dedicated Windows client, codenamed EOS-wnc.

This new, dedicated, command-line Windows client was developed against the EOS server APIs. It now covers all EOS commands from the standard user listing and copy commands to the administrative ones.

On top of this, some EOS commands on Windows have additional functionalities, like improved tab-completion and history features.

Next steps

Together, we will focus on improving the current EOS Windows native client, adding new functionalities, and integrating it in the Windows user interface, developing a dedicated driver.

It is still planned to host dedicated hardware resources at CERN to support prototyping of an EOS-based appliance. This will enable Comtrade to create a first version of a full storage solution and to offer it to potential customers in the future.

Publications

    X. Espinal, M. Lamanna, From Physics to industry: EOS outside HEP. Published in Journal of Physics: Conference Series, Vol. 898, 2017. cern.ch/go/7XWH

Presentations

    L. Mascetti, Comtrade EOS productization (23 January). Presented at CERN openlab technical workshop, Geneva, 2019. cern.ch/go/W6SQ
    G. Molan, EOS Documentation and Tesla Data Box (4 February). Presented at CERN EOS workshop, Geneva, 2019. cern.ch/go/9QbM
    L. Mascetti, EOS Comtrade project (23 January). Presented at CERN openlab Technical workshop, Geneva, 2020. cern.ch/go/l9gc
    L. Mascetti, CERN Disk Storage Services (3 February 2020). Presented at CERN EOS workshop, Geneva, 2020. cern.ch/go/pF97
    G. Molan, Preparing EOS for Enterprise Users (27 January 2020). Presented at Cloud Storage Services for Synchronization and Sharing (CS3), Copenhagen, 2020. cern.ch/go/tQ7d
    G. Molan, EOS Documentation for Enterprise Users (3 February 2020). Presented at CERN EOS workshop, Geneva, 2020. cern.ch/go/swX8
    G. Molan, EOS Windows Native Client (3 February 2020). Presented at CERN EOS workshop, Geneva, 2020. cern.ch/go/P7DX
    G. Molan, EOS Storage Appliance Prototype (5 February 2020). Presented at CERN EOS workshop, Geneva, 2020. cern.ch/go/q8qh
    G. Molan, EOS-wnc demo (30 October 2020). Presented at IT-ST-PDSComtrade workshop, Geneva, 2020. cern.ch/go/q8qh
    L. Mascetti, EOS Open Storage for Science (7 December). Presented at Expo, Dubai, 2021.