High-performance distributed caching technologies (DAQDB)

Project goal

We are exploring the suitability of a new infrastructure for key-value storage in the data-acquisition systems of particle-physics experiments. DAQDB (Data Acquisition Database) is a scalable and distributed key-value store that provides low-latency queries. It exploits Intel® Optane™ DC persistent memory, a cutting-edge non-volatile memory technology that could make it possible to decouple real-time data acquisition from asynchronous event selection.

R&D topic
Computing performance and software
Project coordinator(s)
Giovanna Lehmann Miotto
Team members
Adam Abed Abud, Fabrice Le Goff, Lola Stankovic
Collaborator liaison(s)
Claudio Bellini, Grzegorz Jereczek, Jan Lisowiec, Andrea Luiselli, Maciej Maciejewski, Jakub Radtke, Malgorzata Szychowska, Norbert Szulc, Johann Lombardi, Andrea Luiselli

Collaborators

Project background

Upgrades to the LHC mean that the data rates coming from the detectors will dramatically increase. Data will need to be buffered while waiting for systems to select interesting collision events for analysis. However, the current buffers at the readout nodes can only store a few seconds of data due to capacity constraints and the high cost of DRAM. It is therefore important to explore new, cost-effective solutions — capable of handling large amounts of data — that capitalise on emerging technologies.

Recent progress

The idea of DAQDB has proven to be very interesting. Nevertheless, the effort required to develop a mature product of this complexity was recognised to be too large for the available resources. Work thus continued in two parallel strands:

1) An evaluation of Distributed Asynchronous Object Storage (DAOS) was started, with the aim of assessing whether the future DAQ needs of large experiments, such as ATLAS and CMS, may be addressed through this underlying technology, combined with a custom software layer.

2) Performance measurements of Intel® Optane™ DC persistent memory devices continued, in the context of data-acquisition systems, achieving very promising results.

Next steps

The DAQDB project concluded in 2020. A possible follow-up activity is being discussed between Intel and experiment representatives; this would focus on the further development and use of DAOS.

Publications

    D. Cicalese et al., The design of a distributed key-value store for petascale hot storage in data acquisition systems. Published in EPJ Web Conf. 214, 2019. cern.ch/go/xf9H
    P. Czarnul, G. Gołaszewski, G. Jereczek, M. Maciejewski, Development and benchmarking a parallel Data AcQuisition framework using MPI with hash and hash+tree structures in a cluster environment. Published at the 19th International Symposium on Parallel and Distributed Computing, 2020. cern.ch/go/lK78
    A. A. Abud, D. Cicalese, G. Jereczek, F. L. Goff, G. L. Miotto, J. Love, M. Maciejewski, R. K. Mommsen, J. Radtke, J. Schmiegel, M. Szychowska, Let’s get our hands dirty: a comprehensive evaluation of DAQDB, key-value store for petascale hot storage. Published at the 24th International Conference on Computing in High Energy and Nuclear Physics, 2020. cern.ch/go/7JGF

Presentations

    M. Maciejewski, Persistent Memory based Key-Value Store for Data Acquisition Systems (25 September). Presented at IXPUG 2019 Annual Conference, Geneva, 2019. cern.ch/go/9cFB
    G. Jereczek, Let's get our hands dirty: a comprehensive evaluation of DAQDB, key-value store for petascale hot storage (5 November). Presented at the 4th International Conference on Computing in High-Energy and Nuclear Physics (CHEP), Adelaide, 2019. cern.ch/go/9cpL8
    J. Radtke, A Key-Value Store for Data Acquisition Systems (April). Presented at SPDK, PMDK and VTune(tm) Summit 04'19, Santa Clara, 2019. cern.ch/go/H6Rl
    G. Jereczek, The design of a distributed key-value store for petascale hot storage in data acquisition systems (12 July). Presented at 23rd International Conference on Computing in High Energy and Nuclear Physics (CHEP), Sofia, 2018. cern.ch/go/6hcX
    J. M. Maciejewski, A key-value store for Data Acquisition Systems (12 September). Presented at ATLAS TDAQ week, Cracow, 2018.
    G. Jereczek, M. Maciejewski, Data Acquisition Database (12 November). Presented at The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC18), Dallas, 2018.
    M. Maciejewski, J. Radtke, The Design of Key-Value Store for Data Acquisition Systems (5 December). Presented at NMVe Developer Days, San Diego, 2018.
    P. Czarnul, G. Gołaszewski, G. Jereczek, M. Maciejewski, Development and benchmarking a parallel Data AcQuisition framework using MPI with hash and hash+tree structures in a cluster environment (5-8 July). Presented at the 19th International Symposium on Parallel and Distributed Computing, Warsaw, 2020. cern.ch/go/W9zL
    A. A. Abud, Experience and performance of persistent memory for the DUNE data acquisition system (12-24 October). Presented at the 22nd virtual IEEE Realtime Conference, 2020. cern.ch/go/P8sD