Oracle Cloud technologies for data analytics on industrial control systems

Project goal

CERN’s control systems acquire more than 250 TB of data per day from LHC and its experiments. Managing these extremely complex “Industrial Internet of Things” (IIoT) systems raises important challenges in terms of data management, retrieval, and analytics.

The main objective is to explore the capabilities of Oracle Autonomous Data Warehouse and Oracle Analytics Cloud for integrating heterogeneous control IIoT data, while improving performance and efficiency for the most challenging analytics requirements.

R&D topic
Machine learning and data analytics
Project coordinator(s)
Manuel Martin Marquez, Sébastien Masson
Team members
Manuel Martin Marquez, Sébastien Masson, Aimilios Tsouvelekakis
Collaborator liaison(s)
Çetin Özbütün, Reiner Zimmermann, Michael Connaughton, Cristobal Pedregal-Martin, Engin Senel, Cemil Alper, Giuseppe Calabrese, David Ebert, Dmitrij Dolgušin

Collaborators

Project background

Keeping the LHC and the rest of the accelerator complex at CERN running efficiently requires state-of-the-art control systems. A complex IIOT system has been developed to persist this data, making it possible for engineers to gain insights about temperatures, magnetic-field strengths, and more. This plays a vital role in ensuring the highest levels of operational efficiency.

The current system for persisting, accessing and analysing this data is based on Oracle Database. Today, significant effort is dedicated to improving performance and coping with increasing demand — in terms of data volume, analysis and exploration of bigger data sets.

Recent progress

During 2020, the team focused on three main aspects:

(i) Assessing extended capabilities for supporting custom data types when importing and dealing with complex data schemas on parquet files. The results showed that complex parquet schemas can now be handled and automatically ingested by Oracle Data Warehouse technology. This enabled us to work with data coming from control-system devices based on CERN's custom controls middleware, CMW.

(ii) Deploying a hybrid solution based on standard and external table partitions, with the objective of improving performance for data retrieval. This improved the performance compared to the solution based on external partitions and is thus a good fit for the most demanding real-time applications consuming data from the IIoT control system.

(iii) Increasing data retrieval/analytics complexity using real-life scenarios to explore Oracle Analytics technologies as a front-end solution for control engineers and equipment experts.

We presented the outcomes of this work at events across Asia Pacific, Europe, Middle East, Africa and North America. Through our participation in these events, CERN openlab could share insights obtained with representatives of large companies from various industries, ranging from banking and telecommunications to research and educational institutions.

Next steps

The team will increase the data volume and complexity to assess the capabilities of the new Autonomous Database for drag-and-drop data loading and transformation, as well as for automatic insight discovery. This will be done with the goal of going one step further in terms of automating processes to improve operational efficiency.


Presentations

    E. Grancher, M. Martin, S. Masson, Research Analytics at Scale: CERN’s Experience with Oracle Cloud Solutions (16 January). Presented at Oracle OpenWorld 2019, London, 2019. cern.ch/go/S6qf
    A. Mendelsohn (Oracle), E. Grancher, M. Martin, Oracle Autonomous Database Keynote (16 January). Oracle OpenWorld 2019, London, 2019.
    M. Martin, J. Abel (Oracle), Enterprise Challenges and Outcomes (17 January). Presented at Oracle OpenWorld 2019, London, 2019.
    S. Masson, M. Martin, Managing one of the largest IoT systems in the world with Oracle Autonomous Technologies (18 September). Presented at Oracle OpenWorld 2019, San Francisco, 2019. cern.ch/go/SBc9
    D. Ebert (Oracle), M. Martin, A. Nappi, Advancing research with Oracle Cloud (17 September). Presented at Oracle OpenWorld 2019, San Francisco, 2019. cern.ch/go/9ZCg
    M. Martin, R. Zimmermann (Oracle), J. Otto (IDS GmbH), Oracle Autonomous Data Warehouse: Customer Panel (17 September). Presented at Oracle OpenWorld 2019, San Francisco, 2019. cern.ch/go/nm9B
    S. Masson, M. Martin, Oracle Autonomous Data Warehouse and CERN Accelerator Control Systems (25 November). Presented at Modern Cloud Day, Paris, 2019.
    M. Martin, M. Connaughton (Oracle), Big Data Analytics and the Large Hadron Collider (26 November). Presented at Oracle Digital Days 2019, Dublin, 2019.
    M. Martin, Big Data, AI and Machine Learning at CERN (27 November). Presented at Trinity College Dublin and ADAPT Center, Dublin, 2019.
    M. Martin, M. Connaughton (Oracle), Big Data Analytics and the Large Hadron Collider (27 November). Presented at the National Analytics Summit 2019, Dublin, 2019. cern.ch/go/CF9p
    M. Martin Marquez, Boosting Complex IoT Analysis with Oracle Autonomous Data Warehouse Cloud (June). Presented at Oracle Global Leaders Meeting – EMEA, Budapest, 2018.
    E. Grancher, M. Martin Marquez, S. Masson, Boosting Complex IoT Analysis with Oracle Autonomous Data Warehouse Cloud (23 October). Presented at Oracle Openworld 2018, San Francisco, 2018. cern.ch/go/RBZ6
    E. Grancher, M. Martin Marquez, S. Masson, Managing one of the largest IoT Systems in the world (December). Presented at Oracle Global Leaders Meeting – EMEA, Sevilla, 2018.
    M. M. Marquez, Managing 1 PB of Object Storage in the Oracle Cloud (22 July). Presented at Oracle Cloud Platform Virtual Summit: The Modern Data Warehouse, 2020. cern.ch/go/d7Vj
    M. M. Marquez, Managing 1 PB of Object Storage in the Oracle Cloud (9 July). Presented at Oracle Cloud Platform Virtual Summit: The Modern Data Warehouse North America, 2020. cern.ch/go/QK9Nj
    M. M. Marquez, Managing 1 PB of Object Storage in the Oracle Cloud (6 August). Presented at Oracle Cloud Platform Virtual Summit: The Modern Data Warehouse EMEA, 2020. cern.ch/go/r7cX
    M. M. Marquez, Managing 1 PB of Object Storage in the Oracle Cloud (15 August). Presented at Oracle Cloud Platform Virtual Summit: The Modern Data Warehouse JAPAC, 2020. cern.ch/go/t6JQ
    M. M. Marquez, CERN Industrial IoT data with Oracle Autonomous Data Warehouse (2 June). Presented at the Oracle Global Leaders Summer Meeting, Miami, 2020. cern.ch/go/CH9d
    S. Masson, La gestion des données en gros volume au quotidien (19 May). Presented at the Oracle Technology Day 2020 : Data, innovation et continuité d’activité, 2020. http://cern.ch/go/8xkP