SLoG: a large-scale logging middleware for HPC and Big Data Convergence

Matri, Pierre and Carns, Philip and Ross, Robert and Costan, Alexandru and Pérez Hernández, María de los Santos and Antoniu, Gabriel (2018). SLoG: a large-scale logging middleware for HPC and Big Data Convergence. In: "38th International Conference on Distributed Computing Systems (ICDCS)", 2-6 Jul 2018, Viena, Austria. pp. 1507-1512. https://doi.org/10.1109/ICDCS.2018.00156.

Description

Title: SLoG: a large-scale logging middleware for HPC and Big Data Convergence
Author/s:
  • Matri, Pierre
  • Carns, Philip
  • Ross, Robert
  • Costan, Alexandru
  • Pérez Hernández, María de los Santos
  • Antoniu, Gabriel
Item Type: Presentation at Congress or Conference (Article)
Event Title: 38th International Conference on Distributed Computing Systems (ICDCS)
Event Dates: 2-6 Jul 2018
Event Location: Viena, Austria
Title of Book: ICDCS 2018: 38th IEEE International Conference on Distributed Computing Systems
Título de Revista/Publicación: Proceedings of the IEEE International Conference on Distributed Computing Systems (ICDCS)
Date: 2018
ISSN: 978-1-5386-6871-9
Subjects:
Faculty: E.T.S. de Ingenieros Informáticos (UPM)
Department: Arquitectura y Tecnología de Sistemas Informáticos
Creative Commons Licenses: Recognition - No derivative works - Non commercial

Full text

[img]
Preview
PDF - Requires a PDF viewer, such as GSview, Xpdf or Adobe Acrobat Reader
Download (195kB) | Preview

Abstract

Cloud developers traditionally rely on purposespecific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer very low performance when subject to highly-concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids or discrete event generators. In this paper we tackle this issue. We present SLoG, a shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG design on up to 100,000 cores of the Theta supercomputer: it demonstrates high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.

Funding Projects

TypeCodeAcronymLeaderTitle
Horizon 2020MSCA-ITN-2014-642963BigStorageUnspecifiedBigStorage: Storage-based Convergence between HPC and Cloud to handle Big Data

More information

Item ID: 54692
DC Identifier: http://oa.upm.es/54692/
OAI Identifier: oai:oa.upm.es:54692
DOI: 10.1109/ICDCS.2018.00156
Official URL: https://ieeexplore.ieee.org/document/8416419
Deposited by: Memoria Investigacion
Deposited on: 16 May 2019 11:49
Last Modified: 16 May 2019 11:49
  • Logo InvestigaM (UPM)
  • Logo GEOUP4
  • Logo Open Access
  • Open Access
  • Logo Sherpa/Romeo
    Check whether the anglo-saxon journal in which you have published an article allows you to also publish it under open access.
  • Logo Dulcinea
    Check whether the spanish journal in which you have published an article allows you to also publish it under open access.
  • Logo de Recolecta
  • Logo del Observatorio I+D+i UPM
  • Logo de OpenCourseWare UPM