Optimizing cross-platform data movement

Sebastian Kruse, Zoi Kaoudi, Jorge Arnulfo Quiane-Ruiz, Sanjay Chawla, Felix Naumann, Bertty Contreras-Rojas

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Scopus citations

Abstract

Data analytics are moving beyond the limits of a single data processing platform. A cross-platform query optimizer is necessary to enable applications to run their tasks over multiple platforms efficiently and in a platform-agnostic manner. For the optimizer to be effective, it must consider data movement costs across different data processing platforms. In this paper, we present the graph-based data movement strategy used by Rheem, our open-source cross-platform system. In particular, we (i) model the data movement problem as a new graph problem, which we prove to be NP-hard, and (ii) propose a novel graph exploration algorithm, which allows Rheem to discover multiple hidden opportunities for cross-platform data processing.

Original languageEnglish
Title of host publicationProceedings - 2019 IEEE 35th International Conference on Data Engineering, ICDE 2019
PublisherIEEE Computer Society
Pages1642-1645
Number of pages4
ISBN (Electronic)9781538674741
DOIs
StatePublished - Apr 2019
Externally publishedYes
Event35th IEEE International Conference on Data Engineering, ICDE 2019 - Macau, China
Duration: Apr 8 2019Apr 11 2019

Publication series

NameProceedings - International Conference on Data Engineering
Volume2019-April
ISSN (Print)1084-4627

Conference

Conference35th IEEE International Conference on Data Engineering, ICDE 2019
Country/TerritoryChina
CityMacau
Period04/8/1904/11/19

Keywords

  • Cross-platform
  • Data movement
  • Polystore
  • Query opimization

Fingerprint

Dive into the research topics of 'Optimizing cross-platform data movement'. Together they form a unique fingerprint.

Cite this