Scheduler technologies in support of high performance data analysis

Albert Reuther, Chansup Byun, William Arcand, David Bestor, Bill Bergeron, Matthew Hubbell, Michael Jones, Peter Michaleas, Andrew Prout, Antonio Rosa, Jeremy Kepner

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

17 Scopus citations

Abstract

Job schedulers are a key component of scalable computing infrastructures. They orchestrate all of the work executed on the computing infrastructure and directly impact the effectiveness of the system. Recently, job workloads have diversified from long-running, synchronously-parallel simulations to include short-duration, independently parallel high performance data analysis (HPDA) jobs. Each of these job types requires different features and scheduler tuning to run efficiently. A number of schedulers have been developed to address both job workload and computing system heterogeneity. High performance computing (HPC) schedulers were designed to schedule large-scale scientific modeling and simulations on supercomputers. Big Data schedulers were designed to schedule data processing and analytic jobs on clusters. This paper compares and contrasts the features of HPC and Big Data schedulers with a focus on accommodating both scientific computing and high performance data analytic workloads. Job latency is critical for the efficient utilization of scalable computing infrastructures, and this paper presents the results of job launch benchmarking of several current schedulers: Slurm, Son of Grid Engine, Mesos, and Yarn. We find that all of these schedulers have low utilization for short-running jobs. Furthermore, employing multilevel scheduling significantly improves the utilization across all schedulers.

Original languageEnglish
Title of host publication2016 IEEE High Performance Extreme Computing Conference, HPEC 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781509035250
DOIs
StatePublished - Nov 28 2016
Externally publishedYes
Event2016 IEEE High Performance Extreme Computing Conference, HPEC 2016 - Waltham, United States
Duration: Sep 13 2016Sep 15 2016

Publication series

Name2016 IEEE High Performance Extreme Computing Conference, HPEC 2016

Conference

Conference2016 IEEE High Performance Extreme Computing Conference, HPEC 2016
Country/TerritoryUnited States
CityWaltham
Period09/13/1609/15/16

Keywords

  • Scheduler
  • data analytics
  • high performance computing
  • job scheduler
  • resource manager

Fingerprint

Dive into the research topics of 'Scheduler technologies in support of high performance data analysis'. Together they form a unique fingerprint.

Cite this