LLMapReduce: Multi-level map-reduce for high performance data analysis

Chansup Byun; Jeremy Kepner; William Arcand; David Bestor; Bill Bergeron; Vijay Gadepally; Matthew Hubbell; Peter Michaleas; Julie Mullen; Andrew Prout; Antonio Rosa; Charles Yee; Albert Reuther

doi:10.1109/HPEC.2016.7761618

LLMapReduce: Multi-level map-reduce for high performance data analysis

Chansup Byun, Jeremy Kepner, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Matthew Hubbell, Peter Michaleas, Julie Mullen, Andrew Prout, Antonio Rosa, Charles Yee, Albert Reuther

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

16 Scopus citations

Abstract

The map-reduce parallel programming model has become extremely popular in the big data community. Many big data workloads can benefit from the enhanced performance offered by supercomputers. LLMapReduce provides the familiar map-reduce parallel programming model to big data users running on a supercomputer. LLMapReduce dramatically simplifies map-reduce programming by providing simple parallel programming capability in one line of code. LLMapReduce supports all programming languages and many schedulers. LLMapReduce can work with any application without the need to modify the application. Furthermore, LLMapReduce can overcome scaling limits in the map-reduce parallel programming model via options that allow the user to switch to the more efficient single-program-multiple-data (SPMD) parallel programming model. These features allow users to reduce the computational overhead by more than 10x compared to standard map-reduce for certain applications. LLMapReduce is widely used by hundreds of users at MIT. Currently LLMapReduce works with several schedulers such as SLURM, Grid Engine and LSF.

Original language	English
Title of host publication	2016 IEEE High Performance Extreme Computing Conference, HPEC 2016
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781509035250
DOIs	https://doi.org/10.1109/HPEC.2016.7761618
State	Published - Nov 28 2016
Externally published	Yes
Event	2016 IEEE High Performance Extreme Computing Conference, HPEC 2016 - Waltham, United States Duration: Sep 13 2016 → Sep 15 2016

Publication series

Name	2016 IEEE High Performance Extreme Computing Conference, HPEC 2016

Conference

Conference	2016 IEEE High Performance Extreme Computing Conference, HPEC 2016
Country/Territory	United States
City	Waltham
Period	09/13/16 → 09/15/16

Keywords

Grid Engine
LLMapReduce
LSF
SLURM
map-reduce
performance
scheduler

Access to Document

10.1109/HPEC.2016.7761618

Cite this

Byun, C., Kepner, J., Arcand, W., Bestor, D., Bergeron, B., Gadepally, V., Hubbell, M., Michaleas, P., Mullen, J., Prout, A., Rosa, A., Yee, C., & Reuther, A. (2016). LLMapReduce: Multi-level map-reduce for high performance data analysis. In 2016 IEEE High Performance Extreme Computing Conference, HPEC 2016 Article 7761618 (2016 IEEE High Performance Extreme Computing Conference, HPEC 2016). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/HPEC.2016.7761618

@inproceedings{b3d1a692ca3944d8b716b1207f1b6a53,

title = "LLMapReduce: Multi-level map-reduce for high performance data analysis",

abstract = "The map-reduce parallel programming model has become extremely popular in the big data community. Many big data workloads can benefit from the enhanced performance offered by supercomputers. LLMapReduce provides the familiar map-reduce parallel programming model to big data users running on a supercomputer. LLMapReduce dramatically simplifies map-reduce programming by providing simple parallel programming capability in one line of code. LLMapReduce supports all programming languages and many schedulers. LLMapReduce can work with any application without the need to modify the application. Furthermore, LLMapReduce can overcome scaling limits in the map-reduce parallel programming model via options that allow the user to switch to the more efficient single-program-multiple-data (SPMD) parallel programming model. These features allow users to reduce the computational overhead by more than 10x compared to standard map-reduce for certain applications. LLMapReduce is widely used by hundreds of users at MIT. Currently LLMapReduce works with several schedulers such as SLURM, Grid Engine and LSF.",

keywords = "Grid Engine, LLMapReduce, LSF, SLURM, map-reduce, performance, scheduler",

author = "Chansup Byun and Jeremy Kepner and William Arcand and David Bestor and Bill Bergeron and Vijay Gadepally and Matthew Hubbell and Peter Michaleas and Julie Mullen and Andrew Prout and Antonio Rosa and Charles Yee and Albert Reuther",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 2016 IEEE High Performance Extreme Computing Conference, HPEC 2016 ; Conference date: 13-09-2016 Through 15-09-2016",

year = "2016",

month = nov,

day = "28",

doi = "10.1109/HPEC.2016.7761618",

language = "English",

series = "2016 IEEE High Performance Extreme Computing Conference, HPEC 2016",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2016 IEEE High Performance Extreme Computing Conference, HPEC 2016",

}

Byun, C, Kepner, J, Arcand, W, Bestor, D, Bergeron, B, Gadepally, V, Hubbell, M, Michaleas, P, Mullen, J, Prout, A, Rosa, A, Yee, C & Reuther, A 2016, LLMapReduce: Multi-level map-reduce for high performance data analysis. in 2016 IEEE High Performance Extreme Computing Conference, HPEC 2016., 7761618, 2016 IEEE High Performance Extreme Computing Conference, HPEC 2016, Institute of Electrical and Electronics Engineers Inc., 2016 IEEE High Performance Extreme Computing Conference, HPEC 2016, Waltham, United States, 09/13/16. https://doi.org/10.1109/HPEC.2016.7761618

LLMapReduce: Multi-level map-reduce for high performance data analysis. / Byun, Chansup; Kepner, Jeremy; Arcand, William et al.
2016 IEEE High Performance Extreme Computing Conference, HPEC 2016. Institute of Electrical and Electronics Engineers Inc., 2016. 7761618 (2016 IEEE High Performance Extreme Computing Conference, HPEC 2016).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - LLMapReduce

T2 - 2016 IEEE High Performance Extreme Computing Conference, HPEC 2016

AU - Byun, Chansup

AU - Kepner, Jeremy

AU - Arcand, William

AU - Bestor, David

AU - Bergeron, Bill

AU - Gadepally, Vijay

AU - Hubbell, Matthew

AU - Michaleas, Peter

AU - Mullen, Julie

AU - Prout, Andrew

AU - Rosa, Antonio

AU - Yee, Charles

AU - Reuther, Albert

PY - 2016/11/28

Y1 - 2016/11/28

N2 - The map-reduce parallel programming model has become extremely popular in the big data community. Many big data workloads can benefit from the enhanced performance offered by supercomputers. LLMapReduce provides the familiar map-reduce parallel programming model to big data users running on a supercomputer. LLMapReduce dramatically simplifies map-reduce programming by providing simple parallel programming capability in one line of code. LLMapReduce supports all programming languages and many schedulers. LLMapReduce can work with any application without the need to modify the application. Furthermore, LLMapReduce can overcome scaling limits in the map-reduce parallel programming model via options that allow the user to switch to the more efficient single-program-multiple-data (SPMD) parallel programming model. These features allow users to reduce the computational overhead by more than 10x compared to standard map-reduce for certain applications. LLMapReduce is widely used by hundreds of users at MIT. Currently LLMapReduce works with several schedulers such as SLURM, Grid Engine and LSF.

AB - The map-reduce parallel programming model has become extremely popular in the big data community. Many big data workloads can benefit from the enhanced performance offered by supercomputers. LLMapReduce provides the familiar map-reduce parallel programming model to big data users running on a supercomputer. LLMapReduce dramatically simplifies map-reduce programming by providing simple parallel programming capability in one line of code. LLMapReduce supports all programming languages and many schedulers. LLMapReduce can work with any application without the need to modify the application. Furthermore, LLMapReduce can overcome scaling limits in the map-reduce parallel programming model via options that allow the user to switch to the more efficient single-program-multiple-data (SPMD) parallel programming model. These features allow users to reduce the computational overhead by more than 10x compared to standard map-reduce for certain applications. LLMapReduce is widely used by hundreds of users at MIT. Currently LLMapReduce works with several schedulers such as SLURM, Grid Engine and LSF.

KW - Grid Engine

KW - LLMapReduce

KW - LSF

KW - SLURM

KW - map-reduce

KW - performance

KW - scheduler

UR - http://www.scopus.com/inward/record.url?scp=85007044858&partnerID=8YFLogxK

U2 - 10.1109/HPEC.2016.7761618

DO - 10.1109/HPEC.2016.7761618

M3 - Conference contribution

AN - SCOPUS:85007044858

T3 - 2016 IEEE High Performance Extreme Computing Conference, HPEC 2016

BT - 2016 IEEE High Performance Extreme Computing Conference, HPEC 2016

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 13 September 2016 through 15 September 2016

ER -

Byun C, Kepner J, Arcand W, Bestor D, Bergeron B, Gadepally V et al. LLMapReduce: Multi-level map-reduce for high performance data analysis. In 2016 IEEE High Performance Extreme Computing Conference, HPEC 2016. Institute of Electrical and Electronics Engineers Inc. 2016. 7761618. (2016 IEEE High Performance Extreme Computing Conference, HPEC 2016). doi: 10.1109/HPEC.2016.7761618

LLMapReduce: Multi-level map-reduce for high performance data analysis

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this