Driving big data with big compute

Chansup Byun; William Arcand; David Bestor; Bill Bergeron; Matthew Hubbell; Jeremy Kepner; Andrew McCabe; Peter Michaleas; Julie Mullen; David O'Gwynn; Andrew Prout; Albert Reuther; Antonio Rosa; Charles Yee

doi:10.1109/HPEC.2012.6408678

Driving big data with big compute

Chansup Byun, William Arcand, David Bestor, Bill Bergeron, Matthew Hubbell, Jeremy Kepner, Andrew McCabe, Peter Michaleas, Julie Mullen, David O'Gwynn, Andrew Prout, Albert Reuther, Antonio Rosa, Charles Yee

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

29 Scopus citations

Abstract

Big Data (as embodied by Hadoop clusters) and Big Compute (as embodied by MPI clusters) provide unique capabilities for storing and processing large volumes of data. Hadoop clusters make distributed computing readily accessible to the Java community and MPI clusters provide high parallel efficiency for compute intensive workloads. Bringing the big data and big compute communities together is an active area of research. The LLGrid team has developed and deployed a number of technologies that aim to provide the best of both worlds. LLGrid MapReduce allows the map/reduce parallel programming model to be used quickly and efficiently in any language on any compute cluster. D4M (Dynamic Distributed Dimensional Data Model) provided a high level distributed arrays interface to the Apache Accumulo database. The accessibility of these technologies is assessed by measuring the effort to use these tools and is typically a few lines of code. The performance is assessed by measuring the insert rate into the Accumulo database. Using these tools a database insert rate of 4M inserts/second has been achieved on an 8 node cluster.

Original language	English
Title of host publication	2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012
DOIs	https://doi.org/10.1109/HPEC.2012.6408678
State	Published - 2012
Externally published	Yes
Event	2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012 - Waltham, MA, United States Duration: Sep 10 2012 → Sep 12 2012

Publication series

Name	2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012

Conference

Conference	2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012
Country/Territory	United States
City	Waltham, MA
Period	09/10/12 → 09/12/12

Keywords

LLGridMapReduce
concurrent query
d4m
hdfs
parallel ingestion
parallel matlab
scheduler

Access to Document

10.1109/HPEC.2012.6408678

Cite this

Byun, C., Arcand, W., Bestor, D., Bergeron, B., Hubbell, M., Kepner, J., McCabe, A., Michaleas, P., Mullen, J., O'Gwynn, D., Prout, A., Reuther, A., Rosa, A., & Yee, C. (2012). Driving big data with big compute. In 2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012 [6408678] (2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012). https://doi.org/10.1109/HPEC.2012.6408678

@inproceedings{73dccfe1c79243c6abd2d4a100a5237b,

title = "Driving big data with big compute",

abstract = "Big Data (as embodied by Hadoop clusters) and Big Compute (as embodied by MPI clusters) provide unique capabilities for storing and processing large volumes of data. Hadoop clusters make distributed computing readily accessible to the Java community and MPI clusters provide high parallel efficiency for compute intensive workloads. Bringing the big data and big compute communities together is an active area of research. The LLGrid team has developed and deployed a number of technologies that aim to provide the best of both worlds. LLGrid MapReduce allows the map/reduce parallel programming model to be used quickly and efficiently in any language on any compute cluster. D4M (Dynamic Distributed Dimensional Data Model) provided a high level distributed arrays interface to the Apache Accumulo database. The accessibility of these technologies is assessed by measuring the effort to use these tools and is typically a few lines of code. The performance is assessed by measuring the insert rate into the Accumulo database. Using these tools a database insert rate of 4M inserts/second has been achieved on an 8 node cluster.",

keywords = "LLGridMapReduce, concurrent query, d4m, hdfs, parallel ingestion, parallel matlab, scheduler",

author = "Chansup Byun and William Arcand and David Bestor and Bill Bergeron and Matthew Hubbell and Jeremy Kepner and Andrew McCabe and Peter Michaleas and Julie Mullen and David O'Gwynn and Andrew Prout and Albert Reuther and Antonio Rosa and Charles Yee",

year = "2012",

doi = "10.1109/HPEC.2012.6408678",

language = "English",

isbn = "9781467315760",

series = "2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012",

booktitle = "2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012",

note = "null ; Conference date: 10-09-2012 Through 12-09-2012",

}

Byun, C, Arcand, W, Bestor, D, Bergeron, B, Hubbell, M, Kepner, J, McCabe, A, Michaleas, P, Mullen, J, O'Gwynn, D, Prout, A, Reuther, A, Rosa, A & Yee, C 2012, Driving big data with big compute. in 2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012., 6408678, 2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012, 2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012, Waltham, MA, United States, 09/10/12. https://doi.org/10.1109/HPEC.2012.6408678

TY - GEN

T1 - Driving big data with big compute

AU - Byun, Chansup

AU - Arcand, William

AU - Bestor, David

AU - Bergeron, Bill

AU - Hubbell, Matthew

AU - Kepner, Jeremy

AU - McCabe, Andrew

AU - Michaleas, Peter

AU - Mullen, Julie

AU - O'Gwynn, David

AU - Prout, Andrew

AU - Reuther, Albert

AU - Rosa, Antonio

AU - Yee, Charles

PY - 2012

Y1 - 2012

N2 - Big Data (as embodied by Hadoop clusters) and Big Compute (as embodied by MPI clusters) provide unique capabilities for storing and processing large volumes of data. Hadoop clusters make distributed computing readily accessible to the Java community and MPI clusters provide high parallel efficiency for compute intensive workloads. Bringing the big data and big compute communities together is an active area of research. The LLGrid team has developed and deployed a number of technologies that aim to provide the best of both worlds. LLGrid MapReduce allows the map/reduce parallel programming model to be used quickly and efficiently in any language on any compute cluster. D4M (Dynamic Distributed Dimensional Data Model) provided a high level distributed arrays interface to the Apache Accumulo database. The accessibility of these technologies is assessed by measuring the effort to use these tools and is typically a few lines of code. The performance is assessed by measuring the insert rate into the Accumulo database. Using these tools a database insert rate of 4M inserts/second has been achieved on an 8 node cluster.

AB - Big Data (as embodied by Hadoop clusters) and Big Compute (as embodied by MPI clusters) provide unique capabilities for storing and processing large volumes of data. Hadoop clusters make distributed computing readily accessible to the Java community and MPI clusters provide high parallel efficiency for compute intensive workloads. Bringing the big data and big compute communities together is an active area of research. The LLGrid team has developed and deployed a number of technologies that aim to provide the best of both worlds. LLGrid MapReduce allows the map/reduce parallel programming model to be used quickly and efficiently in any language on any compute cluster. D4M (Dynamic Distributed Dimensional Data Model) provided a high level distributed arrays interface to the Apache Accumulo database. The accessibility of these technologies is assessed by measuring the effort to use these tools and is typically a few lines of code. The performance is assessed by measuring the insert rate into the Accumulo database. Using these tools a database insert rate of 4M inserts/second has been achieved on an 8 node cluster.

KW - LLGridMapReduce

KW - concurrent query

KW - d4m

KW - hdfs

KW - parallel ingestion

KW - parallel matlab

KW - scheduler

UR - http://www.scopus.com/inward/record.url?scp=84873557151&partnerID=8YFLogxK

U2 - 10.1109/HPEC.2012.6408678

DO - 10.1109/HPEC.2012.6408678

M3 - Conference contribution

AN - SCOPUS:84873557151

SN - 9781467315760

T3 - 2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012

BT - 2012 IEEE Conference on High Performance Extreme Computing, HPEC 2012

Y2 - 10 September 2012 through 12 September 2012

ER -

Driving big data with big compute

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this