Achieving 100,000,000 database inserts per second using Accumulo and D4M

Jeremy Kepner; William Arcand; David Bestor; Bill Bergeron; Chansup Byun; Vijay Gadepally; Matthew Hubbell; Peter Michaleas; Julie Mullen; Andrew Prout; Albert Reuther; Antonio Rosa; Charles Yee

doi:10.1109/HPEC.2014.7040945

Achieving 100,000,000 database inserts per second using Accumulo and D4M

Jeremy Kepner, William Arcand, David Bestor, Bill Bergeron, Chansup Byun, Vijay Gadepally, Matthew Hubbell, Peter Michaleas, Julie Mullen, Andrew Prout, Albert Reuther, Antonio Rosa, Charles Yee

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

53 Scopus citations

Abstract

The Apache Accumulo database is an open source relaxed consistency database that is widely used for government applications. Accumulo is designed to deliver high performance on unstructured data such as graphs of network data. This paper tests the performance of Accumulo using data from the Graph500 benchmark. The Dynamic Distributed Dimensional Data Model (D4M) software is used to implement the benchmark on a 216-node cluster running the MIT SuperCloud software stack. A peak performance of over 100,000,000 database inserts per second was achieved which is 100× larger than the highest previously published value for any other database. The performance scales linearly with the number of ingest clients, number of database servers, and data size. The performance was achieved by adapting several supercomputing techniques to this application: distributed arrays, domain decomposition, adaptive load balancing, and single-program-multiple-data programming.

Original language	English
Title of host publication	2014 IEEE High Performance Extreme Computing Conference, HPEC 2014
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781479962334
DOIs	https://doi.org/10.1109/HPEC.2014.7040945
State	Published - Feb 11 2014
Externally published	Yes
Event	2014 IEEE High Performance Extreme Computing Conference, HPEC 2014 - Waltham, United States Duration: Sep 9 2014 → Sep 11 2014

Publication series

Name	2014 IEEE High Performance Extreme Computing Conference, HPEC 2014

Conference

Conference	2014 IEEE High Performance Extreme Computing Conference, HPEC 2014
Country/Territory	United States
City	Waltham
Period	09/9/14 → 09/11/14

Keywords

Accumulo
Big Data
D4M
Graph500
Hadoop
MIT SuperCloud

Access to Document

10.1109/HPEC.2014.7040945

Cite this

Kepner, J., Arcand, W., Bestor, D., Bergeron, B., Byun, C., Gadepally, V., Hubbell, M., Michaleas, P., Mullen, J., Prout, A., Reuther, A., Rosa, A., & Yee, C. (2014). Achieving 100,000,000 database inserts per second using Accumulo and D4M. In 2014 IEEE High Performance Extreme Computing Conference, HPEC 2014 Article 7040945 (2014 IEEE High Performance Extreme Computing Conference, HPEC 2014). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/HPEC.2014.7040945

@inproceedings{1391e5e576084ea99b98c3bcfc7aad3a,

title = "Achieving 100,000,000 database inserts per second using Accumulo and D4M",

abstract = "The Apache Accumulo database is an open source relaxed consistency database that is widely used for government applications. Accumulo is designed to deliver high performance on unstructured data such as graphs of network data. This paper tests the performance of Accumulo using data from the Graph500 benchmark. The Dynamic Distributed Dimensional Data Model (D4M) software is used to implement the benchmark on a 216-node cluster running the MIT SuperCloud software stack. A peak performance of over 100,000,000 database inserts per second was achieved which is 100× larger than the highest previously published value for any other database. The performance scales linearly with the number of ingest clients, number of database servers, and data size. The performance was achieved by adapting several supercomputing techniques to this application: distributed arrays, domain decomposition, adaptive load balancing, and single-program-multiple-data programming.",

keywords = "Accumulo, Big Data, D4M, Graph500, Hadoop, MIT SuperCloud",

author = "Jeremy Kepner and William Arcand and David Bestor and Bill Bergeron and Chansup Byun and Vijay Gadepally and Matthew Hubbell and Peter Michaleas and Julie Mullen and Andrew Prout and Albert Reuther and Antonio Rosa and Charles Yee",

note = "Publisher Copyright: {\textcopyright} 2014 IEEE.; 2014 IEEE High Performance Extreme Computing Conference, HPEC 2014 ; Conference date: 09-09-2014 Through 11-09-2014",

year = "2014",

month = feb,

day = "11",

doi = "10.1109/HPEC.2014.7040945",

language = "English",

series = "2014 IEEE High Performance Extreme Computing Conference, HPEC 2014",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2014 IEEE High Performance Extreme Computing Conference, HPEC 2014",

}

Kepner, J, Arcand, W, Bestor, D, Bergeron, B, Byun, C, Gadepally, V, Hubbell, M, Michaleas, P, Mullen, J, Prout, A, Reuther, A, Rosa, A & Yee, C 2014, Achieving 100,000,000 database inserts per second using Accumulo and D4M. in 2014 IEEE High Performance Extreme Computing Conference, HPEC 2014., 7040945, 2014 IEEE High Performance Extreme Computing Conference, HPEC 2014, Institute of Electrical and Electronics Engineers Inc., 2014 IEEE High Performance Extreme Computing Conference, HPEC 2014, Waltham, United States, 09/9/14. https://doi.org/10.1109/HPEC.2014.7040945

Achieving 100,000,000 database inserts per second using Accumulo and D4M. / Kepner, Jeremy; Arcand, William; Bestor, David et al.
2014 IEEE High Performance Extreme Computing Conference, HPEC 2014. Institute of Electrical and Electronics Engineers Inc., 2014. 7040945 (2014 IEEE High Performance Extreme Computing Conference, HPEC 2014).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Achieving 100,000,000 database inserts per second using Accumulo and D4M

AU - Kepner, Jeremy

AU - Arcand, William

AU - Bestor, David

AU - Bergeron, Bill

AU - Byun, Chansup

AU - Gadepally, Vijay

AU - Hubbell, Matthew

AU - Michaleas, Peter

AU - Mullen, Julie

AU - Prout, Andrew

AU - Reuther, Albert

AU - Rosa, Antonio

AU - Yee, Charles

PY - 2014/2/11

Y1 - 2014/2/11

N2 - The Apache Accumulo database is an open source relaxed consistency database that is widely used for government applications. Accumulo is designed to deliver high performance on unstructured data such as graphs of network data. This paper tests the performance of Accumulo using data from the Graph500 benchmark. The Dynamic Distributed Dimensional Data Model (D4M) software is used to implement the benchmark on a 216-node cluster running the MIT SuperCloud software stack. A peak performance of over 100,000,000 database inserts per second was achieved which is 100× larger than the highest previously published value for any other database. The performance scales linearly with the number of ingest clients, number of database servers, and data size. The performance was achieved by adapting several supercomputing techniques to this application: distributed arrays, domain decomposition, adaptive load balancing, and single-program-multiple-data programming.

AB - The Apache Accumulo database is an open source relaxed consistency database that is widely used for government applications. Accumulo is designed to deliver high performance on unstructured data such as graphs of network data. This paper tests the performance of Accumulo using data from the Graph500 benchmark. The Dynamic Distributed Dimensional Data Model (D4M) software is used to implement the benchmark on a 216-node cluster running the MIT SuperCloud software stack. A peak performance of over 100,000,000 database inserts per second was achieved which is 100× larger than the highest previously published value for any other database. The performance scales linearly with the number of ingest clients, number of database servers, and data size. The performance was achieved by adapting several supercomputing techniques to this application: distributed arrays, domain decomposition, adaptive load balancing, and single-program-multiple-data programming.

KW - Accumulo

KW - Big Data

KW - D4M

KW - Graph500

KW - Hadoop

KW - MIT SuperCloud

UR - http://www.scopus.com/inward/record.url?scp=84946685320&partnerID=8YFLogxK

U2 - 10.1109/HPEC.2014.7040945

DO - 10.1109/HPEC.2014.7040945

M3 - Conference contribution

AN - SCOPUS:84946685320

T3 - 2014 IEEE High Performance Extreme Computing Conference, HPEC 2014

BT - 2014 IEEE High Performance Extreme Computing Conference, HPEC 2014

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2014 IEEE High Performance Extreme Computing Conference, HPEC 2014

Y2 - 9 September 2014 through 11 September 2014

ER -

Kepner J, Arcand W, Bestor D, Bergeron B, Byun C, Gadepally V et al. Achieving 100,000,000 database inserts per second using Accumulo and D4M. In 2014 IEEE High Performance Extreme Computing Conference, HPEC 2014. Institute of Electrical and Electronics Engineers Inc. 2014. 7040945. (2014 IEEE High Performance Extreme Computing Conference, HPEC 2014). doi: 10.1109/HPEC.2014.7040945

Achieving 100,000,000 database inserts per second using Accumulo and D4M

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this