D4M 2.0 schema: A general purpose high performance schema for the Accumulo database

Jeremy Kepner; Christian Anderson; William Arcand; David Bestor; Bill Bergeron; Chansup Byun; Matthew Hubbell; Peter Michaleas; Julie Mullen; David O'Gwynn; Andrew Prout; Albert Reuther; Antonio Rosa; Charles Yee

doi:10.1109/HPEC.2013.6670318

D4M 2.0 schema: A general purpose high performance schema for the Accumulo database

Jeremy Kepner, Christian Anderson, William Arcand, David Bestor, Bill Bergeron, Chansup Byun, Matthew Hubbell, Peter Michaleas, Julie Mullen, David O'Gwynn, Andrew Prout, Albert Reuther, Antonio Rosa, Charles Yee

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

41 Scopus citations

Abstract

Non-traditional, relaxed consistency, triple store databases are the backbone of many web companies (e.g., Google Big Table, Amazon Dynamo, and Facebook Cassandra). The Apache Accumulo database is a high performance open source relaxed consistency database that is widely used for government applications. Obtaining the full benefits of Accumulo requires using novel schemas. The Dynamic Distributed Dimensional Data Model (D4M)[http://www.mit. edu/∼kepner/D4M] provides a uniform mathematical framework based on associative arrays that encompasses both traditional (i.e., SQL) and non-traditional databases. For non-traditional databases D4M naturally leads to a general purpose schema that can be used to fully index and rapidly query every unique string in a dataset. The D4M 2.0 Schema has been applied with little or no customization to cyber, bioinformatics, scientific citation, free text, and social media data. The D4M 2.0 Schema is simple, requires minimal parsing, and achieves the highest published Accumulo ingest rates. The benefits of the D4M 2.0 Schema are independent of the D4M interface. Any interface to Accumulo can achieve these benefits by using the D4M 2.0 Schema.

Original language	English
Title of host publication	2013 IEEE High Performance Extreme Computing Conference, HPEC 2013
Publisher	IEEE Computer Society
ISBN (Print)	9781479913657
DOIs	https://doi.org/10.1109/HPEC.2013.6670318
State	Published - 2013
Externally published	Yes
Event	2013 IEEE High Performance Extreme Computing Conference, HPEC 2013 - Waltham, MA, United States Duration: Sep 10 2013 → Sep 12 2013

Publication series

Name	2013 IEEE High Performance Extreme Computing Conference, HPEC 2013

Conference

Conference	2013 IEEE High Performance Extreme Computing Conference, HPEC 2013
Country/Territory	United States
City	Waltham, MA
Period	09/10/13 → 09/12/13

Keywords

Accumulo
Big Data
D4M
Hadoop
NoSQL
database schema

Access to Document

10.1109/HPEC.2013.6670318

Cite this

Kepner, J., Anderson, C., Arcand, W., Bestor, D., Bergeron, B., Byun, C., Hubbell, M., Michaleas, P., Mullen, J., O'Gwynn, D., Prout, A., Reuther, A., Rosa, A., & Yee, C. (2013). D4M 2.0 schema: A general purpose high performance schema for the Accumulo database. In 2013 IEEE High Performance Extreme Computing Conference, HPEC 2013 [6670318] (2013 IEEE High Performance Extreme Computing Conference, HPEC 2013). IEEE Computer Society. https://doi.org/10.1109/HPEC.2013.6670318

@inproceedings{1e34c08548da4bef98edaf49adf79441,

title = "D4M 2.0 schema: A general purpose high performance schema for the Accumulo database",

abstract = "Non-traditional, relaxed consistency, triple store databases are the backbone of many web companies (e.g., Google Big Table, Amazon Dynamo, and Facebook Cassandra). The Apache Accumulo database is a high performance open source relaxed consistency database that is widely used for government applications. Obtaining the full benefits of Accumulo requires using novel schemas. The Dynamic Distributed Dimensional Data Model (D4M)[http://www.mit. edu/∼kepner/D4M] provides a uniform mathematical framework based on associative arrays that encompasses both traditional (i.e., SQL) and non-traditional databases. For non-traditional databases D4M naturally leads to a general purpose schema that can be used to fully index and rapidly query every unique string in a dataset. The D4M 2.0 Schema has been applied with little or no customization to cyber, bioinformatics, scientific citation, free text, and social media data. The D4M 2.0 Schema is simple, requires minimal parsing, and achieves the highest published Accumulo ingest rates. The benefits of the D4M 2.0 Schema are independent of the D4M interface. Any interface to Accumulo can achieve these benefits by using the D4M 2.0 Schema.",

keywords = "Accumulo, Big Data, D4M, Hadoop, NoSQL, database schema",

author = "Jeremy Kepner and Christian Anderson and William Arcand and David Bestor and Bill Bergeron and Chansup Byun and Matthew Hubbell and Peter Michaleas and Julie Mullen and David O'Gwynn and Andrew Prout and Albert Reuther and Antonio Rosa and Charles Yee",

year = "2013",

doi = "10.1109/HPEC.2013.6670318",

language = "English",

isbn = "9781479913657",

series = "2013 IEEE High Performance Extreme Computing Conference, HPEC 2013",

publisher = "IEEE Computer Society",

booktitle = "2013 IEEE High Performance Extreme Computing Conference, HPEC 2013",

note = "null ; Conference date: 10-09-2013 Through 12-09-2013",

}

Kepner, J, Anderson, C, Arcand, W, Bestor, D, Bergeron, B, Byun, C, Hubbell, M, Michaleas, P, Mullen, J, O'Gwynn, D, Prout, A, Reuther, A, Rosa, A & Yee, C 2013, D4M 2.0 schema: A general purpose high performance schema for the Accumulo database. in 2013 IEEE High Performance Extreme Computing Conference, HPEC 2013., 6670318, 2013 IEEE High Performance Extreme Computing Conference, HPEC 2013, IEEE Computer Society, 2013 IEEE High Performance Extreme Computing Conference, HPEC 2013, Waltham, MA, United States, 09/10/13. https://doi.org/10.1109/HPEC.2013.6670318

D4M 2.0 schema : A general purpose high performance schema for the Accumulo database. / Kepner, Jeremy; Anderson, Christian; Arcand, William et al.

2013 IEEE High Performance Extreme Computing Conference, HPEC 2013. IEEE Computer Society, 2013. 6670318 (2013 IEEE High Performance Extreme Computing Conference, HPEC 2013).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - D4M 2.0 schema

AU - Kepner, Jeremy

AU - Anderson, Christian

AU - Arcand, William

AU - Bestor, David

AU - Bergeron, Bill

AU - Byun, Chansup

AU - Hubbell, Matthew

AU - Michaleas, Peter

AU - Mullen, Julie

AU - O'Gwynn, David

AU - Prout, Andrew

AU - Reuther, Albert

AU - Rosa, Antonio

AU - Yee, Charles

PY - 2013

Y1 - 2013

N2 - Non-traditional, relaxed consistency, triple store databases are the backbone of many web companies (e.g., Google Big Table, Amazon Dynamo, and Facebook Cassandra). The Apache Accumulo database is a high performance open source relaxed consistency database that is widely used for government applications. Obtaining the full benefits of Accumulo requires using novel schemas. The Dynamic Distributed Dimensional Data Model (D4M)[http://www.mit. edu/∼kepner/D4M] provides a uniform mathematical framework based on associative arrays that encompasses both traditional (i.e., SQL) and non-traditional databases. For non-traditional databases D4M naturally leads to a general purpose schema that can be used to fully index and rapidly query every unique string in a dataset. The D4M 2.0 Schema has been applied with little or no customization to cyber, bioinformatics, scientific citation, free text, and social media data. The D4M 2.0 Schema is simple, requires minimal parsing, and achieves the highest published Accumulo ingest rates. The benefits of the D4M 2.0 Schema are independent of the D4M interface. Any interface to Accumulo can achieve these benefits by using the D4M 2.0 Schema.

AB - Non-traditional, relaxed consistency, triple store databases are the backbone of many web companies (e.g., Google Big Table, Amazon Dynamo, and Facebook Cassandra). The Apache Accumulo database is a high performance open source relaxed consistency database that is widely used for government applications. Obtaining the full benefits of Accumulo requires using novel schemas. The Dynamic Distributed Dimensional Data Model (D4M)[http://www.mit. edu/∼kepner/D4M] provides a uniform mathematical framework based on associative arrays that encompasses both traditional (i.e., SQL) and non-traditional databases. For non-traditional databases D4M naturally leads to a general purpose schema that can be used to fully index and rapidly query every unique string in a dataset. The D4M 2.0 Schema has been applied with little or no customization to cyber, bioinformatics, scientific citation, free text, and social media data. The D4M 2.0 Schema is simple, requires minimal parsing, and achieves the highest published Accumulo ingest rates. The benefits of the D4M 2.0 Schema are independent of the D4M interface. Any interface to Accumulo can achieve these benefits by using the D4M 2.0 Schema.

KW - Accumulo

KW - Big Data

KW - D4M

KW - Hadoop

KW - NoSQL

KW - database schema

UR - http://www.scopus.com/inward/record.url?scp=84893498943&partnerID=8YFLogxK

U2 - 10.1109/HPEC.2013.6670318

DO - 10.1109/HPEC.2013.6670318

M3 - Conference contribution

AN - SCOPUS:84893498943

SN - 9781479913657

T3 - 2013 IEEE High Performance Extreme Computing Conference, HPEC 2013

BT - 2013 IEEE High Performance Extreme Computing Conference, HPEC 2013

PB - IEEE Computer Society

Y2 - 10 September 2013 through 12 September 2013

ER -

D4M 2.0 schema: A general purpose high performance schema for the Accumulo database

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this