D4M 2.0 schema: A general purpose high performance schema for the Accumulo database

Jeremy Kepner, Christian Anderson, William Arcand, David Bestor, Bill Bergeron, Chansup Byun, Matthew Hubbell, Peter Michaleas, Julie Mullen, David O'Gwynn, Andrew Prout, Albert Reuther, Antonio Rosa, Charles Yee

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

41 Scopus citations

Abstract

Non-traditional, relaxed consistency, triple store databases are the backbone of many web companies (e.g., Google Big Table, Amazon Dynamo, and Facebook Cassandra). The Apache Accumulo database is a high performance open source relaxed consistency database that is widely used for government applications. Obtaining the full benefits of Accumulo requires using novel schemas. The Dynamic Distributed Dimensional Data Model (D4M)[http://www.mit. edu/∼kepner/D4M] provides a uniform mathematical framework based on associative arrays that encompasses both traditional (i.e., SQL) and non-traditional databases. For non-traditional databases D4M naturally leads to a general purpose schema that can be used to fully index and rapidly query every unique string in a dataset. The D4M 2.0 Schema has been applied with little or no customization to cyber, bioinformatics, scientific citation, free text, and social media data. The D4M 2.0 Schema is simple, requires minimal parsing, and achieves the highest published Accumulo ingest rates. The benefits of the D4M 2.0 Schema are independent of the D4M interface. Any interface to Accumulo can achieve these benefits by using the D4M 2.0 Schema.

Original languageEnglish
Title of host publication2013 IEEE High Performance Extreme Computing Conference, HPEC 2013
PublisherIEEE Computer Society
ISBN (Print)9781479913657
DOIs
StatePublished - 2013
Externally publishedYes
Event2013 IEEE High Performance Extreme Computing Conference, HPEC 2013 - Waltham, MA, United States
Duration: Sep 10 2013Sep 12 2013

Publication series

Name2013 IEEE High Performance Extreme Computing Conference, HPEC 2013

Conference

Conference2013 IEEE High Performance Extreme Computing Conference, HPEC 2013
Country/TerritoryUnited States
CityWaltham, MA
Period09/10/1309/12/13

Keywords

  • Accumulo
  • Big Data
  • D4M
  • Hadoop
  • NoSQL
  • database schema

Fingerprint

Dive into the research topics of 'D4M 2.0 schema: A general purpose high performance schema for the Accumulo database'. Together they form a unique fingerprint.

Cite this