D4M: Bringing associative arrays to database engines

Vijay Gadepally, Jeremy Kepner, William Arcand, David Bestor, Bill Bergeron, Chansup Byun, Lauren Edwards, Matthew Hubbell, Peter Michaleas, Julie Mullen, Andrew Prout, Antonio Rosa, Charles Yee, Albert Reuther

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

29 Scopus citations

Abstract

The ability to collect and analyze large amounts of data is a growing problem within the scientific community. The growing gap between data and users calls for innovative tools that address the challenges faced by big data volume, velocity and variety. Numerous tools exist that allow users to store, query and index these massive quantities of data. Each storage or database engine comes with the promise of dealing with complex data. Scientists and engineers who wish to use these systems often quickly find that there is no single technology that offers a panacea to the complexity of information. When using multiple technologies, however, there is significant trouble in designing the movement of information between storage and database engines to support an end-to-end application along with a steep learning curve associated with learning the nuances of each underlying technology. In this article, we present the Dynamic Distributed Dimensional Data Model (D4M) as a potential tool to unify database and storage engine operations. Previous articles on D4M have showcased the ability of D4M to interact with the popular NoSQL Accumulo database. Recently however, D4M now operates on a variety of backend storage or database engines while providing a federated look to the end user through the use of associative arrays. In order to showcase how new databases may be supported by D4M, we describe the process of building the D4M-SciDB connector and present performance of this connection.

Original languageEnglish
Title of host publication2015 IEEE High Performance Extreme Computing Conference, HPEC 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781467392860
DOIs
StatePublished - Nov 9 2015
Externally publishedYes
EventIEEE High Performance Extreme Computing Conference, HPEC 2015 - Waltham, United States
Duration: Sep 15 2015Sep 17 2015

Publication series

Name2015 IEEE High Performance Extreme Computing Conference, HPEC 2015

Conference

ConferenceIEEE High Performance Extreme Computing Conference, HPEC 2015
Country/TerritoryUnited States
CityWaltham
Period09/15/1509/17/15

Keywords

  • Big Data
  • Data Analytics
  • Dimensional Analysis
  • Federated Databases

Fingerprint

Dive into the research topics of 'D4M: Bringing associative arrays to database engines'. Together they form a unique fingerprint.

Cite this