Python Implementation of the Dynamic Distributed Dimensional Data Model

Hayden Jananthan; Lauren Milechin; Michael Jones; William Arcand; William Bergeron; David Bestor; Chansup Byun; Michael Houle; Matthew Hubbell; Vijay Gadepally; Anna Klein; Peter Michnlons; Guillermo Morales; Julie Mullen; Andrew Prout; Albert Reuther; Antonio Rosa; Siddharth Samsi; Charles Yee; Jeremy Kepner

doi:10.1109/HPEC55821.2022.9926316

Python Implementation of the Dynamic Distributed Dimensional Data Model

Hayden Jananthan, Lauren Milechin, Michael Jones, William Arcand, William Bergeron, David Bestor, Chansup Byun, Michael Houle, Matthew Hubbell, Vijay Gadepally, Anna Klein, Peter Michnlons, Guillermo Morales, Julie Mullen, Andrew Prout, Albert Reuther, Antonio Rosa, Siddharth Samsi, Charles Yee, Jeremy Kepner

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Python has become a standard scientific computing language with fast-growing support of machine learning and data analysis modules, as well as an increasing usage of big data. The Dynamic Distributed Dimensional Data Model (D4M) offers a highly composable, unified data model with strong performance built to handle big data fast and efficiently. In this work we present an implementation of D4M in Python. D4M.py implements all foundational functionality of D4M and includes Accumulo and SQL database support via Graphulo. We describe the mathematical background and motivation, an explanation of the approaches made for its fundamen-tal functions and building blocks, and performance results which compare D4M.py's performance to D4M-MATLAB and D4M.jl.

Original language	English
Title of host publication	2022 IEEE High Performance Extreme Computing Conference, HPEC 2022
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781665497862
DOIs	https://doi.org/10.1109/HPEC55821.2022.9926316
State	Published - 2022
Externally published	Yes
Event	2022 IEEE High Performance Extreme Computing Conference, HPEC 2022 - Virtual, Online, United States Duration: Sep 19 2022 → Sep 23 2022

Publication series

Name	2022 IEEE High Performance Extreme Computing Conference, HPEC 2022

Conference

Conference	2022 IEEE High Performance Extreme Computing Conference, HPEC 2022
Country/Territory	United States
City	Virtual, Online
Period	09/19/22 → 09/23/22

Keywords

Python
array
data science
matrix
sparse linear algebra

Access to Document

10.1109/HPEC55821.2022.9926316

Cite this

Jananthan, H., Milechin, L., Jones, M., Arcand, W., Bergeron, W., Bestor, D., Byun, C., Houle, M., Hubbell, M., Gadepally, V., Klein, A., Michnlons, P., Morales, G., Mullen, J., Prout, A., Reuther, A., Rosa, A., Samsi, S., Yee, C., & Kepner, J. (2022). Python Implementation of the Dynamic Distributed Dimensional Data Model. In 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022 (2022 IEEE High Performance Extreme Computing Conference, HPEC 2022). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/HPEC55821.2022.9926316

@inproceedings{80cb34011e6c42a9ade7471cca1b2ec5,

title = "Python Implementation of the Dynamic Distributed Dimensional Data Model",

abstract = "Python has become a standard scientific computing language with fast-growing support of machine learning and data analysis modules, as well as an increasing usage of big data. The Dynamic Distributed Dimensional Data Model (D4M) offers a highly composable, unified data model with strong performance built to handle big data fast and efficiently. In this work we present an implementation of D4M in Python. D4M.py implements all foundational functionality of D4M and includes Accumulo and SQL database support via Graphulo. We describe the mathematical background and motivation, an explanation of the approaches made for its fundamen-tal functions and building blocks, and performance results which compare D4M.py's performance to D4M-MATLAB and D4M.jl.",

keywords = "Python, array, data science, matrix, sparse linear algebra",

author = "Hayden Jananthan and Lauren Milechin and Michael Jones and William Arcand and William Bergeron and David Bestor and Chansup Byun and Michael Houle and Matthew Hubbell and Vijay Gadepally and Anna Klein and Peter Michnlons and Guillermo Morales and Julie Mullen and Andrew Prout and Albert Reuther and Antonio Rosa and Siddharth Samsi and Charles Yee and Jeremy Kepner",

note = "Funding Information: This material is based upon work supported by the Assistant Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001, National Science Foundation CCF-1533644, and United States Air Force Research Laboratory and Air Force Artificial Intelligence Accelerator Cooperative Agreement Number FA8750-19-2-1000. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Assistant Secretary of Defense for Research and Engineering, the National Science Foundation, or the United States Air Force. Publisher Copyright: {\textcopyright} 2022 IEEE.; 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022 ; Conference date: 19-09-2022 Through 23-09-2022",

year = "2022",

doi = "10.1109/HPEC55821.2022.9926316",

language = "English",

series = "2022 IEEE High Performance Extreme Computing Conference, HPEC 2022",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2022 IEEE High Performance Extreme Computing Conference, HPEC 2022",

}

Jananthan, H, Milechin, L, Jones, M, Arcand, W, Bergeron, W, Bestor, D, Byun, C, Houle, M, Hubbell, M, Gadepally, V, Klein, A, Michnlons, P, Morales, G, Mullen, J, Prout, A, Reuther, A, Rosa, A, Samsi, S, Yee, C & Kepner, J 2022, Python Implementation of the Dynamic Distributed Dimensional Data Model. in 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022. 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022, Institute of Electrical and Electronics Engineers Inc., 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022, Virtual, Online, United States, 09/19/22. https://doi.org/10.1109/HPEC55821.2022.9926316

Python Implementation of the Dynamic Distributed Dimensional Data Model. / Jananthan, Hayden; Milechin, Lauren; Jones, Michael et al.
2022 IEEE High Performance Extreme Computing Conference, HPEC 2022. Institute of Electrical and Electronics Engineers Inc., 2022. (2022 IEEE High Performance Extreme Computing Conference, HPEC 2022).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Python Implementation of the Dynamic Distributed Dimensional Data Model

AU - Jananthan, Hayden

AU - Milechin, Lauren

AU - Jones, Michael

AU - Arcand, William

AU - Bergeron, William

AU - Bestor, David

AU - Byun, Chansup

AU - Houle, Michael

AU - Hubbell, Matthew

AU - Gadepally, Vijay

AU - Klein, Anna

AU - Michnlons, Peter

AU - Morales, Guillermo

AU - Mullen, Julie

AU - Prout, Andrew

AU - Reuther, Albert

AU - Rosa, Antonio

AU - Samsi, Siddharth

AU - Yee, Charles

AU - Kepner, Jeremy

N1 - Funding Information: This material is based upon work supported by the Assistant Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001, National Science Foundation CCF-1533644, and United States Air Force Research Laboratory and Air Force Artificial Intelligence Accelerator Cooperative Agreement Number FA8750-19-2-1000. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Assistant Secretary of Defense for Research and Engineering, the National Science Foundation, or the United States Air Force. Publisher Copyright: © 2022 IEEE.

PY - 2022

Y1 - 2022

N2 - Python has become a standard scientific computing language with fast-growing support of machine learning and data analysis modules, as well as an increasing usage of big data. The Dynamic Distributed Dimensional Data Model (D4M) offers a highly composable, unified data model with strong performance built to handle big data fast and efficiently. In this work we present an implementation of D4M in Python. D4M.py implements all foundational functionality of D4M and includes Accumulo and SQL database support via Graphulo. We describe the mathematical background and motivation, an explanation of the approaches made for its fundamen-tal functions and building blocks, and performance results which compare D4M.py's performance to D4M-MATLAB and D4M.jl.

AB - Python has become a standard scientific computing language with fast-growing support of machine learning and data analysis modules, as well as an increasing usage of big data. The Dynamic Distributed Dimensional Data Model (D4M) offers a highly composable, unified data model with strong performance built to handle big data fast and efficiently. In this work we present an implementation of D4M in Python. D4M.py implements all foundational functionality of D4M and includes Accumulo and SQL database support via Graphulo. We describe the mathematical background and motivation, an explanation of the approaches made for its fundamen-tal functions and building blocks, and performance results which compare D4M.py's performance to D4M-MATLAB and D4M.jl.

KW - Python

KW - array

KW - data science

KW - matrix

KW - sparse linear algebra

UR - http://www.scopus.com/inward/record.url?scp=85142252226&partnerID=8YFLogxK

U2 - 10.1109/HPEC55821.2022.9926316

DO - 10.1109/HPEC55821.2022.9926316

M3 - Conference contribution

AN - SCOPUS:85142252226

T3 - 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022

BT - 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022

Y2 - 19 September 2022 through 23 September 2022

ER -

Jananthan H, Milechin L, Jones M, Arcand W, Bergeron W, Bestor D et al. Python Implementation of the Dynamic Distributed Dimensional Data Model. In 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022. Institute of Electrical and Electronics Engineers Inc. 2022. (2022 IEEE High Performance Extreme Computing Conference, HPEC 2022). doi: 10.1109/HPEC55821.2022.9926316

Python Implementation of the Dynamic Distributed Dimensional Data Model

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this