pPython for Parallel Python Programming

Chansup Byun; William Arcand; David Bestor; Bill Bergeron; Vijay Gadepally; Michael Houle; Matthew Hubbell; Hayden Jananthan; Michael Jones; Kurt Keville; Anna Klein; Peter Michaleas; Lauren Milechin; Guillermo Morales; Julie Mullen; Andrew Prout; Albert Reuther; Antonio Rosa; Siddharth Samsi; Charles Yee; Jeremy Kepner

doi:10.1109/HPEC55821.2022.9926365

pPython for Parallel Python Programming

Chansup Byun, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, Hayden Jananthan, Michael Jones, Kurt Keville, Anna Klein, Peter Michaleas, Lauren Milechin, Guillermo Morales, Julie Mullen, Andrew Prout, Albert Reuther, Antonio Rosa, Siddharth Samsi, Charles YeeJeremy Kepner

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

pPython seeks to provide a parallel capability that provides good speed-up without sacrificing the ease of programming in Python by implementing partitioned global array semantics (PGAS) on top of a simple file-based messaging library (PythonMPI) in pure Python. The core data structure in pPython is a distributed numerical array whose distribution onto multiple processors is specified with a 'map' construct. Communication operations between distributed arrays are abstracted away from the user and pPython transparently supports redistribution between any block-cyclic-overlapped distributions in up to four dimensions. pPython follows a SPMD (single program multiple data) model of computation. pPython runs on any combination of heterogeneous systems that support Python, including Windows, Linux, and MacOS operating systems. In addition to running transparently on single-node (e.g., a laptop), pPython provides a scheduler interface, so that pPython can be executed in a massively parallel computing environment. The initial implementation uses the Slurm scheduler. Performance of pPython on the HPC Challenge benchmark suite demonstrates both ease of programming and scalability.

Original language	English
Title of host publication	2022 IEEE High Performance Extreme Computing Conference, HPEC 2022
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781665497862
DOIs	https://doi.org/10.1109/HPEC55821.2022.9926365
State	Published - 2022
Externally published	Yes
Event	2022 IEEE High Performance Extreme Computing Conference, HPEC 2022 - Virtual, Online, United States Duration: Sep 19 2022 → Sep 23 2022

Publication series

Name	2022 IEEE High Performance Extreme Computing Conference, HPEC 2022

Conference

Conference	2022 IEEE High Performance Extreme Computing Conference, HPEC 2022
Country/Territory	United States
City	Virtual, Online
Period	09/19/22 → 09/23/22

Keywords

MPI
PGAS
Parallel Programming
Python

Access to Document

10.1109/HPEC55821.2022.9926365

Cite this

Byun, C., Arcand, W., Bestor, D., Bergeron, B., Gadepally, V., Houle, M., Hubbell, M., Jananthan, H., Jones, M., Keville, K., Klein, A., Michaleas, P., Milechin, L., Morales, G., Mullen, J., Prout, A., Reuther, A., Rosa, A., Samsi, S., ... Kepner, J. (2022). pPython for Parallel Python Programming. In 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022 (2022 IEEE High Performance Extreme Computing Conference, HPEC 2022). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/HPEC55821.2022.9926365

@inproceedings{7bb6ebe0d1a44f6795176c042377a0ef,

title = "pPython for Parallel Python Programming",

abstract = "pPython seeks to provide a parallel capability that provides good speed-up without sacrificing the ease of programming in Python by implementing partitioned global array semantics (PGAS) on top of a simple file-based messaging library (PythonMPI) in pure Python. The core data structure in pPython is a distributed numerical array whose distribution onto multiple processors is specified with a 'map' construct. Communication operations between distributed arrays are abstracted away from the user and pPython transparently supports redistribution between any block-cyclic-overlapped distributions in up to four dimensions. pPython follows a SPMD (single program multiple data) model of computation. pPython runs on any combination of heterogeneous systems that support Python, including Windows, Linux, and MacOS operating systems. In addition to running transparently on single-node (e.g., a laptop), pPython provides a scheduler interface, so that pPython can be executed in a massively parallel computing environment. The initial implementation uses the Slurm scheduler. Performance of pPython on the HPC Challenge benchmark suite demonstrates both ease of programming and scalability.",

keywords = "MPI, PGAS, Parallel Programming, Python",

author = "Chansup Byun and William Arcand and David Bestor and Bill Bergeron and Vijay Gadepally and Michael Houle and Matthew Hubbell and Hayden Jananthan and Michael Jones and Kurt Keville and Anna Klein and Peter Michaleas and Lauren Milechin and Guillermo Morales and Julie Mullen and Andrew Prout and Albert Reuther and Antonio Rosa and Siddharth Samsi and Charles Yee and Jeremy Kepner",

note = "Funding Information: This material is based upon work supported by the Assistant Secretary of Defense for Research and Engineering under Air Force Contract No. FA8721-05-C-0002 and/or FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Assistant Secretary of Defense for Research and Engineering. Publisher Copyright: {\textcopyright} 2022 IEEE.; null ; Conference date: 19-09-2022 Through 23-09-2022",

year = "2022",

doi = "10.1109/HPEC55821.2022.9926365",

language = "English",

series = "2022 IEEE High Performance Extreme Computing Conference, HPEC 2022",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2022 IEEE High Performance Extreme Computing Conference, HPEC 2022",

}

Byun, C, Arcand, W, Bestor, D, Bergeron, B, Gadepally, V, Houle, M, Hubbell, M, Jananthan, H, Jones, M, Keville, K, Klein, A, Michaleas, P, Milechin, L, Morales, G, Mullen, J, Prout, A, Reuther, A, Rosa, A, Samsi, S, Yee, C & Kepner, J 2022, pPython for Parallel Python Programming. in 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022. 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022, Institute of Electrical and Electronics Engineers Inc., 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022, Virtual, Online, United States, 09/19/22. https://doi.org/10.1109/HPEC55821.2022.9926365

pPython for Parallel Python Programming. / Byun, Chansup; Arcand, William; Bestor, David et al.

2022 IEEE High Performance Extreme Computing Conference, HPEC 2022. Institute of Electrical and Electronics Engineers Inc., 2022. (2022 IEEE High Performance Extreme Computing Conference, HPEC 2022).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - pPython for Parallel Python Programming

AU - Byun, Chansup

AU - Arcand, William

AU - Bestor, David

AU - Bergeron, Bill

AU - Gadepally, Vijay

AU - Houle, Michael

AU - Hubbell, Matthew

AU - Jananthan, Hayden

AU - Jones, Michael

AU - Keville, Kurt

AU - Klein, Anna

AU - Michaleas, Peter

AU - Milechin, Lauren

AU - Morales, Guillermo

AU - Mullen, Julie

AU - Prout, Andrew

AU - Reuther, Albert

AU - Rosa, Antonio

AU - Samsi, Siddharth

AU - Yee, Charles

AU - Kepner, Jeremy

N1 - Funding Information: This material is based upon work supported by the Assistant Secretary of Defense for Research and Engineering under Air Force Contract No. FA8721-05-C-0002 and/or FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Assistant Secretary of Defense for Research and Engineering. Publisher Copyright: © 2022 IEEE.

PY - 2022

Y1 - 2022

N2 - pPython seeks to provide a parallel capability that provides good speed-up without sacrificing the ease of programming in Python by implementing partitioned global array semantics (PGAS) on top of a simple file-based messaging library (PythonMPI) in pure Python. The core data structure in pPython is a distributed numerical array whose distribution onto multiple processors is specified with a 'map' construct. Communication operations between distributed arrays are abstracted away from the user and pPython transparently supports redistribution between any block-cyclic-overlapped distributions in up to four dimensions. pPython follows a SPMD (single program multiple data) model of computation. pPython runs on any combination of heterogeneous systems that support Python, including Windows, Linux, and MacOS operating systems. In addition to running transparently on single-node (e.g., a laptop), pPython provides a scheduler interface, so that pPython can be executed in a massively parallel computing environment. The initial implementation uses the Slurm scheduler. Performance of pPython on the HPC Challenge benchmark suite demonstrates both ease of programming and scalability.

AB - pPython seeks to provide a parallel capability that provides good speed-up without sacrificing the ease of programming in Python by implementing partitioned global array semantics (PGAS) on top of a simple file-based messaging library (PythonMPI) in pure Python. The core data structure in pPython is a distributed numerical array whose distribution onto multiple processors is specified with a 'map' construct. Communication operations between distributed arrays are abstracted away from the user and pPython transparently supports redistribution between any block-cyclic-overlapped distributions in up to four dimensions. pPython follows a SPMD (single program multiple data) model of computation. pPython runs on any combination of heterogeneous systems that support Python, including Windows, Linux, and MacOS operating systems. In addition to running transparently on single-node (e.g., a laptop), pPython provides a scheduler interface, so that pPython can be executed in a massively parallel computing environment. The initial implementation uses the Slurm scheduler. Performance of pPython on the HPC Challenge benchmark suite demonstrates both ease of programming and scalability.

KW - MPI

KW - PGAS

KW - Parallel Programming

KW - Python

UR - http://www.scopus.com/inward/record.url?scp=85138598441&partnerID=8YFLogxK

U2 - 10.1109/HPEC55821.2022.9926365

DO - 10.1109/HPEC55821.2022.9926365

M3 - Conference contribution

AN - SCOPUS:85138598441

T3 - 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022

BT - 2022 IEEE High Performance Extreme Computing Conference, HPEC 2022

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 19 September 2022 through 23 September 2022

ER -

pPython for Parallel Python Programming

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this