Accuracy and Performance Comparison of Video Action Recognition Approaches

Matthew Hutchinson; Siddharth Samsi; William Arcand; David Bestor; Bill Bergeron; Chansup Byun; Micheal Houle; Matthew Hubbell; Micheal Jones; Jeremy Kepner; Andrew Kirby; Peter Michaleas; Lauren Milechin; Julie Mullen; Andrew Prout; Antonio Rosa; Albert Reuther; Charles Yee; Vijay Gadepally

doi:10.1109/HPEC43674.2020.9286249

Accuracy and Performance Comparison of Video Action Recognition Approaches

Matthew Hutchinson, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Chansup Byun, Micheal Houle, Matthew Hubbell, Micheal Jones, Jeremy Kepner, Andrew Kirby, Peter Michaleas, Lauren Milechin, Julie Mullen, Andrew Prout, Antonio Rosa, Albert Reuther, Charles Yee, Vijay Gadepally

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Over the past few years, there has been significant interest in video action recognition systems and models. However, direct comparison of accuracy and computational performance results remain clouded by differing training environments, hardware specifications, hyperparameters, pipelines, and inference methods. This article provides a direct comparison between fourteen 'off-the-shelf' and state-of-the-art models by ensuring consistency in these training characteristics in order to provide readers with a meaningful comparison across different types of video action recognition algorithms. Accuracy of the models is evaluated using standard Top-1 and Top-5 accuracy metrics in addition to a proposed new accuracy metric. Additionally, we compare computational performance of distributed training from two to sixty-four GPUs on a state-of-the-art HPC system.

Original language	English
Title of host publication	2020 IEEE High Performance Extreme Computing Conference, HPEC 2020
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728192192
DOIs	https://doi.org/10.1109/HPEC43674.2020.9286249
State	Published - Sep 22 2020
Externally published	Yes
Event	2020 IEEE High Performance Extreme Computing Conference, HPEC 2020 - Virtual, Waltham, United States Duration: Sep 21 2020 → Sep 25 2020

Publication series

Name	2020 IEEE High Performance Extreme Computing Conference, HPEC 2020

Conference

Conference	2020 IEEE High Performance Extreme Computing Conference, HPEC 2020
Country/Territory	United States
City	Virtual, Waltham
Period	09/21/20 → 09/25/20

Keywords

accuracy metrics
action recognition
computational performance
deep learning
neural network

Access to Document

10.1109/HPEC43674.2020.9286249

Cite this

Hutchinson, M., Samsi, S., Arcand, W., Bestor, D., Bergeron, B., Byun, C., Houle, M., Hubbell, M., Jones, M., Kepner, J., Kirby, A., Michaleas, P., Milechin, L., Mullen, J., Prout, A., Rosa, A., Reuther, A., Yee, C., & Gadepally, V. (2020). Accuracy and Performance Comparison of Video Action Recognition Approaches. In 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020 [9286249] (2020 IEEE High Performance Extreme Computing Conference, HPEC 2020). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/HPEC43674.2020.9286249

@inproceedings{02a9c966b0f54f579b878d62eceb0ad8,

title = "Accuracy and Performance Comparison of Video Action Recognition Approaches",

abstract = "Over the past few years, there has been significant interest in video action recognition systems and models. However, direct comparison of accuracy and computational performance results remain clouded by differing training environments, hardware specifications, hyperparameters, pipelines, and inference methods. This article provides a direct comparison between fourteen 'off-the-shelf' and state-of-the-art models by ensuring consistency in these training characteristics in order to provide readers with a meaningful comparison across different types of video action recognition algorithms. Accuracy of the models is evaluated using standard Top-1 and Top-5 accuracy metrics in addition to a proposed new accuracy metric. Additionally, we compare computational performance of distributed training from two to sixty-four GPUs on a state-of-the-art HPC system.",

keywords = "accuracy metrics, action recognition, computational performance, deep learning, neural network",

author = "Matthew Hutchinson and Siddharth Samsi and William Arcand and David Bestor and Bill Bergeron and Chansup Byun and Micheal Houle and Matthew Hubbell and Micheal Jones and Jeremy Kepner and Andrew Kirby and Peter Michaleas and Lauren Milechin and Julie Mullen and Andrew Prout and Antonio Rosa and Albert Reuther and Charles Yee and Vijay Gadepally",

note = "Funding Information: DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering. {\textcopyright} 2020 Massachusetts Institute of Technology. Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work. Publisher Copyright: {\textcopyright} 2020 IEEE.; null ; Conference date: 21-09-2020 Through 25-09-2020",

year = "2020",

month = sep,

day = "22",

doi = "10.1109/HPEC43674.2020.9286249",

language = "English",

series = "2020 IEEE High Performance Extreme Computing Conference, HPEC 2020",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2020 IEEE High Performance Extreme Computing Conference, HPEC 2020",

}

Hutchinson, M, Samsi, S, Arcand, W, Bestor, D, Bergeron, B, Byun, C, Houle, M, Hubbell, M, Jones, M, Kepner, J, Kirby, A, Michaleas, P, Milechin, L, Mullen, J, Prout, A, Rosa, A, Reuther, A, Yee, C & Gadepally, V 2020, Accuracy and Performance Comparison of Video Action Recognition Approaches. in 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020., 9286249, 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020, Institute of Electrical and Electronics Engineers Inc., 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020, Virtual, Waltham, United States, 09/21/20. https://doi.org/10.1109/HPEC43674.2020.9286249

Accuracy and Performance Comparison of Video Action Recognition Approaches. / Hutchinson, Matthew; Samsi, Siddharth; Arcand, William et al.

2020 IEEE High Performance Extreme Computing Conference, HPEC 2020. Institute of Electrical and Electronics Engineers Inc., 2020. 9286249 (2020 IEEE High Performance Extreme Computing Conference, HPEC 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Accuracy and Performance Comparison of Video Action Recognition Approaches

AU - Hutchinson, Matthew

AU - Samsi, Siddharth

AU - Arcand, William

AU - Bestor, David

AU - Bergeron, Bill

AU - Byun, Chansup

AU - Houle, Micheal

AU - Hubbell, Matthew

AU - Jones, Micheal

AU - Kepner, Jeremy

AU - Kirby, Andrew

AU - Michaleas, Peter

AU - Milechin, Lauren

AU - Mullen, Julie

AU - Prout, Andrew

AU - Rosa, Antonio

AU - Reuther, Albert

AU - Yee, Charles

AU - Gadepally, Vijay

N1 - Funding Information: DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering. © 2020 Massachusetts Institute of Technology. Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work. Publisher Copyright: © 2020 IEEE.

PY - 2020/9/22

Y1 - 2020/9/22

N2 - Over the past few years, there has been significant interest in video action recognition systems and models. However, direct comparison of accuracy and computational performance results remain clouded by differing training environments, hardware specifications, hyperparameters, pipelines, and inference methods. This article provides a direct comparison between fourteen 'off-the-shelf' and state-of-the-art models by ensuring consistency in these training characteristics in order to provide readers with a meaningful comparison across different types of video action recognition algorithms. Accuracy of the models is evaluated using standard Top-1 and Top-5 accuracy metrics in addition to a proposed new accuracy metric. Additionally, we compare computational performance of distributed training from two to sixty-four GPUs on a state-of-the-art HPC system.

AB - Over the past few years, there has been significant interest in video action recognition systems and models. However, direct comparison of accuracy and computational performance results remain clouded by differing training environments, hardware specifications, hyperparameters, pipelines, and inference methods. This article provides a direct comparison between fourteen 'off-the-shelf' and state-of-the-art models by ensuring consistency in these training characteristics in order to provide readers with a meaningful comparison across different types of video action recognition algorithms. Accuracy of the models is evaluated using standard Top-1 and Top-5 accuracy metrics in addition to a proposed new accuracy metric. Additionally, we compare computational performance of distributed training from two to sixty-four GPUs on a state-of-the-art HPC system.

KW - accuracy metrics

KW - action recognition

KW - computational performance

KW - deep learning

KW - neural network

UR - http://www.scopus.com/inward/record.url?scp=85099362128&partnerID=8YFLogxK

U2 - 10.1109/HPEC43674.2020.9286249

DO - 10.1109/HPEC43674.2020.9286249

M3 - Conference contribution

AN - SCOPUS:85099362128

T3 - 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020

BT - 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 21 September 2020 through 25 September 2020

ER -

Hutchinson M, Samsi S, Arcand W, Bestor D, Bergeron B, Byun C et al. Accuracy and Performance Comparison of Video Action Recognition Approaches. In 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020. Institute of Electrical and Electronics Engineers Inc. 2020. 9286249. (2020 IEEE High Performance Extreme Computing Conference, HPEC 2020). doi: 10.1109/HPEC43674.2020.9286249

Accuracy and Performance Comparison of Video Action Recognition Approaches

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this