Optimizing non-decomposable measures with deep networks

Amartya Sanyal, Pawan Kumar, Purushottam Kar, Sanjay Chawla, Fabrizio Sebastiani

Research output: Contribution to journalArticlepeer-review

11 Scopus citations

Abstract

We present a class of algorithms capable of directly training deep neural networks with respect to popular families of task-specific performance measures for binary classification such as the F-measure, QMean and the Kullback–Leibler divergence that are structured and non-decomposable. Our goal is to address tasks such as label-imbalanced learning and quantification. Our techniques present a departure from standard deep learning techniques that typically use squared or cross-entropy loss functions (that are decomposable) to train neural networks. We demonstrate that directly training with task-specific loss functions yields faster and more stable convergence across problems and datasets. Our proposed algorithms and implementations offer several advantages including (i) the use of fewer training samples to achieve a desired level of convergence, (ii) a substantial reduction in training time, (iii) a seamless integration of our implementation into existing symbolic gradient frameworks, and (iv) assurance of convergence to first order stationary points. It is noteworthy that the algorithms achieve this, especially point (iv), despite being asked to optimize complex objective functions. We implement our techniques on a variety of deep architectures including multi-layer perceptrons and recurrent neural networks and show that on a variety of benchmark and real data sets, our algorithms outperform traditional approaches to training deep networks, as well as popular techniques used to handle label imbalance.

Original languageEnglish
Pages (from-to)1597-1620
Number of pages24
JournalMachine Learning
Volume107
Issue number8-10
DOIs
StatePublished - Sep 1 2018
Externally publishedYes

Keywords

  • Deep learning
  • F-measure
  • Optimization
  • Task-specific training

Fingerprint

Dive into the research topics of 'Optimizing non-decomposable measures with deep networks'. Together they form a unique fingerprint.

Cite this