Difference between revisions of "Fix your classifier: the marginal value of training the last weight layer"

From statwiki
Jump to: navigation, search
Line 47: Line 47:
v_i = \frac{e^{y_i}}{\sum_{j}^{C}{e^{y_j}}}, i &isin; </math> { <math> {1, . . . , C} </math> }
v_i = \frac{e^{y_i}}{\sum_{j}^{C}{e^{y_j}}}, i &isin; </math> { <math> {1, . . . , C} </math> }
and reducing the expected negative log likelihood with respect to ground-truth target t {1, . . . , C},
and reducing the expected negative log likelihood with respect to ground-truth target <math> t $isin; </math> { <math> {1, . . . , C} </math> },
by minimizing
by minimizing the loss function:
L(x, t) = − log vt = −wt · x − bt + log
L(x, t) = − log {v_t} = −{w_t}.{x} b_t + log ({\sum_{j}^{C}e^{w_j . x + b_j}})
wj ·x+bj
where wi
where wi
is the i-th column of W.
is the i-th column of W.

Revision as of 11:53, 27 November 2018


Deep neural networks have become a widely used model for machine learning, achieving state-of-the-art results on many tasks. The most common task these models are used for is to perform classification, as in the case of convolutional neural networks (CNNs) used to classify images to a semantic category. Typically, a learned affine transformation is placed at the end of such models, yielding a per-class value used for classification. This classifier can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more computational resources.

Brief Overview

In order to alleviate the aforementioned problem, the authors propose that the final layer of the classifier be fixed (upto a global scale constant). They argue that with little or no loss of accuracy for most classification tasks, the method provides significant memory and computational benefits. In addition, they show that by initializing the classifier with a Hadamard matrix the inference could be made faster as well.

Previous Work

Training NN models and using them for inference requires large amounts of memory and computational resources; thus, extensive amount of research has been done lately to reduce the size of networks which are as follows:

  • Weight sharing and specification (Han et al., 2015)
  • Mixed precision to reduce the size of the neural networks by half (Micikevicius et al., 2017)
  • Low-rank approximations to speed up CNN (Tai et al., 2015)
  • Quantization of weights, activations and gradients to further reduce computation during training (Hubara et al., 2016b; Li et al., 2016 and Zhou et al., 2016)

Some of the past works have also put forward the fact that predefined (Park & Sandberg, 1991) and random (Huang et al., 2006) projections can be used together with a learned affine transformation to achieve competitive results on many of the classification tasks. However, the authors' proposal in the current paper is quite reversed.


Convolutional neural networks (CNNs) are commonly used to solve a variety of spatial and temporal tasks. CNNs are usually composed of a stack of convolutional parameterized layers, spatial pooling layers and fully connected layers, separated by non-linear activation functions. Earlier architectures of CNNs (LeCun et al., 1998; Krizhevsky et al., 2012) used a set of fully-connected layers at later stage of the network, presumably to allow classification based on global features of an image.

Shortcomings of the final classification layer and its solution

Despite the enormous number of trainable parameters these layers added to the model, they are known to have a rather marginal impact on the final performance of the network (Zeiler & Fergus, 2014).

It has been shown previously that these layers could be easily compressed and reduced after a model was trained by simple means such as matrix decomposition and sparsification (Han et al., 2015). Modern architecture choices are characterized with the removal of most of the fully connected layers (Lin et al., 2013; Szegedy et al., 2015; He et al., 2016), that lead to better generalization and overall accuracy, together with a huge decrease in the number of trainable parameters. Additionally, numerous works showed that CNNs can be trained in a metric learning regime (Bromley et al., 1994; Schroff et al., 2015; Hoffer & Ailon, 2015), where no explicit classification layer was introduced and the objective regarded only distance measures between intermediate representations. Hardt & Ma (2017) suggested an all-convolutional network variant, where they kept the original initialization of the classification layer fixed with no negative impact on performance on the CIFAR-10 dataset.

Proposed Method

The aforementioned works provide evidence that fully-connected layers are in fact redundant and play a small role in learning and generalization. In this work, the authors have suggested that parameters used for the final classification transform are completely redundant, and can be replaced with a predetermined linear transform. This holds for even in large-scale models and classification tasks, such as recent architectures trained on the ImageNet benchmark (Deng et al., 2009).

Using a fixed classifier

Suppose the final representation obtained by the network (the last hidden layer) is represented as [math]x = F(z;\theta)[/math] where [math]F[/math] is assumed to be a deep neural network with input z and parameters θ, e.g., a convolutional network, trained by backpropagation.

In common NN models, this representation is followed by an additional affine transformation, [math]y = W^T x + b[/math] ,where [math]W[/math] and [math]b[/math] are also trained by back-propagation.

For input [math]x[/math] of [math]N[/math] length, and [math]C[/math] different possible outputs, [math]W[/math] is required to be a matrix of [math]N × C[/math]. Training is done using cross-entropy loss, by feeding the network outputs through a softmax activation

[math] v_i = \frac{e^{y_i}}{\sum_{j}^{C}{e^{y_j}}}, i ∈ [/math] { [math] {1, . . . , C} [/math] }

and reducing the expected negative log likelihood with respect to ground-truth target [math] t $isin; [/math] { [math] {1, . . . , C} [/math] }, by minimizing the loss function:

[math] L(x, t) = − log {v_t} = −{w_t}.{x} − b_t + log ({\sum_{j}^{C}e^{w_j . x + b_j}}) where wi is the i-th column of W. [/math]