http://wiki.math.uwaterloo.ca/statwiki/api.php?action=feedcontributions&user=C9sharma&feedformat=atom statwiki - User contributions [US] 2021-11-30T09:04:18Z User contributions MediaWiki 1.28.3 http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_Bayesian_Perspective_on_Generalization_and_Stochastic_Gradient_Descent&diff=42098 A Bayesian Perspective on Generalization and Stochastic Gradient Descent 2018-11-30T17:34:54Z <p>C9sharma: /* Introduction */</p> <hr /> <div>==Introduction==<br /> This paper shows Bayesian principles can explain many recent observations in the deep learning literature, and provide practical new insights. This work builds on Zhang et al.(2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. The authors consider two questions: how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? <br /> <br /> The paper shows that the same phenomenon occurs even in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. They also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.<br /> <br /> The authors propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the “noise scale” &lt;math display=&quot;inline&quot;&gt; g \approx \epsilon N/B &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;ε&lt;/math&gt; is the learning rate, &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt; the training set size and &lt;math display=&quot;inline&quot;&gt;B&lt;/math&gt; the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, &lt;math display=&quot;inline&quot;&gt;B_{opt} \propto \epsilon N&lt;/math&gt;. The authors verify these predictions empirically.<br /> <br /> ==Motivation and Related Work==<br /> Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite &amp; Roy, 2017). Nonetheless, their results beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? <br /> <br /> Meanwhile, Keskar et al. (2016) observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking result shows improving the estimate of the full-batch gradient can harm performance. Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Hoffer et al. (2017) proposed a square root rule on theoretical grounds.<br /> <br /> Many authors have suggested “broad minima” whose curvature is small may generalize better than “sharp minima” whose curvature is large (Chaudhari et al., 2016; Hochreiter &amp; Schmidhuber, 1997). Indeed, Dziugaite &amp; Roy (2017) argued the results of Zhang et al. (2016) can be understood using “nonvacuous” PAC-Bayes generalization bounds which penalize sharp minima, while Keskar et al. (2016) showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However, Dinh et al. (2017) challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization.<br /> <br /> ==Contribution==<br /> <br /> The main contributions of this paper are to show that:<br /> * The results of Zhang et al. (2016) are not unique to deep learning; it is observed the same phenomenon in a small “over-parameterized” linear model. Overparameterization occurs when a model is able to effectively “remember” training data. This occurs when there are enough parameters that the system of equations ends up with an infinite number of possible solutions. One can see why this over-training would lead to poor results in test cases, as this “memorization” learns noise as opposed to the inherent structure of different classes. It is demonstrated that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.<br /> * SGD integrates a stochastic differential equation whose “noise scale” &lt;math&gt;g &amp;asymp; &amp;epsilon;N/B&lt;/math&gt;, where<br /> &amp;epsilon; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is '''proportional to the learning rate and training set size'''.<br /> <br /> Zhang et al. (2016) showed high training competency of neural networks under informative labels, but drastic overfitting on improper labels. This implies weak generalizability even when a small proportion of labels are improper. The authors show that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Bayesians tend to make distributional assumptions on gradient updates by adding isotropic Gaussian noise. This paper builds upon these Bayesian principles by driving SGD away from sharp minima, and towards broad minima (the more broad, the better generalization due to less influence from small perturbations within input). The stochastic differential equation used as a component of gradient updates effectively serves as injected noise that improves a network's generalizability.<br /> <br /> ==Main Results==<br /> <br /> The weakly regularized model memorizes random labels, however, generalizes properly on informative labels. Besides, the predictions are overconfident. The authors also showed that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is postulated that the optimum represents a tradeoff between depth and breadth in the Bayesian evidence. However it is the underlying scale of random fluctuations in the SGD dynamics which controls the tradeoff, not the batch size itself. Furthermore, this test accuracy peak shifts as the training set size rises. The authors observed that the best found batch size is proportional to the learning rate. This scaling rule allowed the authors to increase the learning rate by simultaneously increasing the batch size with no loss in test accuracy and no increase in computational cost, thus parallelism across multiple GPU's can be fully leveraged to easily decrease training time. The scaling rule could also be applied to production models by consequentially increasing the batch size as new training data is introduced.<br /> <br /> ==Bayesian Model Comparison==<br /> <br /> ===Introduction to Bayesian Statistics===<br /> Bayes' theorem is a fundamental theorem in Bayesian statistics, as it is used by Bayesian methods to update probabilities, which are degrees of belief, after obtaining new data. Given two events &lt;math&gt;A&lt;/math&gt; and &lt;math&gt;B&lt;/math&gt;, the conditional probability of &lt;math&gt;A&lt;/math&gt; given &lt;math&gt;B &lt;/math&gt; is true, Bayes theorem states that<br /> \begin{align*}\displaystyle P(A\mid B)={\frac {P(B\mid A)P(A)}{P(B)}}\end{align*}<br /> <br /> Bayesian networks are DAGs whose nodes represent variables in the Bayesian sense: they may be observable quantities, latent variables, unknown parameters or hypotheses. Edges represent conditional dependencies; nodes that are not connected (no path connects one node to another) represent variables that are conditionally independent of each other. Each node is associated with a probability function that takes, as input, a particular set of values for the node's parent variables, and gives (as output) the probability (or probability distribution, if applicable) of the variable represented by the node. For example, if &lt;math&gt;m &lt;/math&gt; parent nodes represent &lt;math&gt;m &lt;/math&gt; Boolean variables then the probability function could be represented by a table of &lt;math&gt;2^{m} &lt;/math&gt; entries, one entry for each of the &lt;math&gt;2^{m} &lt;/math&gt; possible parent combinations. <br /> <br /> ===Bayesian Model Comparison in Neural Networks===<br /> MacKay (1992) applied Bayesian model comparison to neural networks. An overview is presented below. <br /> <br /> We first consider a classification model &lt;math&gt;M &lt;/math&gt; with a single parameter &lt;math&gt;\omega &lt;/math&gt;, training inputs &lt;math&gt;x &lt;/math&gt; and training labels &lt;math&gt;y &lt;/math&gt;. We can infer a posterior probability distribution over the parameter by applying Bayes theorem :<br /> <br /> \begin{align*}P(\omega\mid y,x;M) = \frac{P(y\mid \omega,x;M)P(\omega;M) }{P(y\mid x;M)}\end{align*}<br /> <br /> The likelihood, &lt;math&gt;P(y\mid \omega,x;M) = \Pi_i P(y_i\mid \omega,x_i;M) = e^{-H(\omega;M)} &lt;/math&gt;, where &lt;math&gt;H(\omega;M) &lt;/math&gt; denotes the cross-entropy of unique categorical labels. Using a Gaussian prior, &lt;math&gt;P(\omega;M) = \sqrt{\lambda/2\pi e^{-\lambda\omega^2/2}} &lt;/math&gt;, and therefore the posterior probability density of the parameter given the training data, &lt;math&gt;P(\omega\mid y,x;M) \propto \sqrt{\lambda/2\pi e^{-C(\omega;M)}} &lt;/math&gt;, where &lt;math&gt;C(\omega;M) = H(\omega;M) + \lambda\omega^2/2 &lt;/math&gt; denotes the L2 regularized cross entropy, or “cost function”, and &lt;math&gt;\lambda &lt;/math&gt; is the regularization coefficient. <br /> <br /> The value &lt;math&gt;\omega_0 &lt;/math&gt; which minimizes the cost function lies at the maximum of this posterior. To predict an unknown label &lt;math&gt;y_t &lt;/math&gt; of a new input &lt;math&gt;x_t &lt;/math&gt;, we should compute the integral,<br /> <br /> \begin{align*} P(y_t\mid x_t,y,x;M) &amp;= \int \frac{d\omega P(y_t\mid \omega,x_t;M)}{P(\omega\mid y,x;M)}\\ &amp;= \frac{\int d \omega P(y_t \mid \omega ,x_t;M)e^{-C(\omega;M)}}{\int d \omega e^{-C(\omega;M)}} \end{align*}&lt;/math&gt;<br /> <br /> However, these integrals are dominated by the region near &lt;math&gt;\omega_0 &lt;/math&gt; . We usually approximate &lt;math&gt;P(y_t\mid x_t,x,y;M) \approx P(y_t\mid \omega_0,x_t;M) &lt;/math&gt;. Having minimized &lt;math&gt;C(\omega;M) &lt;/math&gt; to find &lt;math&gt;\omega_0 &lt;/math&gt;, we now wish to compare two different models and select the best one. We use the probability ratio<br /> <br /> \begin{align*}\frac{P(M_1\mid y,x)}{P (M_2\mid y, x)} = \frac{P(y\mid x;M_1) P(M_1)}{ P (y\mid x; M_2) P (M_2)} . \end{align*} <br /> <br /> The second factor on the right is the prior ratio, which describes which model is most plausible. To avoid unnecessary subjectivity, we usually set this to 1. Meanwhile the first factor on the right is the evidence ratio, which controls how much the training data changes our prior beliefs<br /> <br /> Germain et al. (2016) showed that maximizing the evidence (or “marginal likelihood”) minimizes a PAC-Bayes generalization bound. To compute it, we evaluate <br /> \begin{align*}P(y\mid x;M) &amp;= \int d\omega P(y\mid \omega,x;M)P(\omega;M) \\ &amp;=\sqrt{\frac{\lambda}{2\pi}}\int d \omega e^{C(\omega;M)}\end{align*}<br /> <br /> Notice that the evidence is computed by integrating out the parameters; and consequently it is invariant to the model parameterization. <br /> Since this integral is dominated by the region near the minimum &lt;math&gt;\omega_0 &lt;/math&gt;, we can estimate the evidence by Taylor expanding &lt;math&gt;C(\omega; M) \approx C(\omega_0) + C′′(\omega_0)(\omega - \omega_0)^2/2&lt;/math&gt;. This gives us<br /> <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2}\\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2}\ln(C (\omega_0)/\lambda) \big\}.\end{align*}<br /> <br /> The evidence is controlled by the value of the cost function at the minimum, and by the logarithm of the ratio of the curvature about this minimum compared to the regularization constant. In models with many parameters <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2} \\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) \big\}.\end{align*}<br /> <br /> Occam’s factor arises from the log ratio &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; The Occam factor describes the fraction of the prior parameter space consistent with the data. Occam’s factor penalizes the amount of information the model must learn about the parameters to accurately model the training data. Since the fraction is always less than one, the authors propose to approximate &lt;math&gt;P(y\mid x;M) &lt;/math&gt; away from local minima by only performing the summation over eigenvalues &lt;math&gt;\lambda_i \geq \lambda &lt;/math&gt;.<br /> <br /> The authors compare evidence against a null model which assumes the labels are entirely random. This model has no parameters, and so the evidence is controlled by the likelihood alone. &lt;math&gt;P(y\mid x;NULL) = (1/n)^N = e^{-N \ln(n)} &lt;/math&gt;, where &lt;math&gt;n &lt;/math&gt; denotes the number of model classes and &lt;math&gt;N&lt;/math&gt; the number of training labels. The evidence ratio :<br /> \begin{equation*}\frac{P(y\mid x;M) }{P(y\mid x;NULL) } = e ^{-E(\omega_0)} \end{equation*}<br /> &lt;math&gt;E(\omega_0) = C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) - N\ln (n) &lt;/math&gt; is the log evidence ratio in favor of the null model.<br /> The authors assign confidence to the predictions of a model iff &lt;math&gt;E(\omega_0 &lt; 0 &lt;/math&gt;.<br /> <br /> The evidence supports the intuition that broad minima generalize better than sharp minima, but unlike the curvature it does not depend on the model parameterization. Dinh et al. (2017) showed one can increase the Hessian eigenvalues by rescaling the parameters, but they must simultaneously rescale the regularization coefficients, otherwise the model changes. Since Occam’s factor arises from the log ratio, &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; , these two effects cancel out. Note however that while the evidence itself is invariant to model parameterization, one can find reparameterizations which change the approximate evidence after the Laplace approximation. . It is difficult to evaluate the evidence for deep networks, as we cannot compute the Hessian of millions of parameters. Additionally, neural networks exhibit many equivalent minima, since we can permute the hidden units without changing the model. To compute the evidence we must carefully account for this “degeneracy”. The authors argue these issues are not a major limitation, since the intuition they build studying the evidence in simple cases will be sufficient to explain the results of both Zhang et al. (2016) and Keskar et al. (2016).<br /> <br /> ==Bayes Theorem and Generalization==<br /> Zhang et al. (2016) showed that deep neural networks generalize well on training inputs with informative labels, but the same model can overfit on the same input images when the labels are randomized; perfectly memorizing the training set. To demonstrate that these observations are not unique to deep network, the authors use logistic regression. They form a small balanced training set comprising 800 images from MNIST, of which half have true label “0” and half true label “1”. The test set is balanced, comprising 5000 MNIST images of zeros and 5000 MNIST images of ones. There are two tasks. In the first task, the labels of both the training and test sets are randomized. In the second task, the labels are informative, matching the true MNIST labels. The model has 784 weights and 1 bias.<br /> <br /> The accuracy of the model predictions on both the training and test sets is shown in figure 1. When trained on the informative labels, the model generalizes well to the test set, so long as it is weakly regularized. However the model also perfectly memorizes the random labels, replicating the obser- vations of Zhang et al. (2016) in deep networks. No significant improvement in model performance is observed as the regularization coefficient increases. For completeness, we also evaluate the mean margin between training examples and the decision boundary. For both random and informative labels, the margin drops significantly as we reduce the regularization coefficient. When weakly regularized, the mean margin is roughly 50% larger for informative labels than for random labels.<br /> <br /> [[File:bg1.png|800px|thumb|center|]]<br /> <br /> Now consider figure 2, where we plot the mean cross-entropy of the model predictions, evaluated on both training and test sets, as well as the Bayesian log evidence ratio defined in the previous section. Looking first at the random label experiment in figure 2a, while the cross-entropy on the training set vanishes when the model is weakly regularized, the cross-entropy on the test set explodes. Not only does the model make random predictions, but it is extremely confident in those predictions. As the regularization coefficient is increased the test set cross-entropy falls, settling at ln 2, the cross- entropy of assigning equal probability to both classes. Now consider the Bayesian evidence, which we evaluate on the training set. The log evidence ratio is large and positive when the model is weakly regularized, indicating that the model is exponentially less plausible than assigning equal probabilities to each class. As the regularization parameter is increased, the log evidence ratio falls, but it is always positive, indicating that the model can never be expected to generalize well.<br /> Now consider figure 2b (informative labels). Once again, the training cross-entropy falls to zero when the model is weakly regularized, while the test cross-entropy is high. Even though the model makes accurate predictions, those predictions are overconfident. As the regularization coefficient increases, the test cross-entropy falls below ln 2, indicating that the model is successfully gener- alizing to the test set. Now consider the Bayesian evidence. The log evidence ratio is large and positive when the model is weakly regularized, but as the regularization coefficient increases, the log evidence ratio drops below zero, indicating that the model is exponentially more plausible than assigning equal probabilities to each class. As we further increase the regularization, the log evi- dence ratio rises to zero while the test cross-entropy rises to ln 2. Test cross-entropy and Bayesian evidence are strongly correlated, with minima at the same regularization strength.<br /> Bayesian model comparison has explained our results in a logistic regression. Meanwhile, Krueger et al. (2017) showed the largest Hessian eigenvalue also increased when training on random labels in deep networks, implying the evidence is falling. We conclude that Bayesian model comparison is quantitatively consistent with the results of Zhang et al. (2016) in linear models where we can compute the evidence, and qualitatively consistent with their results in deep networks where we cannot. Dziugaite &amp; Roy (2017) recently demonstrated the results of Zhang et al. (2016) can also be understood by minimising a PAC-Bayes generalization bound which penalizes sharp minima.<br /> [[File:bg2.png|800px|thumb|center|]]<br /> ==Bayes Theorem and Stochastic Gradient Descent ==<br /> <br /> We showed above that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Consequently Bayesians often add isotropic Gaussian noise to the gradient (Welling &amp; Teh, 2011). In appendix A, we show this drives the parameters towards broad minima whose evidence is large. The noise introduced by small batch training is not isotropic, and its covariance matrix is a function of the parameter values, but empirically Keskar et al. (2016) found it has similar effects, driving the SGD away from sharp minima. This paper therefore proposes Bayesian principles also account for the “generalization gap”, whereby the test set accuracy often falls as the SGD batch size is increased (holding all other hyper-parameters constant). Since the gradient drives the SGD towards deep minima, while noise drives the SGD towards broad minima, we expect the test set performance to show a peak at an optimal batch size, which balances these competing contributions to the evidence.<br /> We were unable to observe a generalization gap in linear models (since linear models are convex there are no sharp minima to avoid). Instead we consider a shallow neural network with 800 hidden units and RELU hidden activations, trained on MNIST without regularization. We use SGD with a momentum parameter of 0.9. Unless otherwise stated, we use a constant learning rate of 1.0 which does not depend on the batch size or decay during training. Furthermore, we train on just 1000 images, selected at random from the MNIST training set. This enables us to compare small batch to full batch training. We emphasize that we are not trying to achieve optimal performance, but to study a simple model which shows a generalization gap between small and large batch training.<br /> In figure 3, we exhibit the evolution of the test accuracy and test cross-entropy during training. Our small batches are composed of 30 images, randomly sampled from the training set. Looking first at figure 3a, small batch training takes longer to converge, but after a thousand gradient updates a clear generalization gap in model accuracy emerges between small and large training batches. Now consider figure 3b. While the test cross-entropy for small batch training is lower at the end of training; the cross-entropy of both small and large training batches is increasing, indicative of over-fitting. Both models exhibit a minimum test cross-entropy, although after different numbers of gradient updates. Intriguingly, we show in appendix B that the generalization gap between small and large batch training shrinks significantly when we introduce L2 regularization.<br /> <br /> [[File:bg3.png|800px|thumb|center|]]<br /> <br /> From now on we focus on the test set accuracy (since this converges as the number of gradient updates increases). In figure 4a, we exhibit training curves for a range of batch sizes between 1 and 1000. We find that the model cannot train when the batch size B 􏰘 10. In figure 4b we plot the mean test set accuracy after 10000 training steps. A clear peak emerges, indicating that there is indeed an optimum batch size which maximizes the test accuracy, consistent with Bayesian intuition. The results of Keskar et al. (2016) focused on the decay in test accuracy above this optimum batch size.<br /> [[File:bg4.png|800px|thumb|center|]]<br /> <br /> ==Stochastic Differential Equations and Scaling Rules==<br /> The results showed above indicate that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is argued that this peak arises from the tradeoff between depth and breadth in the Bayesian evidence. However it is not the batch size itself which controls this tradeoff, but the underlying scale of random fluctuations in the SGD dynamics. The following content identifies this SGD “noise scale”, and uses it to derive three scaling rules which predict how the optimal batch size depends on the learning rate, training set size and momentum coefficient. <br /> First, interpret gradient update, as the discrete update of a stochastic differential equation <br /> \begin{equation*}\frac{d\omega}{dt} = \frac{dC}{d\omega} + \eta(t)\end{equation*}<br /> &lt;math&gt;\eta&lt;/math&gt; represents noise &lt;math&gt;\langle \eta(t) \rangle = 0&lt;/math&gt; and &lt;math&gt; \langle \eta (t)\eta (t')\rangle = gF (\omega)\delta (t-t')&lt;/math&gt;.<br /> &lt;math&gt;t&lt;/math&gt; is a continous variable, and &lt;math&gt;F(\omega)&lt;/math&gt; matrix describing the gradient covariances.<br /> The SGD noise scale is taken to be &lt;math&gt;g \approx \epsilon N/B&lt;/math&gt; where &lt;math&gt;\epsilon&lt;/math&gt; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; the batch size.<br /> [[File:bg5.png|800px|thumb|center|]]<br /> [[File:bg6.png|800px|thumb|center|]]<br /> [[File:bg7.png|800px|thumb|center|]]<br /> The noise scale falls when the batch B<br /> size increases, consistent with our earlier observation of an optimal batch size Bopt while holding the other hyper-parameters fixed. Notice that one would equivalently observe an optimal learning rate if one held the batch size constant. A similar analysis of the SGD was recently performed by Mandt et al. (2017), although their treatment only holds near local minima where the covariances F (ω) are stationary. Our analysis holds throughout training, which is necessary since Keskar et al. (2016) found that the beneficial influence of noise was most pronounced at the start of training.<br /> When we vary the learning rate or the training set size, we should keep the noise scale fixed, which implies that Bopt ∝ εN. In figure 5a, we plot the test accuracy as a function of batch size after (10000/ε) training steps, for a range of learning rates. Exactly as predicted, the peak moves to the right as ε increases. Additionally, the peak test accuracy achieved at a given learning rate does not begin to fall until ε ∼ 3, indicating that there is no significant discretization error in integrating the stochastic differential equation below this point. Above this point, the discretization error begins to dominate and the peak test accuracy falls rapidly. In figure 5b, we plot the best observed batch size as a function of learning rate, observing a clear linear trend, Bopt ∝ ε. The error bars indicate the distance from the best observed batch size to the next batch size sampled in our experiments.<br /> <br /> This scaling rule allows us to increase the learning rate with no loss in test accuracy and no increase in computational cost, simply by simultaneously increasing the batch size. We can then exploit increased parallelism across multiple GPUs, reducing model training times (Goyal et al., 2017). A similar scaling rule was independently proposed by Jastrzebski et al. (2017) and Chaudhari &amp; Soatto (2017), although neither work identifies the existence of an optimal noise scale. A number of authors have proposed adjusting the batch size adaptively during training (Friedlander &amp; Schmidt, 2012; Byrd et al., 2012; De et al., 2017), while Balles et al. (2016) proposed linearly coupling the learning rate and batch size within this framework. In Smith et al. (2017), we show empirically that decaying the learning rate during training and increasing the batch size during training are equivalent.<br /> In figure 6a we exhibit the test set accuracy as a function of batch size, for a range of training set sizes after 10000 steps (ε = 1 everywhere). Once again, the peak shifts right as the training set size rises, although the generalization gap becomes less pronounced as the training set size increases. In figure 6b, we plot the best observed batch size as a function of training set size; observing another linear trend, Bopt ∝ N. This scaling rule could be applied to production models, progressively growing the batch size as new training data is collected. We expect production datasets to grow considerably over time, and consequently large batch training is likely to become increasingly common.<br /> B(1−m)<br /> scale of conventional SGD as m → 0. When m &gt; 0, we obtain an additional scaling rule Bopt ∝ 1/(1 − m). This scaling rule predicts that the optimal batch size will increase when the momentum coefficient is increased. In figure 7a we plot the test set performance as a function of batch size after 10000 gradient updates (ε = 1 everywhere), for a range of momentum coefficients. In figure 7b, we plot the best observed batch size as a function of the momentum coefficient, and fit our results to the scaling rule above; obtaining remarkably good agreement.<br /> <br /> ==Critiques==<br /> <br /> #Bayesian statistics is not provably, at present, a theory that can be used to explain why a learning algorithm works. The Bayesian theory is too optimistic: we introduce a prior and model and then trust both implicitly. Relative to any particular prior and model (likelihood), the Bayesian posterior is the optimal summary of the data, but if either part is misspecified, then the Bayesian posterior carries no optimality guarantee. The prior is chosen for convenience here. <br /> #No discussions with respect to the analysis of information bottleneck which also discuss the generalization ability of the model. <br /> #No discussion on real online learning with streaming data where the total number of data points are unknown?<br /> #The paper presents how mini-batch noises with SGD can improve the performance of neural networks. However, the usefulness of the approach can be described and analyzed in greater details, if the author could provide the performance for various well-known real-life datas.<br /> <br /> ==Conclusion==<br /> <br /> The paper showed that Mini-batch noise helps SGD to go away from sharp minima, and provided an evidence that there is an optimal optimum batch size for a maximum the test accuracy. Based on interpreting SGD as integrating stochastic differential equation, this batch size is proportional to the learning rate and the training set size. Moreover, the authors shown that &lt;math&gt;Bopt \propto 1/(1 − m) &lt;/math&gt;, where m is the momentum coefficient. More analysis was done on the relation between the learning rate, effective learning rate, and batch size is presented in ICLR 2018, where the authors proved by experiments that all the benefits of decaying the learning rate are achieved by increasing the batch size in addition to reducing the number of parameter updates dramatically, and also were able use literature parameters without the need of any hyper parameter tuning (Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le).<br /> <br /> ==References==<br /> <br /> #Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. arXiv preprint arXiv:1612.05086, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012. <br /> #Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference converges to limit cycles for deep networks. arXiv preprint arXiv:1710.11029, 2017.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.<br /> #Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Crispin W Gardiner. Handbook of Stochastic Methods, volume 4. Springer Berlin, 1985.<br /> #Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884– 1892, 2016.<br /> #Priya Goyal, Piotr Dolla ́r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Stephen F Gull. Bayesian inductive inference and maximum entropy. In Maximum-entropy and Bayesian methods in science and engineering, pp. 53–74. Springer, 1988.<br /> #Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM,1993.<br /> #Sepp Hochreiter and Ju ̈rgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Stanisław Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017.<br /> #Robert E Kass and Adrian E Raftery. Bayes factors. Journal of the american statistical association, 90(430):773–795, 1995.<br /> #Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don’t learn via mem- orization. ICLR Workshop, 2017.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101–2110, 2017.<br /> #David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448–472, 1992.<br /> #Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810, 2017.<br /> #Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_Bayesian_Perspective_on_Generalization_and_Stochastic_Gradient_Descent&diff=42097 A Bayesian Perspective on Generalization and Stochastic Gradient Descent 2018-11-30T17:34:16Z <p>C9sharma: /* Motivation and Related Work */</p> <hr /> <div>==Introduction==<br /> This work builds on Zhang et al.(2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. The authors consider two questions: how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? <br /> The paper shows that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. They also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.<br /> <br /> The authors propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the “noise scale” &lt;math display=&quot;inline&quot;&gt; g \approx \epsilon N/B &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;ε&lt;/math&gt; is the learning rate, &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt; the training set size and &lt;math display=&quot;inline&quot;&gt;B&lt;/math&gt; the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, &lt;math display=&quot;inline&quot;&gt;B_{opt} \propto \epsilon N&lt;/math&gt;. The authors verify these predictions empirically.<br /> <br /> ==Motivation and Related Work==<br /> Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite &amp; Roy, 2017). Nonetheless, their results beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? <br /> <br /> Meanwhile, Keskar et al. (2016) observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking result shows improving the estimate of the full-batch gradient can harm performance. Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Hoffer et al. (2017) proposed a square root rule on theoretical grounds.<br /> <br /> Many authors have suggested “broad minima” whose curvature is small may generalize better than “sharp minima” whose curvature is large (Chaudhari et al., 2016; Hochreiter &amp; Schmidhuber, 1997). Indeed, Dziugaite &amp; Roy (2017) argued the results of Zhang et al. (2016) can be understood using “nonvacuous” PAC-Bayes generalization bounds which penalize sharp minima, while Keskar et al. (2016) showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However, Dinh et al. (2017) challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization.<br /> <br /> ==Contribution==<br /> <br /> The main contributions of this paper are to show that:<br /> * The results of Zhang et al. (2016) are not unique to deep learning; it is observed the same phenomenon in a small “over-parameterized” linear model. Overparameterization occurs when a model is able to effectively “remember” training data. This occurs when there are enough parameters that the system of equations ends up with an infinite number of possible solutions. One can see why this over-training would lead to poor results in test cases, as this “memorization” learns noise as opposed to the inherent structure of different classes. It is demonstrated that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.<br /> * SGD integrates a stochastic differential equation whose “noise scale” &lt;math&gt;g &amp;asymp; &amp;epsilon;N/B&lt;/math&gt;, where<br /> &amp;epsilon; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is '''proportional to the learning rate and training set size'''.<br /> <br /> Zhang et al. (2016) showed high training competency of neural networks under informative labels, but drastic overfitting on improper labels. This implies weak generalizability even when a small proportion of labels are improper. The authors show that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Bayesians tend to make distributional assumptions on gradient updates by adding isotropic Gaussian noise. This paper builds upon these Bayesian principles by driving SGD away from sharp minima, and towards broad minima (the more broad, the better generalization due to less influence from small perturbations within input). The stochastic differential equation used as a component of gradient updates effectively serves as injected noise that improves a network's generalizability.<br /> <br /> ==Main Results==<br /> <br /> The weakly regularized model memorizes random labels, however, generalizes properly on informative labels. Besides, the predictions are overconfident. The authors also showed that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is postulated that the optimum represents a tradeoff between depth and breadth in the Bayesian evidence. However it is the underlying scale of random fluctuations in the SGD dynamics which controls the tradeoff, not the batch size itself. Furthermore, this test accuracy peak shifts as the training set size rises. The authors observed that the best found batch size is proportional to the learning rate. This scaling rule allowed the authors to increase the learning rate by simultaneously increasing the batch size with no loss in test accuracy and no increase in computational cost, thus parallelism across multiple GPU's can be fully leveraged to easily decrease training time. The scaling rule could also be applied to production models by consequentially increasing the batch size as new training data is introduced.<br /> <br /> ==Bayesian Model Comparison==<br /> <br /> ===Introduction to Bayesian Statistics===<br /> Bayes' theorem is a fundamental theorem in Bayesian statistics, as it is used by Bayesian methods to update probabilities, which are degrees of belief, after obtaining new data. Given two events &lt;math&gt;A&lt;/math&gt; and &lt;math&gt;B&lt;/math&gt;, the conditional probability of &lt;math&gt;A&lt;/math&gt; given &lt;math&gt;B &lt;/math&gt; is true, Bayes theorem states that<br /> \begin{align*}\displaystyle P(A\mid B)={\frac {P(B\mid A)P(A)}{P(B)}}\end{align*}<br /> <br /> Bayesian networks are DAGs whose nodes represent variables in the Bayesian sense: they may be observable quantities, latent variables, unknown parameters or hypotheses. Edges represent conditional dependencies; nodes that are not connected (no path connects one node to another) represent variables that are conditionally independent of each other. Each node is associated with a probability function that takes, as input, a particular set of values for the node's parent variables, and gives (as output) the probability (or probability distribution, if applicable) of the variable represented by the node. For example, if &lt;math&gt;m &lt;/math&gt; parent nodes represent &lt;math&gt;m &lt;/math&gt; Boolean variables then the probability function could be represented by a table of &lt;math&gt;2^{m} &lt;/math&gt; entries, one entry for each of the &lt;math&gt;2^{m} &lt;/math&gt; possible parent combinations. <br /> <br /> ===Bayesian Model Comparison in Neural Networks===<br /> MacKay (1992) applied Bayesian model comparison to neural networks. An overview is presented below. <br /> <br /> We first consider a classification model &lt;math&gt;M &lt;/math&gt; with a single parameter &lt;math&gt;\omega &lt;/math&gt;, training inputs &lt;math&gt;x &lt;/math&gt; and training labels &lt;math&gt;y &lt;/math&gt;. We can infer a posterior probability distribution over the parameter by applying Bayes theorem :<br /> <br /> \begin{align*}P(\omega\mid y,x;M) = \frac{P(y\mid \omega,x;M)P(\omega;M) }{P(y\mid x;M)}\end{align*}<br /> <br /> The likelihood, &lt;math&gt;P(y\mid \omega,x;M) = \Pi_i P(y_i\mid \omega,x_i;M) = e^{-H(\omega;M)} &lt;/math&gt;, where &lt;math&gt;H(\omega;M) &lt;/math&gt; denotes the cross-entropy of unique categorical labels. Using a Gaussian prior, &lt;math&gt;P(\omega;M) = \sqrt{\lambda/2\pi e^{-\lambda\omega^2/2}} &lt;/math&gt;, and therefore the posterior probability density of the parameter given the training data, &lt;math&gt;P(\omega\mid y,x;M) \propto \sqrt{\lambda/2\pi e^{-C(\omega;M)}} &lt;/math&gt;, where &lt;math&gt;C(\omega;M) = H(\omega;M) + \lambda\omega^2/2 &lt;/math&gt; denotes the L2 regularized cross entropy, or “cost function”, and &lt;math&gt;\lambda &lt;/math&gt; is the regularization coefficient. <br /> <br /> The value &lt;math&gt;\omega_0 &lt;/math&gt; which minimizes the cost function lies at the maximum of this posterior. To predict an unknown label &lt;math&gt;y_t &lt;/math&gt; of a new input &lt;math&gt;x_t &lt;/math&gt;, we should compute the integral,<br /> <br /> \begin{align*} P(y_t\mid x_t,y,x;M) &amp;= \int \frac{d\omega P(y_t\mid \omega,x_t;M)}{P(\omega\mid y,x;M)}\\ &amp;= \frac{\int d \omega P(y_t \mid \omega ,x_t;M)e^{-C(\omega;M)}}{\int d \omega e^{-C(\omega;M)}} \end{align*}&lt;/math&gt;<br /> <br /> However, these integrals are dominated by the region near &lt;math&gt;\omega_0 &lt;/math&gt; . We usually approximate &lt;math&gt;P(y_t\mid x_t,x,y;M) \approx P(y_t\mid \omega_0,x_t;M) &lt;/math&gt;. Having minimized &lt;math&gt;C(\omega;M) &lt;/math&gt; to find &lt;math&gt;\omega_0 &lt;/math&gt;, we now wish to compare two different models and select the best one. We use the probability ratio<br /> <br /> \begin{align*}\frac{P(M_1\mid y,x)}{P (M_2\mid y, x)} = \frac{P(y\mid x;M_1) P(M_1)}{ P (y\mid x; M_2) P (M_2)} . \end{align*} <br /> <br /> The second factor on the right is the prior ratio, which describes which model is most plausible. To avoid unnecessary subjectivity, we usually set this to 1. Meanwhile the first factor on the right is the evidence ratio, which controls how much the training data changes our prior beliefs<br /> <br /> Germain et al. (2016) showed that maximizing the evidence (or “marginal likelihood”) minimizes a PAC-Bayes generalization bound. To compute it, we evaluate <br /> \begin{align*}P(y\mid x;M) &amp;= \int d\omega P(y\mid \omega,x;M)P(\omega;M) \\ &amp;=\sqrt{\frac{\lambda}{2\pi}}\int d \omega e^{C(\omega;M)}\end{align*}<br /> <br /> Notice that the evidence is computed by integrating out the parameters; and consequently it is invariant to the model parameterization. <br /> Since this integral is dominated by the region near the minimum &lt;math&gt;\omega_0 &lt;/math&gt;, we can estimate the evidence by Taylor expanding &lt;math&gt;C(\omega; M) \approx C(\omega_0) + C′′(\omega_0)(\omega - \omega_0)^2/2&lt;/math&gt;. This gives us<br /> <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2}\\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2}\ln(C (\omega_0)/\lambda) \big\}.\end{align*}<br /> <br /> The evidence is controlled by the value of the cost function at the minimum, and by the logarithm of the ratio of the curvature about this minimum compared to the regularization constant. In models with many parameters <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2} \\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) \big\}.\end{align*}<br /> <br /> Occam’s factor arises from the log ratio &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; The Occam factor describes the fraction of the prior parameter space consistent with the data. Occam’s factor penalizes the amount of information the model must learn about the parameters to accurately model the training data. Since the fraction is always less than one, the authors propose to approximate &lt;math&gt;P(y\mid x;M) &lt;/math&gt; away from local minima by only performing the summation over eigenvalues &lt;math&gt;\lambda_i \geq \lambda &lt;/math&gt;.<br /> <br /> The authors compare evidence against a null model which assumes the labels are entirely random. This model has no parameters, and so the evidence is controlled by the likelihood alone. &lt;math&gt;P(y\mid x;NULL) = (1/n)^N = e^{-N \ln(n)} &lt;/math&gt;, where &lt;math&gt;n &lt;/math&gt; denotes the number of model classes and &lt;math&gt;N&lt;/math&gt; the number of training labels. The evidence ratio :<br /> \begin{equation*}\frac{P(y\mid x;M) }{P(y\mid x;NULL) } = e ^{-E(\omega_0)} \end{equation*}<br /> &lt;math&gt;E(\omega_0) = C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) - N\ln (n) &lt;/math&gt; is the log evidence ratio in favor of the null model.<br /> The authors assign confidence to the predictions of a model iff &lt;math&gt;E(\omega_0 &lt; 0 &lt;/math&gt;.<br /> <br /> The evidence supports the intuition that broad minima generalize better than sharp minima, but unlike the curvature it does not depend on the model parameterization. Dinh et al. (2017) showed one can increase the Hessian eigenvalues by rescaling the parameters, but they must simultaneously rescale the regularization coefficients, otherwise the model changes. Since Occam’s factor arises from the log ratio, &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; , these two effects cancel out. Note however that while the evidence itself is invariant to model parameterization, one can find reparameterizations which change the approximate evidence after the Laplace approximation. . It is difficult to evaluate the evidence for deep networks, as we cannot compute the Hessian of millions of parameters. Additionally, neural networks exhibit many equivalent minima, since we can permute the hidden units without changing the model. To compute the evidence we must carefully account for this “degeneracy”. The authors argue these issues are not a major limitation, since the intuition they build studying the evidence in simple cases will be sufficient to explain the results of both Zhang et al. (2016) and Keskar et al. (2016).<br /> <br /> ==Bayes Theorem and Generalization==<br /> Zhang et al. (2016) showed that deep neural networks generalize well on training inputs with informative labels, but the same model can overfit on the same input images when the labels are randomized; perfectly memorizing the training set. To demonstrate that these observations are not unique to deep network, the authors use logistic regression. They form a small balanced training set comprising 800 images from MNIST, of which half have true label “0” and half true label “1”. The test set is balanced, comprising 5000 MNIST images of zeros and 5000 MNIST images of ones. There are two tasks. In the first task, the labels of both the training and test sets are randomized. In the second task, the labels are informative, matching the true MNIST labels. The model has 784 weights and 1 bias.<br /> <br /> The accuracy of the model predictions on both the training and test sets is shown in figure 1. When trained on the informative labels, the model generalizes well to the test set, so long as it is weakly regularized. However the model also perfectly memorizes the random labels, replicating the obser- vations of Zhang et al. (2016) in deep networks. No significant improvement in model performance is observed as the regularization coefficient increases. For completeness, we also evaluate the mean margin between training examples and the decision boundary. For both random and informative labels, the margin drops significantly as we reduce the regularization coefficient. When weakly regularized, the mean margin is roughly 50% larger for informative labels than for random labels.<br /> <br /> [[File:bg1.png|800px|thumb|center|]]<br /> <br /> Now consider figure 2, where we plot the mean cross-entropy of the model predictions, evaluated on both training and test sets, as well as the Bayesian log evidence ratio defined in the previous section. Looking first at the random label experiment in figure 2a, while the cross-entropy on the training set vanishes when the model is weakly regularized, the cross-entropy on the test set explodes. Not only does the model make random predictions, but it is extremely confident in those predictions. As the regularization coefficient is increased the test set cross-entropy falls, settling at ln 2, the cross- entropy of assigning equal probability to both classes. Now consider the Bayesian evidence, which we evaluate on the training set. The log evidence ratio is large and positive when the model is weakly regularized, indicating that the model is exponentially less plausible than assigning equal probabilities to each class. As the regularization parameter is increased, the log evidence ratio falls, but it is always positive, indicating that the model can never be expected to generalize well.<br /> Now consider figure 2b (informative labels). Once again, the training cross-entropy falls to zero when the model is weakly regularized, while the test cross-entropy is high. Even though the model makes accurate predictions, those predictions are overconfident. As the regularization coefficient increases, the test cross-entropy falls below ln 2, indicating that the model is successfully gener- alizing to the test set. Now consider the Bayesian evidence. The log evidence ratio is large and positive when the model is weakly regularized, but as the regularization coefficient increases, the log evidence ratio drops below zero, indicating that the model is exponentially more plausible than assigning equal probabilities to each class. As we further increase the regularization, the log evi- dence ratio rises to zero while the test cross-entropy rises to ln 2. Test cross-entropy and Bayesian evidence are strongly correlated, with minima at the same regularization strength.<br /> Bayesian model comparison has explained our results in a logistic regression. Meanwhile, Krueger et al. (2017) showed the largest Hessian eigenvalue also increased when training on random labels in deep networks, implying the evidence is falling. We conclude that Bayesian model comparison is quantitatively consistent with the results of Zhang et al. (2016) in linear models where we can compute the evidence, and qualitatively consistent with their results in deep networks where we cannot. Dziugaite &amp; Roy (2017) recently demonstrated the results of Zhang et al. (2016) can also be understood by minimising a PAC-Bayes generalization bound which penalizes sharp minima.<br /> [[File:bg2.png|800px|thumb|center|]]<br /> ==Bayes Theorem and Stochastic Gradient Descent ==<br /> <br /> We showed above that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Consequently Bayesians often add isotropic Gaussian noise to the gradient (Welling &amp; Teh, 2011). In appendix A, we show this drives the parameters towards broad minima whose evidence is large. The noise introduced by small batch training is not isotropic, and its covariance matrix is a function of the parameter values, but empirically Keskar et al. (2016) found it has similar effects, driving the SGD away from sharp minima. This paper therefore proposes Bayesian principles also account for the “generalization gap”, whereby the test set accuracy often falls as the SGD batch size is increased (holding all other hyper-parameters constant). Since the gradient drives the SGD towards deep minima, while noise drives the SGD towards broad minima, we expect the test set performance to show a peak at an optimal batch size, which balances these competing contributions to the evidence.<br /> We were unable to observe a generalization gap in linear models (since linear models are convex there are no sharp minima to avoid). Instead we consider a shallow neural network with 800 hidden units and RELU hidden activations, trained on MNIST without regularization. We use SGD with a momentum parameter of 0.9. Unless otherwise stated, we use a constant learning rate of 1.0 which does not depend on the batch size or decay during training. Furthermore, we train on just 1000 images, selected at random from the MNIST training set. This enables us to compare small batch to full batch training. We emphasize that we are not trying to achieve optimal performance, but to study a simple model which shows a generalization gap between small and large batch training.<br /> In figure 3, we exhibit the evolution of the test accuracy and test cross-entropy during training. Our small batches are composed of 30 images, randomly sampled from the training set. Looking first at figure 3a, small batch training takes longer to converge, but after a thousand gradient updates a clear generalization gap in model accuracy emerges between small and large training batches. Now consider figure 3b. While the test cross-entropy for small batch training is lower at the end of training; the cross-entropy of both small and large training batches is increasing, indicative of over-fitting. Both models exhibit a minimum test cross-entropy, although after different numbers of gradient updates. Intriguingly, we show in appendix B that the generalization gap between small and large batch training shrinks significantly when we introduce L2 regularization.<br /> <br /> [[File:bg3.png|800px|thumb|center|]]<br /> <br /> From now on we focus on the test set accuracy (since this converges as the number of gradient updates increases). In figure 4a, we exhibit training curves for a range of batch sizes between 1 and 1000. We find that the model cannot train when the batch size B 􏰘 10. In figure 4b we plot the mean test set accuracy after 10000 training steps. A clear peak emerges, indicating that there is indeed an optimum batch size which maximizes the test accuracy, consistent with Bayesian intuition. The results of Keskar et al. (2016) focused on the decay in test accuracy above this optimum batch size.<br /> [[File:bg4.png|800px|thumb|center|]]<br /> <br /> ==Stochastic Differential Equations and Scaling Rules==<br /> The results showed above indicate that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is argued that this peak arises from the tradeoff between depth and breadth in the Bayesian evidence. However it is not the batch size itself which controls this tradeoff, but the underlying scale of random fluctuations in the SGD dynamics. The following content identifies this SGD “noise scale”, and uses it to derive three scaling rules which predict how the optimal batch size depends on the learning rate, training set size and momentum coefficient. <br /> First, interpret gradient update, as the discrete update of a stochastic differential equation <br /> \begin{equation*}\frac{d\omega}{dt} = \frac{dC}{d\omega} + \eta(t)\end{equation*}<br /> &lt;math&gt;\eta&lt;/math&gt; represents noise &lt;math&gt;\langle \eta(t) \rangle = 0&lt;/math&gt; and &lt;math&gt; \langle \eta (t)\eta (t')\rangle = gF (\omega)\delta (t-t')&lt;/math&gt;.<br /> &lt;math&gt;t&lt;/math&gt; is a continous variable, and &lt;math&gt;F(\omega)&lt;/math&gt; matrix describing the gradient covariances.<br /> The SGD noise scale is taken to be &lt;math&gt;g \approx \epsilon N/B&lt;/math&gt; where &lt;math&gt;\epsilon&lt;/math&gt; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; the batch size.<br /> [[File:bg5.png|800px|thumb|center|]]<br /> [[File:bg6.png|800px|thumb|center|]]<br /> [[File:bg7.png|800px|thumb|center|]]<br /> The noise scale falls when the batch B<br /> size increases, consistent with our earlier observation of an optimal batch size Bopt while holding the other hyper-parameters fixed. Notice that one would equivalently observe an optimal learning rate if one held the batch size constant. A similar analysis of the SGD was recently performed by Mandt et al. (2017), although their treatment only holds near local minima where the covariances F (ω) are stationary. Our analysis holds throughout training, which is necessary since Keskar et al. (2016) found that the beneficial influence of noise was most pronounced at the start of training.<br /> When we vary the learning rate or the training set size, we should keep the noise scale fixed, which implies that Bopt ∝ εN. In figure 5a, we plot the test accuracy as a function of batch size after (10000/ε) training steps, for a range of learning rates. Exactly as predicted, the peak moves to the right as ε increases. Additionally, the peak test accuracy achieved at a given learning rate does not begin to fall until ε ∼ 3, indicating that there is no significant discretization error in integrating the stochastic differential equation below this point. Above this point, the discretization error begins to dominate and the peak test accuracy falls rapidly. In figure 5b, we plot the best observed batch size as a function of learning rate, observing a clear linear trend, Bopt ∝ ε. The error bars indicate the distance from the best observed batch size to the next batch size sampled in our experiments.<br /> <br /> This scaling rule allows us to increase the learning rate with no loss in test accuracy and no increase in computational cost, simply by simultaneously increasing the batch size. We can then exploit increased parallelism across multiple GPUs, reducing model training times (Goyal et al., 2017). A similar scaling rule was independently proposed by Jastrzebski et al. (2017) and Chaudhari &amp; Soatto (2017), although neither work identifies the existence of an optimal noise scale. A number of authors have proposed adjusting the batch size adaptively during training (Friedlander &amp; Schmidt, 2012; Byrd et al., 2012; De et al., 2017), while Balles et al. (2016) proposed linearly coupling the learning rate and batch size within this framework. In Smith et al. (2017), we show empirically that decaying the learning rate during training and increasing the batch size during training are equivalent.<br /> In figure 6a we exhibit the test set accuracy as a function of batch size, for a range of training set sizes after 10000 steps (ε = 1 everywhere). Once again, the peak shifts right as the training set size rises, although the generalization gap becomes less pronounced as the training set size increases. In figure 6b, we plot the best observed batch size as a function of training set size; observing another linear trend, Bopt ∝ N. This scaling rule could be applied to production models, progressively growing the batch size as new training data is collected. We expect production datasets to grow considerably over time, and consequently large batch training is likely to become increasingly common.<br /> B(1−m)<br /> scale of conventional SGD as m → 0. When m &gt; 0, we obtain an additional scaling rule Bopt ∝ 1/(1 − m). This scaling rule predicts that the optimal batch size will increase when the momentum coefficient is increased. In figure 7a we plot the test set performance as a function of batch size after 10000 gradient updates (ε = 1 everywhere), for a range of momentum coefficients. In figure 7b, we plot the best observed batch size as a function of the momentum coefficient, and fit our results to the scaling rule above; obtaining remarkably good agreement.<br /> <br /> ==Critiques==<br /> <br /> #Bayesian statistics is not provably, at present, a theory that can be used to explain why a learning algorithm works. The Bayesian theory is too optimistic: we introduce a prior and model and then trust both implicitly. Relative to any particular prior and model (likelihood), the Bayesian posterior is the optimal summary of the data, but if either part is misspecified, then the Bayesian posterior carries no optimality guarantee. The prior is chosen for convenience here. <br /> #No discussions with respect to the analysis of information bottleneck which also discuss the generalization ability of the model. <br /> #No discussion on real online learning with streaming data where the total number of data points are unknown?<br /> #The paper presents how mini-batch noises with SGD can improve the performance of neural networks. However, the usefulness of the approach can be described and analyzed in greater details, if the author could provide the performance for various well-known real-life datas.<br /> <br /> ==Conclusion==<br /> <br /> The paper showed that Mini-batch noise helps SGD to go away from sharp minima, and provided an evidence that there is an optimal optimum batch size for a maximum the test accuracy. Based on interpreting SGD as integrating stochastic differential equation, this batch size is proportional to the learning rate and the training set size. Moreover, the authors shown that &lt;math&gt;Bopt \propto 1/(1 − m) &lt;/math&gt;, where m is the momentum coefficient. More analysis was done on the relation between the learning rate, effective learning rate, and batch size is presented in ICLR 2018, where the authors proved by experiments that all the benefits of decaying the learning rate are achieved by increasing the batch size in addition to reducing the number of parameter updates dramatically, and also were able use literature parameters without the need of any hyper parameter tuning (Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le).<br /> <br /> ==References==<br /> <br /> #Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. arXiv preprint arXiv:1612.05086, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012. <br /> #Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference converges to limit cycles for deep networks. arXiv preprint arXiv:1710.11029, 2017.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.<br /> #Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Crispin W Gardiner. Handbook of Stochastic Methods, volume 4. Springer Berlin, 1985.<br /> #Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884– 1892, 2016.<br /> #Priya Goyal, Piotr Dolla ́r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Stephen F Gull. Bayesian inductive inference and maximum entropy. In Maximum-entropy and Bayesian methods in science and engineering, pp. 53–74. Springer, 1988.<br /> #Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM,1993.<br /> #Sepp Hochreiter and Ju ̈rgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Stanisław Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017.<br /> #Robert E Kass and Adrian E Raftery. Bayes factors. Journal of the american statistical association, 90(430):773–795, 1995.<br /> #Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don’t learn via mem- orization. ICLR Workshop, 2017.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101–2110, 2017.<br /> #David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448–472, 1992.<br /> #Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810, 2017.<br /> #Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_Bayesian_Perspective_on_Generalization_and_Stochastic_Gradient_Descent&diff=42096 A Bayesian Perspective on Generalization and Stochastic Gradient Descent 2018-11-30T17:32:51Z <p>C9sharma: /* Conclusion */</p> <hr /> <div>==Introduction==<br /> This work builds on Zhang et al.(2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. The authors consider two questions: how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? <br /> The paper shows that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. They also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.<br /> <br /> The authors propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the “noise scale” &lt;math display=&quot;inline&quot;&gt; g \approx \epsilon N/B &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;ε&lt;/math&gt; is the learning rate, &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt; the training set size and &lt;math display=&quot;inline&quot;&gt;B&lt;/math&gt; the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, &lt;math display=&quot;inline&quot;&gt;B_{opt} \propto \epsilon N&lt;/math&gt;. The authors verify these predictions empirically.<br /> <br /> ==Motivation and Related Work==<br /> This paper shows Bayesian principles can explain many recent observations in the deep learning literature, while also discovering practical new insights. Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite &amp; Roy, 2017). Nonetheless, their results beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? Meanwhile, Keskar et al. (2016) observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking result shows improving our estimate of the full-batch gradient can harm performance. Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Hoffer et al. (2017) proposed a square root rule on theoretical grounds.<br /> Many authors have suggested “broad minima” whose curvature is small may generalize better than “sharp minima” whose curvature is large (Chaudhari et al., 2016; Hochreiter &amp; Schmidhuber, 1997). Indeed, Dziugaite &amp; Roy (2017) argued the results of Zhang et al. (2016) can be understood using “nonvacuous” PAC-Bayes generalization bounds which penalize sharp minima, while Keskar et al. (2016) showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However, Dinh et al. (2017) challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization.<br /> <br /> ==Contribution==<br /> <br /> The main contributions of this paper are to show that:<br /> * The results of Zhang et al. (2016) are not unique to deep learning; it is observed the same phenomenon in a small “over-parameterized” linear model. Overparameterization occurs when a model is able to effectively “remember” training data. This occurs when there are enough parameters that the system of equations ends up with an infinite number of possible solutions. One can see why this over-training would lead to poor results in test cases, as this “memorization” learns noise as opposed to the inherent structure of different classes. It is demonstrated that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.<br /> * SGD integrates a stochastic differential equation whose “noise scale” &lt;math&gt;g &amp;asymp; &amp;epsilon;N/B&lt;/math&gt;, where<br /> &amp;epsilon; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is '''proportional to the learning rate and training set size'''.<br /> <br /> Zhang et al. (2016) showed high training competency of neural networks under informative labels, but drastic overfitting on improper labels. This implies weak generalizability even when a small proportion of labels are improper. The authors show that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Bayesians tend to make distributional assumptions on gradient updates by adding isotropic Gaussian noise. This paper builds upon these Bayesian principles by driving SGD away from sharp minima, and towards broad minima (the more broad, the better generalization due to less influence from small perturbations within input). The stochastic differential equation used as a component of gradient updates effectively serves as injected noise that improves a network's generalizability.<br /> <br /> ==Main Results==<br /> <br /> The weakly regularized model memorizes random labels, however, generalizes properly on informative labels. Besides, the predictions are overconfident. The authors also showed that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is postulated that the optimum represents a tradeoff between depth and breadth in the Bayesian evidence. However it is the underlying scale of random fluctuations in the SGD dynamics which controls the tradeoff, not the batch size itself. Furthermore, this test accuracy peak shifts as the training set size rises. The authors observed that the best found batch size is proportional to the learning rate. This scaling rule allowed the authors to increase the learning rate by simultaneously increasing the batch size with no loss in test accuracy and no increase in computational cost, thus parallelism across multiple GPU's can be fully leveraged to easily decrease training time. The scaling rule could also be applied to production models by consequentially increasing the batch size as new training data is introduced.<br /> <br /> ==Bayesian Model Comparison==<br /> <br /> ===Introduction to Bayesian Statistics===<br /> Bayes' theorem is a fundamental theorem in Bayesian statistics, as it is used by Bayesian methods to update probabilities, which are degrees of belief, after obtaining new data. Given two events &lt;math&gt;A&lt;/math&gt; and &lt;math&gt;B&lt;/math&gt;, the conditional probability of &lt;math&gt;A&lt;/math&gt; given &lt;math&gt;B &lt;/math&gt; is true, Bayes theorem states that<br /> \begin{align*}\displaystyle P(A\mid B)={\frac {P(B\mid A)P(A)}{P(B)}}\end{align*}<br /> <br /> Bayesian networks are DAGs whose nodes represent variables in the Bayesian sense: they may be observable quantities, latent variables, unknown parameters or hypotheses. Edges represent conditional dependencies; nodes that are not connected (no path connects one node to another) represent variables that are conditionally independent of each other. Each node is associated with a probability function that takes, as input, a particular set of values for the node's parent variables, and gives (as output) the probability (or probability distribution, if applicable) of the variable represented by the node. For example, if &lt;math&gt;m &lt;/math&gt; parent nodes represent &lt;math&gt;m &lt;/math&gt; Boolean variables then the probability function could be represented by a table of &lt;math&gt;2^{m} &lt;/math&gt; entries, one entry for each of the &lt;math&gt;2^{m} &lt;/math&gt; possible parent combinations. <br /> <br /> ===Bayesian Model Comparison in Neural Networks===<br /> MacKay (1992) applied Bayesian model comparison to neural networks. An overview is presented below. <br /> <br /> We first consider a classification model &lt;math&gt;M &lt;/math&gt; with a single parameter &lt;math&gt;\omega &lt;/math&gt;, training inputs &lt;math&gt;x &lt;/math&gt; and training labels &lt;math&gt;y &lt;/math&gt;. We can infer a posterior probability distribution over the parameter by applying Bayes theorem :<br /> <br /> \begin{align*}P(\omega\mid y,x;M) = \frac{P(y\mid \omega,x;M)P(\omega;M) }{P(y\mid x;M)}\end{align*}<br /> <br /> The likelihood, &lt;math&gt;P(y\mid \omega,x;M) = \Pi_i P(y_i\mid \omega,x_i;M) = e^{-H(\omega;M)} &lt;/math&gt;, where &lt;math&gt;H(\omega;M) &lt;/math&gt; denotes the cross-entropy of unique categorical labels. Using a Gaussian prior, &lt;math&gt;P(\omega;M) = \sqrt{\lambda/2\pi e^{-\lambda\omega^2/2}} &lt;/math&gt;, and therefore the posterior probability density of the parameter given the training data, &lt;math&gt;P(\omega\mid y,x;M) \propto \sqrt{\lambda/2\pi e^{-C(\omega;M)}} &lt;/math&gt;, where &lt;math&gt;C(\omega;M) = H(\omega;M) + \lambda\omega^2/2 &lt;/math&gt; denotes the L2 regularized cross entropy, or “cost function”, and &lt;math&gt;\lambda &lt;/math&gt; is the regularization coefficient. <br /> <br /> The value &lt;math&gt;\omega_0 &lt;/math&gt; which minimizes the cost function lies at the maximum of this posterior. To predict an unknown label &lt;math&gt;y_t &lt;/math&gt; of a new input &lt;math&gt;x_t &lt;/math&gt;, we should compute the integral,<br /> <br /> \begin{align*} P(y_t\mid x_t,y,x;M) &amp;= \int \frac{d\omega P(y_t\mid \omega,x_t;M)}{P(\omega\mid y,x;M)}\\ &amp;= \frac{\int d \omega P(y_t \mid \omega ,x_t;M)e^{-C(\omega;M)}}{\int d \omega e^{-C(\omega;M)}} \end{align*}&lt;/math&gt;<br /> <br /> However, these integrals are dominated by the region near &lt;math&gt;\omega_0 &lt;/math&gt; . We usually approximate &lt;math&gt;P(y_t\mid x_t,x,y;M) \approx P(y_t\mid \omega_0,x_t;M) &lt;/math&gt;. Having minimized &lt;math&gt;C(\omega;M) &lt;/math&gt; to find &lt;math&gt;\omega_0 &lt;/math&gt;, we now wish to compare two different models and select the best one. We use the probability ratio<br /> <br /> \begin{align*}\frac{P(M_1\mid y,x)}{P (M_2\mid y, x)} = \frac{P(y\mid x;M_1) P(M_1)}{ P (y\mid x; M_2) P (M_2)} . \end{align*} <br /> <br /> The second factor on the right is the prior ratio, which describes which model is most plausible. To avoid unnecessary subjectivity, we usually set this to 1. Meanwhile the first factor on the right is the evidence ratio, which controls how much the training data changes our prior beliefs<br /> <br /> Germain et al. (2016) showed that maximizing the evidence (or “marginal likelihood”) minimizes a PAC-Bayes generalization bound. To compute it, we evaluate <br /> \begin{align*}P(y\mid x;M) &amp;= \int d\omega P(y\mid \omega,x;M)P(\omega;M) \\ &amp;=\sqrt{\frac{\lambda}{2\pi}}\int d \omega e^{C(\omega;M)}\end{align*}<br /> <br /> Notice that the evidence is computed by integrating out the parameters; and consequently it is invariant to the model parameterization. <br /> Since this integral is dominated by the region near the minimum &lt;math&gt;\omega_0 &lt;/math&gt;, we can estimate the evidence by Taylor expanding &lt;math&gt;C(\omega; M) \approx C(\omega_0) + C′′(\omega_0)(\omega - \omega_0)^2/2&lt;/math&gt;. This gives us<br /> <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2}\\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2}\ln(C (\omega_0)/\lambda) \big\}.\end{align*}<br /> <br /> The evidence is controlled by the value of the cost function at the minimum, and by the logarithm of the ratio of the curvature about this minimum compared to the regularization constant. In models with many parameters <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2} \\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) \big\}.\end{align*}<br /> <br /> Occam’s factor arises from the log ratio &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; The Occam factor describes the fraction of the prior parameter space consistent with the data. Occam’s factor penalizes the amount of information the model must learn about the parameters to accurately model the training data. Since the fraction is always less than one, the authors propose to approximate &lt;math&gt;P(y\mid x;M) &lt;/math&gt; away from local minima by only performing the summation over eigenvalues &lt;math&gt;\lambda_i \geq \lambda &lt;/math&gt;.<br /> <br /> The authors compare evidence against a null model which assumes the labels are entirely random. This model has no parameters, and so the evidence is controlled by the likelihood alone. &lt;math&gt;P(y\mid x;NULL) = (1/n)^N = e^{-N \ln(n)} &lt;/math&gt;, where &lt;math&gt;n &lt;/math&gt; denotes the number of model classes and &lt;math&gt;N&lt;/math&gt; the number of training labels. The evidence ratio :<br /> \begin{equation*}\frac{P(y\mid x;M) }{P(y\mid x;NULL) } = e ^{-E(\omega_0)} \end{equation*}<br /> &lt;math&gt;E(\omega_0) = C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) - N\ln (n) &lt;/math&gt; is the log evidence ratio in favor of the null model.<br /> The authors assign confidence to the predictions of a model iff &lt;math&gt;E(\omega_0 &lt; 0 &lt;/math&gt;.<br /> <br /> The evidence supports the intuition that broad minima generalize better than sharp minima, but unlike the curvature it does not depend on the model parameterization. Dinh et al. (2017) showed one can increase the Hessian eigenvalues by rescaling the parameters, but they must simultaneously rescale the regularization coefficients, otherwise the model changes. Since Occam’s factor arises from the log ratio, &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; , these two effects cancel out. Note however that while the evidence itself is invariant to model parameterization, one can find reparameterizations which change the approximate evidence after the Laplace approximation. . It is difficult to evaluate the evidence for deep networks, as we cannot compute the Hessian of millions of parameters. Additionally, neural networks exhibit many equivalent minima, since we can permute the hidden units without changing the model. To compute the evidence we must carefully account for this “degeneracy”. The authors argue these issues are not a major limitation, since the intuition they build studying the evidence in simple cases will be sufficient to explain the results of both Zhang et al. (2016) and Keskar et al. (2016).<br /> <br /> ==Bayes Theorem and Generalization==<br /> Zhang et al. (2016) showed that deep neural networks generalize well on training inputs with informative labels, but the same model can overfit on the same input images when the labels are randomized; perfectly memorizing the training set. To demonstrate that these observations are not unique to deep network, the authors use logistic regression. They form a small balanced training set comprising 800 images from MNIST, of which half have true label “0” and half true label “1”. The test set is balanced, comprising 5000 MNIST images of zeros and 5000 MNIST images of ones. There are two tasks. In the first task, the labels of both the training and test sets are randomized. In the second task, the labels are informative, matching the true MNIST labels. The model has 784 weights and 1 bias.<br /> <br /> The accuracy of the model predictions on both the training and test sets is shown in figure 1. When trained on the informative labels, the model generalizes well to the test set, so long as it is weakly regularized. However the model also perfectly memorizes the random labels, replicating the obser- vations of Zhang et al. (2016) in deep networks. No significant improvement in model performance is observed as the regularization coefficient increases. For completeness, we also evaluate the mean margin between training examples and the decision boundary. For both random and informative labels, the margin drops significantly as we reduce the regularization coefficient. When weakly regularized, the mean margin is roughly 50% larger for informative labels than for random labels.<br /> <br /> [[File:bg1.png|800px|thumb|center|]]<br /> <br /> Now consider figure 2, where we plot the mean cross-entropy of the model predictions, evaluated on both training and test sets, as well as the Bayesian log evidence ratio defined in the previous section. Looking first at the random label experiment in figure 2a, while the cross-entropy on the training set vanishes when the model is weakly regularized, the cross-entropy on the test set explodes. Not only does the model make random predictions, but it is extremely confident in those predictions. As the regularization coefficient is increased the test set cross-entropy falls, settling at ln 2, the cross- entropy of assigning equal probability to both classes. Now consider the Bayesian evidence, which we evaluate on the training set. The log evidence ratio is large and positive when the model is weakly regularized, indicating that the model is exponentially less plausible than assigning equal probabilities to each class. As the regularization parameter is increased, the log evidence ratio falls, but it is always positive, indicating that the model can never be expected to generalize well.<br /> Now consider figure 2b (informative labels). Once again, the training cross-entropy falls to zero when the model is weakly regularized, while the test cross-entropy is high. Even though the model makes accurate predictions, those predictions are overconfident. As the regularization coefficient increases, the test cross-entropy falls below ln 2, indicating that the model is successfully gener- alizing to the test set. Now consider the Bayesian evidence. The log evidence ratio is large and positive when the model is weakly regularized, but as the regularization coefficient increases, the log evidence ratio drops below zero, indicating that the model is exponentially more plausible than assigning equal probabilities to each class. As we further increase the regularization, the log evi- dence ratio rises to zero while the test cross-entropy rises to ln 2. Test cross-entropy and Bayesian evidence are strongly correlated, with minima at the same regularization strength.<br /> Bayesian model comparison has explained our results in a logistic regression. Meanwhile, Krueger et al. (2017) showed the largest Hessian eigenvalue also increased when training on random labels in deep networks, implying the evidence is falling. We conclude that Bayesian model comparison is quantitatively consistent with the results of Zhang et al. (2016) in linear models where we can compute the evidence, and qualitatively consistent with their results in deep networks where we cannot. Dziugaite &amp; Roy (2017) recently demonstrated the results of Zhang et al. (2016) can also be understood by minimising a PAC-Bayes generalization bound which penalizes sharp minima.<br /> [[File:bg2.png|800px|thumb|center|]]<br /> ==Bayes Theorem and Stochastic Gradient Descent ==<br /> <br /> We showed above that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Consequently Bayesians often add isotropic Gaussian noise to the gradient (Welling &amp; Teh, 2011). In appendix A, we show this drives the parameters towards broad minima whose evidence is large. The noise introduced by small batch training is not isotropic, and its covariance matrix is a function of the parameter values, but empirically Keskar et al. (2016) found it has similar effects, driving the SGD away from sharp minima. This paper therefore proposes Bayesian principles also account for the “generalization gap”, whereby the test set accuracy often falls as the SGD batch size is increased (holding all other hyper-parameters constant). Since the gradient drives the SGD towards deep minima, while noise drives the SGD towards broad minima, we expect the test set performance to show a peak at an optimal batch size, which balances these competing contributions to the evidence.<br /> We were unable to observe a generalization gap in linear models (since linear models are convex there are no sharp minima to avoid). Instead we consider a shallow neural network with 800 hidden units and RELU hidden activations, trained on MNIST without regularization. We use SGD with a momentum parameter of 0.9. Unless otherwise stated, we use a constant learning rate of 1.0 which does not depend on the batch size or decay during training. Furthermore, we train on just 1000 images, selected at random from the MNIST training set. This enables us to compare small batch to full batch training. We emphasize that we are not trying to achieve optimal performance, but to study a simple model which shows a generalization gap between small and large batch training.<br /> In figure 3, we exhibit the evolution of the test accuracy and test cross-entropy during training. Our small batches are composed of 30 images, randomly sampled from the training set. Looking first at figure 3a, small batch training takes longer to converge, but after a thousand gradient updates a clear generalization gap in model accuracy emerges between small and large training batches. Now consider figure 3b. While the test cross-entropy for small batch training is lower at the end of training; the cross-entropy of both small and large training batches is increasing, indicative of over-fitting. Both models exhibit a minimum test cross-entropy, although after different numbers of gradient updates. Intriguingly, we show in appendix B that the generalization gap between small and large batch training shrinks significantly when we introduce L2 regularization.<br /> <br /> [[File:bg3.png|800px|thumb|center|]]<br /> <br /> From now on we focus on the test set accuracy (since this converges as the number of gradient updates increases). In figure 4a, we exhibit training curves for a range of batch sizes between 1 and 1000. We find that the model cannot train when the batch size B 􏰘 10. In figure 4b we plot the mean test set accuracy after 10000 training steps. A clear peak emerges, indicating that there is indeed an optimum batch size which maximizes the test accuracy, consistent with Bayesian intuition. The results of Keskar et al. (2016) focused on the decay in test accuracy above this optimum batch size.<br /> [[File:bg4.png|800px|thumb|center|]]<br /> <br /> ==Stochastic Differential Equations and Scaling Rules==<br /> The results showed above indicate that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is argued that this peak arises from the tradeoff between depth and breadth in the Bayesian evidence. However it is not the batch size itself which controls this tradeoff, but the underlying scale of random fluctuations in the SGD dynamics. The following content identifies this SGD “noise scale”, and uses it to derive three scaling rules which predict how the optimal batch size depends on the learning rate, training set size and momentum coefficient. <br /> First, interpret gradient update, as the discrete update of a stochastic differential equation <br /> \begin{equation*}\frac{d\omega}{dt} = \frac{dC}{d\omega} + \eta(t)\end{equation*}<br /> &lt;math&gt;\eta&lt;/math&gt; represents noise &lt;math&gt;\langle \eta(t) \rangle = 0&lt;/math&gt; and &lt;math&gt; \langle \eta (t)\eta (t')\rangle = gF (\omega)\delta (t-t')&lt;/math&gt;.<br /> &lt;math&gt;t&lt;/math&gt; is a continous variable, and &lt;math&gt;F(\omega)&lt;/math&gt; matrix describing the gradient covariances.<br /> The SGD noise scale is taken to be &lt;math&gt;g \approx \epsilon N/B&lt;/math&gt; where &lt;math&gt;\epsilon&lt;/math&gt; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; the batch size.<br /> [[File:bg5.png|800px|thumb|center|]]<br /> [[File:bg6.png|800px|thumb|center|]]<br /> [[File:bg7.png|800px|thumb|center|]]<br /> The noise scale falls when the batch B<br /> size increases, consistent with our earlier observation of an optimal batch size Bopt while holding the other hyper-parameters fixed. Notice that one would equivalently observe an optimal learning rate if one held the batch size constant. A similar analysis of the SGD was recently performed by Mandt et al. (2017), although their treatment only holds near local minima where the covariances F (ω) are stationary. Our analysis holds throughout training, which is necessary since Keskar et al. (2016) found that the beneficial influence of noise was most pronounced at the start of training.<br /> When we vary the learning rate or the training set size, we should keep the noise scale fixed, which implies that Bopt ∝ εN. In figure 5a, we plot the test accuracy as a function of batch size after (10000/ε) training steps, for a range of learning rates. Exactly as predicted, the peak moves to the right as ε increases. Additionally, the peak test accuracy achieved at a given learning rate does not begin to fall until ε ∼ 3, indicating that there is no significant discretization error in integrating the stochastic differential equation below this point. Above this point, the discretization error begins to dominate and the peak test accuracy falls rapidly. In figure 5b, we plot the best observed batch size as a function of learning rate, observing a clear linear trend, Bopt ∝ ε. The error bars indicate the distance from the best observed batch size to the next batch size sampled in our experiments.<br /> <br /> This scaling rule allows us to increase the learning rate with no loss in test accuracy and no increase in computational cost, simply by simultaneously increasing the batch size. We can then exploit increased parallelism across multiple GPUs, reducing model training times (Goyal et al., 2017). A similar scaling rule was independently proposed by Jastrzebski et al. (2017) and Chaudhari &amp; Soatto (2017), although neither work identifies the existence of an optimal noise scale. A number of authors have proposed adjusting the batch size adaptively during training (Friedlander &amp; Schmidt, 2012; Byrd et al., 2012; De et al., 2017), while Balles et al. (2016) proposed linearly coupling the learning rate and batch size within this framework. In Smith et al. (2017), we show empirically that decaying the learning rate during training and increasing the batch size during training are equivalent.<br /> In figure 6a we exhibit the test set accuracy as a function of batch size, for a range of training set sizes after 10000 steps (ε = 1 everywhere). Once again, the peak shifts right as the training set size rises, although the generalization gap becomes less pronounced as the training set size increases. In figure 6b, we plot the best observed batch size as a function of training set size; observing another linear trend, Bopt ∝ N. This scaling rule could be applied to production models, progressively growing the batch size as new training data is collected. We expect production datasets to grow considerably over time, and consequently large batch training is likely to become increasingly common.<br /> B(1−m)<br /> scale of conventional SGD as m → 0. When m &gt; 0, we obtain an additional scaling rule Bopt ∝ 1/(1 − m). This scaling rule predicts that the optimal batch size will increase when the momentum coefficient is increased. In figure 7a we plot the test set performance as a function of batch size after 10000 gradient updates (ε = 1 everywhere), for a range of momentum coefficients. In figure 7b, we plot the best observed batch size as a function of the momentum coefficient, and fit our results to the scaling rule above; obtaining remarkably good agreement.<br /> <br /> ==Critiques==<br /> <br /> #Bayesian statistics is not provably, at present, a theory that can be used to explain why a learning algorithm works. The Bayesian theory is too optimistic: we introduce a prior and model and then trust both implicitly. Relative to any particular prior and model (likelihood), the Bayesian posterior is the optimal summary of the data, but if either part is misspecified, then the Bayesian posterior carries no optimality guarantee. The prior is chosen for convenience here. <br /> #No discussions with respect to the analysis of information bottleneck which also discuss the generalization ability of the model. <br /> #No discussion on real online learning with streaming data where the total number of data points are unknown?<br /> #The paper presents how mini-batch noises with SGD can improve the performance of neural networks. However, the usefulness of the approach can be described and analyzed in greater details, if the author could provide the performance for various well-known real-life datas.<br /> <br /> ==Conclusion==<br /> <br /> The paper showed that Mini-batch noise helps SGD to go away from sharp minima, and provided an evidence that there is an optimal optimum batch size for a maximum the test accuracy. Based on interpreting SGD as integrating stochastic differential equation, this batch size is proportional to the learning rate and the training set size. Moreover, the authors shown that &lt;math&gt;Bopt \propto 1/(1 − m) &lt;/math&gt;, where m is the momentum coefficient. More analysis was done on the relation between the learning rate, effective learning rate, and batch size is presented in ICLR 2018, where the authors proved by experiments that all the benefits of decaying the learning rate are achieved by increasing the batch size in addition to reducing the number of parameter updates dramatically, and also were able use literature parameters without the need of any hyper parameter tuning (Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le).<br /> <br /> ==References==<br /> <br /> #Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. arXiv preprint arXiv:1612.05086, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012. <br /> #Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference converges to limit cycles for deep networks. arXiv preprint arXiv:1710.11029, 2017.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.<br /> #Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Crispin W Gardiner. Handbook of Stochastic Methods, volume 4. Springer Berlin, 1985.<br /> #Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884– 1892, 2016.<br /> #Priya Goyal, Piotr Dolla ́r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Stephen F Gull. Bayesian inductive inference and maximum entropy. In Maximum-entropy and Bayesian methods in science and engineering, pp. 53–74. Springer, 1988.<br /> #Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM,1993.<br /> #Sepp Hochreiter and Ju ̈rgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Stanisław Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017.<br /> #Robert E Kass and Adrian E Raftery. Bayes factors. Journal of the american statistical association, 90(430):773–795, 1995.<br /> #Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don’t learn via mem- orization. ICLR Workshop, 2017.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101–2110, 2017.<br /> #David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448–472, 1992.<br /> #Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810, 2017.<br /> #Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_Bayesian_Perspective_on_Generalization_and_Stochastic_Gradient_Descent&diff=42092 A Bayesian Perspective on Generalization and Stochastic Gradient Descent 2018-11-30T17:25:06Z <p>C9sharma: /* Critiques */</p> <hr /> <div>==Introduction==<br /> This work builds on Zhang et al.(2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. The authors consider two questions: how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? <br /> The paper shows that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. They also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.<br /> <br /> The authors propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the “noise scale” &lt;math display=&quot;inline&quot;&gt; g \approx \epsilon N/B &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;ε&lt;/math&gt; is the learning rate, &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt; the training set size and &lt;math display=&quot;inline&quot;&gt;B&lt;/math&gt; the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, &lt;math display=&quot;inline&quot;&gt;B_{opt} \propto \epsilon N&lt;/math&gt;. The authors verify these predictions empirically.<br /> <br /> ==Motivation and Related Work==<br /> This paper shows Bayesian principles can explain many recent observations in the deep learning literature, while also discovering practical new insights. Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite &amp; Roy, 2017). Nonetheless, their results beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? Meanwhile, Keskar et al. (2016) observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking result shows improving our estimate of the full-batch gradient can harm performance. Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Hoffer et al. (2017) proposed a square root rule on theoretical grounds.<br /> Many authors have suggested “broad minima” whose curvature is small may generalize better than “sharp minima” whose curvature is large (Chaudhari et al., 2016; Hochreiter &amp; Schmidhuber, 1997). Indeed, Dziugaite &amp; Roy (2017) argued the results of Zhang et al. (2016) can be understood using “nonvacuous” PAC-Bayes generalization bounds which penalize sharp minima, while Keskar et al. (2016) showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However, Dinh et al. (2017) challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization.<br /> <br /> ==Contribution==<br /> <br /> The main contributions of this paper are to show that:<br /> * The results of Zhang et al. (2016) are not unique to deep learning; it is observed the same phenomenon in a small “over-parameterized” linear model. Overparameterization occurs when a model is able to effectively “remember” training data. This occurs when there are enough parameters that the system of equations ends up with an infinite number of possible solutions. One can see why this over-training would lead to poor results in test cases, as this “memorization” learns noise as opposed to the inherent structure of different classes. It is demonstrated that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.<br /> * SGD integrates a stochastic differential equation whose “noise scale” &lt;math&gt;g &amp;asymp; &amp;epsilon;N/B&lt;/math&gt;, where<br /> &amp;epsilon; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is '''proportional to the learning rate and training set size'''.<br /> <br /> Zhang et al. (2016) showed high training competency of neural networks under informative labels, but drastic overfitting on improper labels. This implies weak generalizability even when a small proportion of labels are improper. The authors show that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Bayesians tend to make distributional assumptions on gradient updates by adding isotropic Gaussian noise. This paper builds upon these Bayesian principles by driving SGD away from sharp minima, and towards broad minima (the more broad, the better generalization due to less influence from small perturbations within input). The stochastic differential equation used as a component of gradient updates effectively serves as injected noise that improves a network's generalizability.<br /> <br /> ==Main Results==<br /> <br /> The weakly regularized model memorizes random labels, however, generalizes properly on informative labels. Besides, the predictions are overconfident. The authors also showed that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is postulated that the optimum represents a tradeoff between depth and breadth in the Bayesian evidence. However it is the underlying scale of random fluctuations in the SGD dynamics which controls the tradeoff, not the batch size itself. Furthermore, this test accuracy peak shifts as the training set size rises. The authors observed that the best found batch size is proportional to the learning rate. This scaling rule allowed the authors to increase the learning rate by simultaneously increasing the batch size with no loss in test accuracy and no increase in computational cost, thus parallelism across multiple GPU's can be fully leveraged to easily decrease training time. The scaling rule could also be applied to production models by consequentially increasing the batch size as new training data is introduced.<br /> <br /> ==Bayesian Model Comparison==<br /> <br /> ===Introduction to Bayesian Statistics===<br /> Bayes' theorem is a fundamental theorem in Bayesian statistics, as it is used by Bayesian methods to update probabilities, which are degrees of belief, after obtaining new data. Given two events &lt;math&gt;A&lt;/math&gt; and &lt;math&gt;B&lt;/math&gt;, the conditional probability of &lt;math&gt;A&lt;/math&gt; given &lt;math&gt;B &lt;/math&gt; is true, Bayes theorem states that<br /> \begin{align*}\displaystyle P(A\mid B)={\frac {P(B\mid A)P(A)}{P(B)}}\end{align*}<br /> <br /> Bayesian networks are DAGs whose nodes represent variables in the Bayesian sense: they may be observable quantities, latent variables, unknown parameters or hypotheses. Edges represent conditional dependencies; nodes that are not connected (no path connects one node to another) represent variables that are conditionally independent of each other. Each node is associated with a probability function that takes, as input, a particular set of values for the node's parent variables, and gives (as output) the probability (or probability distribution, if applicable) of the variable represented by the node. For example, if &lt;math&gt;m &lt;/math&gt; parent nodes represent &lt;math&gt;m &lt;/math&gt; Boolean variables then the probability function could be represented by a table of &lt;math&gt;2^{m} &lt;/math&gt; entries, one entry for each of the &lt;math&gt;2^{m} &lt;/math&gt; possible parent combinations. <br /> <br /> ===Bayesian Model Comparison in Neural Networks===<br /> MacKay (1992) applied Bayesian model comparison to neural networks. An overview is presented below. <br /> <br /> We first consider a classification model &lt;math&gt;M &lt;/math&gt; with a single parameter &lt;math&gt;\omega &lt;/math&gt;, training inputs &lt;math&gt;x &lt;/math&gt; and training labels &lt;math&gt;y &lt;/math&gt;. We can infer a posterior probability distribution over the parameter by applying Bayes theorem :<br /> <br /> \begin{align*}P(\omega\mid y,x;M) = \frac{P(y\mid \omega,x;M)P(\omega;M) }{P(y\mid x;M)}\end{align*}<br /> <br /> The likelihood, &lt;math&gt;P(y\mid \omega,x;M) = \Pi_i P(y_i\mid \omega,x_i;M) = e^{-H(\omega;M)} &lt;/math&gt;, where &lt;math&gt;H(\omega;M) &lt;/math&gt; denotes the cross-entropy of unique categorical labels. Using a Gaussian prior, &lt;math&gt;P(\omega;M) = \sqrt{\lambda/2\pi e^{-\lambda\omega^2/2}} &lt;/math&gt;, and therefore the posterior probability density of the parameter given the training data, &lt;math&gt;P(\omega\mid y,x;M) \propto \sqrt{\lambda/2\pi e^{-C(\omega;M)}} &lt;/math&gt;, where &lt;math&gt;C(\omega;M) = H(\omega;M) + \lambda\omega^2/2 &lt;/math&gt; denotes the L2 regularized cross entropy, or “cost function”, and &lt;math&gt;\lambda &lt;/math&gt; is the regularization coefficient. <br /> <br /> The value &lt;math&gt;\omega_0 &lt;/math&gt; which minimizes the cost function lies at the maximum of this posterior. To predict an unknown label &lt;math&gt;y_t &lt;/math&gt; of a new input &lt;math&gt;x_t &lt;/math&gt;, we should compute the integral,<br /> <br /> \begin{align*} P(y_t\mid x_t,y,x;M) &amp;= \int \frac{d\omega P(y_t\mid \omega,x_t;M)}{P(\omega\mid y,x;M)}\\ &amp;= \frac{\int d \omega P(y_t \mid \omega ,x_t;M)e^{-C(\omega;M)}}{\int d \omega e^{-C(\omega;M)}} \end{align*}&lt;/math&gt;<br /> <br /> However, these integrals are dominated by the region near &lt;math&gt;\omega_0 &lt;/math&gt; . We usually approximate &lt;math&gt;P(y_t\mid x_t,x,y;M) \approx P(y_t\mid \omega_0,x_t;M) &lt;/math&gt;. Having minimized &lt;math&gt;C(\omega;M) &lt;/math&gt; to find &lt;math&gt;\omega_0 &lt;/math&gt;, we now wish to compare two different models and select the best one. We use the probability ratio<br /> <br /> \begin{align*}\frac{P(M_1\mid y,x)}{P (M_2\mid y, x)} = \frac{P(y\mid x;M_1) P(M_1)}{ P (y\mid x; M_2) P (M_2)} . \end{align*} <br /> <br /> The second factor on the right is the prior ratio, which describes which model is most plausible. To avoid unnecessary subjectivity, we usually set this to 1. Meanwhile the first factor on the right is the evidence ratio, which controls how much the training data changes our prior beliefs<br /> <br /> Germain et al. (2016) showed that maximizing the evidence (or “marginal likelihood”) minimizes a PAC-Bayes generalization bound. To compute it, we evaluate <br /> \begin{align*}P(y\mid x;M) &amp;= \int d\omega P(y\mid \omega,x;M)P(\omega;M) \\ &amp;=\sqrt{\frac{\lambda}{2\pi}}\int d \omega e^{C(\omega;M)}\end{align*}<br /> <br /> Notice that the evidence is computed by integrating out the parameters; and consequently it is invariant to the model parameterization. <br /> Since this integral is dominated by the region near the minimum &lt;math&gt;\omega_0 &lt;/math&gt;, we can estimate the evidence by Taylor expanding &lt;math&gt;C(\omega; M) \approx C(\omega_0) + C′′(\omega_0)(\omega - \omega_0)^2/2&lt;/math&gt;. This gives us<br /> <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2}\\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2}\ln(C (\omega_0)/\lambda) \big\}.\end{align*}<br /> <br /> The evidence is controlled by the value of the cost function at the minimum, and by the logarithm of the ratio of the curvature about this minimum compared to the regularization constant. In models with many parameters <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2} \\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) \big\}.\end{align*}<br /> <br /> Occam’s factor arises from the log ratio &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; The Occam factor describes the fraction of the prior parameter space consistent with the data. Occam’s factor penalizes the amount of information the model must learn about the parameters to accurately model the training data. Since the fraction is always less than one, the authors propose to approximate &lt;math&gt;P(y\mid x;M) &lt;/math&gt; away from local minima by only performing the summation over eigenvalues &lt;math&gt;\lambda_i \geq \lambda &lt;/math&gt;.<br /> <br /> The authors compare evidence against a null model which assumes the labels are entirely random. This model has no parameters, and so the evidence is controlled by the likelihood alone. &lt;math&gt;P(y\mid x;NULL) = (1/n)^N = e^{-N \ln(n)} &lt;/math&gt;, where &lt;math&gt;n &lt;/math&gt; denotes the number of model classes and &lt;math&gt;N&lt;/math&gt; the number of training labels. The evidence ratio :<br /> \begin{equation*}\frac{P(y\mid x;M) }{P(y\mid x;NULL) } = e ^{-E(\omega_0)} \end{equation*}<br /> &lt;math&gt;E(\omega_0) = C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) - N\ln (n) &lt;/math&gt; is the log evidence ratio in favor of the null model.<br /> The authors assign confidence to the predictions of a model iff &lt;math&gt;E(\omega_0 &lt; 0 &lt;/math&gt;.<br /> <br /> The evidence supports the intuition that broad minima generalize better than sharp minima, but unlike the curvature it does not depend on the model parameterization. Dinh et al. (2017) showed one can increase the Hessian eigenvalues by rescaling the parameters, but they must simultaneously rescale the regularization coefficients, otherwise the model changes. Since Occam’s factor arises from the log ratio, &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; , these two effects cancel out. Note however that while the evidence itself is invariant to model parameterization, one can find reparameterizations which change the approximate evidence after the Laplace approximation. . It is difficult to evaluate the evidence for deep networks, as we cannot compute the Hessian of millions of parameters. Additionally, neural networks exhibit many equivalent minima, since we can permute the hidden units without changing the model. To compute the evidence we must carefully account for this “degeneracy”. The authors argue these issues are not a major limitation, since the intuition they build studying the evidence in simple cases will be sufficient to explain the results of both Zhang et al. (2016) and Keskar et al. (2016).<br /> <br /> ==Bayes Theorem and Generalization==<br /> Zhang et al. (2016) showed that deep neural networks generalize well on training inputs with informative labels, but the same model can overfit on the same input images when the labels are randomized; perfectly memorizing the training set. To demonstrate that these observations are not unique to deep network, the authors use logistic regression. They form a small balanced training set comprising 800 images from MNIST, of which half have true label “0” and half true label “1”. The test set is balanced, comprising 5000 MNIST images of zeros and 5000 MNIST images of ones. There are two tasks. In the first task, the labels of both the training and test sets are randomized. In the second task, the labels are informative, matching the true MNIST labels. The model has 784 weights and 1 bias.<br /> <br /> The accuracy of the model predictions on both the training and test sets is shown in figure 1. When trained on the informative labels, the model generalizes well to the test set, so long as it is weakly regularized. However the model also perfectly memorizes the random labels, replicating the obser- vations of Zhang et al. (2016) in deep networks. No significant improvement in model performance is observed as the regularization coefficient increases. For completeness, we also evaluate the mean margin between training examples and the decision boundary. For both random and informative labels, the margin drops significantly as we reduce the regularization coefficient. When weakly regularized, the mean margin is roughly 50% larger for informative labels than for random labels.<br /> <br /> [[File:bg1.png|800px|thumb|center|]]<br /> <br /> Now consider figure 2, where we plot the mean cross-entropy of the model predictions, evaluated on both training and test sets, as well as the Bayesian log evidence ratio defined in the previous section. Looking first at the random label experiment in figure 2a, while the cross-entropy on the training set vanishes when the model is weakly regularized, the cross-entropy on the test set explodes. Not only does the model make random predictions, but it is extremely confident in those predictions. As the regularization coefficient is increased the test set cross-entropy falls, settling at ln 2, the cross- entropy of assigning equal probability to both classes. Now consider the Bayesian evidence, which we evaluate on the training set. The log evidence ratio is large and positive when the model is weakly regularized, indicating that the model is exponentially less plausible than assigning equal probabilities to each class. As the regularization parameter is increased, the log evidence ratio falls, but it is always positive, indicating that the model can never be expected to generalize well.<br /> Now consider figure 2b (informative labels). Once again, the training cross-entropy falls to zero when the model is weakly regularized, while the test cross-entropy is high. Even though the model makes accurate predictions, those predictions are overconfident. As the regularization coefficient increases, the test cross-entropy falls below ln 2, indicating that the model is successfully gener- alizing to the test set. Now consider the Bayesian evidence. The log evidence ratio is large and positive when the model is weakly regularized, but as the regularization coefficient increases, the log evidence ratio drops below zero, indicating that the model is exponentially more plausible than assigning equal probabilities to each class. As we further increase the regularization, the log evi- dence ratio rises to zero while the test cross-entropy rises to ln 2. Test cross-entropy and Bayesian evidence are strongly correlated, with minima at the same regularization strength.<br /> Bayesian model comparison has explained our results in a logistic regression. Meanwhile, Krueger et al. (2017) showed the largest Hessian eigenvalue also increased when training on random labels in deep networks, implying the evidence is falling. We conclude that Bayesian model comparison is quantitatively consistent with the results of Zhang et al. (2016) in linear models where we can compute the evidence, and qualitatively consistent with their results in deep networks where we cannot. Dziugaite &amp; Roy (2017) recently demonstrated the results of Zhang et al. (2016) can also be understood by minimising a PAC-Bayes generalization bound which penalizes sharp minima.<br /> [[File:bg2.png|800px|thumb|center|]]<br /> ==Bayes Theorem and Stochastic Gradient Descent ==<br /> <br /> We showed above that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Consequently Bayesians often add isotropic Gaussian noise to the gradient (Welling &amp; Teh, 2011). In appendix A, we show this drives the parameters towards broad minima whose evidence is large. The noise introduced by small batch training is not isotropic, and its covariance matrix is a function of the parameter values, but empirically Keskar et al. (2016) found it has similar effects, driving the SGD away from sharp minima. This paper therefore proposes Bayesian principles also account for the “generalization gap”, whereby the test set accuracy often falls as the SGD batch size is increased (holding all other hyper-parameters constant). Since the gradient drives the SGD towards deep minima, while noise drives the SGD towards broad minima, we expect the test set performance to show a peak at an optimal batch size, which balances these competing contributions to the evidence.<br /> We were unable to observe a generalization gap in linear models (since linear models are convex there are no sharp minima to avoid). Instead we consider a shallow neural network with 800 hidden units and RELU hidden activations, trained on MNIST without regularization. We use SGD with a momentum parameter of 0.9. Unless otherwise stated, we use a constant learning rate of 1.0 which does not depend on the batch size or decay during training. Furthermore, we train on just 1000 images, selected at random from the MNIST training set. This enables us to compare small batch to full batch training. We emphasize that we are not trying to achieve optimal performance, but to study a simple model which shows a generalization gap between small and large batch training.<br /> In figure 3, we exhibit the evolution of the test accuracy and test cross-entropy during training. Our small batches are composed of 30 images, randomly sampled from the training set. Looking first at figure 3a, small batch training takes longer to converge, but after a thousand gradient updates a clear generalization gap in model accuracy emerges between small and large training batches. Now consider figure 3b. While the test cross-entropy for small batch training is lower at the end of training; the cross-entropy of both small and large training batches is increasing, indicative of over-fitting. Both models exhibit a minimum test cross-entropy, although after different numbers of gradient updates. Intriguingly, we show in appendix B that the generalization gap between small and large batch training shrinks significantly when we introduce L2 regularization.<br /> <br /> [[File:bg3.png|800px|thumb|center|]]<br /> <br /> From now on we focus on the test set accuracy (since this converges as the number of gradient updates increases). In figure 4a, we exhibit training curves for a range of batch sizes between 1 and 1000. We find that the model cannot train when the batch size B 􏰘 10. In figure 4b we plot the mean test set accuracy after 10000 training steps. A clear peak emerges, indicating that there is indeed an optimum batch size which maximizes the test accuracy, consistent with Bayesian intuition. The results of Keskar et al. (2016) focused on the decay in test accuracy above this optimum batch size.<br /> [[File:bg4.png|800px|thumb|center|]]<br /> <br /> ==Stochastic Differential Equations and Scaling Rules==<br /> The results showed above indicate that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is argued that this peak arises from the tradeoff between depth and breadth in the Bayesian evidence. However it is not the batch size itself which controls this tradeoff, but the underlying scale of random fluctuations in the SGD dynamics. The following content identifies this SGD “noise scale”, and uses it to derive three scaling rules which predict how the optimal batch size depends on the learning rate, training set size and momentum coefficient. <br /> First, interpret gradient update, as the discrete update of a stochastic differential equation <br /> \begin{equation*}\frac{d\omega}{dt} = \frac{dC}{d\omega} + \eta(t)\end{equation*}<br /> &lt;math&gt;\eta&lt;/math&gt; represents noise &lt;math&gt;\langle \eta(t) \rangle = 0&lt;/math&gt; and &lt;math&gt; \langle \eta (t)\eta (t')\rangle = gF (\omega)\delta (t-t')&lt;/math&gt;.<br /> &lt;math&gt;t&lt;/math&gt; is a continous variable, and &lt;math&gt;F(\omega)&lt;/math&gt; matrix describing the gradient covariances.<br /> The SGD noise scale is taken to be &lt;math&gt;g \approx \epsilon N/B&lt;/math&gt; where &lt;math&gt;\epsilon&lt;/math&gt; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; the batch size.<br /> [[File:bg5.png|800px|thumb|center|]]<br /> [[File:bg6.png|800px|thumb|center|]]<br /> [[File:bg7.png|800px|thumb|center|]]<br /> The noise scale falls when the batch B<br /> size increases, consistent with our earlier observation of an optimal batch size Bopt while holding the other hyper-parameters fixed. Notice that one would equivalently observe an optimal learning rate if one held the batch size constant. A similar analysis of the SGD was recently performed by Mandt et al. (2017), although their treatment only holds near local minima where the covariances F (ω) are stationary. Our analysis holds throughout training, which is necessary since Keskar et al. (2016) found that the beneficial influence of noise was most pronounced at the start of training.<br /> When we vary the learning rate or the training set size, we should keep the noise scale fixed, which implies that Bopt ∝ εN. In figure 5a, we plot the test accuracy as a function of batch size after (10000/ε) training steps, for a range of learning rates. Exactly as predicted, the peak moves to the right as ε increases. Additionally, the peak test accuracy achieved at a given learning rate does not begin to fall until ε ∼ 3, indicating that there is no significant discretization error in integrating the stochastic differential equation below this point. Above this point, the discretization error begins to dominate and the peak test accuracy falls rapidly. In figure 5b, we plot the best observed batch size as a function of learning rate, observing a clear linear trend, Bopt ∝ ε. The error bars indicate the distance from the best observed batch size to the next batch size sampled in our experiments.<br /> <br /> This scaling rule allows us to increase the learning rate with no loss in test accuracy and no increase in computational cost, simply by simultaneously increasing the batch size. We can then exploit increased parallelism across multiple GPUs, reducing model training times (Goyal et al., 2017). A similar scaling rule was independently proposed by Jastrzebski et al. (2017) and Chaudhari &amp; Soatto (2017), although neither work identifies the existence of an optimal noise scale. A number of authors have proposed adjusting the batch size adaptively during training (Friedlander &amp; Schmidt, 2012; Byrd et al., 2012; De et al., 2017), while Balles et al. (2016) proposed linearly coupling the learning rate and batch size within this framework. In Smith et al. (2017), we show empirically that decaying the learning rate during training and increasing the batch size during training are equivalent.<br /> In figure 6a we exhibit the test set accuracy as a function of batch size, for a range of training set sizes after 10000 steps (ε = 1 everywhere). Once again, the peak shifts right as the training set size rises, although the generalization gap becomes less pronounced as the training set size increases. In figure 6b, we plot the best observed batch size as a function of training set size; observing another linear trend, Bopt ∝ N. This scaling rule could be applied to production models, progressively growing the batch size as new training data is collected. We expect production datasets to grow considerably over time, and consequently large batch training is likely to become increasingly common.<br /> B(1−m)<br /> scale of conventional SGD as m → 0. When m &gt; 0, we obtain an additional scaling rule Bopt ∝ 1/(1 − m). This scaling rule predicts that the optimal batch size will increase when the momentum coefficient is increased. In figure 7a we plot the test set performance as a function of batch size after 10000 gradient updates (ε = 1 everywhere), for a range of momentum coefficients. In figure 7b, we plot the best observed batch size as a function of the momentum coefficient, and fit our results to the scaling rule above; obtaining remarkably good agreement.<br /> <br /> ==Critiques==<br /> <br /> #Bayesian statistics is not provably, at present, a theory that can be used to explain why a learning algorithm works. The Bayesian theory is too optimistic: we introduce a prior and model and then trust both implicitly. Relative to any particular prior and model (likelihood), the Bayesian posterior is the optimal summary of the data, but if either part is misspecified, then the Bayesian posterior carries no optimality guarantee. The prior is chosen for convenience here. <br /> #No discussions with respect to the analysis of information bottleneck which also discuss the generalization ability of the model. <br /> #No discussion on real online learning with streaming data where the total number of data points are unknown?<br /> #The paper presents how mini-batch noises with SGD can improve the performance of neural networks. However, the usefulness of the approach can be described and analyzed in greater details, if the author could provide the performance for various well-known real-life datas.<br /> <br /> ==Conclusion==<br /> <br /> The paper showed that Mini-batch noise helps SGD to go away from sharp minima, and provided an evidence that there is an optimal optimum batch size for a maximum the test accuracy. Based on interpreting SGD as integrating stochastic differential equation, this batch size is proportional to the learning rate and the training set size. Moreover, the authors shown that Bopt ∝ 1/(1 − m), where m is the momentum coefficient. More analysis was done on the relation between the learning rate, effective learning rate, and batch size is presented in ICLR 2018, where the authors proved by experiments that all the benefits of decaying the learning rate are achieved by increasing the batch size in addition to reducing the number of parameter updates dramatically, and also were able use literature parameters without the need of any hyper parameter tuning (Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le).<br /> <br /> ==References==<br /> <br /> #Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. arXiv preprint arXiv:1612.05086, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012. <br /> #Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference converges to limit cycles for deep networks. arXiv preprint arXiv:1710.11029, 2017.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.<br /> #Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Crispin W Gardiner. Handbook of Stochastic Methods, volume 4. Springer Berlin, 1985.<br /> #Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884– 1892, 2016.<br /> #Priya Goyal, Piotr Dolla ́r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Stephen F Gull. Bayesian inductive inference and maximum entropy. In Maximum-entropy and Bayesian methods in science and engineering, pp. 53–74. Springer, 1988.<br /> #Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM,1993.<br /> #Sepp Hochreiter and Ju ̈rgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Stanisław Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017.<br /> #Robert E Kass and Adrian E Raftery. Bayes factors. Journal of the american statistical association, 90(430):773–795, 1995.<br /> #Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don’t learn via mem- orization. ICLR Workshop, 2017.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101–2110, 2017.<br /> #David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448–472, 1992.<br /> #Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810, 2017.<br /> #Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_Bayesian_Perspective_on_Generalization_and_Stochastic_Gradient_Descent&diff=42089 A Bayesian Perspective on Generalization and Stochastic Gradient Descent 2018-11-30T17:16:19Z <p>C9sharma: </p> <hr /> <div>==Introduction==<br /> This work builds on Zhang et al.(2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. The authors consider two questions: how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? <br /> The paper shows that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. They also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.<br /> <br /> The authors propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the “noise scale” &lt;math display=&quot;inline&quot;&gt; g \approx \epsilon N/B &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;ε&lt;/math&gt; is the learning rate, &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt; the training set size and &lt;math display=&quot;inline&quot;&gt;B&lt;/math&gt; the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, &lt;math display=&quot;inline&quot;&gt;B_{opt} \propto \epsilon N&lt;/math&gt;. The authors verify these predictions empirically.<br /> <br /> ==Motivation and Related Work==<br /> This paper shows Bayesian principles can explain many recent observations in the deep learning literature, while also discovering practical new insights. Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite &amp; Roy, 2017). Nonetheless, their results beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? Meanwhile, Keskar et al. (2016) observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking result shows improving our estimate of the full-batch gradient can harm performance. Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Hoffer et al. (2017) proposed a square root rule on theoretical grounds.<br /> Many authors have suggested “broad minima” whose curvature is small may generalize better than “sharp minima” whose curvature is large (Chaudhari et al., 2016; Hochreiter &amp; Schmidhuber, 1997). Indeed, Dziugaite &amp; Roy (2017) argued the results of Zhang et al. (2016) can be understood using “nonvacuous” PAC-Bayes generalization bounds which penalize sharp minima, while Keskar et al. (2016) showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However, Dinh et al. (2017) challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization.<br /> <br /> ==Contribution==<br /> <br /> The main contributions of this paper are to show that:<br /> * The results of Zhang et al. (2016) are not unique to deep learning; it is observed the same phenomenon in a small “over-parameterized” linear model. Overparameterization occurs when a model is able to effectively “remember” training data. This occurs when there are enough parameters that the system of equations ends up with an infinite number of possible solutions. One can see why this over-training would lead to poor results in test cases, as this “memorization” learns noise as opposed to the inherent structure of different classes. It is demonstrated that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.<br /> * SGD integrates a stochastic differential equation whose “noise scale” &lt;math&gt;g &amp;asymp; &amp;epsilon;N/B&lt;/math&gt;, where<br /> &amp;epsilon; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is '''proportional to the learning rate and training set size'''.<br /> <br /> Zhang et al. (2016) showed high training competency of neural networks under informative labels, but drastic overfitting on improper labels. This implies weak generalizability even when a small proportion of labels are improper. The authors show that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Bayesians tend to make distributional assumptions on gradient updates by adding isotropic Gaussian noise. This paper builds upon these Bayesian principles by driving SGD away from sharp minima, and towards broad minima (the more broad, the better generalization due to less influence from small perturbations within input). The stochastic differential equation used as a component of gradient updates effectively serves as injected noise that improves a network's generalizability.<br /> <br /> ==Main Results==<br /> <br /> The weakly regularized model memorizes random labels, however, generalizes properly on informative labels. Besides, the predictions are overconfident. The authors also showed that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is postulated that the optimum represents a tradeoff between depth and breadth in the Bayesian evidence. However it is the underlying scale of random fluctuations in the SGD dynamics which controls the tradeoff, not the batch size itself. Furthermore, this test accuracy peak shifts as the training set size rises. The authors observed that the best found batch size is proportional to the learning rate. This scaling rule allowed the authors to increase the learning rate by simultaneously increasing the batch size with no loss in test accuracy and no increase in computational cost, thus parallelism across multiple GPU's can be fully leveraged to easily decrease training time. The scaling rule could also be applied to production models by consequentially increasing the batch size as new training data is introduced.<br /> <br /> ==Bayesian Model Comparison==<br /> <br /> ===Introduction to Bayesian Statistics===<br /> Bayes' theorem is a fundamental theorem in Bayesian statistics, as it is used by Bayesian methods to update probabilities, which are degrees of belief, after obtaining new data. Given two events &lt;math&gt;A&lt;/math&gt; and &lt;math&gt;B&lt;/math&gt;, the conditional probability of &lt;math&gt;A&lt;/math&gt; given &lt;math&gt;B &lt;/math&gt; is true, Bayes theorem states that<br /> \begin{align*}\displaystyle P(A\mid B)={\frac {P(B\mid A)P(A)}{P(B)}}\end{align*}<br /> <br /> Bayesian networks are DAGs whose nodes represent variables in the Bayesian sense: they may be observable quantities, latent variables, unknown parameters or hypotheses. Edges represent conditional dependencies; nodes that are not connected (no path connects one node to another) represent variables that are conditionally independent of each other. Each node is associated with a probability function that takes, as input, a particular set of values for the node's parent variables, and gives (as output) the probability (or probability distribution, if applicable) of the variable represented by the node. For example, if &lt;math&gt;m &lt;/math&gt; parent nodes represent &lt;math&gt;m &lt;/math&gt; Boolean variables then the probability function could be represented by a table of &lt;math&gt;2^{m} &lt;/math&gt; entries, one entry for each of the &lt;math&gt;2^{m} &lt;/math&gt; possible parent combinations. <br /> <br /> ===Bayesian Model Comparison in Neural Networks===<br /> MacKay (1992) applied Bayesian model comparison to neural networks. An overview is presented below. <br /> <br /> We first consider a classification model &lt;math&gt;M &lt;/math&gt; with a single parameter &lt;math&gt;\omega &lt;/math&gt;, training inputs &lt;math&gt;x &lt;/math&gt; and training labels &lt;math&gt;y &lt;/math&gt;. We can infer a posterior probability distribution over the parameter by applying Bayes theorem :<br /> <br /> \begin{align*}P(\omega\mid y,x;M) = \frac{P(y\mid \omega,x;M)P(\omega;M) }{P(y\mid x;M)}\end{align*}<br /> <br /> The likelihood, &lt;math&gt;P(y\mid \omega,x;M) = \Pi_i P(y_i\mid \omega,x_i;M) = e^{-H(\omega;M)} &lt;/math&gt;, where &lt;math&gt;H(\omega;M) &lt;/math&gt; denotes the cross-entropy of unique categorical labels. Using a Gaussian prior, &lt;math&gt;P(\omega;M) = \sqrt{\lambda/2\pi e^{-\lambda\omega^2/2}} &lt;/math&gt;, and therefore the posterior probability density of the parameter given the training data, &lt;math&gt;P(\omega\mid y,x;M) \propto \sqrt{\lambda/2\pi e^{-C(\omega;M)}} &lt;/math&gt;, where &lt;math&gt;C(\omega;M) = H(\omega;M) + \lambda\omega^2/2 &lt;/math&gt; denotes the L2 regularized cross entropy, or “cost function”, and &lt;math&gt;\lambda &lt;/math&gt; is the regularization coefficient. <br /> <br /> The value &lt;math&gt;\omega_0 &lt;/math&gt; which minimizes the cost function lies at the maximum of this posterior. To predict an unknown label &lt;math&gt;y_t &lt;/math&gt; of a new input &lt;math&gt;x_t &lt;/math&gt;, we should compute the integral,<br /> <br /> \begin{align*} P(y_t\mid x_t,y,x;M) &amp;= \int \frac{d\omega P(y_t\mid \omega,x_t;M)}{P(\omega\mid y,x;M)}\\ &amp;= \frac{\int d \omega P(y_t \mid \omega ,x_t;M)e^{-C(\omega;M)}}{\int d \omega e^{-C(\omega;M)}} \end{align*}&lt;/math&gt;<br /> <br /> However, these integrals are dominated by the region near &lt;math&gt;\omega_0 &lt;/math&gt; . We usually approximate &lt;math&gt;P(y_t\mid x_t,x,y;M) \approx P(y_t\mid \omega_0,x_t;M) &lt;/math&gt;. Having minimized &lt;math&gt;C(\omega;M) &lt;/math&gt; to find &lt;math&gt;\omega_0 &lt;/math&gt;, we now wish to compare two different models and select the best one. We use the probability ratio<br /> <br /> \begin{align*}\frac{P(M_1\mid y,x)}{P (M_2\mid y, x)} = \frac{P(y\mid x;M_1) P(M_1)}{ P (y\mid x; M_2) P (M_2)} . \end{align*} <br /> <br /> The second factor on the right is the prior ratio, which describes which model is most plausible. To avoid unnecessary subjectivity, we usually set this to 1. Meanwhile the first factor on the right is the evidence ratio, which controls how much the training data changes our prior beliefs<br /> <br /> Germain et al. (2016) showed that maximizing the evidence (or “marginal likelihood”) minimizes a PAC-Bayes generalization bound. To compute it, we evaluate <br /> \begin{align*}P(y\mid x;M) &amp;= \int d\omega P(y\mid \omega,x;M)P(\omega;M) \\ &amp;=\sqrt{\frac{\lambda}{2\pi}}\int d \omega e^{C(\omega;M)}\end{align*}<br /> <br /> Notice that the evidence is computed by integrating out the parameters; and consequently it is invariant to the model parameterization. <br /> Since this integral is dominated by the region near the minimum &lt;math&gt;\omega_0 &lt;/math&gt;, we can estimate the evidence by Taylor expanding &lt;math&gt;C(\omega; M) \approx C(\omega_0) + C′′(\omega_0)(\omega - \omega_0)^2/2&lt;/math&gt;. This gives us<br /> <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2}\\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2}\ln(C (\omega_0)/\lambda) \big\}.\end{align*}<br /> <br /> The evidence is controlled by the value of the cost function at the minimum, and by the logarithm of the ratio of the curvature about this minimum compared to the regularization constant. In models with many parameters <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2} \\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) \big\}.\end{align*}<br /> <br /> Occam’s factor arises from the log ratio &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; The Occam factor describes the fraction of the prior parameter space consistent with the data. Occam’s factor penalizes the amount of information the model must learn about the parameters to accurately model the training data. Since the fraction is always less than one, the authors propose to approximate &lt;math&gt;P(y\mid x;M) &lt;/math&gt; away from local minima by only performing the summation over eigenvalues &lt;math&gt;\lambda_i \geq \lambda &lt;/math&gt;.<br /> <br /> The authors compare evidence against a null model which assumes the labels are entirely random. This model has no parameters, and so the evidence is controlled by the likelihood alone. &lt;math&gt;P(y\mid x;NULL) = (1/n)^N = e^{-N \ln(n)} &lt;/math&gt;, where &lt;math&gt;n &lt;/math&gt; denotes the number of model classes and &lt;math&gt;N&lt;/math&gt; the number of training labels. The evidence ratio :<br /> \begin{equation*}\frac{P(y\mid x;M) }{P(y\mid x;NULL) } = e ^{-E(\omega_0)} \end{equation*}<br /> &lt;math&gt;E(\omega_0) = C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) - N\ln (n) &lt;/math&gt; is the log evidence ratio in favor of the null model.<br /> The authors assign confidence to the predictions of a model iff &lt;math&gt;E(\omega_0 &lt; 0 &lt;/math&gt;.<br /> <br /> The evidence supports the intuition that broad minima generalize better than sharp minima, but unlike the curvature it does not depend on the model parameterization. Dinh et al. (2017) showed one can increase the Hessian eigenvalues by rescaling the parameters, but they must simultaneously rescale the regularization coefficients, otherwise the model changes. Since Occam’s factor arises from the log ratio, &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; , these two effects cancel out. Note however that while the evidence itself is invariant to model parameterization, one can find reparameterizations which change the approximate evidence after the Laplace approximation. . It is difficult to evaluate the evidence for deep networks, as we cannot compute the Hessian of millions of parameters. Additionally, neural networks exhibit many equivalent minima, since we can permute the hidden units without changing the model. To compute the evidence we must carefully account for this “degeneracy”. The authors argue these issues are not a major limitation, since the intuition they build studying the evidence in simple cases will be sufficient to explain the results of both Zhang et al. (2016) and Keskar et al. (2016).<br /> <br /> ==Bayes Theorem and Generalization==<br /> Zhang et al. (2016) showed that deep neural networks generalize well on training inputs with informative labels, but the same model can overfit on the same input images when the labels are randomized; perfectly memorizing the training set. To demonstrate that these observations are not unique to deep network, the authors use logistic regression. They form a small balanced training set comprising 800 images from MNIST, of which half have true label “0” and half true label “1”. The test set is balanced, comprising 5000 MNIST images of zeros and 5000 MNIST images of ones. There are two tasks. In the first task, the labels of both the training and test sets are randomized. In the second task, the labels are informative, matching the true MNIST labels. The model has 784 weights and 1 bias.<br /> <br /> The accuracy of the model predictions on both the training and test sets is shown in figure 1. When trained on the informative labels, the model generalizes well to the test set, so long as it is weakly regularized. However the model also perfectly memorizes the random labels, replicating the obser- vations of Zhang et al. (2016) in deep networks. No significant improvement in model performance is observed as the regularization coefficient increases. For completeness, we also evaluate the mean margin between training examples and the decision boundary. For both random and informative labels, the margin drops significantly as we reduce the regularization coefficient. When weakly regularized, the mean margin is roughly 50% larger for informative labels than for random labels.<br /> <br /> [[File:bg1.png|800px|thumb|center|]]<br /> <br /> Now consider figure 2, where we plot the mean cross-entropy of the model predictions, evaluated on both training and test sets, as well as the Bayesian log evidence ratio defined in the previous section. Looking first at the random label experiment in figure 2a, while the cross-entropy on the training set vanishes when the model is weakly regularized, the cross-entropy on the test set explodes. Not only does the model make random predictions, but it is extremely confident in those predictions. As the regularization coefficient is increased the test set cross-entropy falls, settling at ln 2, the cross- entropy of assigning equal probability to both classes. Now consider the Bayesian evidence, which we evaluate on the training set. The log evidence ratio is large and positive when the model is weakly regularized, indicating that the model is exponentially less plausible than assigning equal probabilities to each class. As the regularization parameter is increased, the log evidence ratio falls, but it is always positive, indicating that the model can never be expected to generalize well.<br /> Now consider figure 2b (informative labels). Once again, the training cross-entropy falls to zero when the model is weakly regularized, while the test cross-entropy is high. Even though the model makes accurate predictions, those predictions are overconfident. As the regularization coefficient increases, the test cross-entropy falls below ln 2, indicating that the model is successfully gener- alizing to the test set. Now consider the Bayesian evidence. The log evidence ratio is large and positive when the model is weakly regularized, but as the regularization coefficient increases, the log evidence ratio drops below zero, indicating that the model is exponentially more plausible than assigning equal probabilities to each class. As we further increase the regularization, the log evi- dence ratio rises to zero while the test cross-entropy rises to ln 2. Test cross-entropy and Bayesian evidence are strongly correlated, with minima at the same regularization strength.<br /> Bayesian model comparison has explained our results in a logistic regression. Meanwhile, Krueger et al. (2017) showed the largest Hessian eigenvalue also increased when training on random labels in deep networks, implying the evidence is falling. We conclude that Bayesian model comparison is quantitatively consistent with the results of Zhang et al. (2016) in linear models where we can compute the evidence, and qualitatively consistent with their results in deep networks where we cannot. Dziugaite &amp; Roy (2017) recently demonstrated the results of Zhang et al. (2016) can also be understood by minimising a PAC-Bayes generalization bound which penalizes sharp minima.<br /> [[File:bg2.png|800px|thumb|center|]]<br /> ==Bayes Theorem and Stochastic Gradient Descent ==<br /> <br /> We showed above that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Consequently Bayesians often add isotropic Gaussian noise to the gradient (Welling &amp; Teh, 2011). In appendix A, we show this drives the parameters towards broad minima whose evidence is large. The noise introduced by small batch training is not isotropic, and its covariance matrix is a function of the parameter values, but empirically Keskar et al. (2016) found it has similar effects, driving the SGD away from sharp minima. This paper therefore proposes Bayesian principles also account for the “generalization gap”, whereby the test set accuracy often falls as the SGD batch size is increased (holding all other hyper-parameters constant). Since the gradient drives the SGD towards deep minima, while noise drives the SGD towards broad minima, we expect the test set performance to show a peak at an optimal batch size, which balances these competing contributions to the evidence.<br /> We were unable to observe a generalization gap in linear models (since linear models are convex there are no sharp minima to avoid). Instead we consider a shallow neural network with 800 hidden units and RELU hidden activations, trained on MNIST without regularization. We use SGD with a momentum parameter of 0.9. Unless otherwise stated, we use a constant learning rate of 1.0 which does not depend on the batch size or decay during training. Furthermore, we train on just 1000 images, selected at random from the MNIST training set. This enables us to compare small batch to full batch training. We emphasize that we are not trying to achieve optimal performance, but to study a simple model which shows a generalization gap between small and large batch training.<br /> In figure 3, we exhibit the evolution of the test accuracy and test cross-entropy during training. Our small batches are composed of 30 images, randomly sampled from the training set. Looking first at figure 3a, small batch training takes longer to converge, but after a thousand gradient updates a clear generalization gap in model accuracy emerges between small and large training batches. Now consider figure 3b. While the test cross-entropy for small batch training is lower at the end of training; the cross-entropy of both small and large training batches is increasing, indicative of over-fitting. Both models exhibit a minimum test cross-entropy, although after different numbers of gradient updates. Intriguingly, we show in appendix B that the generalization gap between small and large batch training shrinks significantly when we introduce L2 regularization.<br /> <br /> [[File:bg3.png|800px|thumb|center|]]<br /> <br /> From now on we focus on the test set accuracy (since this converges as the number of gradient updates increases). In figure 4a, we exhibit training curves for a range of batch sizes between 1 and 1000. We find that the model cannot train when the batch size B 􏰘 10. In figure 4b we plot the mean test set accuracy after 10000 training steps. A clear peak emerges, indicating that there is indeed an optimum batch size which maximizes the test accuracy, consistent with Bayesian intuition. The results of Keskar et al. (2016) focused on the decay in test accuracy above this optimum batch size.<br /> [[File:bg4.png|800px|thumb|center|]]<br /> <br /> ==Stochastic Differential Equations and Scaling Rules==<br /> The results showed above indicate that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is argued that this peak arises from the tradeoff between depth and breadth in the Bayesian evidence. However it is not the batch size itself which controls this tradeoff, but the underlying scale of random fluctuations in the SGD dynamics. The following content identifies this SGD “noise scale”, and uses it to derive three scaling rules which predict how the optimal batch size depends on the learning rate, training set size and momentum coefficient. <br /> First, interpret gradient update, as the discrete update of a stochastic differential equation <br /> \begin{equation*}\frac{d\omega}{dt} = \frac{dC}{d\omega} + \eta(t)\end{equation*}<br /> &lt;math&gt;\eta&lt;/math&gt; represents noise &lt;math&gt;\langle \eta(t) \rangle = 0&lt;/math&gt; and &lt;math&gt; \langle \eta (t)\eta (t')\rangle = gF (\omega)\delta (t-t')&lt;/math&gt;.<br /> &lt;math&gt;t&lt;/math&gt; is a continous variable, and &lt;math&gt;F(\omega)&lt;/math&gt; matrix describing the gradient covariances.<br /> The SGD noise scale is taken to be &lt;math&gt;g \approx \epsilon N/B&lt;/math&gt; where &lt;math&gt;\epsilon&lt;/math&gt; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; the batch size.<br /> [[File:bg5.png|800px|thumb|center|]]<br /> [[File:bg6.png|800px|thumb|center|]]<br /> [[File:bg7.png|800px|thumb|center|]]<br /> The noise scale falls when the batch B<br /> size increases, consistent with our earlier observation of an optimal batch size Bopt while holding the other hyper-parameters fixed. Notice that one would equivalently observe an optimal learning rate if one held the batch size constant. A similar analysis of the SGD was recently performed by Mandt et al. (2017), although their treatment only holds near local minima where the covariances F (ω) are stationary. Our analysis holds throughout training, which is necessary since Keskar et al. (2016) found that the beneficial influence of noise was most pronounced at the start of training.<br /> When we vary the learning rate or the training set size, we should keep the noise scale fixed, which implies that Bopt ∝ εN. In figure 5a, we plot the test accuracy as a function of batch size after (10000/ε) training steps, for a range of learning rates. Exactly as predicted, the peak moves to the right as ε increases. Additionally, the peak test accuracy achieved at a given learning rate does not begin to fall until ε ∼ 3, indicating that there is no significant discretization error in integrating the stochastic differential equation below this point. Above this point, the discretization error begins to dominate and the peak test accuracy falls rapidly. In figure 5b, we plot the best observed batch size as a function of learning rate, observing a clear linear trend, Bopt ∝ ε. The error bars indicate the distance from the best observed batch size to the next batch size sampled in our experiments.<br /> <br /> This scaling rule allows us to increase the learning rate with no loss in test accuracy and no increase in computational cost, simply by simultaneously increasing the batch size. We can then exploit increased parallelism across multiple GPUs, reducing model training times (Goyal et al., 2017). A similar scaling rule was independently proposed by Jastrzebski et al. (2017) and Chaudhari &amp; Soatto (2017), although neither work identifies the existence of an optimal noise scale. A number of authors have proposed adjusting the batch size adaptively during training (Friedlander &amp; Schmidt, 2012; Byrd et al., 2012; De et al., 2017), while Balles et al. (2016) proposed linearly coupling the learning rate and batch size within this framework. In Smith et al. (2017), we show empirically that decaying the learning rate during training and increasing the batch size during training are equivalent.<br /> In figure 6a we exhibit the test set accuracy as a function of batch size, for a range of training set sizes after 10000 steps (ε = 1 everywhere). Once again, the peak shifts right as the training set size rises, although the generalization gap becomes less pronounced as the training set size increases. In figure 6b, we plot the best observed batch size as a function of training set size; observing another linear trend, Bopt ∝ N. This scaling rule could be applied to production models, progressively growing the batch size as new training data is collected. We expect production datasets to grow considerably over time, and consequently large batch training is likely to become increasingly common.<br /> B(1−m)<br /> scale of conventional SGD as m → 0. When m &gt; 0, we obtain an additional scaling rule Bopt ∝ 1/(1 − m). This scaling rule predicts that the optimal batch size will increase when the momentum coefficient is increased. In figure 7a we plot the test set performance as a function of batch size after 10000 gradient updates (ε = 1 everywhere), for a range of momentum coefficients. In figure 7b, we plot the best observed batch size as a function of the momentum coefficient, and fit our results to the scaling rule above; obtaining remarkably good agreement.<br /> <br /> ==Critiques==<br /> The paper presents how mini-batch noises with SGD can improve. However, the usefulness of the approach can be described and analyzed in greater details, if the author coudl provide the performance for various well-known real-life datas. Note that even without those evidences, the paper's approach and methods are still very interesting. <br /> <br /> ==Conclusion==<br /> <br /> The paper showed that Mini-batch noise helps SGD to go away from sharp minima, and provided an evidence that there is an optimal optimum batch size for a maximum the test accuracy. Based on interpreting SGD as integrating stochastic differential equation, this batch size is proportional to the learning rate and the training set size. Moreover, the authors shown that Bopt ∝ 1/(1 − m), where m is the momentum coefficient. More analysis was done on the relation between the learning rate, effective learning rate, and batch size is presented in ICLR 2018, where the authors proved by experiments that all the benefits of decaying the learning rate are achieved by increasing the batch size in addition to reducing the number of parameter updates dramatically, and also were able use literature parameters without the need of any hyper parameter tuning (Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le).<br /> <br /> ==References==<br /> <br /> #Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. arXiv preprint arXiv:1612.05086, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012. <br /> #Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference converges to limit cycles for deep networks. arXiv preprint arXiv:1710.11029, 2017.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.<br /> #Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Crispin W Gardiner. Handbook of Stochastic Methods, volume 4. Springer Berlin, 1985.<br /> #Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884– 1892, 2016.<br /> #Priya Goyal, Piotr Dolla ́r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Stephen F Gull. Bayesian inductive inference and maximum entropy. In Maximum-entropy and Bayesian methods in science and engineering, pp. 53–74. Springer, 1988.<br /> #Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM,1993.<br /> #Sepp Hochreiter and Ju ̈rgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Stanisław Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017.<br /> #Robert E Kass and Adrian E Raftery. Bayes factors. Journal of the american statistical association, 90(430):773–795, 1995.<br /> #Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don’t learn via mem- orization. ICLR Workshop, 2017.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101–2110, 2017.<br /> #David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448–472, 1992.<br /> #Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810, 2017.<br /> #Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=conditional_neural_process&diff=42081 conditional neural process 2018-11-30T16:37:27Z <p>C9sharma: /* Meta Learning */</p> <hr /> <div>== Motivation ==<br /> <br /> Deep neural networks are good at function approximations, yet they are typically trained from scratch for each new function. While Bayesian methods, such as Gaussian Processes (GPs), exploit prior knowledge to quickly infer the shape of a new function at test time. Yet GPs<br /> are computationally expensive, and it can be hard to design appropriate priors. Hence the authors propose a propose a family of neural models called, Conditional Neural Processes (CNPs), that combine the benefits of both. <br /> <br /> == Introduction ==<br /> <br /> To train a model effectively, deep neural networks typically require large datasets. To mitigate this data efficiency problem, learning in two phases is one approach: the first phase learns the statistics of a generic domain without committing to a specific learning task; the second phase learns a function for a specific task but does so using only a small number of data points by exploiting the domain-wide statistics already learned. Taking a probabilistic stance and specifying a distribution over functions (stochastic processes) is another approach -- Gaussian Processes being a commonly used example of this. Such Bayesian methods can be computationally expensive. <br /> <br /> The authors of the paper propose a family of models that represent solutions to the supervised problem, and an end-to-end training approach to learning them that combines neural networks with features reminiscent of Gaussian Processes. They call this family of models Conditional Neural Processes (CNPs). CNPs can be trained on very few data points to make accurate predictions, while they also have the capacity to scale to complex functions and large datasets.<br /> <br /> == Model ==<br /> Consider a data set &lt;math display=&quot;inline&quot;&gt; \{x_i, y_i\} &lt;/math&gt; with evaluations &lt;math display=&quot;inline&quot;&gt;y_i = f(x_i) &lt;/math&gt; for some unknown function &lt;math display=&quot;inline&quot;&gt;f&lt;/math&gt;. Assume &lt;math display=&quot;inline&quot;&gt;g&lt;/math&gt; is an approximating function of f. The aim is to minimize the loss between &lt;math display=&quot;inline&quot;&gt;f&lt;/math&gt; and &lt;math display=&quot;inline&quot;&gt;g&lt;/math&gt; on the entire space &lt;math display=&quot;inline&quot;&gt;X&lt;/math&gt;. In practice, the routine is evaluated on a finite set of observations.<br /> <br /> <br /> Let training set be &lt;math display=&quot;inline&quot;&gt; O = \{x_i, y_i\}_{i = 0} ^{n-1}&lt;/math&gt;, and test set be &lt;math display=&quot;inline&quot;&gt; T = \{x_i, y_i\}_{i = n} ^ {n + m - 1} \subset X&lt;/math&gt; of unlabelled points.<br /> <br /> P be a probability distribution over functions &lt;math display=&quot;inline&quot;&gt; F : X \to Y&lt;/math&gt;, formally known as a stochastic process. Thus, P defines a joint distribution over the random variables &lt;math display=&quot;inline&quot;&gt; {f(x_i)}_{i = 0} ^{n + m - 1}&lt;/math&gt;. Therefore, for &lt;math display=&quot;inline&quot;&gt; P(f(x)|O, T)&lt;/math&gt;, our task is to predict the output values &lt;math display=&quot;inline&quot;&gt;f(x_i)&lt;/math&gt; for &lt;math display=&quot;inline&quot;&gt; x_i \in T&lt;/math&gt;, given &lt;math display=&quot;inline&quot;&gt; O&lt;/math&gt;. <br /> <br /> A common assumption made on P is that all function evaluations of &lt;math display=&quot;inline&quot;&gt; f &lt;/math&gt; is Gaussian distributed. The random functions class is called Gaussian Processes (GPs). This framework of the stochastic process allows a model to be data efficient, however, it's hard to get appropriate priors and stochastic processes are expensive in computation, scaling poorly with &lt;math&gt;n&lt;/math&gt; and &lt;math&gt;m&lt;/math&gt;. One of the examples is GPs, which has running time &lt;math&gt;O(n+3)^3&lt;/math&gt;.<br /> <br /> [[File:001.jpg|300px|center]]<br /> <br /> == Conditional Neural Process ==<br /> <br /> Conditional Neural Process models directly parametrize conditional stochastic processes without imposing consistency with respect to some prior process. CNP parametrize distributions over &lt;math display=&quot;inline&quot;&gt;f(T)&lt;/math&gt; given a distributed representation of &lt;math display=&quot;inline&quot;&gt;O&lt;/math&gt; of fixed dimensionality. Thus, the mathematical guarantees associated with stochastic processes is traded off for functional flexibility and scalability.<br /> <br /> CNP is a conditional stochastic process &lt;math display=&quot;inline&quot;&gt;Q_\theta&lt;/math&gt; defines distributions over &lt;math display=&quot;inline&quot;&gt;f(x_i)&lt;/math&gt; for &lt;math display=&quot;inline&quot;&gt;x_i \in T&lt;/math&gt;, given a set of observations &lt;math display=&quot;inline&quot;&gt;O&lt;/math&gt;. For stochastic processs, the authors assume that &lt;math display=&quot;inline&quot;&gt;Q_{\theta}&lt;/math&gt; is invariant to permutations, and &lt;math display=&quot;inline&quot;&gt;Q_\theta(f(T) | O, T)= Q_\theta(f(T') | O, T')=Q_\theta(f(T) | O', T) &lt;/math&gt; when &lt;math&gt; O', T'&lt;/math&gt; are permutations of &lt;math display=&quot;inline&quot;&gt;O&lt;/math&gt; and &lt;math display=&quot;inline&quot;&gt;T &lt;/math&gt;. In this work, we generally enforce permutation invariance with respect to &lt;math display=&quot;inline&quot;&gt;T&lt;/math&gt; be assuming a factored structure, which is the easiest way to ensure a valid stochastic process. That is, &lt;math display=&quot;inline&quot;&gt;Q_\theta(f(T) | O, T) = \prod _{x \in T} Q_\theta(f(x) | O, x)&lt;/math&gt;. Moreover, this framework can be extended to non-factored distributions.<br /> <br /> In detail, the following architecture is used<br /> <br /> &lt;math display=&quot;inline&quot;&gt;r_i = h_\theta(x_i, y_i)&lt;/math&gt; &amp;forall; &lt;math display=&quot;inline&quot;&gt;(x_i, y_i) \in O&lt;/math&gt;, where &lt;math display=&quot;inline&quot;&gt;h_\theta : X \times Y \to \mathbb{R} ^ d&lt;/math&gt;<br /> <br /> &lt;math display=&quot;inline&quot;&gt;r = r_i * r_2 * ... * r_n&lt;/math&gt;, where &lt;math display=&quot;inline&quot;&gt;*&lt;/math&gt; is a commutative operation that takes elements in &lt;math display=&quot;inline&quot;&gt;\mathbb{R}^d&lt;/math&gt; and maps them into a single element of &lt;math display=&quot;inline&quot;&gt;\mathbb{R} ^ d&lt;/math&gt;<br /> <br /> &lt;math display=&quot;inline&quot;&gt;\Phi_i = g_\theta&lt;/math&gt; &amp;forall; &lt;math display=&quot;inline&quot;&gt;x_i \in T&lt;/math&gt;, where &lt;math display=&quot;inline&quot;&gt;g_\theta : X \times \mathbb{R} ^ d \to \mathbb{R} ^ e&lt;/math&gt; and &lt;math display=&quot;inline&quot;&gt;\Phi_i&lt;/math&gt; are parameters for &lt;math display=&quot;inline&quot;&gt;Q_\theta&lt;/math&gt;<br /> <br /> Note that this architecture ensures permutation invariance and &lt;math display=&quot;inline&quot;&gt;O(n + m)&lt;/math&gt; scaling for conditional prediction. Also, &lt;math display=&quot;inline&quot;&gt;r = r_i * r_2 * ... * r_n&lt;/math&gt; can be computed in &lt;math display=&quot;inline&quot;&gt;O(n)&lt;/math&gt;, this architecture supports streaming observation with minimal overhead.<br /> <br /> We train &lt;math display=&quot;inline&quot;&gt;Q_\theta&lt;/math&gt; by asking it to predict &lt;math display=&quot;inline&quot;&gt;O&lt;/math&gt; conditioned on a randomly<br /> chosen subset of &lt;math display=&quot;inline&quot;&gt;O&lt;/math&gt;. This gives the model a signal of the uncertainty over the space X inherent in the distribution<br /> P given a set of observations. The authors let &lt;math display=&quot;inline&quot;&gt; f \sim P&lt;/math&gt;, &lt;math display=&quot;inline&quot;&gt; O = \{(x_i, y_i)\}_{i = 0} ^{n-1}&lt;/math&gt;, and N ~ uniform[0, 1, ..... ,n-1]. Subset &lt;math display=&quot;inline&quot;&gt; O = \{(x_i, y_i)\}_{i = 0} ^{N}&lt;/math&gt; that is first N elements of &lt;math display=&quot;inline&quot;&gt;O&lt;/math&gt; is regarded as condition. The negative conditional log probability is given by<br /> $\mathcal{L}(\theta)=-\mathbb{E}_{f \sim p}[\mathbb{E}_{N}[\log Q_\theta(\{y_i\}_{i = 0} ^{n-1}|O_{N}, \{x_i\}_{i = 0} ^{n-1})]]$<br /> Thus, the targets it scores &lt;math display=&quot;inline&quot;&gt;Q_\theta&lt;/math&gt; on include both the observed <br /> and unobserved values. In practice, Monte Carlo estimates of the gradient of this loss is taken by sampling &lt;math display=&quot;inline&quot;&gt;f&lt;/math&gt; and &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt;. <br /> <br /> This approach shifts the burden of imposing prior knowledge from an analytic prior to empirical data. This has the advantage of liberating a practitioner from having to specify an analytic form for the prior, which is ultimately<br /> intended to summarize their empirical experience. Still, we emphasize that the &lt;math display=&quot;inline&quot;&gt;Q_\theta&lt;/math&gt; are not necessarily a consistent set of conditionals for all observation sets, and the training routine does not guarantee that.<br /> <br /> In summary,<br /> <br /> 1. A CNP is a conditional distribution over functions<br /> trained to model the empirical conditional distributions<br /> of functions &lt;math display=&quot;inline&quot;&gt;f \sim P&lt;/math&gt;.<br /> <br /> 2. A CNP is permutation invariant in &lt;math display=&quot;inline&quot;&gt;O&lt;/math&gt; and &lt;math display=&quot;inline&quot;&gt;T&lt;/math&gt;.<br /> <br /> 3. A CNP is scalable, achieving a running time complexity<br /> of &lt;math display=&quot;inline&quot;&gt;O(n + m)&lt;/math&gt; for making &lt;math display=&quot;inline&quot;&gt;m&lt;/math&gt; predictions with &lt;math display=&quot;inline&quot;&gt;n&lt;/math&gt;<br /> observations.<br /> <br /> == Related Work ==<br /> <br /> ===Gaussian Process Framework===<br /> <br /> A Gaussian Process (GP) is a non-parametric method for regression, used extensively for regression and classification problems in the machine learning community. A GP is defined as a collection of random variables, any finite number of which have a joint Gaussian distribution.<br /> A standard approach is to model data as &lt;math&gt;y = m(X, φ) + \epsilon&lt;/math&gt;<br /> where m is the mean function with parameter vector &lt;math&gt;φ&lt;/math&gt;, and &lt;math&gt;\epsilon&lt;/math&gt; represents independent and identically distributed (i.i.d.) Gaussian noise: &lt;math&gt;N\sim (0,\sigma^2)&lt;/math&gt;<br /> <br /> For more info on Gaussian Process Framework:<br /> [https://arxiv.org/abs/1506.07304 A Gaussian process framework for modeling instrumental systematics: application to transmission spectroscopy]<br /> <br /> Several papers attempt to address various issues with GPs. These include:<br /> * Using sparse GPs to aid in scaling (Snelson &amp; Ghahramani, 2006)<br /> * Using Deep GPs to achieve more expressiveness (Damianou &amp; Lawrence, 2013; Salimbeni &amp; Deisenroth, 2017)<br /> * Using neural networks to learn more expressive kernels (Wilson et al., 2016)<br /> <br /> A Python resource for Gaussian Process Framework implementation: [https://github.com/SheffieldML/GPyimplementation Gaussian Process Framework in Python]<br /> <br /> <br /> The goal of this paper is to incorporate ideas from standard neural networks with Gaussian processes in order to overcome drawbacks of both. Bayesian techniques work better with less data, but complex Bayesian networks become intractable on even moderate sized data sizes. NNs on the other hand, cannot make use of prior knowledge and often have to be retrained from scratch. Without sufficient data, they also perform poorly. Combining both frameworks, we get Conditional Neural Processes serves to learn the kernels of the Gaussian Process through neural networks and uses these learned kernels on a framework similar to GPs for prediction.<br /> <br /> ===Meta Learning===<br /> <br /> Meta-Learning attempts to allow neural networks to learn more generalizable functions, as opposed to only approximating one function. This can be done by learning deep generative models which can do few-shot estimations of data. This can be implemented with attention mechanisms (Reed et al., 2017) or additional memory units in a VAE model (Bornschein et al., 2017). Another successful latent variable approach is to explicitly condition on some context during inference (J. Rezende et al., 2016). Given the generative nature of these models they are usually applied to image generation tasks, but models that include a conditioning class-variable can be used for classification as well. Recently meta-learning has also been applied to a wide range of tasks like RL (Wang et al., 2016; Finn et al., 2017) or program induction (Devlin et al., 2017).<br /> <br /> Classification is another common task in meta-learning, few-shot classification algorithms usually rely on some distance metric in feature space to compare target images and the observations (Koch et al., 2015), (Santoro et al., 2016).. Matching networks(Vinyals et al., 2016; Bartunov &amp; Vetrov, 2016) are closely related to CNPs. In their case features of samples are compared with target features using an attention kernel. At a higher level one can interpret this model as a CNP where the aggregator is just the concatenation over all input samples and the decoder &lt;math&gt;g&lt;/math&gt; contains an explicitly defined distance kernel. In this sense matching networks are closer to GPs than to CNPs, since they require the specification of a distance kernel that CNPs learn from the data instead. In addition, as MNs carry out all- to-all comparisons they scale with &lt;math&gt; O(n × m) &lt;/math&gt;, although they can be modified to have the same complexity of &lt;math&gt;O(n + m)&lt;/math&gt; as CNPs (Snell et al., 2017).<br /> <br /> A model that is conceptually very similar to CNPs (and in particular the latent variable version) is the “neural statistician” paper (Edwards &amp; Storkey, 2016) and the related variational homoencoder (Hewitt et al., 2018). As with the<br /> other generative models the neural statistician learns to estimate the density of the observed data but does not allow for targeted sampling at what we have been referring to as input positions &lt;math&gt;x_i&lt;/math&gt;. Instead, one can only generate i.i.d. samples from the estimated density. Finally, the latest variant of Conditional Neural Process can also be seen as an approximated amortized version of Bayesian DL(Gal &amp; Ghahramani, 2016; Blundell et al., 2015; Louizos et al., 2017; Louizos &amp; Welling, 2017). For example, Gal &amp; Ghahramani 2016 develop a new theoretical framework casting dropout training in deep neural networks as approximate Bayesian inference in deep Gaussian processes. Their theory extracts information from existing models and gives us tools to model uncertainty.<br /> <br /> == Experimental Result I: Function Regression ==<br /> <br /> Classical 1D regression task that used as a common baseline for GP is the first example. <br /> They generated two different datasets that consisted of functions<br /> generated from a GP with an exponential kernel. In the first dataset they used a kernel with fixed parameters, and in the second dataset, the function switched at some random point. on the real line between two functions, each sampled with<br /> different kernel parameters. At every training step, they sampled a curve from the GP, select<br /> a subset of n points as observations, and a subset of t points as target points. Using the model, the observed points are encoded using a three-layer MLP encoder h with a 128-dimensional output representation. The representations are aggregated into a single representation<br /> &lt;math display=&quot;inline&quot;&gt;r = \frac{1}{n} \sum r_i&lt;/math&gt;<br /> , which is concatenated to &lt;math display=&quot;inline&quot;&gt;x_t&lt;/math&gt; and passed to a decoder g consisting of a five layer<br /> MLP. The function outputs a Gaussian mean and variance for the target outputs. The model is trained to maximize the log-likelihood of the target points using the Adam optimizer. <br /> <br /> Two examples of the regression results obtained for each<br /> of the datasets are shown in the following figure.<br /> <br /> [[File:007.jpg|300px|center]]<br /> <br /> They compared the model to the predictions generated by a GP with the correct<br /> hyperparameters, which constitutes an upper bound on our<br /> performance. Although the prediction generated by the GP<br /> is smoother than the CNP's prediction both for the mean<br /> and variance, the model is able to learn to regress from a few<br /> context points for both the fixed kernels and switching kernels.<br /> As the number of context points grows, the accuracy<br /> of the model improves and the approximated uncertainty<br /> of the model decreases. Crucially, we see the model learns<br /> to estimate its own uncertainty given the observations very<br /> accurately. Nonetheless, it provides a good approximation<br /> that increases in accuracy as the number of context points<br /> increases.<br /> Furthermore, the model achieves similarly good performance<br /> on the switching kernel task. This type of regression task<br /> is not trivial for GPs whereas in our case we only have to<br /> change the dataset used for training<br /> <br /> == Experimental Result II: Image Completion for Digits ==<br /> <br /> [[File:002.jpg|600px|center]]<br /> <br /> They also tested CNP on the MNIST dataset and use the test<br /> set to evaluate its performance. As shown in the above figure the<br /> model learns to make good predictions of the underlying<br /> digit even for a small number of context points. Crucially,<br /> when conditioned only on one non-informative context point the model’s prediction corresponds<br /> to the average overall MNIST digits. As the number<br /> of context points increases the predictions become more<br /> similar to the underlying ground truth. This demonstrates<br /> the model’s capacity to extract dataset specific prior knowledge.<br /> It is worth mentioning that even with a complete set<br /> of observations, the model does not achieve pixel-perfect<br /> reconstruction, as we have a bottleneck at the representation<br /> level.<br /> Since this implementation of CNP returns factored outputs,<br /> the best prediction it can produce given limited context<br /> information is to average over all possible predictions that<br /> agree with the context. An alternative to this is to add<br /> latent variables in the model such that they can be sampled<br /> conditioned on the context to produce predictions with high<br /> probability in the data distribution. <br /> <br /> <br /> An important aspect of the model is its ability to estimate<br /> the uncertainty of the prediction. As shown in the bottom<br /> row of the above figure, as they added more observations, the variance<br /> shifts from being almost uniformly spread over the digit<br /> positions to being localized around areas that are specific<br /> to the underlying digit, specifically its edges. Being able to<br /> model the uncertainty given some context can be helpful for<br /> many tasks. One example is active exploration, where the<br /> model has a choice over where to observe.<br /> They tested this by<br /> comparing the predictions of CNP when the observations<br /> are chosen according to uncertainty, versus random pixels. This method is a very simple way of doing active<br /> exploration, but it already produces better prediction results<br /> then selecting the conditioning points at random.<br /> <br /> == Experimental Result III: Image Completion for Faces ==<br /> <br /> <br /> [[File:003.jpg|400px|center]]<br /> <br /> <br /> They also applied CNP to CelebA, a dataset of images of<br /> celebrity faces and reported performance obtained on the<br /> test set.<br /> <br /> As shown in the above figure our model is able to capture<br /> the complex shapes and colors of this dataset with predictions<br /> conditioned on less than 10% of the pixels being<br /> already close to the ground truth. As before, given a few contexts<br /> points the model averages over all possible faces, but as<br /> the number of context pairs increases the predictions capture<br /> image-specific details like face orientation and facial<br /> expression. Furthermore, as the number of context points<br /> increases the variance is shifted towards the edges in the<br /> image.<br /> <br /> [[File:004.jpg|400px|center]]<br /> <br /> An important aspect of CNPs demonstrated in the above figure is<br /> it's flexibility not only in the number of observations and<br /> targets it receives but also with regards to their input values.<br /> It is interesting to compare this property to GPs on one hand,<br /> and to trained generative models (van den Oord et al., 2016;<br /> Gregor et al., 2015) on the other hand.<br /> The first type of flexibility can be seen when conditioning on<br /> subsets that the model has not encountered during training.<br /> Consider conditioning the model on one half of the image,<br /> fox example. This forces the model to not only predict the pixel<br /> values according to some stationary smoothness property of<br /> the images, but also according to global spatial properties,<br /> e.g. symmetry and the relative location of different parts of<br /> faces. As seen in the first row of the figure, CNPs are able to<br /> capture those properties. A GP with a stationary kernel cannot<br /> capture this, and in the absence of observations would<br /> revert to its mean (the mean itself can be non-stationary but<br /> usually, this would not be enough to capture the interesting<br /> properties).<br /> <br /> In addition, the model is flexible with regards to the target<br /> input values. This means, e.g., we can query the model<br /> at resolutions it has not seen during training. We take a<br /> model that has only been trained using pixel coordinates of<br /> a specific resolution and predict at test time subpixel values<br /> for targets between the original coordinates. As shown in<br /> Figure 5, with one forward pass we can query the model at<br /> different resolutions. While GPs also exhibit this type of<br /> flexibility, it is not the case for trained generative models,<br /> which can only predict values for the pixel coordinates on<br /> which they were trained. In this sense, CNPs capture the best<br /> of both worlds – it is flexible in regards to the conditioning<br /> and prediction task and has the capacity to extract domain<br /> knowledge from a training set.<br /> <br /> [[File:010.jpg|400px|center]]<br /> <br /> <br /> They compared CNPs quantitatively to two related models:<br /> kNNs and GPs. As shown in the above table CNPs outperform<br /> the latter when a number of context points are small (empirically<br /> when half of the image or less is provided as context).<br /> When the majority of the image is given as context exact<br /> methods like GPs and kNN will perform better. From the table<br /> we can also see that the order in which the context points<br /> are provided is less important for CNPs, since providing the<br /> context points in order from top to bottom still results in<br /> good performance. Both insights point to the fact that CNPs<br /> learn a data-specific ‘prior’ that will generate good samples<br /> even when the number of context points is very small.<br /> <br /> == Experimental Result IV: Classification ==<br /> Finally, they applied the model to one-shot classification using the Omniglot dataset. This dataset consists of 1,623 classes of characters from 50 different alphabets. Each class has only 20 examples and as such this dataset is particularly suitable for few-shot learning algorithms. The authors used 1,200 randomly selected classes as their training set and the remainder as the testing data set.<br /> <br /> Additionally, to apply data augmentation the authors cropped the image from 32 × 32 to 28 × 28, applied small random<br /> translations and rotations to the inputs, and also increased<br /> the number of classes by rotating every character by 90<br /> degrees and defining that to be a new class. They generated<br /> the labels for an N-way classification task by choosing N<br /> random classes at each training step and arbitrarily assigning<br /> the labels 0, ..., N − 1 to each.<br /> <br /> <br /> [[File:008.jpg|400px|center]]<br /> <br /> Given that the input points are images, they modified the architecture<br /> of the encoder h to include convolution layers as<br /> mentioned in section 2. In addition, they only aggregated over<br /> inputs of the same class by using the information provided<br /> by the input label. The aggregated class-specific representations<br /> are then concatenated to form the final representation.<br /> Given that both the size of the class-specific representations<br /> and the number of classes is constant, the size of the final<br /> representation is still constant and thus the O(n + m)<br /> runtime still holds.<br /> The results of the classification are summarized in the following table<br /> CNPs achieve higher accuracy than models that are significantly<br /> more complex (like MANN). While CNPs do not<br /> beat state of the art for one-shot classification our accuracy<br /> values are comparable. Crucially, they reached those values<br /> using a significantly simpler architecture (three convolutional<br /> layers for the encoder and a three-layer MLP for the<br /> decoder) and with a lower runtime of O(n + m) at test time<br /> as opposed to O(nm)<br /> <br /> == Conclusion ==<br /> <br /> The paper introduced Conditional Neural Processes,<br /> a model that is both flexible at test time and has the<br /> capacity to extract prior knowledge from training data.<br /> <br /> The authors had demonstrated its ability to perform a variety of tasks<br /> including regression, classification and image completion.<br /> The paper compared CNP's to Gaussian Processes on one hand, and<br /> deep learning methods on the other, and also discussed the<br /> relation to meta-learning and few-shot learning.<br /> It is important to note that the specific CNP implementations<br /> described here are just simple proofs-of-concept and can<br /> be substantially extended, e.g. by including more elaborate<br /> architectures in line with modern deep learning advances.<br /> To summarize, this work can be seen as a step towards learning<br /> high-level abstractions, one of the grand challenges of<br /> contemporary machine learning. Functions learned by most<br /> Conditional Neural Processes<br /> conventional deep learning models are tied to a specific, constrained<br /> statistical context at any stage of training. A trained<br /> CNP is more general, in that it encapsulates the high-level<br /> statistics of a family of functions. As such it constitutes a<br /> high-level abstraction that can be reused for multiple tasks.<br /> In future work, they are going to explore how far these models can<br /> help in tackling the many key machine learning problems<br /> that seem to hinge on abstraction, such as transfer learning,<br /> meta-learning, and data efficiency.<br /> <br /> == Critiques ==<br /> <br /> This paper introduces a method, for reducing the computational complexity of the more famous Gaussian Processes model, but they have mentioned a complexity of O(n + m) which is almost the same order of RBF kernel GP. With respect to performances in a sequence of tasks, the authors have not made metric comparisons to GP methods to prove the superiority of their approach.<br /> <br /> It appears that the proposed model is effective in making accurate predictions using lower quality inputs. For example, a dataset with fewer data points or an image with fewer pixels. However, it is not clear whether the proposed algorithm can be trained with a smaller amount of input data.<br /> <br /> == Other Sources ==<br /> # Code for this model and a simpler explanation can be found at [https://github.com/deepmind/conditional-neural-process]<br /> # A newer version of the model is described in this paper [https://arxiv.org/pdf/1807.01622.pdf]<br /> # A good blog post on neural processes [https://kasparmartens.rbind.io/post/np/]<br /> <br /> == Reference ==<br /> Bartunov, S. and Vetrov, D. P. Fast adaptation in generative<br /> models with generative matching networks. arXiv<br /> preprint arXiv:1612.02192, 2016.<br /> <br /> Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra,<br /> D. Weight uncertainty in neural networks. arXiv preprint<br /> arXiv:1505.05424, 2015.<br /> <br /> Bornschein, J., Mnih, A., Zoran, D., and J. Rezende, D.<br /> Variational memory addressing in generative models. In<br /> Advances in Neural Information Processing Systems, pp.<br /> 3923–3932, 2017.<br /> <br /> Damianou, A. and Lawrence, N. Deep gaussian processes.<br /> In Artificial Intelligence and Statistics, pp. 207–215,<br /> 2013.<br /> <br /> Devlin, J., Bunel, R. R., Singh, R., Hausknecht, M., and<br /> Kohli, P. Neural program meta-induction. In Advances in<br /> Neural Information Processing Systems, pp. 2077–2085,<br /> 2017.<br /> <br /> Edwards, H. and Storkey, A. Towards a neural statistician.<br /> 2016.<br /> <br /> Finn, C., Abbeel, P., and Levine, S. Model-agnostic metalearning<br /> for fast adaptation of deep networks. arXiv<br /> preprint arXiv:1703.03400, 2017.<br /> <br /> Gal, Y. and Ghahramani, Z. Dropout as a bayesian approximation:<br /> Representing model uncertainty in deep learning.<br /> In international conference on machine learning, pp.<br /> 1050–1059, 2016.<br /> <br /> Garnelo, M., Arulkumaran, K., and Shanahan, M. Towards<br /> deep symbolic reinforcement learning. arXiv preprint<br /> arXiv:1609.05518, 2016.<br /> <br /> Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., and<br /> Wierstra, D. Draw: A recurrent neural network for image<br /> generation. arXiv preprint arXiv:1502.04623, 2015.<br /> <br /> Hewitt, L., Gane, A., Jaakkola, T., and Tenenbaum, J. B. The<br /> variational homoencoder: Learning to infer high-capacity<br /> generative models from few examples. 2018.<br /> <br /> J. Rezende, D., Danihelka, I., Gregor, K., Wierstra, D.,<br /> et al. One-shot generalization in deep generative models.<br /> In International Conference on Machine Learning, pp.<br /> 1521–1529, 2016.<br /> <br /> Kingma, D. P. and Ba, J. Adam: A method for stochastic<br /> optimization. arXiv preprint arXiv:1412.6980, 2014.<br /> <br /> Kingma, D. P. and Welling, M. Auto-encoding variational<br /> bayes. arXiv preprint arXiv:1312.6114, 2013.<br /> <br /> Koch, G., Zemel, R., and Salakhutdinov, R. Siamese neural<br /> networks for one-shot image recognition. In ICML Deep<br /> Learning Workshop, volume 2, 2015.<br /> <br /> Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B.<br /> Human-level concept learning through probabilistic program<br /> induction. Science, 350(6266):1332–1338, 2015.<br /> <br /> Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman,<br /> S. J. Building machines that learn and think like<br /> people. Behavioral and Brain Sciences, 40, 2017.<br /> <br /> LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradientbased<br /> learning applied to document recognition. Proceedings<br /> of the IEEE, 86(11):2278–2324, 1998.<br /> <br /> Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face<br /> attributes in the wild. In Proceedings of International<br /> Conference on Computer Vision (ICCV), December 2015.<br /> <br /> Louizos, C. and Welling, M. Multiplicative normalizing<br /> flows for variational bayesian neural networks. arXiv<br /> preprint arXiv:1703.01961, 2017.<br /> <br /> Louizos, C., Ullrich, K., and Welling, M. Bayesian compression<br /> for deep learning. In Advances in Neural Information<br /> Processing Systems, pp. 3290–3300, 2017.<br /> <br /> Rasmussen, C. E. and Williams, C. K. Gaussian processes<br /> in machine learning. In Advanced lectures on machine<br /> learning, pp. 63–71. Springer, 2004.<br /> <br /> Reed, S., Chen, Y., Paine, T., Oord, A. v. d., Eslami, S.,<br /> J. Rezende, D., Vinyals, O., and de Freitas, N. Few-shot<br /> autoregressive density estimation: Towards learning to<br /> learn distributions. 2017.<br /> <br /> Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic<br /> backpropagation and approximate inference in deep generative<br /> models. arXiv preprint arXiv:1401.4082, 2014.<br /> <br /> Salimbeni, H. and Deisenroth, M. Doubly stochastic variational<br /> inference for deep gaussian processes. In Advances<br /> in Neural Information Processing Systems, pp.<br /> 4591–4602, 2017.<br /> <br /> Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., and<br /> Lillicrap, T. One-shot learning with memory-augmented<br /> neural networks. arXiv preprint arXiv:1605.06065, 2016.<br /> <br /> Snell, J., Swersky, K., and Zemel, R. Prototypical networks<br /> for few-shot learning. In Advances in Neural Information<br /> Processing Systems, pp. 4080–4090, 2017.<br /> <br /> Snelson, E. and Ghahramani, Z. Sparse gaussian processes<br /> using pseudo-inputs. In Advances in neural information<br /> processing systems, pp. 1257–1264, 2006.<br /> <br /> van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals,<br /> O., Graves, A., et al. Conditional image generation with<br /> pixelcnn decoders. In Advances in Neural Information<br /> Processing Systems, pp. 4790–4798, 2016.<br /> <br /> Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.<br /> Matching networks for one shot learning. In Advances in<br /> Neural Information Processing Systems, pp. 3630–3638,<br /> 2016.<br /> <br /> Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H.,<br /> Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and<br /> Botvinick, M. Learning to reinforcement learn. arXiv<br /> preprint arXiv:1611.05763, 2016.<br /> <br /> Wilson, A. G., Hu, Z., Salakhutdinov, R., and Xing, E. P.<br /> Deep kernel learning. In Artificial Intelligence and Statistics,<br /> pp. 370–378, 2016.<br /> <br /> Damianou, A. and Lawrence, N. Deep gaussian processes.<br /> In Artificial Intelligence and Statistics, pp. 207–215,<br /> 2013.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=DON%27T_DECAY_THE_LEARNING_RATE_,_INCREASE_THE_BATCH_SIZE&diff=42069 DON'T DECAY THE LEARNING RATE , INCREASE THE BATCH SIZE 2018-11-30T16:14:47Z <p>C9sharma: /* SIMULATED ANNEALING IN A WIDE RESNET */</p> <hr /> <div>Summary of the ICLR 2018 paper: '''Don't Decay the learning Rate, Increase the Batch Size ''' <br /> <br /> Link: [https://arxiv.org/pdf/1711.00489.pdf]<br /> <br /> Summarized by: Afify, Ahmed [ID: 20700841]<br /> <br /> ==INTUITION==<br /> Nowadays, it is a common practice not to have a singular steady learning rate for the learning phase of neural network models. Instead, we use adaptive learning rates with the standard gradient descent method. The intuition behind this is that when we are far away from the minima, it is beneficial for us to take large steps towards the minima, as it would require a lesser number of steps to converge, but as we approach the minima, our step size should decrease, otherwise we may just keep oscillating around the minima. In practice, this is generally achieved by methods like SGD with momentum, Nesterov momentum, and Adam. However, the core claim of this paper is that the same effect can be achieved by increasing the batch size during the gradient descent process while keeping the learning rate constant throughout. In addition, the paper argues that such an approach also reduces the parameter updates required to reach the minima, thus leading to greater parallelism and shorter training times.<br /> <br /> == INTRODUCTION ==<br /> Stochastic gradient descent (SGD) is the most widely used optimization technique for training deep learning models. The reason for this is that the minima found using this process generalizes well (Zhang et al., 2016; Wilson et al., 2017), but the optimization process is slow and time consuming. According to (Goyal et al., 2017; Hoffer et al., 2017; You et al., 2017a), this has motivated researchers to try to speed up this optimization process by taking bigger steps, and hence reduce the number of parameter updates in training a model. This can be achieved by using large batch training, which can be divided across many machines. <br /> <br /> However, increasing the batch size leads to decreasing the test set accuracy (Keskar et al., 2016; Goyal et al., 2017). Smith and Le (2017) believed that SGD has a scale of random fluctuations &lt;math&gt; g = \epsilon (\frac{N}{B}-1) &lt;/math&gt;, where &lt;math&gt; \epsilon &lt;/math&gt; is the learning rate, N number of training samples, and B batch size. They concluded that there is an optimal batch size proportional to the learning rate when &lt;math&gt; B \ll N &lt;/math&gt;, and optimum fluctuation scale &lt;math&gt;g&lt;/math&gt; for a maximum test set accuracy.<br /> <br /> In this paper, the authors' main goal is to provide evidence that increasing the batch size is quantitatively equivalent to decreasing the learning rate with the same number of training epochs in decreasing the scale of random fluctuations, but with remarkably less number of parameter updates. Moreover, an additional reduction in the number of parameter updates can be attained by increasing the learning rate and scaling &lt;math&gt; B \propto \epsilon &lt;/math&gt; or even more reduction by increasing the momentum coefficient and scaling &lt;math&gt; B \propto \frac{1}{1-m} &lt;/math&gt; although the latter decreases the test accuracy. This has been demonstrated by several experiments on the ImageNet and CIFAR-10 datasets using ResNet-50 and Inception-ResNet-V2 architectures respectively.<br /> <br /> == STOCHASTIC GRADIENT DESCENT AND CONVEX OPTIMIZATION ==<br /> As mentioned in the previous section, the drawback of SGD when compared to full-batch training is the noise that it introduces that hinders optimization. According to (Robbins &amp; Monro, 1951), there are two equations that govern how to reach the minimum of a convex function: (&lt;math&gt; \epsilon_i &lt;/math&gt; denotes the learning rate at the &lt;math&gt; i^{th} &lt;/math&gt; gradient update)<br /> <br /> &lt;math&gt; \sum_{i=1}^{\infty} \epsilon_i = \infty &lt;/math&gt;. This equation guarantees that we will reach the minimum. <br /> <br /> &lt;math&gt; \sum_{i=1}^{\infty} \epsilon^2_i &lt; \infty &lt;/math&gt;. This equation, which is valid only for a fixed batch size, guarantees that learning rate decays fast enough allowing us to reach the minimum rather than bouncing due to noise.<br /> <br /> These equations indicate that the learning rate must decay during training, and second equation is only available when the batch size is constant. To change the batch size, Smith and Le (2017) proposed to interpret SGD as integrating this stochastic differential equation &lt;math&gt; \frac{dw}{dt} = -\frac{dC}{dw} + \eta(t) &lt;/math&gt;, where &lt;math&gt;C&lt;/math&gt; represents cost function, &lt;math&gt;w&lt;/math&gt; represents the parameters, and &lt;math&gt;\eta&lt;/math&gt; represents the Gaussian random noise. Furthermore, they proved that noise scale &lt;math&gt;g&lt;/math&gt; controls the magnitude of random fluctuations in the training dynamics by this formula: &lt;math&gt; g = \epsilon (\frac{N}{B}-1) &lt;/math&gt;, where &lt;math&gt; \epsilon &lt;/math&gt; is the learning rate, N is the training set size and &lt;math&gt;B&lt;/math&gt; is the batch size. As we usually have &lt;math&gt; B \ll N &lt;/math&gt;, we can define &lt;math&gt; g \approx \epsilon \frac{N}{B} &lt;/math&gt;. This explains why when the learning rate decreases, noise &lt;math&gt;g&lt;/math&gt; decreases, enabling us to converge to the minimum of the cost function. However, increasing the batch size has the same effect and makes &lt;math&gt;g&lt;/math&gt; decays with constant learning rate. In this work, the batch size is increased until &lt;math&gt; B \approx \frac{N}{10} &lt;/math&gt;, then the conventional way of decaying the learning rate is followed.<br /> <br /> == SIMULATED ANNEALING AND THE GENERALIZATION GAP ==<br /> '''Simulated Annealing:''' decaying learning rates are empirically successful. To understand this, they note that introducing random fluctuations<br /> whose scale falls during training is also a well established technique in non-convex optimization; simulated annealing. The initial noisy optimization phase allows to explore a larger fraction of the parameter space without becoming trapped in local minima. Once a promising region of parameter space is located, the noise is reduced to fine-tune the parameters.<br /> <br /> For more info: Simulated annealing (SA) is a probabilistic technique for approximating the global optimum of a given function. Specifically, it is a metaheuristic to approximate global optimization in a large search space for an optimization problem. It is often used when the search space is discrete (e.g., all tours that visit a given set of cities). For problems where finding an approximate global optimum is more important than finding a precise local optimum in a fixed amount of time, simulated annealing may be preferable to alternatives such as gradient descent. [https://en.wikipedia.org/wiki/Simulated_annealing [Reference]]<br /> <br /> '''Generalization Gap:''' Small batch data generalizes better to the test set than large batch data.<br /> <br /> Smith and Le (2017) found that there is an optimal batch size which corresponds to optimal noise scale g &lt;math&gt; (g \approx \epsilon \frac{N}{B}) &lt;/math&gt; and concluded that &lt;math&gt; B_{opt} \propto \epsilon N &lt;/math&gt; that corresponds to maximum test set accuracy. This means that gradient noise is helpful as it makes SGD escape sharp minima, which does not generalize well. <br /> <br /> Simulated Annealing is a famous technique in non-convex optimization. Starting with noise in the training process helps us to discover a wide range of parameters then once we are near the optimum value, noise is reduced to fine tune our final parameters. However, more and more researches like to use the sharper decay schedules like cosine decay or step-function drops. In physical sciences, slowly annealing (or decaying) the temperature (which is the noise scale in this situation) helps to converge to the global minimum, which is sharp. But decaying the temperature in discrete steps can make the system stuck in a local minimum, which lead to higher cost and lower curvature. The authors think that deep learning has the same intuition.<br /> .<br /> <br /> == THE EFFECTIVE LEARNING RATE AND THE ACCUMULATION VARIABLE ==<br /> '''The Effective Learning Rate''' : &lt;math&gt; \epsilon_{eff} = \frac{\epsilon}{1-m} &lt;/math&gt;<br /> <br /> Smith and Le (2017) included momentum to the equation of the vanilla SGD noise scale that was defined above to be: &lt;math&gt; g = \frac{\epsilon}{1-m}(\frac{N}{B}-1)\approx \frac{\epsilon N}{B(1-m)} &lt;/math&gt;, which is the same as the previous equation when m goes to 0. They found that increasing the learning rate and momentum coefficient and scaling &lt;math&gt; B \propto \frac{\epsilon }{1-m} &lt;/math&gt; reduces the number of parameter updates, but the test accuracy decreases when the momentum coefficient is increased. <br /> <br /> To understand the reasons behind this, we need to analyze momentum update equations below:<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \Delta A = -(1-m)A + \frac{d\widehat{C}}{dw} <br /> &lt;/math&gt;<br /> <br /> &lt;math&gt;<br /> \Delta w = -A\epsilon<br /> &lt;/math&gt;<br /> &lt;/center&gt;<br /> <br /> We can see that the Accumulation variable A, which is initially set to 0, then increases exponentially to reach its steady state value during &lt;math&gt; \frac{B}{N(1-m)} &lt;/math&gt; training epochs while &lt;math&gt; \Delta w &lt;/math&gt; is suppressed that can reduce the rate of convergence. Moreover, at high momentum, we have three challenges:<br /> <br /> 1- Additional epochs are needed to catch up with the accumulation.<br /> <br /> 2- Accumulation needs more time &lt;math&gt; \frac{B}{N(1-m)} &lt;/math&gt; to forget old gradients. <br /> <br /> 3- After this time, however, the accumulation cannot adapt to changes in the loss landscape.<br /> <br /> 4- In the early stage, large batch size will lead to the instabilities.<br /> <br /> == EXPERIMENTS ==<br /> === SIMULATED ANNEALING IN A WIDE RESNET ===<br /> <br /> '''Dataset:''' CIFAR-10 (50,000 training images)<br /> <br /> '''Network Architecture:''' “16-4” wide ResNet<br /> <br /> '''Training Schedules used as in the below figure:''' . These demonstrate the equivalence between decreasing the learning rate and increasing the batch size.<br /> <br /> - Decaying learning rate: learning rate decays by a factor of 5 at a sequence of “steps”, and the batch size is constant<br /> <br /> - Increasing batch size: learning rate is constant, and the batch size is increased by a factor of 5 at every step.<br /> <br /> - Hybrid: At the beginning, the learning rate is constant and batch size is increased by a factor of 5. Then, the learning rate decays by a factor of 5 at each subsequent step, and the batch size is constant. This is the schedule that will be used if there is a hardware limit affecting a maximum batch size limit.<br /> <br /> If the learning rate itself must decay during training, then these schedules should show different learning curves (as a function of the number of training epochs) and reach different final test set accuracies. Meanwhile if it is the noise scale which should decay, all three schedules should be indistinguishable.<br /> [[File:Paper_40_Fig_1.png | 800px|center]]<br /> <br /> As shown in the below figure: in the left figure (2a), we can observe that for the training set, the three learning curves are exactly the same while in figure 2b, increasing the batch size has a huge advantage of reducing the number of parameter updates.<br /> This concludes that noise scale is the one that needs to be decayed and not the learning rate itself<br /> [[File:Paper_40_Fig_2.png | 800px|center]] <br /> <br /> To make sure that these results are the same for the test set as well, in figure 3, we can see that the three learning curves are exactly the same for SGD with momentum, and Nesterov momentum<br /> [[File:Paper_40_Fig_3.png | 800px|center]]<br /> <br /> To check for other optimizers as well. the below figure shows the same experiment as in figure 3, which is the three learning curves for test set, but for vanilla SGD and Adam, and showing <br /> [[File:Paper_40_Fig_4.png | 800px|center]]<br /> <br /> '''Conclusion:''' Decreasing the learning rate and increasing the batch size during training are equivalent<br /> <br /> === INCREASING THE EFFECTIVE LEARNING RATE===<br /> <br /> Here, the focus is on minimizing the number of parameter updates required to train a model. As shown above, the first step is to replace decaying learning rates by increasing batch sizes. Now, the authors show here that we can also increase the effective learning rate &lt;math&gt;\epsilon_{eff} = \epsilon/(1 − m) &lt;/math&gt; at the start of training, while scaling the initial batch size &lt;math&gt;B \propto \epsilon_{eff} &lt;/math&gt; . All experiments are conducted using SGD with momentum. There are 50000 images in the CIFAR-10 training set, and since the scaling rules only hold when &lt;math&gt;B &lt;&lt; N &lt;/math&gt; , we decided to set a maximum batch size &lt;math&gt;B_{max} &lt;/math&gt;= 5120 .<br /> <br /> '''Dataset:''' CIFAR-10 (50,000 training images)<br /> <br /> '''Network Architecture:''' “16-4” wide ResNet<br /> <br /> '''Training Parameters:''' Optimization Algorithm: SGD with momentum / Maximum batch size = 5120<br /> <br /> '''Training Schedules:''' <br /> <br /> The authors consider four training schedules, all of which decay the noise scale by a factor of five in a series of three steps with the same number of epochs.<br /> <br /> Original training schedule: initial learning rate of 0.1 which decays by a factor of 5 at each step, a momentum coefficient of 0.9, and a batch size of 128. Follows the implementation of Zagoruyko &amp; Komodakis (2016).<br /> <br /> Increasing batch size: learning rate of 0.1, momentum coefficient of 0.9, initial batch size of 128 that increases by a factor of 5 at each step. <br /> <br /> Increased initial learning rate: initial learning rate of 0.5, initial batch size of 640 that increase during training.<br /> <br /> Increased momentum coefficient: increased initial learning rate of 0.5, initial batch size of 3200 that increase during training, and an increased momentum coefficient of 0.98.<br /> <br /> The results of all training schedules, which are presented in the below figure, are documented in the following table:<br /> <br /> [[File:Paper_40_Table_1.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_5.png | 800px|center]]<br /> <br /> <br /> <br /> '''Conclusion:''' Increasing the effective learning rate and scaling the batch size results in further reduction in the number of parameter updates<br /> <br /> === TRAINING IMAGENET IN 2500 PARAMETER UPDATES===<br /> <br /> '''A) Experiment Goal:''' Control Batch Size<br /> <br /> '''Dataset:''' ImageNet (1.28 million training images)<br /> <br /> The paper modified the setup of Goyal et al. (2017), and used the following configuration:<br /> <br /> '''Network Architecture:''' Inception-ResNet-V2 <br /> <br /> '''Training Parameters:''' <br /> <br /> 90 epochs / noise decayed at epoch 30, 60, and 80 by a factor of 10 / Initial ghost batch size = 32 / Learning rate = 3 / momentum coefficient = 0.9 / Initial batch size = 8192<br /> <br /> Two training schedules were used:<br /> <br /> “Decaying learning rate”, where batch size is fixed and the learning rate is decayed<br /> <br /> “Increasing batch size”, where batch size is increased to 81920 then the learning rate is decayed at two steps.<br /> <br /> [[File:Paper_40_Table_2.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_6.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the batch size resulted in reducing the number of parameter updates from 14,000 to 6,000.<br /> <br /> '''B) Experiment Goal:''' Control Batch Size and Momentum Coefficient<br /> <br /> '''Training Parameters:''' Ghost batch size = 64 / noise decayed at epoch 30, 60, and 80 by a factor of 10. <br /> <br /> The below table shows the number of parameter updates and accuracy for different set of training parameters:<br /> <br /> [[File:Paper_40_Table_3.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_7.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the momentum reduces the number of parameter updates, but leads to a drop in the test accuracy.<br /> <br /> === TRAINING IMAGENET IN 30 MINUTES===<br /> <br /> '''Dataset:''' ImageNet (Already introduced in the previous section)<br /> <br /> '''Network Architecture:''' ResNet-50<br /> <br /> The paper replicated the setup of Goyal et al. (2017) while modifying the number of TPU devices, batch size, learning rate, and then calculating the time to complete 90 epochs, and measuring the accuracy, and performed the following experiments below:<br /> <br /> [[File:Paper_40_Table_4.png | 800px|center]]<br /> <br /> '''Conclusion:''' Model training times can be reduced by increasing the batch size during training.<br /> <br /> == RELATED WORK ==<br /> Main related work mentioned in the paper is as follows:<br /> <br /> - Smith &amp; Le (2017) interpreted Stochastic gradient descent as stochastic differential equation; the paper built on this idea to include decaying learning rate.<br /> <br /> - Mandt et al. (2017) analyzed how to modify SGD for the task of Bayesian posterior sampling.<br /> <br /> - Keskar et al. (2016) focused on the analysis of noise once the training is started.<br /> <br /> - Moreover, the proportional relationship between batch size and learning rate was first discovered by Goyal et al. (2017) and successfully trained ResNet-50 on ImageNet in one hour after discovering the proportionality relationship between batch size and learning rate.<br /> <br /> - Furthermore, You et al. (2017a) presented Layer-wise Adaptive Rate Scaling (LARS), which is applying different learning rates to train ImageNet in 14 minutes and 74.9% accuracy. <br /> <br /> - Wilson et al. (2017) argued that adaptive optimization methods tend to generalize less well than SGD and SGD with momentum (although<br /> they did not include K-FAC in their study), while the authors' work reduces the gap in convergence speed.<br /> <br /> - Finally, another strategy called Asynchronous-SGD that allowed (Recht et al., 2011; Dean et al., 2012) to use multiple GPUs even with small batch sizes.<br /> <br /> == CONCLUSIONS ==<br /> Increasing the batch size during training has the same benefits of decaying the learning rate in addition to reducing the number of parameter updates, which corresponds to faster training time. Experiments were performed on different image datasets and various optimizers with different training schedules to prove this result. The paper proposed to increase the learning rate and momentum parameter &lt;math&gt;m&lt;/math&gt;, while scaling &lt;math&gt; B \propto \frac{\epsilon}{1-m} &lt;/math&gt;, which achieves fewer parameter updates, but slightly less test set accuracy as mentioned in detail in the experiments’ section. In summary, on ImageNet dataset, Inception-ResNet-V2 achieved 77% validation accuracy in under 2500 parameter updates, and ResNet-50 achieved 76.1% validation set accuracy on TPU in less than 30 minutes. One of the great finding of this paper is that all the methods use the hyper-parameters directly from previous works in the literature, and no additional hyper-parameter tuning was performed.<br /> <br /> == CRITIQUE ==<br /> '''Pros:'''<br /> <br /> - The paper showed empirically that increasing batch size and decaying learning rate are equivalent.<br /> <br /> - Several experiments were performed on different optimizers such as SGD and Adam.<br /> <br /> - Had several comparisons with previous experimental setups.<br /> <br /> '''Cons:'''<br /> <br /> <br /> - All datasets used are image datasets. Other experiments should have been done on datasets from different domains to ensure generalization. <br /> <br /> - The number of parameter updates was used as a comparison criterion, but wall-clock times could have provided additional measurable judgment although they depend on the hardware used.<br /> <br /> - Special hardware is needed for large batch training, which is not always feasible. As batch-size increases, we generally need more RAM to train the same model. However, if learning rate is decreased, the RAM use remains constant. As a result, learning rate decay will allow us to train bigger models.<br /> <br /> - In section 5.2 (Increasing the Effective Learning rate), the authors did not test a range of learning rate values and used only (0.1 and 0.5). Additional results from varying the initial learning rate values from 0.1 to 3.2 are provided in the appendix, which indicates that the test accuracy begins to fall for initial learning rates greater than ~0.4. The appended results do not show validation set accuracy curves like in Figure 6, however. It would be beneficial to see if they were similar to the original 0.1 and 0.5 initial learning rate baselines.<br /> <br /> - Although the main idea of the paper is interesting, its results does not seem to be too surprising in comparison with other recent papers in the subject.<br /> <br /> - The paper could benefit from using some other models to demonstrate its claim and generalize its idea by adding some comparisons with other models as well as other recent methods to increase batch size.<br /> <br /> - The paper presents interesting ideas. However, it lacks of mathematical and theoretical analysis beyond the idea. Since the experiment is primary on image dataset and it does not provide sufficient theories, the paper itself presents limited applicability to other types. <br /> <br /> - Also, in experimental setting, only single training runs from one random initialization is used. It would be better to take the best of many runs or to show confidence intervals.<br /> <br /> - It is proposed that we should compare learning rate decay with batch-size increase under the setting that total budget / number of training samples is fixed.<br /> <br /> == REFERENCES ==<br /> # Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates.arXiv preprint arXiv:1612.05086, 2016.<br /> #L´eon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning.arXiv preprint arXiv:1606.04838, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in neural information processing systems, pp. 1223–1231, 2012.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting.SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Priya Goyal, Piotr Doll´ar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Sepp Hochreiter and J¨urgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997.<br /> #Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 1–12. ACM, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.<br /> #Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. arXiv preprint arXiv:1511.06251, 2017.<br /> #Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. arXiv preprint arXiv:1608.03983, 2016.<br /> #Stephan Mandt, Matthew D Hoffman, and DavidMBlei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on Machine Learning, pp. 2408–2417, 2015.<br /> #Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, volume 27, pp. 372–376, 1983.<br /> #Lutz Prechelt. Early stopping-but when? Neural Networks: Tricks of the trade, pp. 553–553, 1998.<br /> #Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in neural information processing systems, pp. 693–701, 2011.<br /> #Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400–407, 1951.<br /> #Samuel L. Smith and Quoc V. Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451, 2017.<br /> #Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In AAAI, pp. 4278–4284, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. arXiv preprint arXiv:1705.08292, 2017.<br /> #Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888, 2017a.<br /> #Yang You, Zhao Zhang, C Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. CoRR, abs/1709.05011, 2017b.<br /> #Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=DON%27T_DECAY_THE_LEARNING_RATE_,_INCREASE_THE_BATCH_SIZE&diff=42068 DON'T DECAY THE LEARNING RATE , INCREASE THE BATCH SIZE 2018-11-30T16:14:27Z <p>C9sharma: /* SIMULATED ANNEALING IN A WIDE RESNET */</p> <hr /> <div>Summary of the ICLR 2018 paper: '''Don't Decay the learning Rate, Increase the Batch Size ''' <br /> <br /> Link: [https://arxiv.org/pdf/1711.00489.pdf]<br /> <br /> Summarized by: Afify, Ahmed [ID: 20700841]<br /> <br /> ==INTUITION==<br /> Nowadays, it is a common practice not to have a singular steady learning rate for the learning phase of neural network models. Instead, we use adaptive learning rates with the standard gradient descent method. The intuition behind this is that when we are far away from the minima, it is beneficial for us to take large steps towards the minima, as it would require a lesser number of steps to converge, but as we approach the minima, our step size should decrease, otherwise we may just keep oscillating around the minima. In practice, this is generally achieved by methods like SGD with momentum, Nesterov momentum, and Adam. However, the core claim of this paper is that the same effect can be achieved by increasing the batch size during the gradient descent process while keeping the learning rate constant throughout. In addition, the paper argues that such an approach also reduces the parameter updates required to reach the minima, thus leading to greater parallelism and shorter training times.<br /> <br /> == INTRODUCTION ==<br /> Stochastic gradient descent (SGD) is the most widely used optimization technique for training deep learning models. The reason for this is that the minima found using this process generalizes well (Zhang et al., 2016; Wilson et al., 2017), but the optimization process is slow and time consuming. According to (Goyal et al., 2017; Hoffer et al., 2017; You et al., 2017a), this has motivated researchers to try to speed up this optimization process by taking bigger steps, and hence reduce the number of parameter updates in training a model. This can be achieved by using large batch training, which can be divided across many machines. <br /> <br /> However, increasing the batch size leads to decreasing the test set accuracy (Keskar et al., 2016; Goyal et al., 2017). Smith and Le (2017) believed that SGD has a scale of random fluctuations &lt;math&gt; g = \epsilon (\frac{N}{B}-1) &lt;/math&gt;, where &lt;math&gt; \epsilon &lt;/math&gt; is the learning rate, N number of training samples, and B batch size. They concluded that there is an optimal batch size proportional to the learning rate when &lt;math&gt; B \ll N &lt;/math&gt;, and optimum fluctuation scale &lt;math&gt;g&lt;/math&gt; for a maximum test set accuracy.<br /> <br /> In this paper, the authors' main goal is to provide evidence that increasing the batch size is quantitatively equivalent to decreasing the learning rate with the same number of training epochs in decreasing the scale of random fluctuations, but with remarkably less number of parameter updates. Moreover, an additional reduction in the number of parameter updates can be attained by increasing the learning rate and scaling &lt;math&gt; B \propto \epsilon &lt;/math&gt; or even more reduction by increasing the momentum coefficient and scaling &lt;math&gt; B \propto \frac{1}{1-m} &lt;/math&gt; although the latter decreases the test accuracy. This has been demonstrated by several experiments on the ImageNet and CIFAR-10 datasets using ResNet-50 and Inception-ResNet-V2 architectures respectively.<br /> <br /> == STOCHASTIC GRADIENT DESCENT AND CONVEX OPTIMIZATION ==<br /> As mentioned in the previous section, the drawback of SGD when compared to full-batch training is the noise that it introduces that hinders optimization. According to (Robbins &amp; Monro, 1951), there are two equations that govern how to reach the minimum of a convex function: (&lt;math&gt; \epsilon_i &lt;/math&gt; denotes the learning rate at the &lt;math&gt; i^{th} &lt;/math&gt; gradient update)<br /> <br /> &lt;math&gt; \sum_{i=1}^{\infty} \epsilon_i = \infty &lt;/math&gt;. This equation guarantees that we will reach the minimum. <br /> <br /> &lt;math&gt; \sum_{i=1}^{\infty} \epsilon^2_i &lt; \infty &lt;/math&gt;. This equation, which is valid only for a fixed batch size, guarantees that learning rate decays fast enough allowing us to reach the minimum rather than bouncing due to noise.<br /> <br /> These equations indicate that the learning rate must decay during training, and second equation is only available when the batch size is constant. To change the batch size, Smith and Le (2017) proposed to interpret SGD as integrating this stochastic differential equation &lt;math&gt; \frac{dw}{dt} = -\frac{dC}{dw} + \eta(t) &lt;/math&gt;, where &lt;math&gt;C&lt;/math&gt; represents cost function, &lt;math&gt;w&lt;/math&gt; represents the parameters, and &lt;math&gt;\eta&lt;/math&gt; represents the Gaussian random noise. Furthermore, they proved that noise scale &lt;math&gt;g&lt;/math&gt; controls the magnitude of random fluctuations in the training dynamics by this formula: &lt;math&gt; g = \epsilon (\frac{N}{B}-1) &lt;/math&gt;, where &lt;math&gt; \epsilon &lt;/math&gt; is the learning rate, N is the training set size and &lt;math&gt;B&lt;/math&gt; is the batch size. As we usually have &lt;math&gt; B \ll N &lt;/math&gt;, we can define &lt;math&gt; g \approx \epsilon \frac{N}{B} &lt;/math&gt;. This explains why when the learning rate decreases, noise &lt;math&gt;g&lt;/math&gt; decreases, enabling us to converge to the minimum of the cost function. However, increasing the batch size has the same effect and makes &lt;math&gt;g&lt;/math&gt; decays with constant learning rate. In this work, the batch size is increased until &lt;math&gt; B \approx \frac{N}{10} &lt;/math&gt;, then the conventional way of decaying the learning rate is followed.<br /> <br /> == SIMULATED ANNEALING AND THE GENERALIZATION GAP ==<br /> '''Simulated Annealing:''' decaying learning rates are empirically successful. To understand this, they note that introducing random fluctuations<br /> whose scale falls during training is also a well established technique in non-convex optimization; simulated annealing. The initial noisy optimization phase allows to explore a larger fraction of the parameter space without becoming trapped in local minima. Once a promising region of parameter space is located, the noise is reduced to fine-tune the parameters.<br /> <br /> For more info: Simulated annealing (SA) is a probabilistic technique for approximating the global optimum of a given function. Specifically, it is a metaheuristic to approximate global optimization in a large search space for an optimization problem. It is often used when the search space is discrete (e.g., all tours that visit a given set of cities). For problems where finding an approximate global optimum is more important than finding a precise local optimum in a fixed amount of time, simulated annealing may be preferable to alternatives such as gradient descent. [https://en.wikipedia.org/wiki/Simulated_annealing [Reference]]<br /> <br /> '''Generalization Gap:''' Small batch data generalizes better to the test set than large batch data.<br /> <br /> Smith and Le (2017) found that there is an optimal batch size which corresponds to optimal noise scale g &lt;math&gt; (g \approx \epsilon \frac{N}{B}) &lt;/math&gt; and concluded that &lt;math&gt; B_{opt} \propto \epsilon N &lt;/math&gt; that corresponds to maximum test set accuracy. This means that gradient noise is helpful as it makes SGD escape sharp minima, which does not generalize well. <br /> <br /> Simulated Annealing is a famous technique in non-convex optimization. Starting with noise in the training process helps us to discover a wide range of parameters then once we are near the optimum value, noise is reduced to fine tune our final parameters. However, more and more researches like to use the sharper decay schedules like cosine decay or step-function drops. In physical sciences, slowly annealing (or decaying) the temperature (which is the noise scale in this situation) helps to converge to the global minimum, which is sharp. But decaying the temperature in discrete steps can make the system stuck in a local minimum, which lead to higher cost and lower curvature. The authors think that deep learning has the same intuition.<br /> .<br /> <br /> == THE EFFECTIVE LEARNING RATE AND THE ACCUMULATION VARIABLE ==<br /> '''The Effective Learning Rate''' : &lt;math&gt; \epsilon_{eff} = \frac{\epsilon}{1-m} &lt;/math&gt;<br /> <br /> Smith and Le (2017) included momentum to the equation of the vanilla SGD noise scale that was defined above to be: &lt;math&gt; g = \frac{\epsilon}{1-m}(\frac{N}{B}-1)\approx \frac{\epsilon N}{B(1-m)} &lt;/math&gt;, which is the same as the previous equation when m goes to 0. They found that increasing the learning rate and momentum coefficient and scaling &lt;math&gt; B \propto \frac{\epsilon }{1-m} &lt;/math&gt; reduces the number of parameter updates, but the test accuracy decreases when the momentum coefficient is increased. <br /> <br /> To understand the reasons behind this, we need to analyze momentum update equations below:<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \Delta A = -(1-m)A + \frac{d\widehat{C}}{dw} <br /> &lt;/math&gt;<br /> <br /> &lt;math&gt;<br /> \Delta w = -A\epsilon<br /> &lt;/math&gt;<br /> &lt;/center&gt;<br /> <br /> We can see that the Accumulation variable A, which is initially set to 0, then increases exponentially to reach its steady state value during &lt;math&gt; \frac{B}{N(1-m)} &lt;/math&gt; training epochs while &lt;math&gt; \Delta w &lt;/math&gt; is suppressed that can reduce the rate of convergence. Moreover, at high momentum, we have three challenges:<br /> <br /> 1- Additional epochs are needed to catch up with the accumulation.<br /> <br /> 2- Accumulation needs more time &lt;math&gt; \frac{B}{N(1-m)} &lt;/math&gt; to forget old gradients. <br /> <br /> 3- After this time, however, the accumulation cannot adapt to changes in the loss landscape.<br /> <br /> 4- In the early stage, large batch size will lead to the instabilities.<br /> <br /> == EXPERIMENTS ==<br /> === SIMULATED ANNEALING IN A WIDE RESNET ===<br /> <br /> '''Dataset:''' CIFAR-10 (50,000 training images)<br /> <br /> '''Network Architecture:''' “16-4” wide ResNet<br /> <br /> '''Training Schedules used as in the below figure:''' . These demonstrate the equivalence between decreasing the learning rate and increasing the batch size.<br /> <br /> - Decaying learning rate: learning rate decays by a factor of 5 at a sequence of “steps”, and the batch size is constant<br /> <br /> - Increasing batch size: learning rate is constant, and the batch size is increased by a factor of 5 at every step.<br /> <br /> - Hybrid: At the beginning, the learning rate is constant and batch size is increased by a factor of 5. Then, the learning rate decays by a factor of 5 at each subsequent step, and the batch size is constant. This is the schedule that will be used if there is a hardware limit affecting a maximum batch size limit.<br /> If the learning rate itself must decay during training, then these schedules should show different learning curves (as a function of the number of training epochs) and reach different final test set accuracies. Meanwhile if it is the noise scale which should decay, all three schedules should be indistinguishable.<br /> [[File:Paper_40_Fig_1.png | 800px|center]]<br /> <br /> As shown in the below figure: in the left figure (2a), we can observe that for the training set, the three learning curves are exactly the same while in figure 2b, increasing the batch size has a huge advantage of reducing the number of parameter updates.<br /> This concludes that noise scale is the one that needs to be decayed and not the learning rate itself<br /> [[File:Paper_40_Fig_2.png | 800px|center]] <br /> <br /> To make sure that these results are the same for the test set as well, in figure 3, we can see that the three learning curves are exactly the same for SGD with momentum, and Nesterov momentum<br /> [[File:Paper_40_Fig_3.png | 800px|center]]<br /> <br /> To check for other optimizers as well. the below figure shows the same experiment as in figure 3, which is the three learning curves for test set, but for vanilla SGD and Adam, and showing <br /> [[File:Paper_40_Fig_4.png | 800px|center]]<br /> <br /> '''Conclusion:''' Decreasing the learning rate and increasing the batch size during training are equivalent<br /> <br /> === INCREASING THE EFFECTIVE LEARNING RATE===<br /> <br /> Here, the focus is on minimizing the number of parameter updates required to train a model. As shown above, the first step is to replace decaying learning rates by increasing batch sizes. Now, the authors show here that we can also increase the effective learning rate &lt;math&gt;\epsilon_{eff} = \epsilon/(1 − m) &lt;/math&gt; at the start of training, while scaling the initial batch size &lt;math&gt;B \propto \epsilon_{eff} &lt;/math&gt; . All experiments are conducted using SGD with momentum. There are 50000 images in the CIFAR-10 training set, and since the scaling rules only hold when &lt;math&gt;B &lt;&lt; N &lt;/math&gt; , we decided to set a maximum batch size &lt;math&gt;B_{max} &lt;/math&gt;= 5120 .<br /> <br /> '''Dataset:''' CIFAR-10 (50,000 training images)<br /> <br /> '''Network Architecture:''' “16-4” wide ResNet<br /> <br /> '''Training Parameters:''' Optimization Algorithm: SGD with momentum / Maximum batch size = 5120<br /> <br /> '''Training Schedules:''' <br /> <br /> The authors consider four training schedules, all of which decay the noise scale by a factor of five in a series of three steps with the same number of epochs.<br /> <br /> Original training schedule: initial learning rate of 0.1 which decays by a factor of 5 at each step, a momentum coefficient of 0.9, and a batch size of 128. Follows the implementation of Zagoruyko &amp; Komodakis (2016).<br /> <br /> Increasing batch size: learning rate of 0.1, momentum coefficient of 0.9, initial batch size of 128 that increases by a factor of 5 at each step. <br /> <br /> Increased initial learning rate: initial learning rate of 0.5, initial batch size of 640 that increase during training.<br /> <br /> Increased momentum coefficient: increased initial learning rate of 0.5, initial batch size of 3200 that increase during training, and an increased momentum coefficient of 0.98.<br /> <br /> The results of all training schedules, which are presented in the below figure, are documented in the following table:<br /> <br /> [[File:Paper_40_Table_1.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_5.png | 800px|center]]<br /> <br /> <br /> <br /> '''Conclusion:''' Increasing the effective learning rate and scaling the batch size results in further reduction in the number of parameter updates<br /> <br /> === TRAINING IMAGENET IN 2500 PARAMETER UPDATES===<br /> <br /> '''A) Experiment Goal:''' Control Batch Size<br /> <br /> '''Dataset:''' ImageNet (1.28 million training images)<br /> <br /> The paper modified the setup of Goyal et al. (2017), and used the following configuration:<br /> <br /> '''Network Architecture:''' Inception-ResNet-V2 <br /> <br /> '''Training Parameters:''' <br /> <br /> 90 epochs / noise decayed at epoch 30, 60, and 80 by a factor of 10 / Initial ghost batch size = 32 / Learning rate = 3 / momentum coefficient = 0.9 / Initial batch size = 8192<br /> <br /> Two training schedules were used:<br /> <br /> “Decaying learning rate”, where batch size is fixed and the learning rate is decayed<br /> <br /> “Increasing batch size”, where batch size is increased to 81920 then the learning rate is decayed at two steps.<br /> <br /> [[File:Paper_40_Table_2.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_6.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the batch size resulted in reducing the number of parameter updates from 14,000 to 6,000.<br /> <br /> '''B) Experiment Goal:''' Control Batch Size and Momentum Coefficient<br /> <br /> '''Training Parameters:''' Ghost batch size = 64 / noise decayed at epoch 30, 60, and 80 by a factor of 10. <br /> <br /> The below table shows the number of parameter updates and accuracy for different set of training parameters:<br /> <br /> [[File:Paper_40_Table_3.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_7.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the momentum reduces the number of parameter updates, but leads to a drop in the test accuracy.<br /> <br /> === TRAINING IMAGENET IN 30 MINUTES===<br /> <br /> '''Dataset:''' ImageNet (Already introduced in the previous section)<br /> <br /> '''Network Architecture:''' ResNet-50<br /> <br /> The paper replicated the setup of Goyal et al. (2017) while modifying the number of TPU devices, batch size, learning rate, and then calculating the time to complete 90 epochs, and measuring the accuracy, and performed the following experiments below:<br /> <br /> [[File:Paper_40_Table_4.png | 800px|center]]<br /> <br /> '''Conclusion:''' Model training times can be reduced by increasing the batch size during training.<br /> <br /> == RELATED WORK ==<br /> Main related work mentioned in the paper is as follows:<br /> <br /> - Smith &amp; Le (2017) interpreted Stochastic gradient descent as stochastic differential equation; the paper built on this idea to include decaying learning rate.<br /> <br /> - Mandt et al. (2017) analyzed how to modify SGD for the task of Bayesian posterior sampling.<br /> <br /> - Keskar et al. (2016) focused on the analysis of noise once the training is started.<br /> <br /> - Moreover, the proportional relationship between batch size and learning rate was first discovered by Goyal et al. (2017) and successfully trained ResNet-50 on ImageNet in one hour after discovering the proportionality relationship between batch size and learning rate.<br /> <br /> - Furthermore, You et al. (2017a) presented Layer-wise Adaptive Rate Scaling (LARS), which is applying different learning rates to train ImageNet in 14 minutes and 74.9% accuracy. <br /> <br /> - Wilson et al. (2017) argued that adaptive optimization methods tend to generalize less well than SGD and SGD with momentum (although<br /> they did not include K-FAC in their study), while the authors' work reduces the gap in convergence speed.<br /> <br /> - Finally, another strategy called Asynchronous-SGD that allowed (Recht et al., 2011; Dean et al., 2012) to use multiple GPUs even with small batch sizes.<br /> <br /> == CONCLUSIONS ==<br /> Increasing the batch size during training has the same benefits of decaying the learning rate in addition to reducing the number of parameter updates, which corresponds to faster training time. Experiments were performed on different image datasets and various optimizers with different training schedules to prove this result. The paper proposed to increase the learning rate and momentum parameter &lt;math&gt;m&lt;/math&gt;, while scaling &lt;math&gt; B \propto \frac{\epsilon}{1-m} &lt;/math&gt;, which achieves fewer parameter updates, but slightly less test set accuracy as mentioned in detail in the experiments’ section. In summary, on ImageNet dataset, Inception-ResNet-V2 achieved 77% validation accuracy in under 2500 parameter updates, and ResNet-50 achieved 76.1% validation set accuracy on TPU in less than 30 minutes. One of the great finding of this paper is that all the methods use the hyper-parameters directly from previous works in the literature, and no additional hyper-parameter tuning was performed.<br /> <br /> == CRITIQUE ==<br /> '''Pros:'''<br /> <br /> - The paper showed empirically that increasing batch size and decaying learning rate are equivalent.<br /> <br /> - Several experiments were performed on different optimizers such as SGD and Adam.<br /> <br /> - Had several comparisons with previous experimental setups.<br /> <br /> '''Cons:'''<br /> <br /> <br /> - All datasets used are image datasets. Other experiments should have been done on datasets from different domains to ensure generalization. <br /> <br /> - The number of parameter updates was used as a comparison criterion, but wall-clock times could have provided additional measurable judgment although they depend on the hardware used.<br /> <br /> - Special hardware is needed for large batch training, which is not always feasible. As batch-size increases, we generally need more RAM to train the same model. However, if learning rate is decreased, the RAM use remains constant. As a result, learning rate decay will allow us to train bigger models.<br /> <br /> - In section 5.2 (Increasing the Effective Learning rate), the authors did not test a range of learning rate values and used only (0.1 and 0.5). Additional results from varying the initial learning rate values from 0.1 to 3.2 are provided in the appendix, which indicates that the test accuracy begins to fall for initial learning rates greater than ~0.4. The appended results do not show validation set accuracy curves like in Figure 6, however. It would be beneficial to see if they were similar to the original 0.1 and 0.5 initial learning rate baselines.<br /> <br /> - Although the main idea of the paper is interesting, its results does not seem to be too surprising in comparison with other recent papers in the subject.<br /> <br /> - The paper could benefit from using some other models to demonstrate its claim and generalize its idea by adding some comparisons with other models as well as other recent methods to increase batch size.<br /> <br /> - The paper presents interesting ideas. However, it lacks of mathematical and theoretical analysis beyond the idea. Since the experiment is primary on image dataset and it does not provide sufficient theories, the paper itself presents limited applicability to other types. <br /> <br /> - Also, in experimental setting, only single training runs from one random initialization is used. It would be better to take the best of many runs or to show confidence intervals.<br /> <br /> - It is proposed that we should compare learning rate decay with batch-size increase under the setting that total budget / number of training samples is fixed.<br /> <br /> == REFERENCES ==<br /> # Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates.arXiv preprint arXiv:1612.05086, 2016.<br /> #L´eon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning.arXiv preprint arXiv:1606.04838, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in neural information processing systems, pp. 1223–1231, 2012.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting.SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Priya Goyal, Piotr Doll´ar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Sepp Hochreiter and J¨urgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997.<br /> #Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 1–12. ACM, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.<br /> #Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. arXiv preprint arXiv:1511.06251, 2017.<br /> #Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. arXiv preprint arXiv:1608.03983, 2016.<br /> #Stephan Mandt, Matthew D Hoffman, and DavidMBlei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on Machine Learning, pp. 2408–2417, 2015.<br /> #Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, volume 27, pp. 372–376, 1983.<br /> #Lutz Prechelt. Early stopping-but when? Neural Networks: Tricks of the trade, pp. 553–553, 1998.<br /> #Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in neural information processing systems, pp. 693–701, 2011.<br /> #Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400–407, 1951.<br /> #Samuel L. Smith and Quoc V. Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451, 2017.<br /> #Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In AAAI, pp. 4278–4284, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. arXiv preprint arXiv:1705.08292, 2017.<br /> #Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888, 2017a.<br /> #Yang You, Zhao Zhang, C Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. CoRR, abs/1709.05011, 2017b.<br /> #Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=DON%27T_DECAY_THE_LEARNING_RATE_,_INCREASE_THE_BATCH_SIZE&diff=42066 DON'T DECAY THE LEARNING RATE , INCREASE THE BATCH SIZE 2018-11-30T16:12:31Z <p>C9sharma: /* INCREASING THE EFFECTIVE LEARNING RATE */</p> <hr /> <div>Summary of the ICLR 2018 paper: '''Don't Decay the learning Rate, Increase the Batch Size ''' <br /> <br /> Link: [https://arxiv.org/pdf/1711.00489.pdf]<br /> <br /> Summarized by: Afify, Ahmed [ID: 20700841]<br /> <br /> ==INTUITION==<br /> Nowadays, it is a common practice not to have a singular steady learning rate for the learning phase of neural network models. Instead, we use adaptive learning rates with the standard gradient descent method. The intuition behind this is that when we are far away from the minima, it is beneficial for us to take large steps towards the minima, as it would require a lesser number of steps to converge, but as we approach the minima, our step size should decrease, otherwise we may just keep oscillating around the minima. In practice, this is generally achieved by methods like SGD with momentum, Nesterov momentum, and Adam. However, the core claim of this paper is that the same effect can be achieved by increasing the batch size during the gradient descent process while keeping the learning rate constant throughout. In addition, the paper argues that such an approach also reduces the parameter updates required to reach the minima, thus leading to greater parallelism and shorter training times.<br /> <br /> == INTRODUCTION ==<br /> Stochastic gradient descent (SGD) is the most widely used optimization technique for training deep learning models. The reason for this is that the minima found using this process generalizes well (Zhang et al., 2016; Wilson et al., 2017), but the optimization process is slow and time consuming. According to (Goyal et al., 2017; Hoffer et al., 2017; You et al., 2017a), this has motivated researchers to try to speed up this optimization process by taking bigger steps, and hence reduce the number of parameter updates in training a model. This can be achieved by using large batch training, which can be divided across many machines. <br /> <br /> However, increasing the batch size leads to decreasing the test set accuracy (Keskar et al., 2016; Goyal et al., 2017). Smith and Le (2017) believed that SGD has a scale of random fluctuations &lt;math&gt; g = \epsilon (\frac{N}{B}-1) &lt;/math&gt;, where &lt;math&gt; \epsilon &lt;/math&gt; is the learning rate, N number of training samples, and B batch size. They concluded that there is an optimal batch size proportional to the learning rate when &lt;math&gt; B \ll N &lt;/math&gt;, and optimum fluctuation scale &lt;math&gt;g&lt;/math&gt; for a maximum test set accuracy.<br /> <br /> In this paper, the authors' main goal is to provide evidence that increasing the batch size is quantitatively equivalent to decreasing the learning rate with the same number of training epochs in decreasing the scale of random fluctuations, but with remarkably less number of parameter updates. Moreover, an additional reduction in the number of parameter updates can be attained by increasing the learning rate and scaling &lt;math&gt; B \propto \epsilon &lt;/math&gt; or even more reduction by increasing the momentum coefficient and scaling &lt;math&gt; B \propto \frac{1}{1-m} &lt;/math&gt; although the latter decreases the test accuracy. This has been demonstrated by several experiments on the ImageNet and CIFAR-10 datasets using ResNet-50 and Inception-ResNet-V2 architectures respectively.<br /> <br /> == STOCHASTIC GRADIENT DESCENT AND CONVEX OPTIMIZATION ==<br /> As mentioned in the previous section, the drawback of SGD when compared to full-batch training is the noise that it introduces that hinders optimization. According to (Robbins &amp; Monro, 1951), there are two equations that govern how to reach the minimum of a convex function: (&lt;math&gt; \epsilon_i &lt;/math&gt; denotes the learning rate at the &lt;math&gt; i^{th} &lt;/math&gt; gradient update)<br /> <br /> &lt;math&gt; \sum_{i=1}^{\infty} \epsilon_i = \infty &lt;/math&gt;. This equation guarantees that we will reach the minimum. <br /> <br /> &lt;math&gt; \sum_{i=1}^{\infty} \epsilon^2_i &lt; \infty &lt;/math&gt;. This equation, which is valid only for a fixed batch size, guarantees that learning rate decays fast enough allowing us to reach the minimum rather than bouncing due to noise.<br /> <br /> These equations indicate that the learning rate must decay during training, and second equation is only available when the batch size is constant. To change the batch size, Smith and Le (2017) proposed to interpret SGD as integrating this stochastic differential equation &lt;math&gt; \frac{dw}{dt} = -\frac{dC}{dw} + \eta(t) &lt;/math&gt;, where &lt;math&gt;C&lt;/math&gt; represents cost function, &lt;math&gt;w&lt;/math&gt; represents the parameters, and &lt;math&gt;\eta&lt;/math&gt; represents the Gaussian random noise. Furthermore, they proved that noise scale &lt;math&gt;g&lt;/math&gt; controls the magnitude of random fluctuations in the training dynamics by this formula: &lt;math&gt; g = \epsilon (\frac{N}{B}-1) &lt;/math&gt;, where &lt;math&gt; \epsilon &lt;/math&gt; is the learning rate, N is the training set size and &lt;math&gt;B&lt;/math&gt; is the batch size. As we usually have &lt;math&gt; B \ll N &lt;/math&gt;, we can define &lt;math&gt; g \approx \epsilon \frac{N}{B} &lt;/math&gt;. This explains why when the learning rate decreases, noise &lt;math&gt;g&lt;/math&gt; decreases, enabling us to converge to the minimum of the cost function. However, increasing the batch size has the same effect and makes &lt;math&gt;g&lt;/math&gt; decays with constant learning rate. In this work, the batch size is increased until &lt;math&gt; B \approx \frac{N}{10} &lt;/math&gt;, then the conventional way of decaying the learning rate is followed.<br /> <br /> == SIMULATED ANNEALING AND THE GENERALIZATION GAP ==<br /> '''Simulated Annealing:''' decaying learning rates are empirically successful. To understand this, they note that introducing random fluctuations<br /> whose scale falls during training is also a well established technique in non-convex optimization; simulated annealing. The initial noisy optimization phase allows to explore a larger fraction of the parameter space without becoming trapped in local minima. Once a promising region of parameter space is located, the noise is reduced to fine-tune the parameters.<br /> <br /> For more info: Simulated annealing (SA) is a probabilistic technique for approximating the global optimum of a given function. Specifically, it is a metaheuristic to approximate global optimization in a large search space for an optimization problem. It is often used when the search space is discrete (e.g., all tours that visit a given set of cities). For problems where finding an approximate global optimum is more important than finding a precise local optimum in a fixed amount of time, simulated annealing may be preferable to alternatives such as gradient descent. [https://en.wikipedia.org/wiki/Simulated_annealing [Reference]]<br /> <br /> '''Generalization Gap:''' Small batch data generalizes better to the test set than large batch data.<br /> <br /> Smith and Le (2017) found that there is an optimal batch size which corresponds to optimal noise scale g &lt;math&gt; (g \approx \epsilon \frac{N}{B}) &lt;/math&gt; and concluded that &lt;math&gt; B_{opt} \propto \epsilon N &lt;/math&gt; that corresponds to maximum test set accuracy. This means that gradient noise is helpful as it makes SGD escape sharp minima, which does not generalize well. <br /> <br /> Simulated Annealing is a famous technique in non-convex optimization. Starting with noise in the training process helps us to discover a wide range of parameters then once we are near the optimum value, noise is reduced to fine tune our final parameters. However, more and more researches like to use the sharper decay schedules like cosine decay or step-function drops. In physical sciences, slowly annealing (or decaying) the temperature (which is the noise scale in this situation) helps to converge to the global minimum, which is sharp. But decaying the temperature in discrete steps can make the system stuck in a local minimum, which lead to higher cost and lower curvature. The authors think that deep learning has the same intuition.<br /> .<br /> <br /> == THE EFFECTIVE LEARNING RATE AND THE ACCUMULATION VARIABLE ==<br /> '''The Effective Learning Rate''' : &lt;math&gt; \epsilon_{eff} = \frac{\epsilon}{1-m} &lt;/math&gt;<br /> <br /> Smith and Le (2017) included momentum to the equation of the vanilla SGD noise scale that was defined above to be: &lt;math&gt; g = \frac{\epsilon}{1-m}(\frac{N}{B}-1)\approx \frac{\epsilon N}{B(1-m)} &lt;/math&gt;, which is the same as the previous equation when m goes to 0. They found that increasing the learning rate and momentum coefficient and scaling &lt;math&gt; B \propto \frac{\epsilon }{1-m} &lt;/math&gt; reduces the number of parameter updates, but the test accuracy decreases when the momentum coefficient is increased. <br /> <br /> To understand the reasons behind this, we need to analyze momentum update equations below:<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \Delta A = -(1-m)A + \frac{d\widehat{C}}{dw} <br /> &lt;/math&gt;<br /> <br /> &lt;math&gt;<br /> \Delta w = -A\epsilon<br /> &lt;/math&gt;<br /> &lt;/center&gt;<br /> <br /> We can see that the Accumulation variable A, which is initially set to 0, then increases exponentially to reach its steady state value during &lt;math&gt; \frac{B}{N(1-m)} &lt;/math&gt; training epochs while &lt;math&gt; \Delta w &lt;/math&gt; is suppressed that can reduce the rate of convergence. Moreover, at high momentum, we have three challenges:<br /> <br /> 1- Additional epochs are needed to catch up with the accumulation.<br /> <br /> 2- Accumulation needs more time &lt;math&gt; \frac{B}{N(1-m)} &lt;/math&gt; to forget old gradients. <br /> <br /> 3- After this time, however, the accumulation cannot adapt to changes in the loss landscape.<br /> <br /> 4- In the early stage, large batch size will lead to the instabilities.<br /> <br /> == EXPERIMENTS ==<br /> === SIMULATED ANNEALING IN A WIDE RESNET ===<br /> <br /> '''Dataset:''' CIFAR-10 (50,000 training images)<br /> <br /> '''Network Architecture:''' “16-4” wide ResNet<br /> <br /> '''Training Schedules used as in the below figure:''' <br /> <br /> - Decaying learning rate: learning rate decays by a factor of 5 at a sequence of “steps”, and the batch size is constant<br /> <br /> - Increasing batch size: learning rate is constant, and the batch size is increased by a factor of 5 at every step.<br /> <br /> - Hybrid: At the beginning, the learning rate is constant and batch size is increased by a factor of 5. Then, the learning rate decays by a factor of 5 at each subsequent step, and the batch size is constant. This is the schedule that will be used if there is a hardware limit affecting a maximum batch size limit.<br /> <br /> [[File:Paper_40_Fig_1.png | 800px|center]]<br /> <br /> As shown in the below figure: in the left figure (2a), we can observe that for the training set, the three learning curves are exactly the same while in figure 2b, increasing the batch size has a huge advantage of reducing the number of parameter updates.<br /> This concludes that noise scale is the one that needs to be decayed and not the learning rate itself<br /> [[File:Paper_40_Fig_2.png | 800px|center]] <br /> <br /> To make sure that these results are the same for the test set as well, in figure 3, we can see that the three learning curves are exactly the same for SGD with momentum, and Nesterov momentum<br /> [[File:Paper_40_Fig_3.png | 800px|center]]<br /> <br /> To check for other optimizers as well. the below figure shows the same experiment as in figure 3, which is the three learning curves for test set, but for vanilla SGD and Adam, and showing <br /> [[File:Paper_40_Fig_4.png | 800px|center]]<br /> <br /> '''Conclusion:''' Decreasing the learning rate and increasing the batch size during training are equivalent<br /> <br /> === INCREASING THE EFFECTIVE LEARNING RATE===<br /> <br /> Here, the focus is on minimizing the number of parameter updates required to train a model. As shown above, the first step is to replace decaying learning rates by increasing batch sizes. Now, the authors show here that we can also increase the effective learning rate &lt;math&gt;\epsilon_{eff} = \epsilon/(1 − m) &lt;/math&gt; at the start of training, while scaling the initial batch size &lt;math&gt;B \propto \epsilon_{eff} &lt;/math&gt; . All experiments are conducted using SGD with momentum. There are 50000 images in the CIFAR-10 training set, and since the scaling rules only hold when &lt;math&gt;B &lt;&lt; N &lt;/math&gt; , we decided to set a maximum batch size &lt;math&gt;B_{max} &lt;/math&gt;= 5120 .<br /> <br /> '''Dataset:''' CIFAR-10 (50,000 training images)<br /> <br /> '''Network Architecture:''' “16-4” wide ResNet<br /> <br /> '''Training Parameters:''' Optimization Algorithm: SGD with momentum / Maximum batch size = 5120<br /> <br /> '''Training Schedules:''' <br /> <br /> The authors consider four training schedules, all of which decay the noise scale by a factor of five in a series of three steps with the same number of epochs.<br /> <br /> Original training schedule: initial learning rate of 0.1 which decays by a factor of 5 at each step, a momentum coefficient of 0.9, and a batch size of 128. Follows the implementation of Zagoruyko &amp; Komodakis (2016).<br /> <br /> Increasing batch size: learning rate of 0.1, momentum coefficient of 0.9, initial batch size of 128 that increases by a factor of 5 at each step. <br /> <br /> Increased initial learning rate: initial learning rate of 0.5, initial batch size of 640 that increase during training.<br /> <br /> Increased momentum coefficient: increased initial learning rate of 0.5, initial batch size of 3200 that increase during training, and an increased momentum coefficient of 0.98.<br /> <br /> The results of all training schedules, which are presented in the below figure, are documented in the following table:<br /> <br /> [[File:Paper_40_Table_1.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_5.png | 800px|center]]<br /> <br /> <br /> <br /> '''Conclusion:''' Increasing the effective learning rate and scaling the batch size results in further reduction in the number of parameter updates<br /> <br /> === TRAINING IMAGENET IN 2500 PARAMETER UPDATES===<br /> <br /> '''A) Experiment Goal:''' Control Batch Size<br /> <br /> '''Dataset:''' ImageNet (1.28 million training images)<br /> <br /> The paper modified the setup of Goyal et al. (2017), and used the following configuration:<br /> <br /> '''Network Architecture:''' Inception-ResNet-V2 <br /> <br /> '''Training Parameters:''' <br /> <br /> 90 epochs / noise decayed at epoch 30, 60, and 80 by a factor of 10 / Initial ghost batch size = 32 / Learning rate = 3 / momentum coefficient = 0.9 / Initial batch size = 8192<br /> <br /> Two training schedules were used:<br /> <br /> “Decaying learning rate”, where batch size is fixed and the learning rate is decayed<br /> <br /> “Increasing batch size”, where batch size is increased to 81920 then the learning rate is decayed at two steps.<br /> <br /> [[File:Paper_40_Table_2.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_6.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the batch size resulted in reducing the number of parameter updates from 14,000 to 6,000.<br /> <br /> '''B) Experiment Goal:''' Control Batch Size and Momentum Coefficient<br /> <br /> '''Training Parameters:''' Ghost batch size = 64 / noise decayed at epoch 30, 60, and 80 by a factor of 10. <br /> <br /> The below table shows the number of parameter updates and accuracy for different set of training parameters:<br /> <br /> [[File:Paper_40_Table_3.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_7.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the momentum reduces the number of parameter updates, but leads to a drop in the test accuracy.<br /> <br /> === TRAINING IMAGENET IN 30 MINUTES===<br /> <br /> '''Dataset:''' ImageNet (Already introduced in the previous section)<br /> <br /> '''Network Architecture:''' ResNet-50<br /> <br /> The paper replicated the setup of Goyal et al. (2017) while modifying the number of TPU devices, batch size, learning rate, and then calculating the time to complete 90 epochs, and measuring the accuracy, and performed the following experiments below:<br /> <br /> [[File:Paper_40_Table_4.png | 800px|center]]<br /> <br /> '''Conclusion:''' Model training times can be reduced by increasing the batch size during training.<br /> <br /> == RELATED WORK ==<br /> Main related work mentioned in the paper is as follows:<br /> <br /> - Smith &amp; Le (2017) interpreted Stochastic gradient descent as stochastic differential equation; the paper built on this idea to include decaying learning rate.<br /> <br /> - Mandt et al. (2017) analyzed how to modify SGD for the task of Bayesian posterior sampling.<br /> <br /> - Keskar et al. (2016) focused on the analysis of noise once the training is started.<br /> <br /> - Moreover, the proportional relationship between batch size and learning rate was first discovered by Goyal et al. (2017) and successfully trained ResNet-50 on ImageNet in one hour after discovering the proportionality relationship between batch size and learning rate.<br /> <br /> - Furthermore, You et al. (2017a) presented Layer-wise Adaptive Rate Scaling (LARS), which is applying different learning rates to train ImageNet in 14 minutes and 74.9% accuracy. <br /> <br /> - Wilson et al. (2017) argued that adaptive optimization methods tend to generalize less well than SGD and SGD with momentum (although<br /> they did not include K-FAC in their study), while the authors' work reduces the gap in convergence speed.<br /> <br /> - Finally, another strategy called Asynchronous-SGD that allowed (Recht et al., 2011; Dean et al., 2012) to use multiple GPUs even with small batch sizes.<br /> <br /> == CONCLUSIONS ==<br /> Increasing the batch size during training has the same benefits of decaying the learning rate in addition to reducing the number of parameter updates, which corresponds to faster training time. Experiments were performed on different image datasets and various optimizers with different training schedules to prove this result. The paper proposed to increase the learning rate and momentum parameter &lt;math&gt;m&lt;/math&gt;, while scaling &lt;math&gt; B \propto \frac{\epsilon}{1-m} &lt;/math&gt;, which achieves fewer parameter updates, but slightly less test set accuracy as mentioned in detail in the experiments’ section. In summary, on ImageNet dataset, Inception-ResNet-V2 achieved 77% validation accuracy in under 2500 parameter updates, and ResNet-50 achieved 76.1% validation set accuracy on TPU in less than 30 minutes. One of the great finding of this paper is that all the methods use the hyper-parameters directly from previous works in the literature, and no additional hyper-parameter tuning was performed.<br /> <br /> == CRITIQUE ==<br /> '''Pros:'''<br /> <br /> - The paper showed empirically that increasing batch size and decaying learning rate are equivalent.<br /> <br /> - Several experiments were performed on different optimizers such as SGD and Adam.<br /> <br /> - Had several comparisons with previous experimental setups.<br /> <br /> '''Cons:'''<br /> <br /> <br /> - All datasets used are image datasets. Other experiments should have been done on datasets from different domains to ensure generalization. <br /> <br /> - The number of parameter updates was used as a comparison criterion, but wall-clock times could have provided additional measurable judgment although they depend on the hardware used.<br /> <br /> - Special hardware is needed for large batch training, which is not always feasible. As batch-size increases, we generally need more RAM to train the same model. However, if learning rate is decreased, the RAM use remains constant. As a result, learning rate decay will allow us to train bigger models.<br /> <br /> - In section 5.2 (Increasing the Effective Learning rate), the authors did not test a range of learning rate values and used only (0.1 and 0.5). Additional results from varying the initial learning rate values from 0.1 to 3.2 are provided in the appendix, which indicates that the test accuracy begins to fall for initial learning rates greater than ~0.4. The appended results do not show validation set accuracy curves like in Figure 6, however. It would be beneficial to see if they were similar to the original 0.1 and 0.5 initial learning rate baselines.<br /> <br /> - Although the main idea of the paper is interesting, its results does not seem to be too surprising in comparison with other recent papers in the subject.<br /> <br /> - The paper could benefit from using some other models to demonstrate its claim and generalize its idea by adding some comparisons with other models as well as other recent methods to increase batch size.<br /> <br /> - The paper presents interesting ideas. However, it lacks of mathematical and theoretical analysis beyond the idea. Since the experiment is primary on image dataset and it does not provide sufficient theories, the paper itself presents limited applicability to other types. <br /> <br /> - Also, in experimental setting, only single training runs from one random initialization is used. It would be better to take the best of many runs or to show confidence intervals.<br /> <br /> - It is proposed that we should compare learning rate decay with batch-size increase under the setting that total budget / number of training samples is fixed.<br /> <br /> == REFERENCES ==<br /> # Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates.arXiv preprint arXiv:1612.05086, 2016.<br /> #L´eon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning.arXiv preprint arXiv:1606.04838, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in neural information processing systems, pp. 1223–1231, 2012.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting.SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Priya Goyal, Piotr Doll´ar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Sepp Hochreiter and J¨urgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997.<br /> #Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 1–12. ACM, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.<br /> #Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. arXiv preprint arXiv:1511.06251, 2017.<br /> #Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. arXiv preprint arXiv:1608.03983, 2016.<br /> #Stephan Mandt, Matthew D Hoffman, and DavidMBlei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on Machine Learning, pp. 2408–2417, 2015.<br /> #Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, volume 27, pp. 372–376, 1983.<br /> #Lutz Prechelt. Early stopping-but when? Neural Networks: Tricks of the trade, pp. 553–553, 1998.<br /> #Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in neural information processing systems, pp. 693–701, 2011.<br /> #Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400–407, 1951.<br /> #Samuel L. Smith and Quoc V. Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451, 2017.<br /> #Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In AAAI, pp. 4278–4284, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. arXiv preprint arXiv:1705.08292, 2017.<br /> #Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888, 2017a.<br /> #Yang You, Zhao Zhang, C Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. CoRR, abs/1709.05011, 2017b.<br /> #Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=DON%27T_DECAY_THE_LEARNING_RATE_,_INCREASE_THE_BATCH_SIZE&diff=42063 DON'T DECAY THE LEARNING RATE , INCREASE THE BATCH SIZE 2018-11-30T16:12:05Z <p>C9sharma: /* INCREASING THE EFFECTIVE LEARNING RATE */</p> <hr /> <div>Summary of the ICLR 2018 paper: '''Don't Decay the learning Rate, Increase the Batch Size ''' <br /> <br /> Link: [https://arxiv.org/pdf/1711.00489.pdf]<br /> <br /> Summarized by: Afify, Ahmed [ID: 20700841]<br /> <br /> ==INTUITION==<br /> Nowadays, it is a common practice not to have a singular steady learning rate for the learning phase of neural network models. Instead, we use adaptive learning rates with the standard gradient descent method. The intuition behind this is that when we are far away from the minima, it is beneficial for us to take large steps towards the minima, as it would require a lesser number of steps to converge, but as we approach the minima, our step size should decrease, otherwise we may just keep oscillating around the minima. In practice, this is generally achieved by methods like SGD with momentum, Nesterov momentum, and Adam. However, the core claim of this paper is that the same effect can be achieved by increasing the batch size during the gradient descent process while keeping the learning rate constant throughout. In addition, the paper argues that such an approach also reduces the parameter updates required to reach the minima, thus leading to greater parallelism and shorter training times.<br /> <br /> == INTRODUCTION ==<br /> Stochastic gradient descent (SGD) is the most widely used optimization technique for training deep learning models. The reason for this is that the minima found using this process generalizes well (Zhang et al., 2016; Wilson et al., 2017), but the optimization process is slow and time consuming. According to (Goyal et al., 2017; Hoffer et al., 2017; You et al., 2017a), this has motivated researchers to try to speed up this optimization process by taking bigger steps, and hence reduce the number of parameter updates in training a model. This can be achieved by using large batch training, which can be divided across many machines. <br /> <br /> However, increasing the batch size leads to decreasing the test set accuracy (Keskar et al., 2016; Goyal et al., 2017). Smith and Le (2017) believed that SGD has a scale of random fluctuations &lt;math&gt; g = \epsilon (\frac{N}{B}-1) &lt;/math&gt;, where &lt;math&gt; \epsilon &lt;/math&gt; is the learning rate, N number of training samples, and B batch size. They concluded that there is an optimal batch size proportional to the learning rate when &lt;math&gt; B \ll N &lt;/math&gt;, and optimum fluctuation scale &lt;math&gt;g&lt;/math&gt; for a maximum test set accuracy.<br /> <br /> In this paper, the authors' main goal is to provide evidence that increasing the batch size is quantitatively equivalent to decreasing the learning rate with the same number of training epochs in decreasing the scale of random fluctuations, but with remarkably less number of parameter updates. Moreover, an additional reduction in the number of parameter updates can be attained by increasing the learning rate and scaling &lt;math&gt; B \propto \epsilon &lt;/math&gt; or even more reduction by increasing the momentum coefficient and scaling &lt;math&gt; B \propto \frac{1}{1-m} &lt;/math&gt; although the latter decreases the test accuracy. This has been demonstrated by several experiments on the ImageNet and CIFAR-10 datasets using ResNet-50 and Inception-ResNet-V2 architectures respectively.<br /> <br /> == STOCHASTIC GRADIENT DESCENT AND CONVEX OPTIMIZATION ==<br /> As mentioned in the previous section, the drawback of SGD when compared to full-batch training is the noise that it introduces that hinders optimization. According to (Robbins &amp; Monro, 1951), there are two equations that govern how to reach the minimum of a convex function: (&lt;math&gt; \epsilon_i &lt;/math&gt; denotes the learning rate at the &lt;math&gt; i^{th} &lt;/math&gt; gradient update)<br /> <br /> &lt;math&gt; \sum_{i=1}^{\infty} \epsilon_i = \infty &lt;/math&gt;. This equation guarantees that we will reach the minimum. <br /> <br /> &lt;math&gt; \sum_{i=1}^{\infty} \epsilon^2_i &lt; \infty &lt;/math&gt;. This equation, which is valid only for a fixed batch size, guarantees that learning rate decays fast enough allowing us to reach the minimum rather than bouncing due to noise.<br /> <br /> These equations indicate that the learning rate must decay during training, and second equation is only available when the batch size is constant. To change the batch size, Smith and Le (2017) proposed to interpret SGD as integrating this stochastic differential equation &lt;math&gt; \frac{dw}{dt} = -\frac{dC}{dw} + \eta(t) &lt;/math&gt;, where &lt;math&gt;C&lt;/math&gt; represents cost function, &lt;math&gt;w&lt;/math&gt; represents the parameters, and &lt;math&gt;\eta&lt;/math&gt; represents the Gaussian random noise. Furthermore, they proved that noise scale &lt;math&gt;g&lt;/math&gt; controls the magnitude of random fluctuations in the training dynamics by this formula: &lt;math&gt; g = \epsilon (\frac{N}{B}-1) &lt;/math&gt;, where &lt;math&gt; \epsilon &lt;/math&gt; is the learning rate, N is the training set size and &lt;math&gt;B&lt;/math&gt; is the batch size. As we usually have &lt;math&gt; B \ll N &lt;/math&gt;, we can define &lt;math&gt; g \approx \epsilon \frac{N}{B} &lt;/math&gt;. This explains why when the learning rate decreases, noise &lt;math&gt;g&lt;/math&gt; decreases, enabling us to converge to the minimum of the cost function. However, increasing the batch size has the same effect and makes &lt;math&gt;g&lt;/math&gt; decays with constant learning rate. In this work, the batch size is increased until &lt;math&gt; B \approx \frac{N}{10} &lt;/math&gt;, then the conventional way of decaying the learning rate is followed.<br /> <br /> == SIMULATED ANNEALING AND THE GENERALIZATION GAP ==<br /> '''Simulated Annealing:''' decaying learning rates are empirically successful. To understand this, they note that introducing random fluctuations<br /> whose scale falls during training is also a well established technique in non-convex optimization; simulated annealing. The initial noisy optimization phase allows to explore a larger fraction of the parameter space without becoming trapped in local minima. Once a promising region of parameter space is located, the noise is reduced to fine-tune the parameters.<br /> <br /> For more info: Simulated annealing (SA) is a probabilistic technique for approximating the global optimum of a given function. Specifically, it is a metaheuristic to approximate global optimization in a large search space for an optimization problem. It is often used when the search space is discrete (e.g., all tours that visit a given set of cities). For problems where finding an approximate global optimum is more important than finding a precise local optimum in a fixed amount of time, simulated annealing may be preferable to alternatives such as gradient descent. [https://en.wikipedia.org/wiki/Simulated_annealing [Reference]]<br /> <br /> '''Generalization Gap:''' Small batch data generalizes better to the test set than large batch data.<br /> <br /> Smith and Le (2017) found that there is an optimal batch size which corresponds to optimal noise scale g &lt;math&gt; (g \approx \epsilon \frac{N}{B}) &lt;/math&gt; and concluded that &lt;math&gt; B_{opt} \propto \epsilon N &lt;/math&gt; that corresponds to maximum test set accuracy. This means that gradient noise is helpful as it makes SGD escape sharp minima, which does not generalize well. <br /> <br /> Simulated Annealing is a famous technique in non-convex optimization. Starting with noise in the training process helps us to discover a wide range of parameters then once we are near the optimum value, noise is reduced to fine tune our final parameters. However, more and more researches like to use the sharper decay schedules like cosine decay or step-function drops. In physical sciences, slowly annealing (or decaying) the temperature (which is the noise scale in this situation) helps to converge to the global minimum, which is sharp. But decaying the temperature in discrete steps can make the system stuck in a local minimum, which lead to higher cost and lower curvature. The authors think that deep learning has the same intuition.<br /> .<br /> <br /> == THE EFFECTIVE LEARNING RATE AND THE ACCUMULATION VARIABLE ==<br /> '''The Effective Learning Rate''' : &lt;math&gt; \epsilon_{eff} = \frac{\epsilon}{1-m} &lt;/math&gt;<br /> <br /> Smith and Le (2017) included momentum to the equation of the vanilla SGD noise scale that was defined above to be: &lt;math&gt; g = \frac{\epsilon}{1-m}(\frac{N}{B}-1)\approx \frac{\epsilon N}{B(1-m)} &lt;/math&gt;, which is the same as the previous equation when m goes to 0. They found that increasing the learning rate and momentum coefficient and scaling &lt;math&gt; B \propto \frac{\epsilon }{1-m} &lt;/math&gt; reduces the number of parameter updates, but the test accuracy decreases when the momentum coefficient is increased. <br /> <br /> To understand the reasons behind this, we need to analyze momentum update equations below:<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \Delta A = -(1-m)A + \frac{d\widehat{C}}{dw} <br /> &lt;/math&gt;<br /> <br /> &lt;math&gt;<br /> \Delta w = -A\epsilon<br /> &lt;/math&gt;<br /> &lt;/center&gt;<br /> <br /> We can see that the Accumulation variable A, which is initially set to 0, then increases exponentially to reach its steady state value during &lt;math&gt; \frac{B}{N(1-m)} &lt;/math&gt; training epochs while &lt;math&gt; \Delta w &lt;/math&gt; is suppressed that can reduce the rate of convergence. Moreover, at high momentum, we have three challenges:<br /> <br /> 1- Additional epochs are needed to catch up with the accumulation.<br /> <br /> 2- Accumulation needs more time &lt;math&gt; \frac{B}{N(1-m)} &lt;/math&gt; to forget old gradients. <br /> <br /> 3- After this time, however, the accumulation cannot adapt to changes in the loss landscape.<br /> <br /> 4- In the early stage, large batch size will lead to the instabilities.<br /> <br /> == EXPERIMENTS ==<br /> === SIMULATED ANNEALING IN A WIDE RESNET ===<br /> <br /> '''Dataset:''' CIFAR-10 (50,000 training images)<br /> <br /> '''Network Architecture:''' “16-4” wide ResNet<br /> <br /> '''Training Schedules used as in the below figure:''' <br /> <br /> - Decaying learning rate: learning rate decays by a factor of 5 at a sequence of “steps”, and the batch size is constant<br /> <br /> - Increasing batch size: learning rate is constant, and the batch size is increased by a factor of 5 at every step.<br /> <br /> - Hybrid: At the beginning, the learning rate is constant and batch size is increased by a factor of 5. Then, the learning rate decays by a factor of 5 at each subsequent step, and the batch size is constant. This is the schedule that will be used if there is a hardware limit affecting a maximum batch size limit.<br /> <br /> [[File:Paper_40_Fig_1.png | 800px|center]]<br /> <br /> As shown in the below figure: in the left figure (2a), we can observe that for the training set, the three learning curves are exactly the same while in figure 2b, increasing the batch size has a huge advantage of reducing the number of parameter updates.<br /> This concludes that noise scale is the one that needs to be decayed and not the learning rate itself<br /> [[File:Paper_40_Fig_2.png | 800px|center]] <br /> <br /> To make sure that these results are the same for the test set as well, in figure 3, we can see that the three learning curves are exactly the same for SGD with momentum, and Nesterov momentum<br /> [[File:Paper_40_Fig_3.png | 800px|center]]<br /> <br /> To check for other optimizers as well. the below figure shows the same experiment as in figure 3, which is the three learning curves for test set, but for vanilla SGD and Adam, and showing <br /> [[File:Paper_40_Fig_4.png | 800px|center]]<br /> <br /> '''Conclusion:''' Decreasing the learning rate and increasing the batch size during training are equivalent<br /> <br /> === INCREASING THE EFFECTIVE LEARNING RATE===<br /> <br /> Here, the focus is on minimizing the number of parameter updates required to train a model. As shown above, the first step is to replace decaying learning rates by increasing batch sizes. Now, the authors show here that we can also increase the effective learning rate &lt;math&gt;\epsilon_{eff} = \epsilon/(1 − m) &lt;/math&gt; at the start of training, while scaling the initial batch size &lt;math&gt;B \propto \epsilon_{eff} &lt;/math&gt; . All experiments are conducted using SGD with momentum. There are 50000 images in the CIFAR-10 training set, and since the scaling rules only hold when &lt;math&gt;B &lt;&lt; N &lt;/math&gt; , we decided to set a maximum batch size &lt;math&gt;B_{max} = 5120 &lt;/math&gt;.<br /> <br /> '''Dataset:''' CIFAR-10 (50,000 training images)<br /> <br /> '''Network Architecture:''' “16-4” wide ResNet<br /> <br /> '''Training Parameters:''' Optimization Algorithm: SGD with momentum / Maximum batch size = 5120<br /> <br /> '''Training Schedules:''' <br /> <br /> The authors consider four training schedules, all of which decay the noise scale by a factor of five in a series of three steps with the same number of epochs.<br /> <br /> Original training schedule: initial learning rate of 0.1 which decays by a factor of 5 at each step, a momentum coefficient of 0.9, and a batch size of 128. Follows the implementation of Zagoruyko &amp; Komodakis (2016).<br /> <br /> Increasing batch size: learning rate of 0.1, momentum coefficient of 0.9, initial batch size of 128 that increases by a factor of 5 at each step. <br /> <br /> Increased initial learning rate: initial learning rate of 0.5, initial batch size of 640 that increase during training.<br /> <br /> Increased momentum coefficient: increased initial learning rate of 0.5, initial batch size of 3200 that increase during training, and an increased momentum coefficient of 0.98.<br /> <br /> The results of all training schedules, which are presented in the below figure, are documented in the following table:<br /> <br /> [[File:Paper_40_Table_1.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_5.png | 800px|center]]<br /> <br /> <br /> <br /> '''Conclusion:''' Increasing the effective learning rate and scaling the batch size results in further reduction in the number of parameter updates<br /> <br /> === TRAINING IMAGENET IN 2500 PARAMETER UPDATES===<br /> <br /> '''A) Experiment Goal:''' Control Batch Size<br /> <br /> '''Dataset:''' ImageNet (1.28 million training images)<br /> <br /> The paper modified the setup of Goyal et al. (2017), and used the following configuration:<br /> <br /> '''Network Architecture:''' Inception-ResNet-V2 <br /> <br /> '''Training Parameters:''' <br /> <br /> 90 epochs / noise decayed at epoch 30, 60, and 80 by a factor of 10 / Initial ghost batch size = 32 / Learning rate = 3 / momentum coefficient = 0.9 / Initial batch size = 8192<br /> <br /> Two training schedules were used:<br /> <br /> “Decaying learning rate”, where batch size is fixed and the learning rate is decayed<br /> <br /> “Increasing batch size”, where batch size is increased to 81920 then the learning rate is decayed at two steps.<br /> <br /> [[File:Paper_40_Table_2.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_6.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the batch size resulted in reducing the number of parameter updates from 14,000 to 6,000.<br /> <br /> '''B) Experiment Goal:''' Control Batch Size and Momentum Coefficient<br /> <br /> '''Training Parameters:''' Ghost batch size = 64 / noise decayed at epoch 30, 60, and 80 by a factor of 10. <br /> <br /> The below table shows the number of parameter updates and accuracy for different set of training parameters:<br /> <br /> [[File:Paper_40_Table_3.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_7.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the momentum reduces the number of parameter updates, but leads to a drop in the test accuracy.<br /> <br /> === TRAINING IMAGENET IN 30 MINUTES===<br /> <br /> '''Dataset:''' ImageNet (Already introduced in the previous section)<br /> <br /> '''Network Architecture:''' ResNet-50<br /> <br /> The paper replicated the setup of Goyal et al. (2017) while modifying the number of TPU devices, batch size, learning rate, and then calculating the time to complete 90 epochs, and measuring the accuracy, and performed the following experiments below:<br /> <br /> [[File:Paper_40_Table_4.png | 800px|center]]<br /> <br /> '''Conclusion:''' Model training times can be reduced by increasing the batch size during training.<br /> <br /> == RELATED WORK ==<br /> Main related work mentioned in the paper is as follows:<br /> <br /> - Smith &amp; Le (2017) interpreted Stochastic gradient descent as stochastic differential equation; the paper built on this idea to include decaying learning rate.<br /> <br /> - Mandt et al. (2017) analyzed how to modify SGD for the task of Bayesian posterior sampling.<br /> <br /> - Keskar et al. (2016) focused on the analysis of noise once the training is started.<br /> <br /> - Moreover, the proportional relationship between batch size and learning rate was first discovered by Goyal et al. (2017) and successfully trained ResNet-50 on ImageNet in one hour after discovering the proportionality relationship between batch size and learning rate.<br /> <br /> - Furthermore, You et al. (2017a) presented Layer-wise Adaptive Rate Scaling (LARS), which is applying different learning rates to train ImageNet in 14 minutes and 74.9% accuracy. <br /> <br /> - Wilson et al. (2017) argued that adaptive optimization methods tend to generalize less well than SGD and SGD with momentum (although<br /> they did not include K-FAC in their study), while the authors' work reduces the gap in convergence speed.<br /> <br /> - Finally, another strategy called Asynchronous-SGD that allowed (Recht et al., 2011; Dean et al., 2012) to use multiple GPUs even with small batch sizes.<br /> <br /> == CONCLUSIONS ==<br /> Increasing the batch size during training has the same benefits of decaying the learning rate in addition to reducing the number of parameter updates, which corresponds to faster training time. Experiments were performed on different image datasets and various optimizers with different training schedules to prove this result. The paper proposed to increase the learning rate and momentum parameter &lt;math&gt;m&lt;/math&gt;, while scaling &lt;math&gt; B \propto \frac{\epsilon}{1-m} &lt;/math&gt;, which achieves fewer parameter updates, but slightly less test set accuracy as mentioned in detail in the experiments’ section. In summary, on ImageNet dataset, Inception-ResNet-V2 achieved 77% validation accuracy in under 2500 parameter updates, and ResNet-50 achieved 76.1% validation set accuracy on TPU in less than 30 minutes. One of the great finding of this paper is that all the methods use the hyper-parameters directly from previous works in the literature, and no additional hyper-parameter tuning was performed.<br /> <br /> == CRITIQUE ==<br /> '''Pros:'''<br /> <br /> - The paper showed empirically that increasing batch size and decaying learning rate are equivalent.<br /> <br /> - Several experiments were performed on different optimizers such as SGD and Adam.<br /> <br /> - Had several comparisons with previous experimental setups.<br /> <br /> '''Cons:'''<br /> <br /> <br /> - All datasets used are image datasets. Other experiments should have been done on datasets from different domains to ensure generalization. <br /> <br /> - The number of parameter updates was used as a comparison criterion, but wall-clock times could have provided additional measurable judgment although they depend on the hardware used.<br /> <br /> - Special hardware is needed for large batch training, which is not always feasible. As batch-size increases, we generally need more RAM to train the same model. However, if learning rate is decreased, the RAM use remains constant. As a result, learning rate decay will allow us to train bigger models.<br /> <br /> - In section 5.2 (Increasing the Effective Learning rate), the authors did not test a range of learning rate values and used only (0.1 and 0.5). Additional results from varying the initial learning rate values from 0.1 to 3.2 are provided in the appendix, which indicates that the test accuracy begins to fall for initial learning rates greater than ~0.4. The appended results do not show validation set accuracy curves like in Figure 6, however. It would be beneficial to see if they were similar to the original 0.1 and 0.5 initial learning rate baselines.<br /> <br /> - Although the main idea of the paper is interesting, its results does not seem to be too surprising in comparison with other recent papers in the subject.<br /> <br /> - The paper could benefit from using some other models to demonstrate its claim and generalize its idea by adding some comparisons with other models as well as other recent methods to increase batch size.<br /> <br /> - The paper presents interesting ideas. However, it lacks of mathematical and theoretical analysis beyond the idea. Since the experiment is primary on image dataset and it does not provide sufficient theories, the paper itself presents limited applicability to other types. <br /> <br /> - Also, in experimental setting, only single training runs from one random initialization is used. It would be better to take the best of many runs or to show confidence intervals.<br /> <br /> - It is proposed that we should compare learning rate decay with batch-size increase under the setting that total budget / number of training samples is fixed.<br /> <br /> == REFERENCES ==<br /> # Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates.arXiv preprint arXiv:1612.05086, 2016.<br /> #L´eon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning.arXiv preprint arXiv:1606.04838, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in neural information processing systems, pp. 1223–1231, 2012.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting.SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Priya Goyal, Piotr Doll´ar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Sepp Hochreiter and J¨urgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997.<br /> #Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 1–12. ACM, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.<br /> #Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. arXiv preprint arXiv:1511.06251, 2017.<br /> #Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. arXiv preprint arXiv:1608.03983, 2016.<br /> #Stephan Mandt, Matthew D Hoffman, and DavidMBlei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on Machine Learning, pp. 2408–2417, 2015.<br /> #Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, volume 27, pp. 372–376, 1983.<br /> #Lutz Prechelt. Early stopping-but when? Neural Networks: Tricks of the trade, pp. 553–553, 1998.<br /> #Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in neural information processing systems, pp. 693–701, 2011.<br /> #Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400–407, 1951.<br /> #Samuel L. Smith and Quoc V. Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451, 2017.<br /> #Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In AAAI, pp. 4278–4284, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. arXiv preprint arXiv:1705.08292, 2017.<br /> #Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888, 2017a.<br /> #Yang You, Zhao Zhang, C Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. CoRR, abs/1709.05011, 2017b.<br /> #Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=DON%27T_DECAY_THE_LEARNING_RATE_,_INCREASE_THE_BATCH_SIZE&diff=42062 DON'T DECAY THE LEARNING RATE , INCREASE THE BATCH SIZE 2018-11-30T16:11:48Z <p>C9sharma: /* INCREASING THE EFFECTIVE LEARNING RATE */</p> <hr /> <div>Summary of the ICLR 2018 paper: '''Don't Decay the learning Rate, Increase the Batch Size ''' <br /> <br /> Link: [https://arxiv.org/pdf/1711.00489.pdf]<br /> <br /> Summarized by: Afify, Ahmed [ID: 20700841]<br /> <br /> ==INTUITION==<br /> Nowadays, it is a common practice not to have a singular steady learning rate for the learning phase of neural network models. Instead, we use adaptive learning rates with the standard gradient descent method. The intuition behind this is that when we are far away from the minima, it is beneficial for us to take large steps towards the minima, as it would require a lesser number of steps to converge, but as we approach the minima, our step size should decrease, otherwise we may just keep oscillating around the minima. In practice, this is generally achieved by methods like SGD with momentum, Nesterov momentum, and Adam. However, the core claim of this paper is that the same effect can be achieved by increasing the batch size during the gradient descent process while keeping the learning rate constant throughout. In addition, the paper argues that such an approach also reduces the parameter updates required to reach the minima, thus leading to greater parallelism and shorter training times.<br /> <br /> == INTRODUCTION ==<br /> Stochastic gradient descent (SGD) is the most widely used optimization technique for training deep learning models. The reason for this is that the minima found using this process generalizes well (Zhang et al., 2016; Wilson et al., 2017), but the optimization process is slow and time consuming. According to (Goyal et al., 2017; Hoffer et al., 2017; You et al., 2017a), this has motivated researchers to try to speed up this optimization process by taking bigger steps, and hence reduce the number of parameter updates in training a model. This can be achieved by using large batch training, which can be divided across many machines. <br /> <br /> However, increasing the batch size leads to decreasing the test set accuracy (Keskar et al., 2016; Goyal et al., 2017). Smith and Le (2017) believed that SGD has a scale of random fluctuations &lt;math&gt; g = \epsilon (\frac{N}{B}-1) &lt;/math&gt;, where &lt;math&gt; \epsilon &lt;/math&gt; is the learning rate, N number of training samples, and B batch size. They concluded that there is an optimal batch size proportional to the learning rate when &lt;math&gt; B \ll N &lt;/math&gt;, and optimum fluctuation scale &lt;math&gt;g&lt;/math&gt; for a maximum test set accuracy.<br /> <br /> In this paper, the authors' main goal is to provide evidence that increasing the batch size is quantitatively equivalent to decreasing the learning rate with the same number of training epochs in decreasing the scale of random fluctuations, but with remarkably less number of parameter updates. Moreover, an additional reduction in the number of parameter updates can be attained by increasing the learning rate and scaling &lt;math&gt; B \propto \epsilon &lt;/math&gt; or even more reduction by increasing the momentum coefficient and scaling &lt;math&gt; B \propto \frac{1}{1-m} &lt;/math&gt; although the latter decreases the test accuracy. This has been demonstrated by several experiments on the ImageNet and CIFAR-10 datasets using ResNet-50 and Inception-ResNet-V2 architectures respectively.<br /> <br /> == STOCHASTIC GRADIENT DESCENT AND CONVEX OPTIMIZATION ==<br /> As mentioned in the previous section, the drawback of SGD when compared to full-batch training is the noise that it introduces that hinders optimization. According to (Robbins &amp; Monro, 1951), there are two equations that govern how to reach the minimum of a convex function: (&lt;math&gt; \epsilon_i &lt;/math&gt; denotes the learning rate at the &lt;math&gt; i^{th} &lt;/math&gt; gradient update)<br /> <br /> &lt;math&gt; \sum_{i=1}^{\infty} \epsilon_i = \infty &lt;/math&gt;. This equation guarantees that we will reach the minimum. <br /> <br /> &lt;math&gt; \sum_{i=1}^{\infty} \epsilon^2_i &lt; \infty &lt;/math&gt;. This equation, which is valid only for a fixed batch size, guarantees that learning rate decays fast enough allowing us to reach the minimum rather than bouncing due to noise.<br /> <br /> These equations indicate that the learning rate must decay during training, and second equation is only available when the batch size is constant. To change the batch size, Smith and Le (2017) proposed to interpret SGD as integrating this stochastic differential equation &lt;math&gt; \frac{dw}{dt} = -\frac{dC}{dw} + \eta(t) &lt;/math&gt;, where &lt;math&gt;C&lt;/math&gt; represents cost function, &lt;math&gt;w&lt;/math&gt; represents the parameters, and &lt;math&gt;\eta&lt;/math&gt; represents the Gaussian random noise. Furthermore, they proved that noise scale &lt;math&gt;g&lt;/math&gt; controls the magnitude of random fluctuations in the training dynamics by this formula: &lt;math&gt; g = \epsilon (\frac{N}{B}-1) &lt;/math&gt;, where &lt;math&gt; \epsilon &lt;/math&gt; is the learning rate, N is the training set size and &lt;math&gt;B&lt;/math&gt; is the batch size. As we usually have &lt;math&gt; B \ll N &lt;/math&gt;, we can define &lt;math&gt; g \approx \epsilon \frac{N}{B} &lt;/math&gt;. This explains why when the learning rate decreases, noise &lt;math&gt;g&lt;/math&gt; decreases, enabling us to converge to the minimum of the cost function. However, increasing the batch size has the same effect and makes &lt;math&gt;g&lt;/math&gt; decays with constant learning rate. In this work, the batch size is increased until &lt;math&gt; B \approx \frac{N}{10} &lt;/math&gt;, then the conventional way of decaying the learning rate is followed.<br /> <br /> == SIMULATED ANNEALING AND THE GENERALIZATION GAP ==<br /> '''Simulated Annealing:''' decaying learning rates are empirically successful. To understand this, they note that introducing random fluctuations<br /> whose scale falls during training is also a well established technique in non-convex optimization; simulated annealing. The initial noisy optimization phase allows to explore a larger fraction of the parameter space without becoming trapped in local minima. Once a promising region of parameter space is located, the noise is reduced to fine-tune the parameters.<br /> <br /> For more info: Simulated annealing (SA) is a probabilistic technique for approximating the global optimum of a given function. Specifically, it is a metaheuristic to approximate global optimization in a large search space for an optimization problem. It is often used when the search space is discrete (e.g., all tours that visit a given set of cities). For problems where finding an approximate global optimum is more important than finding a precise local optimum in a fixed amount of time, simulated annealing may be preferable to alternatives such as gradient descent. [https://en.wikipedia.org/wiki/Simulated_annealing [Reference]]<br /> <br /> '''Generalization Gap:''' Small batch data generalizes better to the test set than large batch data.<br /> <br /> Smith and Le (2017) found that there is an optimal batch size which corresponds to optimal noise scale g &lt;math&gt; (g \approx \epsilon \frac{N}{B}) &lt;/math&gt; and concluded that &lt;math&gt; B_{opt} \propto \epsilon N &lt;/math&gt; that corresponds to maximum test set accuracy. This means that gradient noise is helpful as it makes SGD escape sharp minima, which does not generalize well. <br /> <br /> Simulated Annealing is a famous technique in non-convex optimization. Starting with noise in the training process helps us to discover a wide range of parameters then once we are near the optimum value, noise is reduced to fine tune our final parameters. However, more and more researches like to use the sharper decay schedules like cosine decay or step-function drops. In physical sciences, slowly annealing (or decaying) the temperature (which is the noise scale in this situation) helps to converge to the global minimum, which is sharp. But decaying the temperature in discrete steps can make the system stuck in a local minimum, which lead to higher cost and lower curvature. The authors think that deep learning has the same intuition.<br /> .<br /> <br /> == THE EFFECTIVE LEARNING RATE AND THE ACCUMULATION VARIABLE ==<br /> '''The Effective Learning Rate''' : &lt;math&gt; \epsilon_{eff} = \frac{\epsilon}{1-m} &lt;/math&gt;<br /> <br /> Smith and Le (2017) included momentum to the equation of the vanilla SGD noise scale that was defined above to be: &lt;math&gt; g = \frac{\epsilon}{1-m}(\frac{N}{B}-1)\approx \frac{\epsilon N}{B(1-m)} &lt;/math&gt;, which is the same as the previous equation when m goes to 0. They found that increasing the learning rate and momentum coefficient and scaling &lt;math&gt; B \propto \frac{\epsilon }{1-m} &lt;/math&gt; reduces the number of parameter updates, but the test accuracy decreases when the momentum coefficient is increased. <br /> <br /> To understand the reasons behind this, we need to analyze momentum update equations below:<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \Delta A = -(1-m)A + \frac{d\widehat{C}}{dw} <br /> &lt;/math&gt;<br /> <br /> &lt;math&gt;<br /> \Delta w = -A\epsilon<br /> &lt;/math&gt;<br /> &lt;/center&gt;<br /> <br /> We can see that the Accumulation variable A, which is initially set to 0, then increases exponentially to reach its steady state value during &lt;math&gt; \frac{B}{N(1-m)} &lt;/math&gt; training epochs while &lt;math&gt; \Delta w &lt;/math&gt; is suppressed that can reduce the rate of convergence. Moreover, at high momentum, we have three challenges:<br /> <br /> 1- Additional epochs are needed to catch up with the accumulation.<br /> <br /> 2- Accumulation needs more time &lt;math&gt; \frac{B}{N(1-m)} &lt;/math&gt; to forget old gradients. <br /> <br /> 3- After this time, however, the accumulation cannot adapt to changes in the loss landscape.<br /> <br /> 4- In the early stage, large batch size will lead to the instabilities.<br /> <br /> == EXPERIMENTS ==<br /> === SIMULATED ANNEALING IN A WIDE RESNET ===<br /> <br /> '''Dataset:''' CIFAR-10 (50,000 training images)<br /> <br /> '''Network Architecture:''' “16-4” wide ResNet<br /> <br /> '''Training Schedules used as in the below figure:''' <br /> <br /> - Decaying learning rate: learning rate decays by a factor of 5 at a sequence of “steps”, and the batch size is constant<br /> <br /> - Increasing batch size: learning rate is constant, and the batch size is increased by a factor of 5 at every step.<br /> <br /> - Hybrid: At the beginning, the learning rate is constant and batch size is increased by a factor of 5. Then, the learning rate decays by a factor of 5 at each subsequent step, and the batch size is constant. This is the schedule that will be used if there is a hardware limit affecting a maximum batch size limit.<br /> <br /> [[File:Paper_40_Fig_1.png | 800px|center]]<br /> <br /> As shown in the below figure: in the left figure (2a), we can observe that for the training set, the three learning curves are exactly the same while in figure 2b, increasing the batch size has a huge advantage of reducing the number of parameter updates.<br /> This concludes that noise scale is the one that needs to be decayed and not the learning rate itself<br /> [[File:Paper_40_Fig_2.png | 800px|center]] <br /> <br /> To make sure that these results are the same for the test set as well, in figure 3, we can see that the three learning curves are exactly the same for SGD with momentum, and Nesterov momentum<br /> [[File:Paper_40_Fig_3.png | 800px|center]]<br /> <br /> To check for other optimizers as well. the below figure shows the same experiment as in figure 3, which is the three learning curves for test set, but for vanilla SGD and Adam, and showing <br /> [[File:Paper_40_Fig_4.png | 800px|center]]<br /> <br /> '''Conclusion:''' Decreasing the learning rate and increasing the batch size during training are equivalent<br /> <br /> === INCREASING THE EFFECTIVE LEARNING RATE===<br /> <br /> Here, the focus is on minimizing the number of parameter updates required to train a model. As shown above, the first step is to replace decaying learning rates by increasing batch sizes. Now, the authors show here that we can also increase the effective learning rate &lt;math&gt;\epsilon_{eff} = \epsion/(1 − m) &lt;/math&gt; at the start of training, while scaling the initial batch size &lt;math&gt;B \propto \epsilon_{eff} &lt;/math&gt; . All experiments are conducted using SGD with momentum. There are 50000 images in the CIFAR-10 training set, and since the scaling rules only hold when &lt;math&gt;B &lt;&lt; N &lt;/math&gt; , we decided to set a maximum batch size &lt;math&gt;B_{max} = 5120 &lt;/math&gt;.<br /> <br /> '''Dataset:''' CIFAR-10 (50,000 training images)<br /> <br /> '''Network Architecture:''' “16-4” wide ResNet<br /> <br /> '''Training Parameters:''' Optimization Algorithm: SGD with momentum / Maximum batch size = 5120<br /> <br /> '''Training Schedules:''' <br /> <br /> The authors consider four training schedules, all of which decay the noise scale by a factor of five in a series of three steps with the same number of epochs.<br /> <br /> Original training schedule: initial learning rate of 0.1 which decays by a factor of 5 at each step, a momentum coefficient of 0.9, and a batch size of 128. Follows the implementation of Zagoruyko &amp; Komodakis (2016).<br /> <br /> Increasing batch size: learning rate of 0.1, momentum coefficient of 0.9, initial batch size of 128 that increases by a factor of 5 at each step. <br /> <br /> Increased initial learning rate: initial learning rate of 0.5, initial batch size of 640 that increase during training.<br /> <br /> Increased momentum coefficient: increased initial learning rate of 0.5, initial batch size of 3200 that increase during training, and an increased momentum coefficient of 0.98.<br /> <br /> The results of all training schedules, which are presented in the below figure, are documented in the following table:<br /> <br /> [[File:Paper_40_Table_1.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_5.png | 800px|center]]<br /> <br /> <br /> <br /> '''Conclusion:''' Increasing the effective learning rate and scaling the batch size results in further reduction in the number of parameter updates<br /> <br /> === TRAINING IMAGENET IN 2500 PARAMETER UPDATES===<br /> <br /> '''A) Experiment Goal:''' Control Batch Size<br /> <br /> '''Dataset:''' ImageNet (1.28 million training images)<br /> <br /> The paper modified the setup of Goyal et al. (2017), and used the following configuration:<br /> <br /> '''Network Architecture:''' Inception-ResNet-V2 <br /> <br /> '''Training Parameters:''' <br /> <br /> 90 epochs / noise decayed at epoch 30, 60, and 80 by a factor of 10 / Initial ghost batch size = 32 / Learning rate = 3 / momentum coefficient = 0.9 / Initial batch size = 8192<br /> <br /> Two training schedules were used:<br /> <br /> “Decaying learning rate”, where batch size is fixed and the learning rate is decayed<br /> <br /> “Increasing batch size”, where batch size is increased to 81920 then the learning rate is decayed at two steps.<br /> <br /> [[File:Paper_40_Table_2.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_6.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the batch size resulted in reducing the number of parameter updates from 14,000 to 6,000.<br /> <br /> '''B) Experiment Goal:''' Control Batch Size and Momentum Coefficient<br /> <br /> '''Training Parameters:''' Ghost batch size = 64 / noise decayed at epoch 30, 60, and 80 by a factor of 10. <br /> <br /> The below table shows the number of parameter updates and accuracy for different set of training parameters:<br /> <br /> [[File:Paper_40_Table_3.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_7.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the momentum reduces the number of parameter updates, but leads to a drop in the test accuracy.<br /> <br /> === TRAINING IMAGENET IN 30 MINUTES===<br /> <br /> '''Dataset:''' ImageNet (Already introduced in the previous section)<br /> <br /> '''Network Architecture:''' ResNet-50<br /> <br /> The paper replicated the setup of Goyal et al. (2017) while modifying the number of TPU devices, batch size, learning rate, and then calculating the time to complete 90 epochs, and measuring the accuracy, and performed the following experiments below:<br /> <br /> [[File:Paper_40_Table_4.png | 800px|center]]<br /> <br /> '''Conclusion:''' Model training times can be reduced by increasing the batch size during training.<br /> <br /> == RELATED WORK ==<br /> Main related work mentioned in the paper is as follows:<br /> <br /> - Smith &amp; Le (2017) interpreted Stochastic gradient descent as stochastic differential equation; the paper built on this idea to include decaying learning rate.<br /> <br /> - Mandt et al. (2017) analyzed how to modify SGD for the task of Bayesian posterior sampling.<br /> <br /> - Keskar et al. (2016) focused on the analysis of noise once the training is started.<br /> <br /> - Moreover, the proportional relationship between batch size and learning rate was first discovered by Goyal et al. (2017) and successfully trained ResNet-50 on ImageNet in one hour after discovering the proportionality relationship between batch size and learning rate.<br /> <br /> - Furthermore, You et al. (2017a) presented Layer-wise Adaptive Rate Scaling (LARS), which is applying different learning rates to train ImageNet in 14 minutes and 74.9% accuracy. <br /> <br /> - Wilson et al. (2017) argued that adaptive optimization methods tend to generalize less well than SGD and SGD with momentum (although<br /> they did not include K-FAC in their study), while the authors' work reduces the gap in convergence speed.<br /> <br /> - Finally, another strategy called Asynchronous-SGD that allowed (Recht et al., 2011; Dean et al., 2012) to use multiple GPUs even with small batch sizes.<br /> <br /> == CONCLUSIONS ==<br /> Increasing the batch size during training has the same benefits of decaying the learning rate in addition to reducing the number of parameter updates, which corresponds to faster training time. Experiments were performed on different image datasets and various optimizers with different training schedules to prove this result. The paper proposed to increase the learning rate and momentum parameter &lt;math&gt;m&lt;/math&gt;, while scaling &lt;math&gt; B \propto \frac{\epsilon}{1-m} &lt;/math&gt;, which achieves fewer parameter updates, but slightly less test set accuracy as mentioned in detail in the experiments’ section. In summary, on ImageNet dataset, Inception-ResNet-V2 achieved 77% validation accuracy in under 2500 parameter updates, and ResNet-50 achieved 76.1% validation set accuracy on TPU in less than 30 minutes. One of the great finding of this paper is that all the methods use the hyper-parameters directly from previous works in the literature, and no additional hyper-parameter tuning was performed.<br /> <br /> == CRITIQUE ==<br /> '''Pros:'''<br /> <br /> - The paper showed empirically that increasing batch size and decaying learning rate are equivalent.<br /> <br /> - Several experiments were performed on different optimizers such as SGD and Adam.<br /> <br /> - Had several comparisons with previous experimental setups.<br /> <br /> '''Cons:'''<br /> <br /> <br /> - All datasets used are image datasets. Other experiments should have been done on datasets from different domains to ensure generalization. <br /> <br /> - The number of parameter updates was used as a comparison criterion, but wall-clock times could have provided additional measurable judgment although they depend on the hardware used.<br /> <br /> - Special hardware is needed for large batch training, which is not always feasible. As batch-size increases, we generally need more RAM to train the same model. However, if learning rate is decreased, the RAM use remains constant. As a result, learning rate decay will allow us to train bigger models.<br /> <br /> - In section 5.2 (Increasing the Effective Learning rate), the authors did not test a range of learning rate values and used only (0.1 and 0.5). Additional results from varying the initial learning rate values from 0.1 to 3.2 are provided in the appendix, which indicates that the test accuracy begins to fall for initial learning rates greater than ~0.4. The appended results do not show validation set accuracy curves like in Figure 6, however. It would be beneficial to see if they were similar to the original 0.1 and 0.5 initial learning rate baselines.<br /> <br /> - Although the main idea of the paper is interesting, its results does not seem to be too surprising in comparison with other recent papers in the subject.<br /> <br /> - The paper could benefit from using some other models to demonstrate its claim and generalize its idea by adding some comparisons with other models as well as other recent methods to increase batch size.<br /> <br /> - The paper presents interesting ideas. However, it lacks of mathematical and theoretical analysis beyond the idea. Since the experiment is primary on image dataset and it does not provide sufficient theories, the paper itself presents limited applicability to other types. <br /> <br /> - Also, in experimental setting, only single training runs from one random initialization is used. It would be better to take the best of many runs or to show confidence intervals.<br /> <br /> - It is proposed that we should compare learning rate decay with batch-size increase under the setting that total budget / number of training samples is fixed.<br /> <br /> == REFERENCES ==<br /> # Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates.arXiv preprint arXiv:1612.05086, 2016.<br /> #L´eon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning.arXiv preprint arXiv:1606.04838, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in neural information processing systems, pp. 1223–1231, 2012.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting.SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Priya Goyal, Piotr Doll´ar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Sepp Hochreiter and J¨urgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997.<br /> #Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 1–12. ACM, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.<br /> #Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. arXiv preprint arXiv:1511.06251, 2017.<br /> #Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. arXiv preprint arXiv:1608.03983, 2016.<br /> #Stephan Mandt, Matthew D Hoffman, and DavidMBlei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on Machine Learning, pp. 2408–2417, 2015.<br /> #Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, volume 27, pp. 372–376, 1983.<br /> #Lutz Prechelt. Early stopping-but when? Neural Networks: Tricks of the trade, pp. 553–553, 1998.<br /> #Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in neural information processing systems, pp. 693–701, 2011.<br /> #Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400–407, 1951.<br /> #Samuel L. Smith and Quoc V. Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451, 2017.<br /> #Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In AAAI, pp. 4278–4284, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. arXiv preprint arXiv:1705.08292, 2017.<br /> #Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888, 2017a.<br /> #Yang You, Zhao Zhang, C Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. CoRR, abs/1709.05011, 2017b.<br /> #Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=DON%27T_DECAY_THE_LEARNING_RATE_,_INCREASE_THE_BATCH_SIZE&diff=42059 DON'T DECAY THE LEARNING RATE , INCREASE THE BATCH SIZE 2018-11-30T15:59:07Z <p>C9sharma: /* REFERENCES */</p> <hr /> <div>Summary of the ICLR 2018 paper: '''Don't Decay the learning Rate, Increase the Batch Size ''' <br /> <br /> Link: [https://arxiv.org/pdf/1711.00489.pdf]<br /> <br /> Summarized by: Afify, Ahmed [ID: 20700841]<br /> <br /> ==INTUITION==<br /> Nowadays, it is a common practice not to have a singular steady learning rate for the learning phase of neural network models. Instead, we use adaptive learning rates with the standard gradient descent method. The intuition behind this is that when we are far away from the minima, it is beneficial for us to take large steps towards the minima, as it would require a lesser number of steps to converge, but as we approach the minima, our step size should decrease, otherwise we may just keep oscillating around the minima. In practice, this is generally achieved by methods like SGD with momentum, Nesterov momentum, and Adam. However, the core claim of this paper is that the same effect can be achieved by increasing the batch size during the gradient descent process while keeping the learning rate constant throughout. In addition, the paper argues that such an approach also reduces the parameter updates required to reach the minima, thus leading to greater parallelism and shorter training times.<br /> <br /> == INTRODUCTION ==<br /> Stochastic gradient descent (SGD) is the most widely used optimization technique for training deep learning models. The reason for this is that the minima found using this process generalizes well (Zhang et al., 2016; Wilson et al., 2017), but the optimization process is slow and time consuming. According to (Goyal et al., 2017; Hoffer et al., 2017; You et al., 2017a), this has motivated researchers to try to speed up this optimization process by taking bigger steps, and hence reduce the number of parameter updates in training a model. This can be achieved by using large batch training, which can be divided across many machines. <br /> <br /> However, increasing the batch size leads to decreasing the test set accuracy (Keskar et al., 2016; Goyal et al., 2017). Smith and Le (2017) believed that SGD has a scale of random fluctuations &lt;math&gt; g = \epsilon (\frac{N}{B}-1) &lt;/math&gt;, where &lt;math&gt; \epsilon &lt;/math&gt; is the learning rate, N number of training samples, and B batch size. They concluded that there is an optimal batch size proportional to the learning rate when &lt;math&gt; B \ll N &lt;/math&gt;, and optimum fluctuation scale &lt;math&gt;g&lt;/math&gt; for a maximum test set accuracy.<br /> <br /> In this paper, the authors' main goal is to provide evidence that increasing the batch size is quantitatively equivalent to decreasing the learning rate with the same number of training epochs in decreasing the scale of random fluctuations, but with remarkably less number of parameter updates. Moreover, an additional reduction in the number of parameter updates can be attained by increasing the learning rate and scaling &lt;math&gt; B \propto \epsilon &lt;/math&gt; or even more reduction by increasing the momentum coefficient and scaling &lt;math&gt; B \propto \frac{1}{1-m} &lt;/math&gt; although the latter decreases the test accuracy. This has been demonstrated by several experiments on the ImageNet and CIFAR-10 datasets using ResNet-50 and Inception-ResNet-V2 architectures respectively.<br /> <br /> == STOCHASTIC GRADIENT DESCENT AND CONVEX OPTIMIZATION ==<br /> As mentioned in the previous section, the drawback of SGD when compared to full-batch training is the noise that it introduces that hinders optimization. According to (Robbins &amp; Monro, 1951), there are two equations that govern how to reach the minimum of a convex function: (&lt;math&gt; \epsilon_i &lt;/math&gt; denotes the learning rate at the &lt;math&gt; i^{th} &lt;/math&gt; gradient update)<br /> <br /> &lt;math&gt; \sum_{i=1}^{\infty} \epsilon_i = \infty &lt;/math&gt;. This equation guarantees that we will reach the minimum. <br /> <br /> &lt;math&gt; \sum_{i=1}^{\infty} \epsilon^2_i &lt; \infty &lt;/math&gt;. This equation, which is valid only for a fixed batch size, guarantees that learning rate decays fast enough allowing us to reach the minimum rather than bouncing due to noise.<br /> <br /> These equations indicate that the learning rate must decay during training, and second equation is only available when the batch size is constant. To change the batch size, Smith and Le (2017) proposed to interpret SGD as integrating this stochastic differential equation &lt;math&gt; \frac{dw}{dt} = -\frac{dC}{dw} + \eta(t) &lt;/math&gt;, where &lt;math&gt;C&lt;/math&gt; represents cost function, &lt;math&gt;w&lt;/math&gt; represents the parameters, and &lt;math&gt;\eta&lt;/math&gt; represents the Gaussian random noise. Furthermore, they proved that noise scale &lt;math&gt;g&lt;/math&gt; controls the magnitude of random fluctuations in the training dynamics by this formula: &lt;math&gt; g = \epsilon (\frac{N}{B}-1) &lt;/math&gt;, where &lt;math&gt; \epsilon &lt;/math&gt; is the learning rate, N is the training set size and &lt;math&gt;B&lt;/math&gt; is the batch size. As we usually have &lt;math&gt; B \ll N &lt;/math&gt;, we can define &lt;math&gt; g \approx \epsilon \frac{N}{B} &lt;/math&gt;. This explains why when the learning rate decreases, noise &lt;math&gt;g&lt;/math&gt; decreases, enabling us to converge to the minimum of the cost function. However, increasing the batch size has the same effect and makes &lt;math&gt;g&lt;/math&gt; decays with constant learning rate. In this work, the batch size is increased until &lt;math&gt; B \approx \frac{N}{10} &lt;/math&gt;, then the conventional way of decaying the learning rate is followed.<br /> <br /> == SIMULATED ANNEALING AND THE GENERALIZATION GAP ==<br /> '''Simulated Annealing:''' decaying learning rates are empirically successful. To understand this, they note that introducing random fluctuations<br /> whose scale falls during training is also a well established technique in non-convex optimization; simulated annealing. The initial noisy optimization phase allows to explore a larger fraction of the parameter space without becoming trapped in local minima. Once a promising region of parameter space is located, the noise is reduced to fine-tune the parameters.<br /> <br /> For more info: Simulated annealing (SA) is a probabilistic technique for approximating the global optimum of a given function. Specifically, it is a metaheuristic to approximate global optimization in a large search space for an optimization problem. It is often used when the search space is discrete (e.g., all tours that visit a given set of cities). For problems where finding an approximate global optimum is more important than finding a precise local optimum in a fixed amount of time, simulated annealing may be preferable to alternatives such as gradient descent. [https://en.wikipedia.org/wiki/Simulated_annealing [Reference]]<br /> <br /> '''Generalization Gap:''' Small batch data generalizes better to the test set than large batch data.<br /> <br /> Smith and Le (2017) found that there is an optimal batch size which corresponds to optimal noise scale g &lt;math&gt; (g \approx \epsilon \frac{N}{B}) &lt;/math&gt; and concluded that &lt;math&gt; B_{opt} \propto \epsilon N &lt;/math&gt; that corresponds to maximum test set accuracy. This means that gradient noise is helpful as it makes SGD escape sharp minima, which does not generalize well. <br /> <br /> Simulated Annealing is a famous technique in non-convex optimization. Starting with noise in the training process helps us to discover a wide range of parameters then once we are near the optimum value, noise is reduced to fine tune our final parameters. However, more and more researches like to use the sharper decay schedules like cosine decay or step-function drops. In physical sciences, slowly annealing (or decaying) the temperature (which is the noise scale in this situation) helps to converge to the global minimum, which is sharp. But decaying the temperature in discrete steps can make the system stuck in a local minimum, which lead to higher cost and lower curvature. The authors think that deep learning has the same intuition.<br /> .<br /> <br /> == THE EFFECTIVE LEARNING RATE AND THE ACCUMULATION VARIABLE ==<br /> '''The Effective Learning Rate''' : &lt;math&gt; \epsilon_{eff} = \frac{\epsilon}{1-m} &lt;/math&gt;<br /> <br /> Smith and Le (2017) included momentum to the equation of the vanilla SGD noise scale that was defined above to be: &lt;math&gt; g = \frac{\epsilon}{1-m}(\frac{N}{B}-1)\approx \frac{\epsilon N}{B(1-m)} &lt;/math&gt;, which is the same as the previous equation when m goes to 0. They found that increasing the learning rate and momentum coefficient and scaling &lt;math&gt; B \propto \frac{\epsilon }{1-m} &lt;/math&gt; reduces the number of parameter updates, but the test accuracy decreases when the momentum coefficient is increased. <br /> <br /> To understand the reasons behind this, we need to analyze momentum update equations below:<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \Delta A = -(1-m)A + \frac{d\widehat{C}}{dw} <br /> &lt;/math&gt;<br /> <br /> &lt;math&gt;<br /> \Delta w = -A\epsilon<br /> &lt;/math&gt;<br /> &lt;/center&gt;<br /> <br /> We can see that the Accumulation variable A, which is initially set to 0, then increases exponentially to reach its steady state value during &lt;math&gt; \frac{B}{N(1-m)} &lt;/math&gt; training epochs while &lt;math&gt; \Delta w &lt;/math&gt; is suppressed that can reduce the rate of convergence. Moreover, at high momentum, we have three challenges:<br /> <br /> 1- Additional epochs are needed to catch up with the accumulation.<br /> <br /> 2- Accumulation needs more time &lt;math&gt; \frac{B}{N(1-m)} &lt;/math&gt; to forget old gradients. <br /> <br /> 3- After this time, however, the accumulation cannot adapt to changes in the loss landscape.<br /> <br /> 4- In the early stage, large batch size will lead to the instabilities.<br /> <br /> == EXPERIMENTS ==<br /> === SIMULATED ANNEALING IN A WIDE RESNET ===<br /> <br /> '''Dataset:''' CIFAR-10 (50,000 training images)<br /> <br /> '''Network Architecture:''' “16-4” wide ResNet<br /> <br /> '''Training Schedules used as in the below figure:''' <br /> <br /> - Decaying learning rate: learning rate decays by a factor of 5 at a sequence of “steps”, and the batch size is constant<br /> <br /> - Increasing batch size: learning rate is constant, and the batch size is increased by a factor of 5 at every step.<br /> <br /> - Hybrid: At the beginning, the learning rate is constant and batch size is increased by a factor of 5. Then, the learning rate decays by a factor of 5 at each subsequent step, and the batch size is constant. This is the schedule that will be used if there is a hardware limit affecting a maximum batch size limit.<br /> <br /> [[File:Paper_40_Fig_1.png | 800px|center]]<br /> <br /> As shown in the below figure: in the left figure (2a), we can observe that for the training set, the three learning curves are exactly the same while in figure 2b, increasing the batch size has a huge advantage of reducing the number of parameter updates.<br /> This concludes that noise scale is the one that needs to be decayed and not the learning rate itself<br /> [[File:Paper_40_Fig_2.png | 800px|center]] <br /> <br /> To make sure that these results are the same for the test set as well, in figure 3, we can see that the three learning curves are exactly the same for SGD with momentum, and Nesterov momentum<br /> [[File:Paper_40_Fig_3.png | 800px|center]]<br /> <br /> To check for other optimizers as well. the below figure shows the same experiment as in figure 3, which is the three learning curves for test set, but for vanilla SGD and Adam, and showing <br /> [[File:Paper_40_Fig_4.png | 800px|center]]<br /> <br /> '''Conclusion:''' Decreasing the learning rate and increasing the batch size during training are equivalent<br /> <br /> === INCREASING THE EFFECTIVE LEARNING RATE===<br /> <br /> '''Dataset:''' CIFAR-10 (50,000 training images)<br /> <br /> '''Network Architecture:''' “16-4” wide ResNet<br /> <br /> '''Training Parameters:''' Optimization Algorithm: SGD with momentum / Maximum batch size = 5120<br /> <br /> '''Training Schedules:''' <br /> <br /> The authors consider four training schedules, all of which decay the noise scale by a factor of five in a series of three steps with the same number of epochs.<br /> <br /> Original training schedule: initial learning rate of 0.1 which decays by a factor of 5 at each step, a momentum coefficient of 0.9, and a batch size of 128. <br /> <br /> Increasing batch size: learning rate of 0.1, momentum coefficient of 0.9, initial batch size of 128 that increases by a factor of 5 at each step. <br /> <br /> Increased initial learning rate: initial learning rate of 0.5, initial batch size of 640 that increase during training.<br /> <br /> Increased momentum coefficient: increased initial learning rate of 0.5, initial batch size of 3200 that increase during training, and an increased momentum coefficient of 0.98.<br /> <br /> The results of all training schedules, which are presented in the below figure, are documented in the following table:<br /> <br /> [[File:Paper_40_Table_1.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_5.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the effective learning rate and scaling the batch size results in further reduction in the number of parameter updates<br /> <br /> === TRAINING IMAGENET IN 2500 PARAMETER UPDATES===<br /> <br /> '''A) Experiment Goal:''' Control Batch Size<br /> <br /> '''Dataset:''' ImageNet (1.28 million training images)<br /> <br /> The paper modified the setup of Goyal et al. (2017), and used the following configuration:<br /> <br /> '''Network Architecture:''' Inception-ResNet-V2 <br /> <br /> '''Training Parameters:''' <br /> <br /> 90 epochs / noise decayed at epoch 30, 60, and 80 by a factor of 10 / Initial ghost batch size = 32 / Learning rate = 3 / momentum coefficient = 0.9 / Initial batch size = 8192<br /> <br /> Two training schedules were used:<br /> <br /> “Decaying learning rate”, where batch size is fixed and the learning rate is decayed<br /> <br /> “Increasing batch size”, where batch size is increased to 81920 then the learning rate is decayed at two steps.<br /> <br /> [[File:Paper_40_Table_2.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_6.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the batch size resulted in reducing the number of parameter updates from 14,000 to 6,000.<br /> <br /> '''B) Experiment Goal:''' Control Batch Size and Momentum Coefficient<br /> <br /> '''Training Parameters:''' Ghost batch size = 64 / noise decayed at epoch 30, 60, and 80 by a factor of 10. <br /> <br /> The below table shows the number of parameter updates and accuracy for different set of training parameters:<br /> <br /> [[File:Paper_40_Table_3.png | 800px|center]]<br /> <br /> [[File:Paper_40_Fig_7.png | 800px|center]]<br /> <br /> '''Conclusion:''' Increasing the momentum reduces the number of parameter updates, but leads to a drop in the test accuracy.<br /> <br /> === TRAINING IMAGENET IN 30 MINUTES===<br /> <br /> '''Dataset:''' ImageNet (Already introduced in the previous section)<br /> <br /> '''Network Architecture:''' ResNet-50<br /> <br /> The paper replicated the setup of Goyal et al. (2017) while modifying the number of TPU devices, batch size, learning rate, and then calculating the time to complete 90 epochs, and measuring the accuracy, and performed the following experiments below:<br /> <br /> [[File:Paper_40_Table_4.png | 800px|center]]<br /> <br /> '''Conclusion:''' Model training times can be reduced by increasing the batch size during training.<br /> <br /> == RELATED WORK ==<br /> Main related work mentioned in the paper is as follows:<br /> <br /> - Smith &amp; Le (2017) interpreted Stochastic gradient descent as stochastic differential equation; the paper built on this idea to include decaying learning rate.<br /> <br /> - Mandt et al. (2017) analyzed how to modify SGD for the task of Bayesian posterior sampling.<br /> <br /> - Keskar et al. (2016) focused on the analysis of noise once the training is started.<br /> <br /> - Moreover, the proportional relationship between batch size and learning rate was first discovered by Goyal et al. (2017) and successfully trained ResNet-50 on ImageNet in one hour after discovering the proportionality relationship between batch size and learning rate.<br /> <br /> - Furthermore, You et al. (2017a) presented Layer-wise Adaptive Rate Scaling (LARS), which is applying different learning rates to train ImageNet in 14 minutes and 74.9% accuracy. <br /> <br /> - Wilson et al. (2017) argued that adaptive optimization methods tend to generalize less well than SGD and SGD with momentum (although<br /> they did not include K-FAC in their study), while the authors' work reduces the gap in convergence speed.<br /> <br /> - Finally, another strategy called Asynchronous-SGD that allowed (Recht et al., 2011; Dean et al., 2012) to use multiple GPUs even with small batch sizes.<br /> <br /> == CONCLUSIONS ==<br /> Increasing the batch size during training has the same benefits of decaying the learning rate in addition to reducing the number of parameter updates, which corresponds to faster training time. Experiments were performed on different image datasets and various optimizers with different training schedules to prove this result. The paper proposed to increase the learning rate and momentum parameter &lt;math&gt;m&lt;/math&gt;, while scaling &lt;math&gt; B \propto \frac{\epsilon}{1-m} &lt;/math&gt;, which achieves fewer parameter updates, but slightly less test set accuracy as mentioned in detail in the experiments’ section. In summary, on ImageNet dataset, Inception-ResNet-V2 achieved 77% validation accuracy in under 2500 parameter updates, and ResNet-50 achieved 76.1% validation set accuracy on TPU in less than 30 minutes. One of the great finding of this paper is that all the methods use the hyper-parameters directly from previous works in the literature, and no additional hyper-parameter tuning was performed.<br /> <br /> == CRITIQUE ==<br /> '''Pros:'''<br /> <br /> - The paper showed empirically that increasing batch size and decaying learning rate are equivalent.<br /> <br /> - Several experiments were performed on different optimizers such as SGD and Adam.<br /> <br /> - Had several comparisons with previous experimental setups.<br /> <br /> '''Cons:'''<br /> <br /> <br /> - All datasets used are image datasets. Other experiments should have been done on datasets from different domains to ensure generalization. <br /> <br /> - The number of parameter updates was used as a comparison criterion, but wall-clock times could have provided additional measurable judgment although they depend on the hardware used.<br /> <br /> - Special hardware is needed for large batch training, which is not always feasible. As batch-size increases, we generally need more RAM to train the same model. However, if learning rate is decreased, the RAM use remains constant. As a result, learning rate decay will allow us to train bigger models.<br /> <br /> - In section 5.2 (Increasing the Effective Learning rate), the authors did not test a range of learning rate values and used only (0.1 and 0.5). Additional results from varying the initial learning rate values from 0.1 to 3.2 are provided in the appendix, which indicates that the test accuracy begins to fall for initial learning rates greater than ~0.4. The appended results do not show validation set accuracy curves like in Figure 6, however. It would be beneficial to see if they were similar to the original 0.1 and 0.5 initial learning rate baselines.<br /> <br /> - Although the main idea of the paper is interesting, its results does not seem to be too surprising in comparison with other recent papers in the subject.<br /> <br /> - The paper could benefit from using some other models to demonstrate its claim and generalize its idea by adding some comparisons with other models as well as other recent methods to increase batch size.<br /> <br /> - The paper presents interesting ideas. However, it lacks of mathematical and theoretical analysis beyond the idea. Since the experiment is primary on image dataset and it does not provide sufficient theories, the paper itself presents limited applicability to other types. <br /> <br /> - Also, in experimental setting, only single training runs from one random initialization is used. It would be better to take the best of many runs or to show confidence intervals.<br /> <br /> - It is proposed that we should compare learning rate decay with batch-size increase under the setting that total budget / number of training samples is fixed.<br /> <br /> == REFERENCES ==<br /> # Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates.arXiv preprint arXiv:1612.05086, 2016.<br /> #L´eon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning.arXiv preprint arXiv:1606.04838, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in neural information processing systems, pp. 1223–1231, 2012.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting.SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Priya Goyal, Piotr Doll´ar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Sepp Hochreiter and J¨urgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997.<br /> #Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 1–12. ACM, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.<br /> #Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. arXiv preprint arXiv:1511.06251, 2017.<br /> #Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. arXiv preprint arXiv:1608.03983, 2016.<br /> #Stephan Mandt, Matthew D Hoffman, and DavidMBlei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on Machine Learning, pp. 2408–2417, 2015.<br /> #Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, volume 27, pp. 372–376, 1983.<br /> #Lutz Prechelt. Early stopping-but when? Neural Networks: Tricks of the trade, pp. 553–553, 1998.<br /> #Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in neural information processing systems, pp. 693–701, 2011.<br /> #Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400–407, 1951.<br /> #Samuel L. Smith and Quoc V. Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451, 2017.<br /> #Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In AAAI, pp. 4278–4284, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. arXiv preprint arXiv:1705.08292, 2017.<br /> #Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888, 2017a.<br /> #Yang You, Zhao Zhang, C Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. CoRR, abs/1709.05011, 2017b.<br /> #Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Reinforcement_Learning_in_Continuous_Action_Spaces_a_Case_Study_in_the_Game_of_Simulated_Curling&diff=42058 Deep Reinforcement Learning in Continuous Action Spaces a Case Study in the Game of Simulated Curling 2018-11-30T15:54:26Z <p>C9sharma: </p> <hr /> <div>This page provides a summary and critique of the paper '''Deep Reinforcement Learning in Continuous Action Spaces: a Case Study in the Game of Simulated Curling''' [[http://proceedings.mlr.press/v80/lee18b/lee18b.pdf Online Source]], published in ICML 2018. The source code for this paper is available [https://github.com/leekwoon/KR-DL-UCT here]<br /> <br /> = Introduction and Motivation =<br /> <br /> In recent years, Reinforcement Learning methods have been applied to many different games, such as chess and checkers. More recently, the use of CNN's has allowed neural networks to out-perform humans in many difficult games, such as Go. However, many of these cases involve a discrete state or action space; the number of actions a player can take and/or the number of possible game states are finite. <br /> <br /> Interacting with the real world (e.g.; a scenario that involves moving physical objects) typically involves working with a continuous action space. It is thus important to develop strategies for dealing with continuous action spaces. Deep neural networks that are designed to succeed in finite action spaces are not necessarily suitable for continuous action space problems. This is due to the fact that deterministic discretization of a continuous action space causes strong biases in policy evaluation and improvement. <br /> <br /> This paper introduces a method to allow learning with continuous action spaces. A CNN is used to perform learning on a discretion state and action spaces, and then a continuous action search is performed on these discrete results.<br /> <br /> Curling is chosen as a domain to test the network on. Curling was chosen due to its large action space, potential for complicated strategies, and need for precise interactions.<br /> <br /> == Curling ==<br /> <br /> Curling is a sport played by two teams on a long sheet of ice. Roughly, the goal is for each time to slide rocks closer to the target on the other end of the sheet than the other team. The next sections will provide a background on the game play, and potential challenges/concerns for learning algorithms. A terminology section follows.<br /> <br /> === Game play ===<br /> <br /> A game of curling is divided into ends. In each end, players from both teams alternate throwing (sliding) eight rocks to the other end of the ice sheet, known as the house. Rocks must land in a certain area in order to stay in play, and must touch or be inside concentric rings (12ft diameter and smaller) in order to score points. At the end of each end, the team with rocks closest to the center of the house scores points.<br /> <br /> When throwing a rock, the curling can spin the rock. This allows the rock to 'curl' its path towards the house and can allow rocks to travel around other rocks. Team members are also able to sweep the ice in front of a moving rock in order to decrease friction, which allows for fine-tuning of distance (though the physics of sweeping are not implemented in the simulation used).<br /> <br /> Curling offers many possible high-level actions, which are directed by a team member to the throwing member. An example set of these includes:<br /> <br /> * Draw: Throw a rock to a target location<br /> * Freeze: Draw a rock up against another rock<br /> * Takeout: Knock another rock out of the house. Can be combined with different ricochet directions<br /> * Guard: Place a rock in front of another, to block other rocks (ex: takeouts)<br /> <br /> === Challenges for AI ===<br /> <br /> Curling offers many challenges for curling based on its physics and rules. This section lists a few concerns.<br /> <br /> The effect of changing actions can be highly nonlinear and discontinuous. This can be seen when considering that a 1-cm deviation in a path can make the difference between a high-speed collision, or lack of collision.<br /> <br /> Curling will require both offensive and defensive strategies. For example, consider the fact that the last team to throw a rock each end only needs to place that rock closer than the opposing team's rocks to score a point and invalidate any opposing rocks in the house. The opposing team should thus be considering how to prevent this from happening, in addition to scoring points themselves.<br /> <br /> Curling also has a concept known as 'the hammer'. The hammer belongs to the team which throws the last rock each end, providing an advantage, and is given to the team that does not score points each end. It could very well be a good strategy to try not to win a single point in an end (if already ahead in points, etc), as this would give the advantage to the opposing team.<br /> <br /> Finally, curling has a rule known as the 'Free Guard Zone'. This applies to the first 4 rocks thrown (2 from each team). If they land short of the house, but still in play, then the rocks are not allowed to be removed (via collisions) until all of the first 4 rocks have been thrown.<br /> <br /> === Terminology ===<br /> <br /> * End: A round of the game<br /> * House: The end of the sheet of ice, which contains<br /> * Hammer: The team that throws the last rock of an end 'has the hammer'<br /> * Hog Line: thick line that is drawn in front of the house, orthogonal to the length of the ice sheet. Rocks must pass this line to remain in play.<br /> * Back Line: think line drawn just behind the house. Rocks that pass this line are removed from play.<br /> <br /> <br /> == Related Work ==<br /> <br /> === AlphaGo Lee ===<br /> <br /> AlphaGo Lee (Silver et al., 2016, ) refers to an algorithm used to play the game Go, which was able to defeat international champion Lee Sedol. <br /> <br /> <br /> Go game:<br /> * Start with 19x19 empty board<br /> * One player take black stones and the other take white stones<br /> * Two players take turns to put stones on the board<br /> * Rules:<br /> 1. If one connected part is completely surrounded by the opponents stones, remove it from the board<br /> <br /> 2. Ko rule: Forbids a board play to repeat a board position<br /> * End when there is no valuable moves on the board.<br /> * Count the territory of both players.<br /> * Add 7.5 points to whites points (called Komi).<br /> [[File:go.JPG|700px|center]]<br /> <br /> Two neural networks were trained on the moves of human experts, to act as both a policy network and a value network. A Monte Carlo Tree Search algorithm was used for policy improvement.<br /> <br /> The AlphaGo Lee policy network predicts the best move given a board configuration. It has a CNN architecture with 13 hidden layers, and it is trained using expert game play data and improved through self-play.<br /> <br /> The value network evaluates the probability of winning given a board configuration. It consists of a CNN with 14 hidden layers, and it is trained using self-play data from the policy network. <br /> <br /> Finally, the two networks are combined using Monte-Carlo Tree Search, which performs look ahead search to select the actions for game play.<br /> <br /> The use of both policy and value networks are reflected in this paper's work.<br /> <br /> === AlphaGo Zero ===<br /> <br /> AlphaGo Zero (Silver et al., 2017, ) is an improvement on the AlphaGo Lee algorithm. AlphaGo Zero uses a unified neural network in place of the separate policy and value networks and is trained on self-play, without the need of expert training.<br /> Previous versions of AlphaGo initially trained on thousands of human amateur and professional games to learn how to play Go. AlphaGo Zero skips this step and learns to play simply by playing games against itself, starting from completely random play. In doing so, it quickly surpassed human level of play and defeated the previously published champion-defeating version of AlphaGo by 100 games to 0.<br /> It is able to do this by using a novel form of reinforcement learning, in which AlphaGo Zero becomes its own teacher. The system starts off with a neural network that knows nothing about the game of Go. It then plays games against itself, by combining this neural network with a powerful search algorithm. As it plays, the neural network is tuned and updated to predict moves, as well as the eventual winner of the games.<br /> <br /> This updated neural network is then recombined with the search algorithm to create a new, stronger version of AlphaGo Zero, and the process begins again. In each iteration, the performance of the system improves by a small amount, and the quality of the self-play games increases, leading to more and more accurate neural networks and ever stronger versions of AlphaGo Zero.<br /> <br /> This technique is more powerful than previous versions of AlphaGo because it is no longer constrained by the limits of human knowledge. Instead, it is able to learn tabula rasa from the strongest player in the world: AlphaGo itself.<br /> <br /> Other differences from the previous AlphaGo iterations are as follows. AlphaGo Zero only uses the black and white stones from the Go board as its input, whereas previous versions of AlphaGo included a small number of hand-engineered features. It uses one neural network rather than two. Earlier versions of AlphaGo used a “policy network” to select the next move to play and a ”value network” to predict the winner of the game from each position. These are combined in AlphaGo Zero, allowing it to be trained and evaluated more efficiently. AlphaGo Zero does not use “rollouts” - fast, random games used by other Go programs to predict which player will win from the current board position. Instead, it relies on its high quality neural networks to evaluate positions. All of these differences help improve the performance of the system and make it more general. But it is the algorithmic change that makes the system much more powerful and efficient.<br /> <br /> The unification of networks and self-play are also reflected in this paper.<br /> <br /> === Curling Algorithms ===<br /> <br /> Some past algorithms have been proposed to deal with continuous action spaces. For example, (Yammamoto et al, 2015, ) use game tree search methods in a discretized space. The value of an action is taken as the average of nearby values, with respect to some knowledge of execution uncertainty.<br /> <br /> === Monte Carlo Tree Search ===<br /> <br /> Monte Carlo Tree Search algorithms have been applied to continuous action spaces. These algorithms, to be discussed in further detail, balance exploration of different states, with knowledge of paths of execution through past games. An MCTS called &lt;math&gt;KR-UCT&lt;/math&gt; which is able to find effective selections and use kernel regression (KR) and kernel density estimation(KDE) to estimate rewards using neighborhood information has been applied to continuous action space by researchers. <br /> <br /> With bandit problem, scholars used hierarchical optimistic optimization(HOO) to create a cover tree and divide the action space into small ranges at different depths, where the most promising node will create fine granularity estimates.<br /> <br /> === Curling Physics and Simulation ===<br /> <br /> Several references in the paper refer to the study and simulation of curling physics. Scholars have analyzed friction coefficients between curling stones and ice. While modelling the changes in friction on ice is not possible, a fixed friction coefficient was predefined in the simulation. The behavior of the stones was also modeled. Important parameters are trained from professional players. The authors used the same parameters in this paper.<br /> <br /> == General Background of Algorithms ==<br /> <br /> === Policy and Value Functions ===<br /> <br /> A policy function is trained to provide the best action to take, given a current state. Policy iteration is an algorithm used to improve a policy over time. This is done by alternating between policy evaluation and policy improvement.<br /> <br /> POLICY IMPROVEMENT: LEARNING ACTION POLICY<br /> <br /> Action policy &lt;math&gt; p_{\sigma}(a|s) &lt;/math&gt; outputs a probability distribution over all eligible moves &lt;math&gt; a &lt;/math&gt;. Here &lt;math&gt; \sigma &lt;/math&gt; denotes the weights of a neural network that approximates the policy. &lt;math&gt;s&lt;/math&gt; denotes the set of states and &lt;math&gt;a&lt;/math&gt; denotes the set of actions taken in the environment. The policy is a function that returns a action given the state at which the agent is present. The policy gradient reinforcement learning can be used to train action policy. It is updated by stochastic gradient ascent in the direction that maximizes the expected outcome at each time step t,<br /> $\Delta \rho \propto \frac{\partial p_{\rho}(a_t|s_t)}{\partial \rho} r(s_t)$<br /> where &lt;math&gt; r(s_t) &lt;/math&gt; is the return.<br /> <br /> POLICY EVALUATION: LEARNING VALUE FUNCTIONS<br /> <br /> A value function is trained to estimate the value of a value of being in a certain state with parameter &lt;math&gt; \theta &lt;/math&gt;. It is trained based on records of state-action-reward sets &lt;math&gt; (s, r(s)) &lt;/math&gt; by using stochastic gradient de- scent to minimize the mean squared error (MSE) between the predicted regression value and the corresponding outcome,<br /> $\Delta \theta \propto \frac{\partial v_{\theta}(s)}{\partial \theta}(r(s)-v_{\theta}(s))$<br /> <br /> === Monte Carlo Tree Search ===<br /> <br /> Monte Carlo Tree Search (MCTS) is a search algorithm used for finite-horizon tasks (ex: in curling, only 16 moves, or throw stones, are taken each end).<br /> <br /> MCTS is a tree search algorithm similar to minimax. However, MCTS is probabilistic and does not need to explore a full game tree or even a tree reduced with alpha-beta pruning. This makes it tractable for games such as GO, and curling.<br /> <br /> Nodes of the tree are game states, and branches represent actions. Each node stores statistics on how many times it has been visited by the MCTS, as well as the number of wins encountered by playouts from that position. A node has been considered 'visited' if a full playout has started from that node. A node is considered 'expanded' if all its children have been visited.<br /> <br /> MCTS begins with the '''selection''' phase, which involves traversing known states/actions. This involves expanding the tree by beginning at the root node, and selecting the child/score with the highest 'score'. From each successive node, a path down to a root node is explored in a similar fashion.<br /> <br /> The next phase, '''expansion''', begins when the algorithm reaches a node where not all children have been visited (ie: the node has not been fully expanded). In the expansion phase, children of the node are visited, and '''simulations''' run from their states.<br /> <br /> Once the new child is expanded, '''simulation''' takes place. This refers to a full playout of the game from the point of the current node, and can involve many strategies, such as randomly taken moves, the use of heuristics, etc.<br /> <br /> The final phase is '''update''' or '''back-propagation''' (unrelated to the neural network algorithm). In this phase, the result of the '''simulation''' (ie: win/lose) is update in the statistics of all parent nodes.<br /> <br /> A selection function known as Upper Confidence Bound (UCT) can be used for selecting which node to select. The formula for this equation is shown below [[https://www.baeldung.com/java-monte-carlo-tree-search source]]. Note that the first term essentially acts as an average score of games played from a certain node. The second term, meanwhile, will grow when sibling nodes are expanded. This means that unexplored nodes will gradually increase their UCT score, and be selected in the future.<br /> <br /> &lt;math&gt; \frac{w_i}{n_i} + c \sqrt{\frac{\ln t}{n_i}} &lt;/math&gt;<br /> <br /> In which<br /> <br /> * &lt;math&gt; w_i = &lt;/math&gt; number of wins after &lt;math&gt; i&lt;/math&gt;th move<br /> * &lt;math&gt; n_i = &lt;/math&gt; number of simulations after &lt;math&gt; i&lt;/math&gt;th move<br /> * &lt;math&gt; c = &lt;/math&gt; exploration parameter (theoritically eqal to &lt;math&gt; \sqrt{2}&lt;/math&gt;)<br /> * &lt;math&gt; t = &lt;/math&gt; total number of simulations for the parent node<br /> <br /> <br /> Sources: 2,3,4<br /> <br /> [[File:MCTS_Diagram.jpg | 500px|center]]<br /> <br /> === Kernel Regression ===<br /> <br /> Kernel regression is a form of weighted averaging which uses a kernel function as a weight to estimate the conditional expectation of a random variable. Given two items of data, '''x''', each of which has a value '''y''' associated with them, and a choice of Kernel '''K''', the kernel functions outputs a weighting factor. An estimate of the value of a new, unseen point, is then calculated as the weighted average of values of surrounding points.<br /> <br /> A typical kernel is a Gaussian kernel, shown below. The formula for calculating estimated value is shown below as well (sources: Lee et al.).<br /> <br /> [[File:gaussian_kernel.png | 400 px]]<br /> <br /> [[File:kernel_regression.png | 250 px]]<br /> <br /> The denominator of the conditional expectation is related to kernel density estimation, which is defined as &lt;math display=&quot;inline&quot;&gt;W(x)=\sum_{i=0}^n K(x,x_i)&lt;/math&gt;.<br /> <br /> In this case, the combination of the two-act to weigh scores of samples closest to '''x''' more strongly.<br /> <br /> = Methods =<br /> <br /> == Variable Definitions ==<br /> <br /> The following variables are used often in the paper:<br /> <br /> * &lt;math&gt;s&lt;/math&gt;: A state in the game, as described below as the input to the network.<br /> * &lt;math&gt;s_t&lt;/math&gt;: The state at a certain time-step of the game. Time-steps refer to full turns in the game<br /> * &lt;math&gt;a_t&lt;/math&gt;: The action taken in state &lt;math&gt;s_t&lt;/math&gt;<br /> * &lt;math&gt;A_t&lt;/math&gt;: The actions taken for sibling nodes related to &lt;math&gt;a_t&lt;/math&gt; in MCTS<br /> * &lt;math&gt;n_{a_t}&lt;/math&gt;: The number of visits to node a in MCTS<br /> * &lt;math&gt;v_{a_t}&lt;/math&gt;: The MCTS value estimate of a node<br /> <br /> == Network Design ==<br /> <br /> The authors design a CNN called the 'policy-value' network. The network consists of a common network structure, which is then split into 'policy' and 'value' outputs. This network is trained to learn a probability distribution of actions to take, and expected rewards, given an input state.<br /> <br /> === Shared Structure ===<br /> <br /> The network consists of 1 convolutional layer followed by 9 residual blocks, each block consisting of 2 convolutional layers with 32 3x3 filters. The structure of this network is shown below:<br /> <br /> <br /> [[File:curling_network_layers.png|600px|thumb|center|Figure 2. A detail description of our policy-value network. The shared network is composed of one convolutional layer and nine residual blocks. Each residual block (explained in b) has two convolutional layer with batch normalization (Ioffe &amp; Szegedy, 2015) followed by the addition of the input and the residual block. Each layer in the shared network uses 3x3 filters. The policy head<br /> has two more convolutional layers, while the value head has two fully connected layers on top of a convolutional layer. For the activation function of each convolutional layer, ReLU (Nair &amp; Hinton) is used.]]<br /> <br /> <br /> <br /> the input to this network is the following:<br /> * Location of stones<br /> * Order to tee (the center of the sheet)<br /> * A 32x32 grid of representation of the ice sheet, representing which stones are present in each grid cell.<br /> <br /> The authors do not describe how the stone-based information is added to the 32x32 grid as input to the network.<br /> <br /> === Policy Network ===<br /> <br /> The policy head is created by adding 2 convolutional layers with 2 (two) 3x3 filters to the main body of the network. The output of the policy head is a distribution of probabilities of the actions to select the best shot out of a 32x32x2 set of actions. The actions represent target locations in the grid and spin direction of the stone.<br /> <br /> [[File:policy-value-net.PNG | 700px]]<br /> <br /> === Value Network ===<br /> <br /> The valve head is created by adding a convolution layer with 1 3x3 filter, and dense layers of 256 and 17 units, to the shared network. The 17 output units represent a probability of scores in the range of [-8,8], which are the possible scores at each end of a curling game.<br /> <br /> == Continuous Action Search ==<br /> <br /> The policy head of the network only outputs actions from a discretized action space. For real-life interactions, and especially in curling, this will not suffice, as very fine adjustments to actions can make significant differences in outcomes.<br /> <br /> Actions in the continuous space are generated using an MCTS algorithm, with the following steps:<br /> <br /> === Selection ===<br /> <br /> From a given state, the list of already-visited actions is denoted as A&lt;sub&gt;t&lt;/sub&gt;. Scores and the number of visits to each node are estimated using the equations below (the first equation shows the expectation of the end value for one-end games). These are likely estimated rather than simply taken from the MCTS statistics to help account for the differences in a continuous action space.<br /> <br /> [[File:curling_kernel_equations.png | 400px]]<br /> <br /> The UCB formula is then used to select an action to expand.<br /> <br /> The actions that are taken in the simulator appear to be drawn from a Gaussian centered around &lt;math&gt;a_t&lt;/math&gt;. This allows exploration in the continuous action space.<br /> <br /> === Expansion ===<br /> <br /> The authors use a variant of regular UCT for expansion. In this case, they expand a new node only when existing nodes have been visited a certain number of times. The authors utilize a widening approach to overcome problems with standard UCT performing a shallow search when there is a large action space.<br /> <br /> === Simulation ===<br /> <br /> Instead of simulating with a random game playout, the authors use the value network to estimate the likely score associated with a state. This speeds up simulation (assuming the network is well trained), as the game does not actually need to be simulated.<br /> <br /> === Backpropogation ===<br /> <br /> Standard backpropagation is used, updating both the values and number of visits stored in the path of parent nodes.<br /> <br /> <br /> == Supervised Learning ==<br /> <br /> During supervised training, data is gathered from the program AyumuGAT'16 (). This program is also based on both an MCTS algorithm, and a high-performance AI curling program. 400 000 state-action pairs were generated during this training.<br /> <br /> === Policy Network ===<br /> <br /> The policy network was trained to learn the action taken in each state. Here, the likelihood of the taken action was set to be 1, and the likelihood of other actions to be 0.<br /> <br /> === Value Network ===<br /> <br /> The value network was trained by 'd-depth simulations and bootstrapping of the prediction to handle the high variance in rewards resulting from a sequence of stochastic moves' (quote taken from paper). In this case, ''m'' state-action pairs were sampled from the training data. For each pair, &lt;math&gt;(s_t, a_t)&lt;/math&gt;, a state d' steps ahead was generated, &lt;math&gt;s_{t+d}&lt;/math&gt;. This process dealt with uncertainty by considering all actions in this rollout to have no uncertainty, and allowing uncertainty in the last action, ''a&lt;sub&gt;t+d-1&lt;/sub&gt;''. The value network is used to predict the value for this state, &lt;math&gt;z_t&lt;/math&gt;, and the value is used for learning the value at ''s&lt;sub&gt;t&lt;/sub&gt;''.<br /> <br /> === Policy-Value Network ===<br /> <br /> The policy-value network was trained to maximize the similarity of the predicted policy and value, and the actual policy and value from a state. The learning algorithm parameters are:<br /> <br /> * Algorithm: stochastic gradient descent<br /> * Batch size: 256<br /> * Momentum: 0.9<br /> * L2 regularization: 0.0001<br /> * Training time: ~100 epochs<br /> * Learning rate: initialized at 0.01, reduced twice<br /> <br /> A multi-task loss function was used. This takes the summation of the cross-entropy losses of each prediction:<br /> <br /> [[File:curling_loss_function.png | 300px]]<br /> <br /> == Self-Play Reinforcement Learning ==<br /> <br /> After initialization by supervised learning, the algorithm uses self-play to further train itself. During this training, the policy network learns probabilities from the MCTS process, while the value network learns from game outcomes.<br /> <br /> At a game state ''s&lt;sub&gt;t&lt;/sub&gt;'':<br /> <br /> 1) the algorithm outputs a prediction ''z&lt;sub&gt;t&lt;/sub&gt;''. This is en estimate of game score probabilities. It is based on similar past actions, and computed using kernel regression.<br /> <br /> 2) the algorithm outputs a prediction &lt;math&gt;\pi_t&lt;/math&gt;, representing a probability distribution of actions. These are proportional to estimated visit counts from MCTS, based on kernel density estimation.<br /> <br /> It is not clear how these predictions are created. It would seem likely that the policy-value network generates these, but the wording of the paper suggests they are generated from MCTS statistics.<br /> <br /> The policy-value network is updated by sampling data &lt;math&gt;(s, \pi, z)&lt;/math&gt; from recent history of self-play. The same loss function is used as before.<br /> <br /> It is not clear how the improved network is used, as MCTS seems to be the driving process at this point.<br /> <br /> == Long-Term Strategy Learning ==<br /> <br /> Finally, the authors implement a new strategy to augment their algorithm for long-term play. In this context, this refers to playing a game over many ends, where the strategy to win a single end may not be a good strategy to win a full game. For example, scoring one point in an end, while being one point ahead, gives the advantage to the other team in the next round (as they will throw the last stone). The other team could then use the advantage to score two points, taking the lead.<br /> <br /> The authors build a 'winning percentage' table. This table stores the percentage of games won, based on the number of ends left, and the difference in score (current team - opposing team). This can be computed iteratively and using the probability distribution estimation of one-end scores.<br /> <br /> == Final Algorithms ==<br /> <br /> The authors make use of the following versions of their algorithm:<br /> <br /> === KR-DL ===<br /> <br /> ''Kernel regression-deep learning'': This algorithm is trained only by supervised learning.<br /> <br /> === KR-DRL ===<br /> <br /> ''Kernel regression-deep reinforcement learning'': This algorithm is trained by supervised learning (ie: initialized as the KR-DL algorithm), and again on self-play. During self-play, each shot is selected after 400 MCTS simulations of k=20 randomly selected actions. Data for self-play was collected over a week on 5 GPUS and generated 5 million game positions. The policy-value network was continually updated using samples from the latest 1 million game positions.<br /> <br /> === KR-DRL-MES ===<br /> <br /> ''Kernel regression-deep reinforcement learning-multi-ends-strategy'': This algorithm makes use of the winning percentage table generated from self-play.<br /> <br /> = Testing and Results =<br /> The authors use data from the public program AyumuGAT’16 to test. Testing is done with a simulated curling program . This simulator does not deal with changing ice conditions, or sweeping, but does deal with stone trajectories and collisions.<br /> <br /> == Comparison of KR-DL-UCT and DL-UCT ==<br /> <br /> The first test compares an algorithm trained with kernel regression with an algorithm trained without kernel regression, to show the contribution that kernel regression adds to the performance. Both algorithms have networks initialised with the supervised learning, and then trained with two different algorithms for self-play. KR-DL-UCT uses the algorithm described above. The authors do not go into detail on how DL-UCT selects shots, but state that a constant is set to allow exploration.<br /> <br /> As an evaluation, both algorithms play 2000 games against the DL-UCT algorithm, which is frozen after supervised training. 1000 games are played with the algorithm taking the first, and 100 taking the 2nd, shots. The games were two-end games. The figure below shows each algorithm's winning percentage given different amounts of training data. While the DL-UCT outperforms the supervised-training-only-DL-UCT algorithm, the KR-DL-UCT algorithm performs much better.<br /> <br /> &lt;center&gt;[[File:curling_KR_test.png | 400px]]&lt;/center&gt;<br /> <br /> == Matches ==<br /> <br /> Finally, to test the performance of their multiple algorithms, the authors run matches between their algorithms and other existing programs. Each algorithm plays 200 matches against each other program, 100 of which are played as the first-playing team, and 100 as the second-playing team. Only 1 program was able to out-perform the KR-DRL algorithm. The authors state that this program, ''JiritsukunGAT'17'' also uses a deep network and hand-crafted features. However, the KR-DRL-MES algorithm was still able to out-perform this. Figure 4 shows the Elo ratings of the different programs. Note that the programs in blue are those created by the authors. They also played some games between their KR-DRL-MES and notable<br /> programs. Table 1, shows the details of the match results. ''JiritsukunGAT'17'' shows a similar level of performance but KR-DRL-MES is still the winner.<br /> <br /> <br /> <br /> [[File:curling_ratings.png|600px|thumb|center|Figure 4. Elo rating and winning percentages of our models and GAT rankers. Each match has 200 games (each program plays 100 pre-ordered games), because the player which has the last shot (the hammer shot) in each end would have an advantage.]]<br /> <br /> <br /> [[File:ttt.png|600px|thumb|center|Table 1. The 8-end game results for KR-DRL-MES against other programs alternating the opening player each game. The matches are held by following the rules of the latest GAT competition.]]<br /> <br /> = Conclusion &amp; Critique =<br /> <br /> The authors have presented a new framework which incorporates a deep neural network for learning game strategy with a kernel-based Monte Carlo tree search from a continuous space. Without the use of any hand-crafted feature, their policy-value network is successfully trained using supervised learning followed by reinforcement learning with a high-fidelity simulator for the Olympic sport of curling. Following are my critiques on the paper:<br /> <br /> == Strengths ==<br /> <br /> This algorithm out-performs other high-performance algorithms (including past competition champions).<br /> <br /> I think the paper does a decent job of comparing the performance of their algorithm to others. They are able to clearly show the benefits of many of their additions.<br /> <br /> The authors do seem to be able to adopt strategies similar to those used in Go and other games to the continuous action-space domain. In addition, the final strategy needs no hand-crafted features for learning.<br /> <br /> == Weaknesses ==<br /> <br /> Somtimes, I found this paper difficult to follow. One problem was that the algorithms were introduced first, and then how they were used was described. So when the paper stated that self-play shots were taken after 400 simulations, it seemed unclear what simulations were being run and at what stage of the algorithm (ex: MCTS simulations, simulations sped up by using the value network, full simulations on the curling simulator). In particular, both the MCTS statistics and the policy-value network could be used to estimate both action probabilities and state values, so it is difficult to tell which is used in which case. There was also no clear distinction between discrete-space actions and continuous-space actions.<br /> <br /> While I think the comparison of different algorithms was done well, I believe it still lacked significant details. There were one-off mentioned in the paper which would have been nice to see as results. These include the statement that having a policy-value network in place of two networks lead to better performance.<br /> <br /> At this point, the algorithms used still rely on initialization by a pre-made program.<br /> <br /> There was little theoretical development or justification done in this paper.<br /> <br /> While curling is an interesting choice for demonstrating the algorithm, the fact that the simulations used did not support many of the key points of curling (ice conditions, sweeping) seems very limited. Another game, such as pool, would likely have offered some of the same challenges but offered more high-fidelity simulations/training.<br /> <br /> While the spatial placements of stones were discretized in a grid, the curl of thrown stones was discretized to only +/-1. This seems like it may limit learning high- and low-spin moves. It should be noted that having zero spins is not commonly used, to the best of my knowledge.<br /> <br /> =References=<br /> # Lee, K., Kim, S., Choi, J. &amp; Lee, S. &quot;Deep Reinforcement Learning in Continuous Action Spaces: a Case Study in the Game of Simulated Curling.&quot; Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:2937-2946 (2018)<br /> # https://www.baeldung.com/java-monte-carlo-tree-search<br /> # https://jeffbradberry.com/posts/2015/09/intro-to-monte-carlo-tree-search/<br /> # https://int8.io/monte-carlo-tree-search-beginners-guide/<br /> # https://en.wikipedia.org/wiki/Monte_Carlo_tree_search<br /> # Silver, D., Huang, A., Maddison, C., Guez, A., Sifre, L.,Van Den Driessche, G., Schrittwieser, J., Antonoglou, I.,Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe,D., Nham, J., Kalchbrenner, N.,Sutskever, I., Lillicrap, T.,Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis,D. Mastering the game of go with deep neural networksand tree search. Nature, pp. 484–489, 2016.<br /> # Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou,I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L.,van den Driessche, G., Graepel, T., and Hassabis, D.Mastering the game of go without human knowledge.Nature, pp. 354–359, 2017.<br /> # Yamamoto, M., Kato, S., and Iizuka, H. Digital curling strategy based on game tree search. In Proceedings of the IEEE Conference on Computational Intelligence and Games, CIG, pp. 474–480, 2015.<br /> # Ohto, K. and Tanaka, T. A curling agent based on the montecarlo tree search considering the similarity of the best action among similar states. In Proceedings of Advances in Computer Games, ACG, pp. 151–164, 2017.<br /> # Ito, T. and Kitasei, Y. Proposal and implementation of digital curling. In Proceedings of the IEEE Conference on Computational Intelligence and Games, CIG, pp. 469–473, 2015.<br /> # Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning, ICML, pp. 448–456, 2015.<br /> # Nair, V. and Hinton, G. Rectified linear units improve restricted boltzmann machines.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Fix_your_classifier:_the_marginal_value_of_training_the_last_weight_layer&diff=42057 Fix your classifier: the marginal value of training the last weight layer 2018-11-30T15:43:26Z <p>C9sharma: /* Background */</p> <hr /> <div><br /> The code for the proposed model is available at https://github.com/eladhoffer/fix_your_classifier.<br /> <br /> =Introduction=<br /> <br /> Deep neural networks have become widely used for machine learning, achieving state-of-the-art results on many tasks. One of the most common tasks they are used for is classification. For example, convolutional neural networks (CNNs) are used to classify images to a semantic category. Typically, a learned affine transformation is placed at the end of such models, yielding a per-class value used for classification. This classifier can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more computational resources.<br /> <br /> =Brief Overview=<br /> <br /> In order to alleviate the aforementioned problem, the authors propose that the final layer of the classifier be fixed (upto a global scale constant). They argue that with little or no loss of accuracy for most classification tasks, the method provides significant memory and computational benefits. In addition, they show that by initializing the classifier with a Hadamard matrix the inference could be made faster as well.<br /> <br /> =Previous Work=<br /> <br /> Training NN models and using them for inference requires large amounts of memory and computational resources; thus, extensive amount of research has been done lately to reduce the size of networks which are as follows:<br /> <br /> * Weight sharing and specification (Han et al., 2015)<br /> <br /> * Mixed precision to reduce the size of the neural networks by half (Micikevicius et al., 2017)<br /> <br /> * Low-rank approximations to speed up CNN (Tai et al., 2015)<br /> <br /> * Quantization of weights, activations and gradients to further reduce computation during training (Hubara et al., 2016b; Li et al., 2016 and Zhou et al., 2016)<br /> <br /> Some of the past works have also put forward the fact that predefined (Park &amp; Sandberg, 1991) and random (Huang et al., 2006) projections can be used together with a learned affine transformation to achieve competitive results on many of the classification tasks. However, the authors' proposal in the current paper is quite reversed.<br /> <br /> =Background=<br /> <br /> A Convolutional Neural Network (CNN) is comprised of one or more convolutional layers (often with a subsampling step) and then followed by one or more fully connected layers as in a standard multilayer neural network. The architecture of a CNN is designed to take advantage of the 2D structure of an input image (or other 2D input such as a speech signal). This is achieved with local connections and tied weights followed by some form of pooling which results in translation invariant features. Another benefit of CNNs is that they are easier to train and have many fewer parameters than fully connected networks with the same number of hidden units. <br /> <br /> A CNN consists of a number of convolutional and subsampling layers optionally followed by fully connected layers. The input to a convolutional layer is a &lt;math&gt;m \times m \times r&lt;/math&gt; image where m is the height and width of the image and &lt;math&gt;r&lt;/math&gt; is the number of channels, e.g. an RGB image has &lt;math&gt;r=3&lt;/math&gt;. The convolutional layer will have &lt;math&gt;k&lt;/math&gt; filters (or kernels) of size &lt;math&gt;n \times n \times q&lt;/math&gt; where &lt;math&gt;n&lt;/math&gt; is smaller than the dimension of the image and &lt;math&gt;q&lt;/math&gt; can either be the same as the number of channels &lt;math&gt;r&lt;/math&gt; or smaller and may vary for each kernel. The size of the filters gives rise to the locally connected structure which are each convolved with the image to produce &lt;math&gt;k&lt;/math&gt; feature maps of size &lt;math&gt;m−n+1&lt;/math&gt;. Each map is then subsampled typically with mean or max pooling over &lt;math&gt;p \times p&lt;/math&gt; contiguous regions where &lt;math&gt;p&lt;/math&gt; ranges between 2 for small images (e.g. MNIST) and is usually not more than 5 for larger inputs. Either before or after the subsampling layer an additive bias and sigmoidal nonlinearity is applied to each feature map. <br /> <br /> CNNs are commonly used to solve a variety of spatial and temporal tasks. Earlier architectures of CNNs (LeCun et al., 1998; Krizhevsky et al., 2012) used a set of fully-connected layers at later stage of the network, presumably to allow classification based on global features of an image.<br /> <br /> <br /> <br /> == Shortcomings of the Final Classification Layer and its Solution ==<br /> <br /> Zeiler &amp; Fergus, 2014 showed that despite the enormous number of trainable parameters these layers add to the model, they are known to have a rather marginal impact on the final performance of the network.<br /> <br /> It has been shown previously that these layers could be easily compressed and reduced after a model was trained by simple means such as matrix decomposition and sparsification (Han et al., 2015). Modern architecture choices are characterized with the removal of most of the fully connected layers (Lin et al., 2013; Szegedy et al., 2015; He et al., 2016), that lead to better generalization and overall accuracy, together with a huge decrease in the number of trainable parameters. Additionally, numerous works showed that CNNs can be trained in a metric learning regime (Bromley et al., 1994; Schroff et al., 2015; Hoffer &amp; Ailon, 2015), where no explicit classification layer was introduced and the objective regarded only distance measures between intermediate representations. Hardt &amp; Ma (2017) suggested an all-convolutional network variant, where they kept the original initialization of the classification layer fixed with no negative impact on performance on the CIFAR-10 dataset.<br /> <br /> =Proposed Method=<br /> <br /> The aforementioned works provide evidence that fully-connected layers are in fact redundant and play a small role in learning and generalization. In this work, the authors have suggested that parameters used for the final classification transform are completely redundant, and can be replaced with a predetermined linear transform. This holds for even in large-scale models and classification tasks, such as recent architectures trained on the ImageNet benchmark (Deng et al., 2009).<br /> <br /> ==Using a Fixed Classifier==<br /> <br /> Suppose the final representation obtained by the network (the last hidden layer) is represented as &lt;math&gt;x = F(z;\theta)&lt;/math&gt; where &lt;math&gt;F&lt;/math&gt; is assumed to be a deep neural network with input z and parameters θ, e.g., a convolutional network, trained by backpropagation.<br /> <br /> In common NN models, this representation is followed by an additional affine transformation, &lt;math&gt;y = W^T x + b&lt;/math&gt; ,where &lt;math&gt;W&lt;/math&gt; and &lt;math&gt;b&lt;/math&gt; are also trained by back-propagation.<br /> <br /> For input &lt;math&gt;x&lt;/math&gt; of &lt;math&gt;N&lt;/math&gt; length, and &lt;math&gt;C&lt;/math&gt; different possible outputs, &lt;math&gt;W&lt;/math&gt; is required to be a matrix of &lt;math&gt;N ×<br /> C&lt;/math&gt;. Training is done using cross-entropy loss, by feeding the network outputs through a softmax activation<br /> <br /> &lt;math&gt;<br /> v_i = \frac{e^{y_i}}{\sum_{j}^{C}{e^{y_j}}}, i &amp;isin; &lt;/math&gt; { &lt;math&gt; {1, . . . , C} &lt;/math&gt; }<br /> <br /> and reducing the expected negative log likelihood with respect to ground-truth target &lt;math&gt; t &amp;isin; &lt;/math&gt; { &lt;math&gt; {1, . . . , C} &lt;/math&gt; },<br /> by minimizing the loss function:<br /> <br /> &lt;math&gt;<br /> L(x, t) = −\text{log}\ {v_t} = −{w_t}·{x} − b_t + \text{log} ({\sum_{j}^{C}e^{w_j . x + b_j}})<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;w_i&lt;/math&gt; is the &lt;math&gt;i&lt;/math&gt;-th column of &lt;math&gt;W&lt;/math&gt;.<br /> <br /> ==Choosing the Projection Matrix==<br /> <br /> To evaluate the conjecture regarding the importance of the final classification transformation, the trainable parameter matrix &lt;math&gt;W&lt;/math&gt; is replaced with a fixed orthonormal projection &lt;math&gt; Q &amp;isin; R^{N×C} &lt;/math&gt;, such that &lt;math&gt; &amp;forall; i &amp;ne; j : q_i · q_j = 0 &lt;/math&gt; and &lt;math&gt; || q_i ||_{2} = 1 &lt;/math&gt;, where &lt;math&gt;q_i&lt;/math&gt; is the &lt;math&gt;i&lt;/math&gt;th column of &lt;math&gt;Q&lt;/math&gt;. This is ensured by a simple random sampling and singular-value decomposition<br /> <br /> As the rows of classifier weight matrix are fixed with an equally valued &lt;math&gt;L_{2}&lt;/math&gt; norm, we find it beneficial<br /> to also restrict the representation of &lt;math&gt;x&lt;/math&gt; by normalizing it to reside on the &lt;math&gt;n&lt;/math&gt;-dimensional sphere:<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \hat{x} = \frac{x}{||x||_{2}}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This allows faster training and convergence, as the network does not need to account for changes in the scale of its weights. However, it has now an issue that &lt;math&gt;q_i · \hat{x} &lt;/math&gt; is bounded between −1 and 1. This causes convergence issues, as the softmax function is scale sensitive, and the network is affected by the inability to re-scale its input. This issue is amended with a fixed scale &lt;math&gt;T&lt;/math&gt; applied to softmax inputs &lt;math&gt;f(y) = softmax(\frac{1}{T}y)&lt;/math&gt;, also known as the ''softmax temperature''. However, this introduces an additional hyper-parameter which may differ between networks and datasets. So, the authors propose to introduce a single scalar parameter &lt;math&gt;\alpha&lt;/math&gt; to learn the softmax scale, effectively functioning as an inverse of the softmax temperature &lt;math&gt;\frac{1}{T}&lt;/math&gt;. The normalized weights and an additional scale coefficient are also used, specially using a single scale for all entries in the weight matrix. The additional vector of bias parameters &lt;math&gt;b &amp;isin; \mathbb{R}^{C}&lt;/math&gt; is kept the same and the model is trained using the traditional negative-log-likelihood criterion. Explicitly, the classifier output is now:<br /> <br /> &lt;center&gt;<br /> &lt;math&gt;<br /> v_i=\frac{e^{\alpha q_i &amp;middot; \hat{x} + b_i}}{\sum_{j}^{C} e^{\alpha q_j &amp;middot; \hat{x} + b_j}}, i &amp;isin; &lt;/math&gt; { &lt;math&gt; {1,...,C} &lt;/math&gt;}<br /> &lt;/center&gt;<br /> <br /> and the loss to be minimized is:<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> L(x, t) = -\alpha q_t &amp;middot; \frac{x}{||x||_{2}} + b_t + \text{log} (\sum_{i=1}^{C} \text{exp}((\alpha q_i &amp;middot; \frac{x}{||x||_{2}} + b_i)))<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;x&lt;/math&gt; is the final representation obtained by the network for a specific sample, and &lt;math&gt; t &amp;isin; &lt;/math&gt; { &lt;math&gt; {1, . . . , C} &lt;/math&gt; } is the ground-truth label for that sample. The behaviour of the parameter &lt;math&gt; \alpha &lt;/math&gt; over time, which is logarithmic in nature and has the same behavior exhibited by the norm of a learned classifier, is shown in<br /> [[Media: figure1_log_behave.png| Figure 1]].<br /> <br /> &lt;center&gt;[[File:figure1_log_behave.png]]&lt;/center&gt;<br /> <br /> When &lt;math&gt; -1 \le q_i · \hat{x} \le 1 &lt;/math&gt;, a possible cosine angle loss is <br /> <br /> &lt;center&gt;[[File:caloss.png]]&lt;/center&gt;<br /> <br /> But its final validation accuracy has slight decrease, compared to original models.<br /> <br /> ==Using a Hadmard Matrix==<br /> <br /> To recall, Hadmard matrix (Hedayat et al., 1978) &lt;math&gt; H &lt;/math&gt; is an &lt;math&gt; n × n &lt;/math&gt; matrix, where all of its entries are either +1 or −1.<br /> Furthermore, &lt;math&gt; H &lt;/math&gt; is orthogonal, such that &lt;math&gt; HH^{T} = nI_n &lt;/math&gt; where &lt;math&gt;I_n&lt;/math&gt; is the identity matrix. Instead of using the entire Hadmard matrix &lt;math&gt;H&lt;/math&gt;, a truncated version, &lt;math&gt; \hat{H} &amp;isin; &lt;/math&gt; {&lt;math&gt; {-1, 1}&lt;/math&gt;}&lt;math&gt;^{C \times N}&lt;/math&gt; where all &lt;math&gt;C&lt;/math&gt; rows are orthogonal as the final classification layer is such that:<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> y = \hat{H} \hat{x} + b<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This usage allows two main benefits:<br /> * A deterministic, low-memory and easily generated matrix that can be used for classification.<br /> * Removal of the need to perform a full matrix-matrix multiplication - as multiplying by a Hadamard matrix can be done by simple sign manipulation and addition.<br /> <br /> Here, &lt;math&gt;n&lt;/math&gt; must be a multiple of 4, but it can be easily truncated to fit normally defined networks. Also, as the classifier weights are fixed to need only 1-bit precision, it is now possible to focus our attention on the features preceding it.<br /> <br /> =Experimental Results=<br /> <br /> The authors have evaluated their proposed model on the following datasets:<br /> <br /> ==CIFAR-10/100==<br /> <br /> ===About the Dataset===<br /> <br /> CIFAR-10 is an image classification benchmark dataset containing 50,000 training images and 10,000 test images. The images are in color and contain 32×32 pixels. There are 10 possible classes of various animals and vehicles. CIFAR-100 holds the same number of images of same size, but contains 100 different classes.<br /> <br /> ===Training Details===<br /> <br /> The authors trained a residual network ( He et al., 2016) on the CIFAR-10 dataset. The network depth was 56 and the same hyper-parameters as in the original work were used. A comparison of the two variants, i.e., the learned classifier and the proposed classifier with a fixed transformation is shown in [[Media: figure1_resnet_cifar10.png | Figure 2]].<br /> <br /> &lt;center&gt;[[File: figure1_resnet_cifar10.png]]&lt;/center&gt;<br /> <br /> These results demonstrate that although the training error is considerably lower for the network with learned classifier, both models achieve the same classification accuracy on the validation set. The authors' conjecture is that with the new fixed parameterization, the network can no longer increase the norm of a given sample’s representation - thus learning its label requires more effort. As this may happen for specific seen samples - it affects only training error.<br /> <br /> The authors also compared using a fixed scale variable &lt;math&gt;\alpha &lt;/math&gt; at different values vs. the learned parameter. Results for &lt;math&gt; \alpha = &lt;/math&gt; {0.1, 1, 10} are depicted in [[Media: figure3_alpha_resnet_cifar.png| Figure 3]] for both training and validation error and as can be seen, similar validation accuracy can be obtained using a fixed scale value (in this case &lt;math&gt;\alpha &lt;/math&gt;= 1 or 10 will suffice) at the expense of another hyper-parameter to seek. In all the further experiments the scaling parameter &lt;math&gt; \alpha &lt;/math&gt; was regularized with the same weight decay coefficient used on original classifier. Although learning the scale is not necessary, but it will help convergence during training.<br /> <br /> &lt;center&gt;[[File: figure3_alpha_resnet_cifar.png]]&lt;/center&gt;<br /> <br /> The authors then train the model on CIFAR-100 dataset. They used the DenseNet-BC model from Huang et al. (2017) with depth of 100 layers and k = 12. The higher number of classes caused the number of parameters to grow and encompassed about 4% of the whole model. However, validation accuracy for the fixed-classifier model remained equally good as the original model, and the same training curve was observed as earlier.<br /> <br /> ==IMAGENET==<br /> <br /> ===About the Dataset===<br /> <br /> The Imagenet dataset introduced by Deng et al. (2009) spans over 1000 visual classes, and over 1.2 million samples. This is supposedly a more challenging dataset to work on as compared to CIFAR-10/100.<br /> <br /> ===Experiment Details===<br /> <br /> The authors evaluated their fixed classifier method on Imagenet using Resnet50 by He et al. (2016) and Densenet169 model (Huang et al., 2017) as described in the original work. Using a fixed classifier removed approximately 2-million parameters were from the model, accounting for about 8% and 12 % of the model parameters respectively. The experiments revealed similar trends as observed on CIFAR-10.<br /> <br /> For a more stricter evaluation, the authors also trained a Shufflenet architecture (Zhang et al., 2017b), which was designed to be used in low memory and limited computing platforms and has parameters making up the majority of the model. They were able to reduce the parameters to 0.86 million as compared to 0.96 million parameters in the final layer of the original model. Again, the proposed modification in the original model gave similar convergence results on validation accuracy. Interestingly, this method allowed Imagenet training in an under-specified regime, where there are<br /> more training samples than number of parameters. This is an unconventional regime for modern deep networks, which are usually over-specified to have many more parameters than training samples (Zhang et al., 2017a).<br /> <br /> The overall results of the fixed-classifier are summarized in [[Media: table1_fixed_results.png | Table 1]].<br /> <br /> &lt;center&gt;[[File: table1_fixed_results.png]]&lt;/center&gt;<br /> <br /> ==Language Modelling==<br /> <br /> Recent works have empirically found that using the same weights for both word embedding and classifier can yield equal or better results than using a separate pair of weights. So the authors experimented with fix-classifiers on language modelling as it also requires classification of all possible tokens available in the task vocabulary. They trained a recurrent model with 2-layers of LSTM (Hochreiter &amp; Schmidhuber, 1997) and embedding + hidden size of 512 on the WikiText2 dataset (Merity et al., 2016), using same settings as in Merity et al. (2017). WikiText2 dataset contains about 33K different words, so the number of parameters expected in the embedding and classifier layer was about 34-million. This number is about 89% of the total number of parameters used for the whole model which is 38-million. However, using a random orthogonal transform yielded poor results compared to learned embedding. This was suspected to be due to semantic relationships captured in the embedding layer of language models, which is not the case in image classification task. The intuition was further confirmed by the much better results when pre-trained embeddings using word2vec algorithm by Mikolov et al. (2013) or PMI factorization as suggested by Levy &amp; Goldberg (2014), were used.<br /> <br /> &lt;center&gt;[[File: language.png]]&lt;/center&gt;<br /> <br /> =Discussion=<br /> <br /> ==Implications and Use Cases==<br /> <br /> With the increasing number of classes in the benchmark datasets, computational demands for the final classifier will increase as well. In order to understand the problem better, the authors observe the work by Sun et al. (2017), which introduced JFT-300M - an internal Google dataset with over 18K different classes. Using a Resnet50 (He et al., 2016), with a 2048 sized representation led to a model with over 36M parameters meaning that over 60% of the model parameters resided in the final classification layer. Sun et al. (2017) also describe the difficulty in distributing so many parameters over the training servers involving a non-trivial overhead during synchronization of the model for update. The authors claim that the fixed-classifier would help considerably in this kind of scenario - where using a fixed classifier removes the need to do any gradient synchronization for the final layer. Furthermore, introduction of Hadamard matrix removes the need to save the transformation altogether, thereby, making it more efficient and allowing considerable memory and computational savings.<br /> <br /> ==Possible Caveats==<br /> <br /> The good performance of fixed-classifiers relies on the ability of the preceding layers to learn separable representations. This could be affected when when the ratio between learned features and number of classes is small – that is, when &lt;math&gt; C &gt; N&lt;/math&gt;. However, they tested their method in such cases and their model performed well and provided good results.<br /> Another factor that can affect the performance of their model using a fixed classifier is when the classes are highly correlated. In that case, the fixed classifier actually cannot support correlated classes and thus, the network could have some difficulty to learn. For a language model, word classes tend to have highly correlated instances, which also lead to difficult learning process.<br /> <br /> Also, this proposed approach will only eliminate the computation of the classifier weights, so when the classes are fewer, the computation saving effect will not be readily apparent.<br /> <br /> ==Future Work==<br /> <br /> <br /> The use of fixed classifiers might be further simplified in Binarized Neural Networks (Hubara et al., 2016a), where the activations and weights are restricted to ±1 during propagations. In that case the norm of the last hidden layer would be constant for all samples (equal to the square root of the hidden layer width). The constant could then be absorbed into the scale constant &lt;math&gt;\alpha&lt;/math&gt;, and there is no need in a per-sample normalization.<br /> <br /> Additionally, more efficient ways to learn a word embedding should also be explored where similar redundancy in classifier weights may suggest simpler forms of token representations - such as low-rank or sparse versions.<br /> <br /> A related paper was published that claims that fixing most of the parameters of the neural network achieves comparable results with learning all of them [A. Rosenfeld and J. K. Tsotsos]<br /> <br /> =Conclusion=<br /> <br /> In this work, the authors argue that the final classification layer in deep neural networks is redundant and suggest removing the parameters from the classification layer. The empirical results from experiments on the CIFAR and IMAGENET datasets suggest that such a change lead to little or almost no decline in the performance of the architecture. Furthermore, using a Hadmard matrix as classifier might lead to some computational benefits when properly implemented, and save memory otherwise spent on large amount of transformation coefficients.<br /> <br /> Another possible scope of research that could be pointed out for future could be to find new efficient methods to create pre-defined word embeddings, which require huge amount of parameters that can possibly be avoided when learning a new task. Therefore, more emphasis should be given to the representations learned by the non-linear parts of the neural networks - upto the final classifier, as it seems highly redundant.<br /> <br /> =Critique=<br /> <br /> The paper proposes an interesting idea that has a potential use case when designing memory-efficient neural networks. The experiments shown in the paper are quite rigorous and provide support to the authors' claim. However, it would have been more helpful if the authors had described a bit more about efficient implementation of the Hadamard matrix and how to scale this method for larger datasets (cases with &lt;math&gt; C &gt;N&lt;/math&gt;).<br /> <br /> Moreover, one of the main intuitions of the paper has introduced to be computational cost but it has left out to compare a fixed and learned classifier based on the computational cost and then investigate whether it worth the drop in performance or not considering the fact that not always the output can be degraded because of need for speed! At least a discussion on this issue is expected.<br /> <br /> On the other hand, the computational cost and performance change after fixation of classifier could be related to dataset and the nature and complexity of it. Mostly, having 1000 classes makes the classification more crucial than 2 classes. An evaluation on this topic is also needed.<br /> <br /> =References=<br /> <br /> Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXiv:1710.03667, 2017.<br /> <br /> Peter Bartlett, Dylan J Foster, and Matus Telgarsky. Spectrally-normalized margin bounds for neural networks. arXiv preprint arXiv:1706.08498, 2017.<br /> <br /> Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Sackinger, and Roopak Shah. Signature verification using a” siamese” time delay neural network. In Advances in Neural Information Processing Systems, pp. 737–744, 1994.<br /> <br /> Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in Neural Information Processing Systems, pp. 3123–3131, 2015.<br /> <br /> Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248–255. IEEE, 2009.<br /> <br /> Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization. arXiv preprint arXiv:1705.09280, 2017.<br /> <br /> Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.<br /> <br /> Moritz Hardt and Tengyu Ma. Identity matters in deep learning. 2017.<br /> <br /> Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.<br /> <br /> A Hedayat, WD Wallis, et al. Hadamard matrices and their applications. The Annals of Statistics, 6<br /> (6):1184–1238, 1978.<br /> <br /> Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. ¨ Neural computation, 9(8): 1735–1780, 1997.<br /> <br /> Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In International Workshop on Similarity-Based Pattern Recognition, pp. 84–92. Springer, 2015.<br /> <br /> Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. 2017.<br /> <br /> Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.<br /> <br /> Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.<br /> <br /> Guang-Bin Huang, Qin-Yu Zhu, and Chee-Kheong Siew. Extreme learning machine: theory and applications. Neurocomputing, 70(1):489–501, 2006.<br /> <br /> Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. In Advances in Neural Information Processing Systems 29 (NIPS’16), 2016a.<br /> <br /> Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. arXiv preprint arXiv:1609.07061, 2016b.<br /> <br /> Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. arXiv preprint arXiv:1611.01462, 2016.<br /> <br /> Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014.<br /> <br /> Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.<br /> <br /> Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.<br /> <br /> Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to ´ document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.<br /> <br /> Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Advances in neural information processing systems, pp. 2177–2185, 2014.<br /> <br /> Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint arXiv:1605.04711, 2016.<br /> <br /> Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.<br /> <br /> Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.<br /> <br /> Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and Optimizing LSTM Language Models. arXiv preprint arXiv:1708.02182, 2017.<br /> <br /> Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017.<br /> <br /> Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed tations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013.<br /> <br /> Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring generalization in deep learning. arXiv preprint arXiv:1706.08947, 2017.<br /> Jooyoung Park and Irwin W Sandberg. Universal approximation using radial-basis-function networks. Neural computation, 3(2):246–257, 1991.<br /> <br /> Ofir Press and Lior Wolf. Using the output embedding to improve language models. EACL 2017,<br /> pp. 157, 2017.<br /> <br /> Itay Safran and Ohad Shamir. On the quality of the initial basin in overspecified neural networks. In International Conference on Machine Learning, pp. 774–782, 2016.<br /> <br /> Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901–909, 2016.<br /> <br /> Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823, 2015.<br /> <br /> Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. Theoretical insights into the optimization landscape of over-parameterized shallow neural networks. arXiv preprint arXiv:1707.04926, 2017.<br /> <br /> Daniel Soudry and Yair Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks. arXiv preprint arXiv:1605.08361, 2016.<br /> <br /> Daniel Soudry and Elad Hoffer. Exponentially vanishing sub-optimal local minima in multilayer neural networks. arXiv preprint arXiv:1702.05777, 2017.<br /> <br /> Daniel Soudry, Elad Hoffer, and Nathan Srebro. The implicit bias of gradient descent on separable data. 2018.<br /> <br /> Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014.<br /> <br /> Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. arXiv preprint arXiv:1707.02968, 2017.<br /> <br /> Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9, 2015.<br /> <br /> Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826, 2016.<br /> <br /> Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. Convolutional neural networks with lowrank regularization. arXiv preprint arXiv:1511.06067, 2015.<br /> <br /> Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. 2017.<br /> Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. arXiv preprint arXiv:1705.08292, 2017.<br /> <br /> Bo Xie, Yingyu Liang, and Le Song. Diversity leads to generalization in neural networks. arXiv preprint arXiv:1611.03131, 2016.<br /> <br /> Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818–833. Springer, 2014. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017a. URL https://arxiv.org/abs/1611.03530.<br /> <br /> Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. arXiv preprint arXiv:1707.01083, 2017b.<br /> <br /> Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016.<br /> <br /> A. Rosenfeld and J. K. Tsotsos, “Intriguing properties of randomly weighted networks: Generalizing while learning next to nothing,” arXiv preprint arXiv:1802.00844, 2018.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=File:griddomain.png&diff=42056 File:griddomain.png 2018-11-30T15:32:55Z <p>C9sharma: </p> <hr /> <div></div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=learn_what_not_to_learn&diff=42055 learn what not to learn 2018-11-30T15:32:07Z <p>C9sharma: /* Experiment */</p> <hr /> <div>=Introduction=<br /> <br /> In reinforcement learning, it is often difficult for an agent to learn when the action space is large, especially the difficulties from function approximation and exploration. In some cases many actions are irrelevant and it is sometimes easier for the algorithm to learn which action not to take. The paper proposes a new reinforcement learning approach for dealing with large action spaces based on action elimination by restricting the available actions in each state to a subset of the most likely ones. There is a core assumption being made in the proposed method that it is easier to predict which actions in each state are invalid or inferior and use that information for control. More specifically, it proposes a system that learns the approximation of a Q-function and concurrently learns to eliminate actions. The method utilizes an external elimination signal which incorporates domain-specific prior knowledge. For example, in parser-based text games, the parser gives feedback regarding irrelevant actions after the action is played (e.g., Player: &quot;Climb the tree.&quot; Parser: &quot;There are no trees to climb&quot;). Then a machine learning model can be trained to generalize to unseen states. <br /> <br /> The paper focuses on tasks where both states and the actions are natural language. It introduces a novel deep reinforcement learning approach which has a Deep Q-Network (DQN) and an Action Elimination Network (AEN), both using the Convolutional Neural Networks (CNN) for Natural Language Processing (NLP) tasks. The AEN is trained to predict invalid actions, supervised by the elimination signal from the environment. The proposed method uses the final layer activations of AEN to build a linear contextual bandit model which allows the elimination of sub-optimal actions with high probability. '''Note that the core assumption is that it is easy to predict which actions are invalid or inferior in each state and leverage that information for control.'''<br /> <br /> The text-based game called &quot;Zork&quot;, which lets players to interact with a virtual world through a text based interface, is tested by using the elimination framework. <br /> In this game, the player explores an environment using imagination of the text he/she reads. For more info, you can watch this video: [https://www.youtube.com/watch?v=xzUagi41Wo0 Zork].<br /> <br /> The AEN algorithm has achieved faster learning rate than the baseline agents through eliminating irrelevant actions.<br /> <br /> Below shows an example for the Zork interface:<br /> <br /> [[File:lnottol_fig1.png|500px|center]]<br /> <br /> All states and actions are given in natural language. Input for the game contains more than a thousand possible actions in each state since player can type anything.<br /> <br /> =Related Work=<br /> <br /> Text-Based Games(TBG): The state of the environment in TBG is described by simple language. The player interacts with the environment with text command which respects a pre-defined grammar. A popular example is Zork which has been tested in the paper. TBG is a good research intersection of RL and NLP, it requires language understanding, long-term memory, planning, exploration, affordability extraction and common sense. It also often introduce stochastic dynamics to increase randomness.<br /> <br /> Representations for TBG: Good word representation is necessary in order to learn control policies from texts. Previous work on TBG used pre-trained embeddings directly for control, other works combined pre-trained embedding with neural networks.<br /> <br /> DRL with linear function approximation: DRL methods such as the DQN have achieved state-of-the-art results in a variety of challenging, high-dimensional domains. This is mainly because neural networks can learn rich domain representations for value function and policy. On the other hand, linear representation batch reinforcement learning methods are more stable and accurate, while feature engineering is necessary.<br /> <br /> RL in Large Action Spaces: Prior work concentrated on factorizing the action space into binary subspace(Pazis and Parr, 2011; Dulac-Arnold et al., 2012; Lagoudakis and Parr, 2003), other works proposed to embed the discrete actions into a continuous space, then choose the nearest discrete action according to the optimal actions in the continuous space(Dulac-Arnold et al., 2015; Van Hasselt and Wiering, 2009). He et. al. (2015)extended DQN to unbounded(natural language) action spaces.<br /> Learning to eliminate actions was first mentioned by (Even-Dar, Mannor, and Mansour, 2003). They proposed to learn confidence intervals around the value function in each state. Lipton et al.(2016a) proposed to learn a classifier that detects hazardous state and then use it to shape the reward. Fulda et al.(2017) presented a method for affordability extraction via inner products of pre-trained word embedding.<br /> <br /> =Action Elimination=<br /> <br /> The approach in the paper builds on the standard Reinforcement Learning formulation. At each time step &lt;math&gt;t&lt;/math&gt;, the agent observes state &lt;math display=&quot;inline&quot;&gt;s_t &lt;/math&gt; and chooses a discrete action &lt;math display=&quot;inline&quot;&gt;a_t\in\{1,...,|A|\} &lt;/math&gt;. Then, after action execution, the agent obtains a reward &lt;math display=&quot;inline&quot;&gt;r_t(s_t,a_t) &lt;/math&gt; and observes next state &lt;math display=&quot;inline&quot;&gt;s_{t+1} &lt;/math&gt; according to a transition kernel &lt;math&gt;P(s_{t+1}|s_t,a_t)&lt;/math&gt;. The goal of the algorithm is to learn a policy &lt;math display=&quot;inline&quot;&gt;\pi(a|s) &lt;/math&gt; which maximizes the expected future discounted cumulative return &lt;math display=&quot;inline&quot;&gt;V^\pi(s)=E^\pi[\sum_{t=0}^{\infty}\gamma^tr(s_t,a_t)|s_0=s]&lt;/math&gt;, where &lt;math&gt; 0&lt; \gamma &lt;1 &lt;/math&gt;. The Q-function is &lt;math display=&quot;inline&quot;&gt;Q^\pi(s,a)=E^\pi[\sum_{t=0}^{\infty}\gamma^tr(s_t,a_t)|s_0=s,a_0=a]&lt;/math&gt;, and it can be optimized by Q-learning algorithm.<br /> <br /> After executing an action, the agent observes a binary elimination signal &lt;math&gt;e(s, a)&lt;/math&gt; to determine which actions not to take. It equals 1 if action &lt;math&gt;a&lt;/math&gt; may be eliminated in state &lt;math&gt;s&lt;/math&gt; (and 0 otherwise). The signal helps mitigating the problem of large discrete action spaces. We start with the following definitions:<br /> <br /> '''Definition 1:''' <br /> <br /> Valid state-action pairs with respect to an elimination signal are state action pairs which the elimination process should not eliminate. <br /> <br /> The set of valid state-action pairs contains all of the state-action pairs that are a part of some optimal policy, i.e., only strictly suboptimal state-actions can be invalid.<br /> <br /> '''Definition 2:'''<br /> <br /> Admissible state-action pairs with respect to an elimination algorithm are state action pairs which the elimination algorithm does not eliminate.<br /> <br /> '''Definition 3:'''<br /> <br /> Action Elimination Q-learning is a Q-learning algorithm which updates only admissible state-action pairs and chooses the best action in the next state from its admissible actions. We allow the base Q-learning algorithm to be any algorithm that converges to &lt;math display=&quot;inline&quot;&gt;Q^*&lt;/math&gt; with probability 1 after observing each state-action infinitely often.<br /> <br /> ==Advantages of Action Elimination==<br /> <br /> The main advantages of action elimination is that it allows the agent to overcome some of the main difficulties in large action spaces which are Function Approximation and Sample Complexity. <br /> <br /> Function approximation: Errors in the Q-function estimates may cause the learning algorithm to converge to a suboptimal policy, this phenomenon becomes more noticeable when the action space is large. Action elimination mitigates this effect by taking the max operator only on valid actions, thus, reducing potential overestimation errors. Besides, by ignoring the invalid actions, the function approximation can also learn a simpler mapping (i.e., only the Q-values of the valid state-action pairs) leading to faster convergence and better solution.<br /> <br /> Sample complexity: The sample complexity measures the number of steps during learning, in which the policy is not &lt;math display=&quot;inline&quot;&gt;\epsilon&lt;/math&gt;-optimal. Assume that there are &lt;math&gt;A'&lt;/math&gt; actions that should be eliminated and are &lt;math&gt;\epsilon&lt;/math&gt;-optimal, i.e. their value is at least &lt;math&gt;V^*(s)-\epsilon&lt;/math&gt;. The invalid action often returns no reward and doesn't change the state, (Lattimore and Hutter, 2012)resulting in an action gap of &lt;math display=&quot;inline&quot;&gt;\epsilon=(1-\gamma)V^*(s)&lt;/math&gt;, and this translates to &lt;math display=&quot;inline&quot;&gt;V^*(s)^{-2}(1-\gamma)^{-5}log(1/\delta)&lt;/math&gt; wasted samples for learning each invalid state-action pair. Practically, elimination algorithm can eliminate these invalid actions and therefore speed up the learning process approximately by &lt;math display=&quot;inline&quot;&gt;A/A'&lt;/math&gt;.<br /> <br /> Because it is difficult to embed the elimination signal into the MDP, the authors use contextual multi-armed bandits to decouple the elimination signal from the MDP, which can correctly eliminate actions when applying standard Q learning into learning process.<br /> <br /> ==Action elimination with contextual bandits==<br /> <br /> Contextual bandit problem is a famous probability problem, and is a natural extension from the multi-arm bandit problem.<br /> <br /> Let &lt;math display=&quot;inline&quot;&gt;x(s_t)\in R^d &lt;/math&gt; be the feature representation of &lt;math display=&quot;inline&quot;&gt;s_t &lt;/math&gt;. We assume that under this representation there exists a set of parameters &lt;math display=&quot;inline&quot;&gt;\theta_a^*\in \mathbb{R}^d &lt;/math&gt; such that the elimination signal in state &lt;math display=&quot;inline&quot;&gt;s_t &lt;/math&gt; is &lt;math display=&quot;inline&quot;&gt;e_t(s_t,a) = \theta_a^{*T}x(s_t)+\eta_t &lt;/math&gt;, where &lt;math display=&quot;inline&quot;&gt; \Vert\theta_a^*\Vert_2\leq S&lt;/math&gt;. &lt;math display=&quot;inline&quot;&gt;\eta_t&lt;/math&gt; is an R-subgaussian random variable with zero mean that models additive noise to the elimination signal. When there is no noise in the elimination signal, R=0. Otherwise, &lt;math display=&quot;inline&quot;&gt;R\leq 1&lt;/math&gt; since the elimination signal is bounded in [0,1]. Assume the elimination signal satisfies: &lt;math display=&quot;inline&quot;&gt;0\leq E[e_t(s_t,a)]\leq l &lt;/math&gt; for any valid action and &lt;math display=&quot;inline&quot;&gt; u\leq E[e_t(s_t, a)]\leq 1&lt;/math&gt; for any invalid action. And &lt;math display=&quot;inline&quot;&gt; l\leq u&lt;/math&gt;. Denote by &lt;math display=&quot;inline&quot;&gt;X_{t,a}&lt;/math&gt; as the matrix whose rows are the observed state representation vectors in which action a was chosen, up to time t. &lt;math display=&quot;inline&quot;&gt;E_{t,a}&lt;/math&gt; as the vector whose elements are the observed state representation elimination signals in which action a was chosen, up to time t. Denote the solution to the regularized linear regression &lt;math display=&quot;inline&quot;&gt;\Vert X_{t,a}\theta_{t,a}-E_{t,a}\Vert_2^2+\lambda\Vert \theta_{t,a}\Vert_2^2 &lt;/math&gt; (for some &lt;math display=&quot;inline&quot;&gt;\lambda&gt;0&lt;/math&gt;) by &lt;math display=&quot;inline&quot;&gt;\hat{\theta}_{t,a}=\bar{V}_{t,a}^{-1}X_{t,a}^TE_{t,a} &lt;/math&gt;, where &lt;math display=&quot;inline&quot;&gt;\bar{V}_{t,a}=\lambda I + X_{t,a}^TX_{t,a}&lt;/math&gt;.<br /> <br /> <br /> According to Theorem 2 in (Abbasi-Yadkori, Pal, and Szepesvari, 2011), &lt;math display=&quot;inline&quot;&gt;|\hat{\theta}_{t,a}^{T}x(s_t)-\theta_a^{*T}x(s_t)|\leq\sqrt{\beta_t(\delta)x(s_t)^T\bar{V}_{t,a}^{-1}x(s_t)}\ \forall t&gt;0&lt;/math&gt;, where &lt;math display=&quot;inline&quot;&gt;\sqrt{\beta_t(\delta)}=R\sqrt{2\ \text{log}(\text{det}(\bar{V}_{t,a})^{1/2}\text{det}(\lambda I)^{-1/2}/\delta)}+\lambda^{1/2}S&lt;/math&gt;, with probability of at least &lt;math display=&quot;inline&quot;&gt;1-\delta&lt;/math&gt;. If &lt;math display=&quot;inline&quot;&gt;\forall s\ ,\Vert x(s)\Vert_2 \leq L&lt;/math&gt;, then &lt;math display=&quot;inline&quot;&gt;\beta_t&lt;/math&gt; can be bounded by &lt;math display=&quot;inline&quot;&gt;\sqrt{\beta_t(\delta)} \leq R \sqrt{d\ \text{log}(1+tL^2/\lambda/\delta)}+\lambda^{1/2}S&lt;/math&gt;. Next, define &lt;math display=&quot;inline&quot;&gt;\tilde{\delta}=\delta/k&lt;/math&gt; and bound this probability for all the actions. i.e., &lt;math display=&quot;inline&quot;&gt;\forall a,t&gt;0&lt;/math&gt;<br /> <br /> &lt;math display=&quot;inline&quot;&gt;Pr(|\hat{\theta}_{t-1,a}^{T}x(s_t)-\theta_{t-1, a}^{*T}x(s_t)|\leq\sqrt{\beta_t(\tilde\delta)x(s_t)^T\bar{V}_{t - 1,a}^{-1}x(s_t)}) \leq 1-\delta&lt;/math&gt;<br /> <br /> Recall that &lt;math display=&quot;inline&quot;&gt;E[e_t(s,a)]=\theta_a^{*T}x(s_t)\leq l&lt;/math&gt; if a is a valid action. Then we can eliminate action a at state &lt;math display=&quot;inline&quot;&gt;s_t&lt;/math&gt; if it satisfies:<br /> <br /> &lt;math display=&quot;inline&quot;&gt;\hat{\theta}_{t-1,a}^{T}x(s_t)-\sqrt{\beta_{t-1}(\tilde\delta)x(s_t)^T\bar{V}_{t-1,a}^{-1}x(s_t)})&gt;l&lt;/math&gt;<br /> <br /> with probability &lt;math display=&quot;inline&quot;&gt;1-\delta&lt;/math&gt; that we never eliminate any valid action. Note that &lt;math display=&quot;inline&quot;&gt;l, u&lt;/math&gt; are not known. In practice, choosing &lt;math display=&quot;inline&quot;&gt;l&lt;/math&gt; to be 0.5 should suffice.<br /> <br /> ==Concurrent Learning==<br /> In fact, Q-learning and contextual bandit algorithms can learn simultaneously, resulting in the convergence of both algorithms, i.e., finding an optimal policy and a minimal valid action space. <br /> <br /> If the elimination is done based on the concentration bounds of the linear contextual bandits, it can be ensured that Action Elimination Q-learning converges, as shown in Proposition 1.<br /> <br /> '''Proposition 1:'''<br /> <br /> Assume that all state action pairs (s,a) are visited infinitely often, unless eliminated according to &lt;math display=&quot;inline&quot;&gt;\hat{\theta}_{t-1,a}^Tx(s)-\sqrt{\beta_{t-1}(\tilde{\delta})x(s)^T\bar{V}_{t-1,a}^{-1}x(s))}&gt;l&lt;/math&gt;. Then, with a probability of at least &lt;math display=&quot;inline&quot;&gt;1-\delta&lt;/math&gt;, action elimination Q-learning converges to the optimal Q-function for any valid state-action pairs. In addition, actions which should be eliminated are visited at most &lt;math display=&quot;inline&quot;&gt;T_{s,a}(t)\leq 4\beta_t/(u-l)^2<br /> +1&lt;/math&gt; times.<br /> <br /> Notice that when there is no noise in the elimination signal(R=0), we correctly eliminate actions with probability 1. so invalid actions will be sampled a finite number of times.<br /> <br /> =Method=<br /> <br /> The assumption that &lt;math display=&quot;inline&quot;&gt;e_t(s_t,a)=\theta_a^{*T}x(s_t)+\eta_t &lt;/math&gt; generally does not hold when using raw features like word2vec. So the paper proposes to use the neural network's last layer as feature representation of states. A practical challenge here is that the features must be fixed over time when used by the contextual bandit. So batch-updates framework(Levine et al., 2017;Riquelme, Tucker, and Snoek, 2018) is used, where a new contextual bandit model is learned for every few steps that uses the last layer activation of the AEN as features.<br /> <br /> ==Architecture of action elimination framework==<br /> <br /> [[File:lnottol_fig1b.png|300px|center]]<br /> <br /> After taking action &lt;math display=&quot;inline&quot;&gt;a_t&lt;/math&gt;, the agent observes &lt;math display=&quot;inline&quot;&gt;(r_t,s_{t+1},e_t)&lt;/math&gt;. The agent use it to learn two function approximation deep neural networks: A DQN and an AEN. AEN provides an admissible actions set &lt;math display=&quot;inline&quot;&gt;A'&lt;/math&gt; to the DQN, which uses this set to decide how to act and learn. The architecture for both the AEN and DQN is an NLP CNN(100 convolutional filters for AEN and 500 for DQN, with three different 1D kernels of length (1,2,3)), based on(Kim, 2014). The state is represented as a sequence of words, composed of the game descriptor and the player's inventory. These are truncated or zero padded to a length of 50 descriptor + 15 inventory words and each word is embedded into continuous vectors using word2vec in &lt;math display=&quot;inline&quot;&gt;R^{300}&lt;/math&gt;. The features of the last four states are then concatenated together such that the final state representations s are in &lt;math display=&quot;inline&quot;&gt;R^{78000}&lt;/math&gt;. The AEN is trained to minimize the MSE loss, using the elimination signal as a label. The code, the Zork domain, and the implementation of the elimination signal can be found [https://github.com/TomZahavy/CB_AE_DQN here.]<br /> <br /> ==Psuedocode of the Algorithm==<br /> <br /> [[File:lnottol_fig2.png|750px|center]]<br /> <br /> AE-DQN trains two networks: a DQN denoted by Q and an AEN denoted by E. The algorithm creates a linear contextual bandit model from it every L iterations with procedure AENUpdate(). This procedure uses the activations of the last hidden layer of E as features, which are then used to create a contextual linear bandit model.AENUpdate() then solved this model and plugin it into the target AEN. The contextual linear bandit model &lt;math display=&quot;inline&quot;&gt;(E^-,V)&lt;/math&gt; is then used to eliminate actions via the ACT() and Target() functions. ACT() follows an &lt;math display=&quot;inline&quot;&gt;\epsilon&lt;/math&gt;-greedy mechanism on the admissible actions set. For exploitation, it selects the action with highest Q-value by taking an argmax on Q-values among &lt;math display=&quot;inline&quot;&gt;A'&lt;/math&gt;. For exploration, it selects an action uniformly from &lt;math display=&quot;inline&quot;&gt;A'&lt;/math&gt;. The targets() procedure is estimating the value function by taking max over Q-values only among admissible actions, hence, reducing function approximation errors.<br /> <br /> =Experiments=<br /> ==Grid Domain==<br /> The authors start by evaluating our algorithm on a small grid world domain with 9 rooms, where they ca analyze the effect of the action elimination (visualization can be found in the appendix). In this domain, the agent starts at the center of the grid and needs to navigate to its upper-left corner. On every step, the agent suffers a penalty of (−1), with a terminal reward of 0. Prior to the game, the states are randomly divided into K categories. The environment has 4K navigation actions, 4 for each category, each with a probability to move in a random direction. If the chosen action belongs to the same category as the state, the action is performed correctly in probability pTc = 0.75. Otherwise, it will be performed correctly in probability pFc = 0.5. If the action does not fit the state category, the elimination signal equals 1, and if the action and state belong to the same category, then e = 0. The optimal policy will only use the navigation actions from the same type as the state, and all of the other actions are strictly suboptimal. A basic comparison between vanilla Q-learning without action elimination (green) and a tabular version of the action elimination Q-learning (blue) can be found in the figure below. In all of the figures, the results are compared to the case with one category (red), i.e., only 4 basic navigation actions, which forms an upper bound on performance with multiple categories. In Figure (a),(c), the episode length is T = 150, and in Figure (b) it is T = 300, to allow sufficient exploration for the vanilla Q-Learning. It is clear from the simulations that the action elimination dramatically improves the results in large action spaces. Also note that the gain from action elimination increases with the grid size since the elimination allows the agent to reach the goal earlier.<br /> <br /> <br /> [[File:griddomain.png|1200px|thumb|center|Performance of agents in grid world]]<br /> ==Zork domain==<br /> <br /> The world of Zork presents a rich environment with a large state and action space. <br /> Zork players describe their actions using natural language instructions. For example, &quot;open the mailbox&quot;. Then their actions were processed by a sophisticated natural language parser. Based on the results, the game presents the outcome of the action. The goal of Zork is to collect the Twenty Treasures of Zork and install them in the trophy case. Points that are generated from the game's scoring system are given to the agent as the reward. For example, the player gets the points when solving the puzzles. Placing all treasures in the trophy will get 350 points. The elimination signal is given in two forms, &quot;wrong parse&quot; flag, and text feedback &quot;you cannot take that&quot;. These two signals are grouped together into a single binary signal which then provided to the algorithm. <br /> <br /> Experiments begin with the two subdomains of Zork domains: Egg Quest and the Troll Quest. For these subdomains, an additional reward signal is provided to guide the agent towards solving specific tasks and make the results more visible. A reward of -1 is applied at every time step to encourage the agent to favor short paths. Each trajectory terminates is upon completing the quest or after T steps are taken. The discounted factor for training is &lt;math display=&quot;inline&quot;&gt;\gamma=0.8&lt;/math&gt; and &lt;math display=&quot;inline&quot;&gt;\gamma=1&lt;/math&gt; during evaluation. Also &lt;math display=&quot;inline&quot;&gt;\beta=0.5, l=0.6&lt;/math&gt; in all experiments. <br /> <br /> ===Egg Quest===<br /> <br /> The goal for this quest is to find and open the jewel-encrusted egg hidden on a tree in the forest. An egg-splorer goes on an adventure to find a mystical ancient relic with his furry companion. You can have a look at the game at [https://scratch.mit.edu/projects/212838126/ EggQuest]<br /> <br /> The agent will get 100 points upon completing this task. For action space, there are 9 fixed actions for navigation, and a second subset which consisting &lt;math display=&quot;inline&quot;&gt;N_{Take}&lt;/math&gt; actions for taking possible objects in the game. &lt;math display=&quot;inline&quot;&gt;N_{Take}=200 (set A_1), N_{Take}=300 (set A_2)&lt;/math&gt; has been tested separately.<br /> AE-DQN (blue) and a vanilla DQN agent (green) has been tested in this quest.<br /> <br /> [[File:AEF_zork_comparison.png|1200px|thumb|center|Performance of agents in the egg quest.]]<br /> <br /> Figure a) corresponds to the set &lt;math display=&quot;inline&quot;&gt;A_1&lt;/math&gt;, with T=100, b) corresponds to the set &lt;math display=&quot;inline&quot;&gt;A_2&lt;/math&gt;, with T=100, and c) corresponds to the set &lt;math display=&quot;inline&quot;&gt;A_2&lt;/math&gt;, with T=200. Both agents has performed well on sets a and c. However the AE-DQN agent has learned much faster than the DQN on set b, which implies that action elimination is more robust to hyperparameter optimization when the action space is large. One important observation to note is that the three figures have different scales for the cumulative reward. While the AE-DQN outperformed the standard DQN in figure b, both models performed significantly better with the hyperparameter configuration in figure c.<br /> <br /> ===Troll Quest===<br /> <br /> The goal of this quest is to find the troll. To do it the agent need to find the way to the house, use a lantern to expose the hidden entrance to the underworld. It will get 100 points upon achieving the goal. This quest is a larger problem than Egg Quest. The action set &lt;math display=&quot;inline&quot;&gt;A_1&lt;/math&gt; is 200 take actions and 15 necessary actions, 215 in total.<br /> <br /> [[File:AEF_troll_comparison.png|400px|thumb|center|Results in the Troll Quest.]]<br /> <br /> The red line above is an &quot;optimal elimination&quot; baseline which consists of only 35 actions(15 essential, and 20 relevant take actions). We can see that AE-DQN still outperforms DQN and its improvement over DQN is more significant in the Troll Quest than the Egg quest. Also, it achieves compatible performance to the &quot;optimal elimination&quot; baseline.<br /> <br /> ===Open Zork===<br /> <br /> Lastly, the &quot;Open Zork&quot; domain has been tested which only the environment reward has been used. 1M steps has been trained. Each trajectory terminates after T=200 steps. Two action sets have been used:&lt;math display=&quot;inline&quot;&gt;A_3&lt;/math&gt;, the &quot;Minimal Zork&quot; action set, which is the minimal set of actions (131) that is required to solve the game. &lt;math display=&quot;inline&quot;&gt;A_4&lt;/math&gt;, the &quot;Open Zork&quot; action set (1227) which composed of {Verb, Object} tuples for all the verbs and objects in the game.<br /> <br /> [[]]<br /> <br /> [[File:AEF_open_zork_comparison.png|600px|thumb|center|Results in &quot;Open Zork&quot;.]]<br /> <br /> <br /> The above Figure shows the learning curve for both AE-DQN and DQN. We can see that AE-DQN (blue) still outperform the DQN (blue) in terms of speed and cumulative reward.<br /> <br /> =Conclusion=<br /> In this paper, the authors proposed a Deep Reinforcement Learning model for sub-optimal actions while performing Q-learning. Moreover, they showed that by eliminating actions, using linear contextual bandits with theoretical guarantees of convergence, the size of the action space is reduced, exploration is more effective, and learning is improved when tested on Zork, a text-based game.<br /> <br /> For future work the authors aim to investigate more sophisticated architectures and tackle learning shared representations for elimination and control which may boost performance on both tasks.<br /> <br /> They also hope to to investigate other mechanisms for action elimination, such as eliminating actions that result from low Q-values as in Even-Dar, Mannor, and Mansour, 2003.<br /> <br /> The authors also hope to generate elimination signals in real-world domains and achieve the purpose of eliminating the signal implicitly.<br /> <br /> =Critique=<br /> The paper is not a significant algorithmic contribution and it merely adds an extra layer of complexity to the very famous DQN algorithm. All the experimental domains considered in the paper are discrete action problems that have so many actions that it could have been easily extended to a continuous action problem. In continuous action space there are several policy gradient based RL algorithms that have provided stronger performances. The authors should have ideally compared their methods to such algorithms like PPO or DRPO.<br /> <br /> Even with the critique above, the paper presents mathematical/theoretical justifications of the methodology. Moreover, since the methodology is built on the standard RL framework, this means that other variant RL algorithms can apply the idea to decrease the complexity and increase the performance. Moreover, the there are some rooms for applying technical variations for the algorithm.<br /> <br /> Also, since we are utilizing the system's response to irrelevant actions, an intuitive approach to eliminate such irrelevant actions is to add a huge negative reward for such actions, which will be much easier than the approach suggested by this paper. However, the in experiments, the author only compares AE-DQN to traditional DQN, not traditional DQN with negative rewards assigned to irrelevant actions.<br /> <br /> After all, the name that the authors have chosen is a good and attractive choice and matches our brain's structure which in so many real-world scenarios detects what not to learn.<br /> <br /> =Reference=<br /> 1. Chu, W.; Li, L.; Reyzin, L.; and Schapire, R. 2011. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artiﬁcial Intelligence and Statistics.<br /> <br /> 2. Côté,M.-A.;Kádár,Á.;Yuan,X.;Kybartas,B.;Barnes,T.;Fine,E.;Moore,J.;Hausknecht,M.;Asri, L. E.; Adada, M.; et al. 2018. Textworld: A learning environment for text-based games. arXiv.<br /> <br /> 3. Dulac-Arnold, G.; Evans, R.; van Hasselt, H.; Sunehag, P.; Lillicrap, T.; Hunt, J.; Mann, T.; Weber, T.; Degris, T.; and Coppin, B. 2015. Deep reinforcement learning in large discrete action spaces. arXiv.<br /> <br /> 4. He, J.; Chen, J.; He, X.; Gao, J.; Li, L.; Deng, L.; and Ostendorf, M. 2015. Deep reinforcement learning with an unbounded action space. CoRR abs/1511.04636.<br /> <br /> 5. Kim, Y. 2014. Convolutional neural networks for sentence classiﬁcation. [https://arxiv.org/abs/1408.5882 arXiv preprint].<br /> <br /> 6. VanHasselt,H.,andWiering,M.A. 2009. Usingcontinuousactionspacestosolvediscreteproblems. In Neural Networks, 2009. IJCNN 2009. International Joint Conference on, 1149–1156. IEEE.<br /> <br /> 7. Watkins, C. J., and Dayan, P. 1992. Q-learning. Machine learning 8(3-4):279–292.<br /> <br /> 8. Su, P.-H.; Gasic, M.; Mrksic, N.; Rojas-Barahona, L.; Ultes, S.; Vandyke, D.; Wen, T.-H.; and Young, S. 2016. Continuously learning neural dialogue management. arXiv preprint.<br /> <br /> 9. Wu, Y.; Schuster, M.; Chen, Z.; Le, Q. V.; Norouzi, M.; Macherey, W.; Krikun, M.; Cao, Y.; Gao, Q.; Macherey, K.; et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint.<br /> <br /> 10. Yuan, X.; Côté, M.-A.; Sordoni, A.; Laroche, R.; Combes, R. T. d.; Hausknecht, M.; and Trischler, A. 2018. Counting to explore and generalize in text-based games. arXiv preprint arXiv:1806.1152<br /> <br /> 11. Zahavy, T.; Haroush, M.; Merlis, N.; Mankowitz, D. J.; 2018. Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning. arXiv:1809.02121v1</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Learning_to_Teach&diff=42054 Learning to Teach 2018-11-30T14:58:36Z <p>C9sharma: /* Experiments */</p> <hr /> <div><br /> <br /> =Introduction=<br /> <br /> This paper proposed the &quot;learning to teach&quot; (L2T) framework with two intelligent agents: a student model/agent, corresponding to the learner in traditional machine learning algorithms, and a teacher model/agent, determining the appropriate data, loss function, and hypothesis space to facilitate the learning of the student model.<br /> <br /> In modern human society, the role of teaching is heavily implicated in our education system; the goal is to equip students with the necessary knowledge and skills in an efficient manner. This is the fundamental ''student'' and ''teacher'' framework on which education stands. However, in the field of artificial intelligence (AI) and specifically machine learning, researchers have focused most of their efforts on the ''student'' (ie. designing various optimization algorithms to enhance the learning ability of intelligent agents). The paper argues that a formal study on the role of ‘teaching’ in AI is required. Analogous to teaching in human society, the teaching framework can: select training data that corresponds to the appropriate teaching materials (e.g. textbooks selected for the right difficulty), design loss functions that correspond to targeted examinations, and define the hypothesis space that corresponds to imparting the proper methodologies. Furthermore, an optimization framework (instead of heuristics) should be used to update the teaching skills based on the feedback from students, so as to achieve teacher-student co-evolution.<br /> <br /> Thus, the training phase of L2T would have several episodes of interactions between the teacher and the student model. Based on the state information in each step, the teacher model would update the teaching actions so that the student model could perform better on the Machine Learning problem. The student model would then provide reward signals back to the teacher model. These reward signals are used by the teacher model as part of the Reinforcement Learning process to update its parameters. This process is end-to-end trainable and the authors are convinced that once converged, the teacher model could be applied to new learning scenarios and even new students, without extra efforts on re-training.<br /> <br /> To demonstrate the practical value of the proposed approach, the '''training data scheduling''' problem is chosen as an example. The authors show that by using the proposed method to adaptively select the most<br /> suitable training data, they can significantly improve the accuracy and convergence speed of various neural networks including multi-layer perceptron (MLP), convolutional neural networks (CNNs)<br /> and recurrent neural networks (RNNs), for different applications including image classification and text understanding.<br /> Further more , the teacher model obtained by the paper from one task can be smoothly transferred to other tasks. As an example, the teacher model trained on MNIST with the MLP learner, one can achieve a satisfactory performance on CIFAR-10 only using roughly half<br /> of the training data to train a ResNet model as the student.<br /> <br /> =Related Work=<br /> The L2T framework connects with two emerging trends in machine learning. The first is the movement from simple to advanced learning. This includes meta-learning (Schmidhuber, 1987; Thrun &amp; Pratt, 2012) which explores automatic learning by transferring learned knowledge from meta tasks . This approach has been applied to few-shot learning scenarios and in designing general optimizers and neural network architectures. (Hochreiter et al., 2001; Andrychowicz et al., 2016; Li &amp; Malik, 2016; Zoph &amp; Le, 2017)<br /> <br /> The second is the teaching, which can be classified into either machine-teaching (Zhu, 2015)  or hardness based methods. The former seeks to construct a minimal training set for the student to learn a target model (ie. an oracle). The latter assumes an order of data from easy instances to hard ones, hardness being determined in different ways. In curriculum learning (CL) (Bengio et al, 2009; Spitkovsky et al. 2010; Tsvetkov et al, 2016)  measures hardness through heuristics of the data while self-paced learning (SPL) (Kumar et al., 2010; Lee &amp; Grauman, 2011; Jiang et al., 2014; Supancic &amp; Ramanan, 2013)  measures hardness by loss on data. <br /> <br /> The limitations of these works include the lack of a formally defined teaching problem, and the reliance on heuristics and fixed rules, which hinders generalization of the teaching task.<br /> <br /> =Learning to Teach=<br /> To introduce the problem and framework, without loss of generality, consider the setting of supervised learning.<br /> <br /> In supervised learning, each sample &lt;math&gt;x&lt;/math&gt; is from a fixed but unknown distribution &lt;math&gt;P(x)&lt;/math&gt;, and the corresponding label &lt;math&gt; y &lt;/math&gt; is from a fixed but unknown distribution &lt;math&gt;P(y|x) &lt;/math&gt;. The goal is to find a function &lt;math&gt;f_\omega(x)&lt;/math&gt; with parameter vector &lt;math&gt;\omega&lt;/math&gt; that minimizes the gap between the predicted label and the actual label.<br /> <br /> <br /> <br /> ==Problem Definition==<br /> The student model, denoted &amp;mu;(), takes the set of training data &lt;math&gt; D &lt;/math&gt;, the function class &lt;math&gt; Ω &lt;/math&gt;, and loss function &lt;math&gt; L &lt;/math&gt; as input to output a function, &lt;math&gt; f(ω) &lt;/math&gt;, with parameter &lt;math&gt;ω^*&lt;/math&gt; which minimizes risk &lt;math&gt;R(ω)&lt;/math&gt; as in:<br /> <br /> \begin{align*}<br /> ω^* = arg min_{w \in \Omega} \sum_{x,y \in D} L(y, f_ω(x)) =: \mu (D, L, \Omega)<br /> \end{align*}<br /> <br /> The teaching model, denoted φ, tries to provide &lt;math&gt; D &lt;/math&gt;, &lt;math&gt; L &lt;/math&gt;, and &lt;math&gt; Ω &lt;/math&gt; (or any combination, denoted &lt;math&gt; A &lt;/math&gt;) to the student model such that the student model either achieves lower risk R(ω) or progresses as fast as possible.<br /> In contrast to traditional machine learning, which is only concerned with the student model in the<br /> learning to teach framework, the problem in the paper is also concerned with a teacher model, which tries to provide<br /> appropriate inputs to the student model so that it can achieve low risk functional as efficiently<br /> as possible.<br /> <br /> <br /> ::'''Training Data''': Outputting a good training set &lt;math&gt; D &lt;/math&gt;, analogous to human teachers providing students with proper learning materials such as textbooks.<br /> ::'''Loss Function''': Designing a good loss function &lt;math&gt; L &lt;/math&gt; , analogous to providing useful assessment criteria for students.<br /> ::'''Hypothesis Space''': Defining a good function class &lt;math&gt; Ω &lt;/math&gt; which the student model can select from. This is analogous to human teachers providing appropriate context, eg. middle school students taught math with basic algebra while undergraduate students are taught with calculus. Different Ω leads to different errors and optimization problem (Mohri et al., 2012).<br /> <br /> ==Framework==<br /> The training phase consists of the teacher providing the student with the subset &lt;math&gt; A_{train} &lt;/math&gt; of &lt;math&gt; A &lt;/math&gt; and then taking feedback to improve its own parameters.After the convergence of the training process,<br /> the teacher model can be used to teach either<br /> new student models, or the same student<br /> models in new learning scenarios such as another<br /> subset &lt;math&gt; A_{test} &lt;/math&gt;is provided.Such a generalization is feasible as long as the state representations<br /> S are the same across different student<br /> models and different scenarios. The L2T process is outlined in figure below:<br /> <br /> [[File: L2T_process.png | 500px|center]]<br /> <br /> * &lt;math&gt; s_t &amp;isin; S &lt;/math&gt; represents information available to the teacher model at time &lt;math&gt; t &lt;/math&gt;. &lt;math&gt; s_t &lt;/math&gt; is typically constructed from the current student model &lt;math&gt; f_{t−1} &lt;/math&gt; and the past teaching history of the teacher model. &lt;math&gt; S &lt;/math&gt; represents the set of states.<br /> * &lt;math&gt; a_t &amp;isin; A &lt;/math&gt; represents action taken the teacher model at time &lt;math&gt; t &lt;/math&gt;, given state &lt;math&gt;s_t&lt;/math&gt;. &lt;math&gt; A &lt;/math&gt; represents the set of actions, where the action(s) can be any combination of teaching tasks involving the training data, loss function, and hypothesis space. <br /> * &lt;math&gt; φ_θ : S → A &lt;/math&gt; is policy used by the teacher model to generate its action &lt;math&gt; φ_θ(s_t) = a_t &lt;/math&gt;<br /> * Student model takes &lt;math&gt; a_t &lt;/math&gt; as input and outputs function &lt;math&gt; f_t &lt;/math&gt;, by using the conventional ML techniques.<br /> <br /> Once the training process converges, the teacher model may be utilized to teach a different subset of &lt;math&gt; A &lt;/math&gt; or teach a different student model.<br /> <br /> =Application=<br /> <br /> There are different approaches to training the teacher model, this paper will apply reinforcement learning with &lt;math&gt; φ_θ &lt;/math&gt; being the ''policy'' that interacts with &lt;math&gt; S &lt;/math&gt;, the ''environment''. The paper applies data teaching to train a deep neural network student, &lt;math&gt; f &lt;/math&gt;, for several classification tasks. Thus the student feedback measure will be classification accuracy. Its learning rule will be mini-batch stochastic gradient descent, where batches of data will arrive sequentially in random order. The teacher model is responsible for providing the training data, which in this case means it must determine which instances (subset) of the mini-batch of data will be fed to the student. In order to reach the convergence faster, the reward was set to relate to the speed the student model learns. <br /> <br /> The authors also designed a state feature vector &lt;math&gt; g(s) &lt;/math&gt; in order to efficiently represent the current states which include arrived training data and the student model. Within the State Features, there are three categories including Data features, student model features and the combination of both data and learner model. This state feature will be computed when each mini-batch of data arrives.<br /> <br /> <br /> The optimizer for training the teacher model is the maximum expected reward: <br /> <br /> \begin{align} <br /> J(θ) = E_{φ_θ(a|s)}[R(s,a)]<br /> \end{align}<br /> <br /> Which is non-differentiable w.r.t. &lt;math&gt; θ &lt;/math&gt;, thus a likelihood ratio policy gradient algorithm is used to optimize &lt;math&gt; J(θ) &lt;/math&gt; (Williams, 1992) <br /> <br /> ==Experiments==<br /> <br /> The L2T framework is tested on the following student models: multi-layer perceptron (MLP), ResNet (CNN), and Long-Short-Term-Memory network (RNN). <br /> <br /> The student tasks are Image classification for MNIST, for CIFAR-10, and sentiment classification for IMDB movie review dataset. <br /> <br /> The strategy will be benchmarked against the following teaching strategies:<br /> <br /> ::'''NoTeach''': NoTeach removes the entire Teacher-Student paradigm and reverts back to the classical machine learning paradigm. In the context of data teaching, we consider the architecture fixed, and feed data in a pre-determined way. One would pre-define batch-size and cross-validation procedures as needed.<br /> ::'''Self-Paced Learning (SPL)''': Teaching by ''hardness'' of data, defined as the loss. This strategy begins by filtering out data with larger loss value to train the student with &quot;easy&quot; data and gradually increases the hardness. Mathematically speaking, those training data &lt;math&gt;d &lt;/math&gt; satisfying loss value &lt;math&gt;l(d) &gt; \eta &lt;/math&gt; will be filtered out, where the threshold &lt;math&gt; \eta &lt;/math&gt; grows from smaller to larger during the training process. To improve the robustness of SPL, following the widely used trick in common SPL implementation (Jiang et al., 2014b), the authors filter training data using its loss rank in one mini-batch rather than the absolute loss value: they filter data instances with top &lt;math&gt;K &lt;/math&gt;largest training loss values within a &lt;math&gt;M&lt;/math&gt;-sized mini-batch, where &lt;math&gt;K&lt;/math&gt; linearly drops from &lt;math&gt;M − 1 &lt;/math&gt;to 0 during training.<br /> <br /> ::'''L2T''': The Learning to Teach framework.<br /> ::'''RandTeach''': Randomly filter data in each epoch according to the logged ratio of filtered data instances per epoch (as opposed to deliberate and dynamic filtering by L2T).<br /> <br /> For all teaching strategies, they make sure that the base neural network model will not be updated until &lt;math&gt;M &lt;/math&gt; un-trained, yet selected data instances are accumulated. That is to guarantee that the convergence speed is only determined by the quality of taught data, not by different model updating frequencies. The model is implemented with Theano and run on one NVIDIA Tesla K40 GPU for each training/testing process.<br /> ===Training a New Student===<br /> <br /> In the first set of experiments, the datasets or divided into two folds. The first folder is used to train the teacher; This is done by having the teacher train a student network on that half of the data, with a certain portion being used for computing rewards. After training, the teacher parameters are fixed, and used to train a new student network (with the same structure) on the second half of the dataset. When teaching a new student with the same model architecture, we observe that L2T achieves significantly faster convergence than other strategies across all tasks, especially compared to the NoTeach and RandTeach methods:<br /> <br /> [[File: L2T_speed.png | 1100px|center]]<br /> <br /> ===Filtration Number===<br /> <br /> When investigating the details of filtered data instances per epoch, for the two image classification tasks, the L2T teacher filters an increasing amount of data as training goes on. The authors' intuition for the two image classification tasks is that the student model can learn from harder instances of data from the beginning, and thus the teacher can filter redundant data. In contrast, for training while for the natural language task, the student model must first learn from easy data instances.<br /> <br /> [[File: L2T_fig3.png | 1100px|center]]<br /> <br /> ===Teaching New Student with Different Model Architecture===<br /> <br /> In this part, first a teacher model is trained by interacting with a student model. Then using the teacher model, another student model<br /> which has a different model architecture is taught.<br /> The results of Applying the teacher trained on ResNet32 to teach other architectures is shown below. The L2T algorithm can be seen to obtain higher accuracies earlier than the SPL, RandTeach, or NoTeach algorithms.<br /> <br /> [[File: L2T_fig4.png | 1100px|center]]<br /> <br /> ===Training Time Analysis===<br /> <br /> The learning curves demonstrate the efficiency in accuracy achieved by the L2T over the other strategies. This is especially evident during the earlier training stages.<br /> <br /> [[File: L2T_fig5.png | 600px|center]]<br /> <br /> ===Accuracy Improvement===<br /> <br /> When comparing training accuracy on the IMDB sentiment classification task, L2T improves on teaching policy over NoTeach and SPL.<br /> <br /> [[File: L2T_t1.png | 500px|center]]<br /> <br /> Table 1 shows that we boost the convergence speed, while the teacher model improves final accuracy. The student model is the LSTM network trained on IMDB. Prior to teaching the student model, we train the teacher model on half of the training data, and define the terminal reward as the set accuracy after the teacher model trains the student for 15 epochs. Then the teacher model is applied to train the student model on the full dataset till its convergence. The state features are kept the same as those in previous experiments. We can see that L2T achieves better classification accuracy for training LSTM network, surpassing the SPL baseline by more than 0.6 point (with p value &lt; 0.001).<br /> <br /> =Future Work=<br /> <br /> There is some useful future work that can be extended from this work: <br /> <br /> 1) Recent advances in multi-agent reinforcement learning could be tried on the Reinforcement Learning problem formulation of this paper. <br /> <br /> 2) Some human in the loop architectures like CHAT and HAT (https://www.ijcai.org/proceedings/2017/0422.pdf) should give better results for the same framework. <br /> <br /> 3) It would be interesting to try out the framework suggested in this paper (L2T) in Imperfect information and partially observable settings. <br /> <br /> 4) As they have focused on data teaching exploring loss function teaching would be interesting.<br /> <br /> =Critique=<br /> <br /> While the conceptual framework of L2T is sound, the paper only experimentally demonstrates efficacy for ''data teaching'' which would seem to be the simplest to implement. The feasibility and effectiveness of teaching the loss function and hypothesis space are not explored in a real-world scenario. Furthermore, the experimental results for data teaching suggest that the speed of convergence is the main improvement over other teaching strategies whereas the difference in accuracy less remarkable. The paper also assesses accuracy only by comparing L2T with NoTeach and SPL on the IMDB classification task, the improvement (or lack thereof) on the other classification tasks and teaching strategies is omitted. Again, this distinction is not possible to assess in loss function or hypothesis space teaching within the scope of this paper. They could have included larger datasets such as ImageNet and CIFAR100 in their experiments which would have provided some more insight.<br /> <br /> The idea of having a generalizable teacher model to enhance student learning is admirable. In fact, the L2T framework is similar to the reinforcement learning actor-critic model, which is known to be effective. In general, one expects an effective teacher model would facilitate transfer learning and can significantly reduce student model training time. However, the T2L framework seems to fall short of that goal. Consider the CIFAR10 training scenario, the L2T model achieve 85% accuracy after 2 million training data, which is only about 3% more accuracy than a no-teacher model. Perhaps in the future, the L2T framework can improve and produce better performance.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_Bayesian_Perspective_on_Generalization_and_Stochastic_Gradient_Descent&diff=42053 A Bayesian Perspective on Generalization and Stochastic Gradient Descent 2018-11-30T14:47:36Z <p>C9sharma: /* Bayes Theorem and Generalization */</p> <hr /> <div>==Introduction==<br /> This work builds on Zhang et al.(2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. The authors consider two questions: how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? <br /> They show that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. We also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.<br /> The authors propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the “noise scale” &lt;math display=&quot;inline&quot;&gt; g \approx \epsilon N/B &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;ε&lt;/math&gt; is the learning rate, &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt; the training set size and &lt;math display=&quot;inline&quot;&gt;B&lt;/math&gt; the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, &lt;math display=&quot;inline&quot;&gt;B_{opt} \propto \epsilon N&lt;/math&gt; . They verify these predictions empirically.<br /> <br /> ==Motivation==<br /> This paper shows Bayesian principles can explain many recent observations in the deep learning literature, while also discovering practical new insights. Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite &amp; Roy, 2017). Nonetheless, their results beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? Meanwhile, Keskar et al. (2016) observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking result shows improving our estimate of the full-batch gradient can harm performance. Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Hoffer et al. (2017) proposed a square root rule on theoretical grounds.<br /> Many authors have suggested “broad minima” whose curvature is small may generalize better than “sharp minima” whose curvature is large (Chaudhari et al., 2016; Hochreiter &amp; Schmidhuber, 1997). Indeed, Dziugaite &amp; Roy (2017) argued the results of Zhang et al. (2016) can be understood using “nonvacuous” PAC-Bayes generalization bounds which penalize sharp minima, while Keskar et al. (2016) showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However, Dinh et al. (2017) challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization.<br /> <br /> ==Contribution==<br /> <br /> The main contributions of this paper are to show that:<br /> * The results of Zhang et al. (2016) are not unique to deep learning; it is observed the same phenomenon in a small “over-parameterized” linear model. Overparameterization occurs when a model is able to effectively “remember” training data. This occurs when there are enough parameters that the system of equations ends up with an infinite number of possible solutions. One can see why this over-training would lead to poor results in test cases, as this “memorization” learns noise as opposed to the inherent structure of different classes. It is demonstrated that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.<br /> * SGD integrates a stochastic differential equation whose “noise scale” &lt;math&gt;g &amp;asymp; &amp;epsilon;N/B&lt;/math&gt;, where<br /> &amp;epsilon; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is '''proportional to the learning rate and training set size'''.<br /> <br /> Zhang et al. (2016) showed high training competency of neural networks under informative labels, but drastic overfitting on improper labels. This implies weak generalizability even when a small proportion of labels are improper. The authors show that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Bayesians tend to make distributional assumptions on gradient updates by adding isotropic Gaussian noise. This paper builds upon these Bayesian principles by driving SGD away from sharp minima, and towards broad minima (the more broad, the better generalization due to less influence from small perturbations within input). The stochastic differential equation used as a component of gradient updates effectively serves as injected noise that improves a network's generalizability.<br /> <br /> ==Main Results==<br /> <br /> The weakly regularized model memorizes random labels, however, generalizes properly on informative labels. Besides, the predictions are overconfident. The authors also showed that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is postulated that the optimum represents a tradeoff between depth and breadth in the Bayesian evidence. However it is the underlying scale of random fluctuations in the SGD dynamics which controls the tradeoff, not the batch size itself. Furthermore, this test accuracy peak shifts as the training set size rises. The authors observed that the best found batch size is proportional to the learning rate. This scaling rule allowed the authors to increase the learning rate by simultaneously increasing the batch size with no loss in test accuracy and no increase in computational cost, thus parallelism across multiple GPU's can be fully leveraged to easily decrease training time. The scaling rule could also be applied to production models by consequentially increasing the batch size as new training data is introduced.<br /> <br /> ==Bayesian Model Comparison==<br /> <br /> We first consider a classification model &lt;math&gt;M &lt;/math&gt; with a single parameter &lt;math&gt;\omega &lt;/math&gt;, training inputs &lt;math&gt;x &lt;/math&gt; and training labels &lt;math&gt;y &lt;/math&gt;. We can infer a posterior probability distribution over the parameter by applying Bayes theorem :<br /> <br /> \begin{align*}P(\omega\mid y,x;M) = \frac{P(y\mid \omega,x;M)P(\omega;M) }{P(y\mid x;M)}\end{align*}<br /> <br /> The likelihood, &lt;math&gt;P(y\mid \omega,x;M) = \Pi_i P(y_i\mid \omega,x_i;M) = e^{-H(\omega;M)} &lt;/math&gt;, where &lt;math&gt;H(\omega;M) &lt;/math&gt; denotes the cross-entropy of unique categorical labels. Using a Gaussian prior, &lt;math&gt;P(\omega;M) = \sqrt{\lambda/2\pi e^{-\lambda\omega^2/2}} &lt;/math&gt;, and therefore the posterior probability density of the parameter given the training data, &lt;math&gt;P(\omega\mid y,x;M) \propto \sqrt{\lambda/2\pi e^{-C(\omega;M)}} &lt;/math&gt;, where &lt;math&gt;C(\omega;M) = H(\omega;M) + \lambda\omega^2/2 &lt;/math&gt; denotes the L2 regularized cross entropy, or “cost function”, and &lt;math&gt;\lambda &lt;/math&gt; is the regularization coefficient. <br /> <br /> The value &lt;math&gt;\omega_0 &lt;/math&gt; which minimizes the cost function lies at the maximum of this posterior. To predict an unknown label &lt;math&gt;y_t &lt;/math&gt; of a new input &lt;math&gt;x_t &lt;/math&gt;, we should compute the integral,<br /> <br /> \begin{align*} P(y_t\mid x_t,y,x;M) &amp;= \int \frac{d\omega P(y_t\mid \omega,x_t;M)}{P(\omega\mid y,x;M)}\\ &amp;= \frac{\int d \omega P(y_t \mid \omega ,x_t;M)e^{-C(\omega;M)}}{\int d \omega e^{-C(\omega;M)}} \end{align*}&lt;/math&gt;<br /> <br /> However, these integrals are dominated by the region near &lt;math&gt;\omega_0 &lt;/math&gt; . We usually approximate &lt;math&gt;P(y_t\mid x_t,x,y;M) \approx P(y_t\mid \omega_0,x_t;M) &lt;/math&gt;. Having minimized &lt;math&gt;C(\omega;M) &lt;/math&gt; to find &lt;math&gt;\omega_0 &lt;/math&gt;, we now wish to compare two different models and select the best one. We use the probability ratio<br /> <br /> \begin{align*}\frac{P(M_1\mid y,x)}{P (M_2\mid y, x)} = \frac{P(y\mid x;M_1) P(M_1)}{ P (y\mid x; M_2) P (M_2)} . \end{align*} <br /> <br /> The second factor on the right is the prior ratio, which describes which model is most plausible. To avoid unnecessary subjectivity, we usually set this to 1. Meanwhile the first factor on the right is the evidence ratio, which controls how much the training data changes our prior beliefs<br /> <br /> Germain et al. (2016) showed that maximizing the evidence (or “marginal likelihood”) minimizes a PAC-Bayes generalization bound. To compute it, we evaluate <br /> \begin{align*}P(y\mid x;M) &amp;= \int d\omega P(y\mid \omega,x;M)P(\omega;M) \\ &amp;=\sqrt{\frac{\lambda}{2\pi}}\int d \omega e^{C(\omega;M)}\end{align*}<br /> <br /> Notice that the evidence is computed by integrating out the parameters; and consequently it is invariant to the model parameterization. <br /> Since this integral is dominated by the region near the minimum &lt;math&gt;\omega_0 &lt;/math&gt;, we can estimate the evidence by Taylor expanding &lt;math&gt;C(\omega; M) \approx C(\omega_0) + C′′(\omega_0)(\omega - \omega_0)^2/2&lt;/math&gt;. This gives us<br /> <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2}\\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2}\ln(C (\omega_0)/\lambda) \big\}.\end{align*}<br /> <br /> The evidence is controlled by the value of the cost function at the minimum, and by the logarithm of the ratio of the curvature about this minimum compared to the regularization constant. In models with many parameters <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2} \\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) \big\}.\end{align*}<br /> <br /> Occam’s factor arises from the log ratio &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; The Occam factor describes the fraction of the prior parameter space consistent with the data. Occam’s factor penalizes the amount of information the model must learn about the parameters to accurately model the training data. Since the fraction is always less than one, the authors propose to approximate &lt;math&gt;P(y\mid x;M) &lt;/math&gt; away from local minima by only performing the summation over eigenvalues &lt;math&gt;\lambda_i \geq \lambda &lt;/math&gt;.<br /> <br /> The authors compare evidence against a null model which assumes the labels are entirely random. This model has no parameters, and so the evidence is controlled by the likelihood alone. &lt;math&gt;P(y\mid x;NULL) = (1/n)^N = e^{-N \ln(n)} &lt;/math&gt;, where &lt;math&gt;n &lt;/math&gt; denotes the number of model classes and &lt;/math&gt;N &lt;/math&gt; the number of training labels. The evidence ratio :<br /> \begin{equation*}\frac{P(y\mid x;M) }{P(y\mid x;NULL) } = e ^{-E(\omega_0)} \end{equation*}<br /> &lt;math&gt;E(\omega_0) = C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) - N\ln (n) &lt;/math&gt; is the log evidence ratio in favor of the null model.<br /> The authors assign confidence to the predictions of a model iff &lt;math&gt;E(\omega_0 &lt; 0 &lt;/math&gt;.<br /> <br /> The evidence supports the intuition that broad minima generalize better than sharp minima, but unlike the curvature it does not depend on the model parameterization. Dinh et al. (2017) showed one can increase the Hessian eigenvalues by rescaling the parameters, but they must simultaneously rescale the regularization coefficients, otherwise the model changes. Since Occam’s factor arises from the log ratio, &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; , these two effects cancel out. Note however that while the evidence itself is invariant to model parameterization, one can find reparameterizations which change the approximate evidence after the Laplace approximation. . It is difficult to evaluate the evidence for deep networks, as we cannot compute the Hessian of millions of parameters. Additionally, neural networks exhibit many equivalent minima, since we can permute the hidden units without changing the model. To compute the evidence we must carefully account for this “degeneracy”. The authors argue these issues are not a major limitation, since the intuition they build studying the evidence in simple cases will be sufficient to explain the results of both Zhang et al. (2016) and Keskar et al. (2016).<br /> <br /> ==Bayes Theorem and Generalization==<br /> Zhang et al. (2016) showed that deep neural networks generalize well on training inputs with informative labels, but the same model can overfit on the same input images when the labels are randomized; perfectly memorizing the training set. To demonstrate that these observations are not unique to deep network, the authors use logistic regression. They form a small balanced training set comprising 800 images from MNIST, of which half have true label “0” and half true label “1”. The test set is balanced, comprising 5000 MNIST images of zeros and 5000 MNIST images of ones. There are two tasks. In the first task, the labels of both the training and test sets are randomized. In the second task, the labels are informative, matching the true MNIST labels. The model has 784 weights and 1 bias.<br /> <br /> The accuracy of the model predictions on both the training and test sets is shown in figure 1. When trained on the informative labels, the model generalizes well to the test set, so long as it is weakly regularized. However the model also perfectly memorizes the random labels, replicating the obser- vations of Zhang et al. (2016) in deep networks. No significant improvement in model performance is observed as the regularization coefficient increases. For completeness, we also evaluate the mean margin between training examples and the decision boundary. For both random and informative labels, the margin drops significantly as we reduce the regularization coefficient. When weakly regularized, the mean margin is roughly 50% larger for informative labels than for random labels.<br /> <br /> [[File:bg1.png|800px|thumb|center|]]<br /> <br /> Now consider figure 2, where we plot the mean cross-entropy of the model predictions, evaluated on both training and test sets, as well as the Bayesian log evidence ratio defined in the previous section. Looking first at the random label experiment in figure 2a, while the cross-entropy on the training set vanishes when the model is weakly regularized, the cross-entropy on the test set explodes. Not only does the model make random predictions, but it is extremely confident in those predictions. As the regularization coefficient is increased the test set cross-entropy falls, settling at ln 2, the cross- entropy of assigning equal probability to both classes. Now consider the Bayesian evidence, which we evaluate on the training set. The log evidence ratio is large and positive when the model is weakly regularized, indicating that the model is exponentially less plausible than assigning equal probabilities to each class. As the regularization parameter is increased, the log evidence ratio falls, but it is always positive, indicating that the model can never be expected to generalize well.<br /> Now consider figure 2b (informative labels). Once again, the training cross-entropy falls to zero when the model is weakly regularized, while the test cross-entropy is high. Even though the model makes accurate predictions, those predictions are overconfident. As the regularization coefficient increases, the test cross-entropy falls below ln 2, indicating that the model is successfully gener- alizing to the test set. Now consider the Bayesian evidence. The log evidence ratio is large and positive when the model is weakly regularized, but as the regularization coefficient increases, the log evidence ratio drops below zero, indicating that the model is exponentially more plausible than assigning equal probabilities to each class. As we further increase the regularization, the log evi- dence ratio rises to zero while the test cross-entropy rises to ln 2. Test cross-entropy and Bayesian evidence are strongly correlated, with minima at the same regularization strength.<br /> Bayesian model comparison has explained our results in a logistic regression. Meanwhile, Krueger et al. (2017) showed the largest Hessian eigenvalue also increased when training on random labels in deep networks, implying the evidence is falling. We conclude that Bayesian model comparison is quantitatively consistent with the results of Zhang et al. (2016) in linear models where we can compute the evidence, and qualitatively consistent with their results in deep networks where we cannot. Dziugaite &amp; Roy (2017) recently demonstrated the results of Zhang et al. (2016) can also be understood by minimising a PAC-Bayes generalization bound which penalizes sharp minima.<br /> [[File:bg2.png|800px|thumb|center|]]<br /> ==Bayes Theorem and Stochastic Gradient Descent ==<br /> <br /> [[File:bg3.png|800px|thumb|center|]]<br /> [[File:bg4.png|800px|thumb|center|]]<br /> <br /> ==Stochastic Differential Equations and Scaling Rules==<br /> The results showed above indicate that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is argued that this peak arises from the tradeoff between depth and breadth in the Bayesian evidence. However it is not the batch size itself which controls this tradeoff, but the underlying scale of random fluctuations in the SGD dynamics. The following content identifies this SGD “noise scale”, and uses it to derive three scaling rules which predict how the optimal batch size depends on the learning rate, training set size and momentum coefficient. <br /> First, interpret gradient update, as the discrete update of a stochastic differential equation <br /> \begin{equation*}\frac{d\omega}{dt} = \frac{dC}{d\omega} + \eta(t)\end{equation*}<br /> &lt;math&gt;\eta&lt;/math&gt; represents noise &lt;math&gt;\langle \eta(t) \rangle = 0&lt;/math&gt; and &lt;math&gt; \langle \eta (t)\eta (t')\rangle = gF (\omega)\delta (t-t')&lt;/math&gt;.<br /> &lt;math&gt;t&lt;/math&gt; is a continous variable, and &lt;math&gt;F(\omega)&lt;/math&gt; matrix describing the gradient covariances.<br /> The SGD noise scale is taken to be &lt;math&gt;g \approx \epsilon N/B&lt;/math&gt; where &lt;math&gt;\epsilon&lt;/math&gt; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; the batch size.<br /> [[File:bg5.png|800px|thumb|center|]]<br /> [[File:bg6.png|800px|thumb|center|]]<br /> [[File:bg7.png|800px|thumb|center|]]<br /> <br /> ==Critiques==<br /> The paper presents how mini-batch noises with SGD can improve. However, the usefulness of the approach can be described and analyzed in greater details, if the author coudl provide the performance for various well-known real-life datas. Note that even without those evidences, the paper's approach and methods are still very interesting. <br /> <br /> ==Conclusion==<br /> <br /> The paper showed that Mini-batch noise helps SGD to go away from sharp minima, and provided an evidence that there is an optimal optimum batch size for a maximum the test accuracy. Based on interpreting SGD as integrating stochastic differential equation, this batch size is proportional to the learning rate and the training set size. Moreover, the authors shown that Bopt ∝ 1/(1 − m), where m is the momentum coefficient. More analysis was done on the relation between the learning rate, effective learning rate, and batch size is presented in ICLR 2018, where the authors proved by experiments that all the benefits of decaying the learning rate are achieved by increasing the batch size in addition to reducing the number of parameter updates dramatically, and also were able use literature parameters without the need of any hyper parameter tuning (Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le).<br /> <br /> ==References==<br /> <br /> #Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. arXiv preprint arXiv:1612.05086, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012. <br /> #Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference converges to limit cycles for deep networks. arXiv preprint arXiv:1710.11029, 2017.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.<br /> #Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Crispin W Gardiner. Handbook of Stochastic Methods, volume 4. Springer Berlin, 1985.<br /> #Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884– 1892, 2016.<br /> #Priya Goyal, Piotr Dolla ́r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Stephen F Gull. Bayesian inductive inference and maximum entropy. In Maximum-entropy and Bayesian methods in science and engineering, pp. 53–74. Springer, 1988.<br /> #Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM,1993.<br /> #Sepp Hochreiter and Ju ̈rgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Stanisław Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017.<br /> #Robert E Kass and Adrian E Raftery. Bayes factors. Journal of the american statistical association, 90(430):773–795, 1995.<br /> #Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don’t learn via mem- orization. ICLR Workshop, 2017.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101–2110, 2017.<br /> #David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448–472, 1992.<br /> #Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810, 2017.<br /> #Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_Bayesian_Perspective_on_Generalization_and_Stochastic_Gradient_Descent&diff=42052 A Bayesian Perspective on Generalization and Stochastic Gradient Descent 2018-11-30T14:28:33Z <p>C9sharma: /* Bayes Theorem and Generalization */</p> <hr /> <div>==Introduction==<br /> This work builds on Zhang et al.(2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. The authors consider two questions: how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? <br /> They show that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. We also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.<br /> The authors propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the “noise scale” &lt;math display=&quot;inline&quot;&gt; g \approx \epsilon N/B &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;ε&lt;/math&gt; is the learning rate, &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt; the training set size and &lt;math display=&quot;inline&quot;&gt;B&lt;/math&gt; the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, &lt;math display=&quot;inline&quot;&gt;B_{opt} \propto \epsilon N&lt;/math&gt; . They verify these predictions empirically.<br /> <br /> ==Motivation==<br /> This paper shows Bayesian principles can explain many recent observations in the deep learning literature, while also discovering practical new insights. Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite &amp; Roy, 2017). Nonetheless, their results beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? Meanwhile, Keskar et al. (2016) observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking result shows improving our estimate of the full-batch gradient can harm performance. Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Hoffer et al. (2017) proposed a square root rule on theoretical grounds.<br /> Many authors have suggested “broad minima” whose curvature is small may generalize better than “sharp minima” whose curvature is large (Chaudhari et al., 2016; Hochreiter &amp; Schmidhuber, 1997). Indeed, Dziugaite &amp; Roy (2017) argued the results of Zhang et al. (2016) can be understood using “nonvacuous” PAC-Bayes generalization bounds which penalize sharp minima, while Keskar et al. (2016) showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However, Dinh et al. (2017) challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization.<br /> <br /> ==Contribution==<br /> <br /> The main contributions of this paper are to show that:<br /> * The results of Zhang et al. (2016) are not unique to deep learning; it is observed the same phenomenon in a small “over-parameterized” linear model. Overparameterization occurs when a model is able to effectively “remember” training data. This occurs when there are enough parameters that the system of equations ends up with an infinite number of possible solutions. One can see why this over-training would lead to poor results in test cases, as this “memorization” learns noise as opposed to the inherent structure of different classes. It is demonstrated that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.<br /> * SGD integrates a stochastic differential equation whose “noise scale” &lt;math&gt;g &amp;asymp; &amp;epsilon;N/B&lt;/math&gt;, where<br /> &amp;epsilon; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is '''proportional to the learning rate and training set size'''.<br /> <br /> Zhang et al. (2016) showed high training competency of neural networks under informative labels, but drastic overfitting on improper labels. This implies weak generalizability even when a small proportion of labels are improper. The authors show that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Bayesians tend to make distributional assumptions on gradient updates by adding isotropic Gaussian noise. This paper builds upon these Bayesian principles by driving SGD away from sharp minima, and towards broad minima (the more broad, the better generalization due to less influence from small perturbations within input). The stochastic differential equation used as a component of gradient updates effectively serves as injected noise that improves a network's generalizability.<br /> <br /> ==Main Results==<br /> <br /> The weakly regularized model memorizes random labels, however, generalizes properly on informative labels. Besides, the predictions are overconfident. The authors also showed that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is postulated that the optimum represents a tradeoff between depth and breadth in the Bayesian evidence. However it is the underlying scale of random fluctuations in the SGD dynamics which controls the tradeoff, not the batch size itself. Furthermore, this test accuracy peak shifts as the training set size rises. The authors observed that the best found batch size is proportional to the learning rate. This scaling rule allowed the authors to increase the learning rate by simultaneously increasing the batch size with no loss in test accuracy and no increase in computational cost, thus parallelism across multiple GPU's can be fully leveraged to easily decrease training time. The scaling rule could also be applied to production models by consequentially increasing the batch size as new training data is introduced.<br /> <br /> ==Bayesian Model Comparison==<br /> <br /> We first consider a classification model &lt;math&gt;M &lt;/math&gt; with a single parameter &lt;math&gt;\omega &lt;/math&gt;, training inputs &lt;math&gt;x &lt;/math&gt; and training labels &lt;math&gt;y &lt;/math&gt;. We can infer a posterior probability distribution over the parameter by applying Bayes theorem :<br /> <br /> \begin{align*}P(\omega\mid y,x;M) = \frac{P(y\mid \omega,x;M)P(\omega;M) }{P(y\mid x;M)}\end{align*}<br /> <br /> The likelihood, &lt;math&gt;P(y\mid \omega,x;M) = \Pi_i P(y_i\mid \omega,x_i;M) = e^{-H(\omega;M)} &lt;/math&gt;, where &lt;math&gt;H(\omega;M) &lt;/math&gt; denotes the cross-entropy of unique categorical labels. Using a Gaussian prior, &lt;math&gt;P(\omega;M) = \sqrt{\lambda/2\pi e^{-\lambda\omega^2/2}} &lt;/math&gt;, and therefore the posterior probability density of the parameter given the training data, &lt;math&gt;P(\omega\mid y,x;M) \propto \sqrt{\lambda/2\pi e^{-C(\omega;M)}} &lt;/math&gt;, where &lt;math&gt;C(\omega;M) = H(\omega;M) + \lambda\omega^2/2 &lt;/math&gt; denotes the L2 regularized cross entropy, or “cost function”, and &lt;math&gt;\lambda &lt;/math&gt; is the regularization coefficient. <br /> <br /> The value &lt;math&gt;\omega_0 &lt;/math&gt; which minimizes the cost function lies at the maximum of this posterior. To predict an unknown label &lt;math&gt;y_t &lt;/math&gt; of a new input &lt;math&gt;x_t &lt;/math&gt;, we should compute the integral,<br /> <br /> \begin{align*} P(y_t\mid x_t,y,x;M) &amp;= \int \frac{d\omega P(y_t\mid \omega,x_t;M)}{P(\omega\mid y,x;M)}\\ &amp;= \frac{\int d \omega P(y_t \mid \omega ,x_t;M)e^{-C(\omega;M)}}{\int d \omega e^{-C(\omega;M)}} \end{align*}&lt;/math&gt;<br /> <br /> However, these integrals are dominated by the region near &lt;math&gt;\omega_0 &lt;/math&gt; . We usually approximate &lt;math&gt;P(y_t\mid x_t,x,y;M) \approx P(y_t\mid \omega_0,x_t;M) &lt;/math&gt;. Having minimized &lt;math&gt;C(\omega;M) &lt;/math&gt; to find &lt;math&gt;\omega_0 &lt;/math&gt;, we now wish to compare two different models and select the best one. We use the probability ratio<br /> <br /> \begin{align*}\frac{P(M_1\mid y,x)}{P (M_2\mid y, x)} = \frac{P(y\mid x;M_1) P(M_1)}{ P (y\mid x; M_2) P (M_2)} . \end{align*} <br /> <br /> The second factor on the right is the prior ratio, which describes which model is most plausible. To avoid unnecessary subjectivity, we usually set this to 1. Meanwhile the first factor on the right is the evidence ratio, which controls how much the training data changes our prior beliefs<br /> <br /> Germain et al. (2016) showed that maximizing the evidence (or “marginal likelihood”) minimizes a PAC-Bayes generalization bound. To compute it, we evaluate <br /> \begin{align*}P(y\mid x;M) &amp;= \int d\omega P(y\mid \omega,x;M)P(\omega;M) \\ &amp;=\sqrt{\frac{\lambda}{2\pi}}\int d \omega e^{C(\omega;M)}\end{align*}<br /> <br /> Notice that the evidence is computed by integrating out the parameters; and consequently it is invariant to the model parameterization. <br /> Since this integral is dominated by the region near the minimum &lt;math&gt;\omega_0 &lt;/math&gt;, we can estimate the evidence by Taylor expanding &lt;math&gt;C(\omega; M) \approx C(\omega_0) + C′′(\omega_0)(\omega - \omega_0)^2/2&lt;/math&gt;. This gives us<br /> <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2}\\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2}\ln(C (\omega_0)/\lambda) \big\}.\end{align*}<br /> <br /> The evidence is controlled by the value of the cost function at the minimum, and by the logarithm of the ratio of the curvature about this minimum compared to the regularization constant. In models with many parameters <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2} \\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) \big\}.\end{align*}<br /> <br /> Occam’s factor arises from the log ratio &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; The Occam factor describes the fraction of the prior parameter space consistent with the data. Occam’s factor penalizes the amount of information the model must learn about the parameters to accurately model the training data. Since the fraction is always less than one, the authors propose to approximate &lt;math&gt;P(y\mid x;M) &lt;/math&gt; away from local minima by only performing the summation over eigenvalues &lt;math&gt;\lambda_i \geq \lambda &lt;/math&gt;.<br /> <br /> The authors compare evidence against a null model which assumes the labels are entirely random. This model has no parameters, and so the evidence is controlled by the likelihood alone. &lt;math&gt;P(y\mid x;NULL) = (1/n)^N = e^{-N \ln(n)} &lt;/math&gt;, where &lt;math&gt;n &lt;/math&gt; denotes the number of model classes and &lt;/math&gt;N &lt;/math&gt; the number of training labels. The evidence ratio :<br /> \begin{equation*}\frac{P(y\mid x;M) }{P(y\mid x;NULL) } = e ^{-E(\omega_0)} \end{equation*}<br /> &lt;math&gt;E(\omega_0) = C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) - N\ln (n) &lt;/math&gt; is the log evidence ratio in favor of the null model.<br /> The authors assign confidence to the predictions of a model iff &lt;math&gt;E(\omega_0 &lt; 0 &lt;/math&gt;.<br /> <br /> The evidence supports the intuition that broad minima generalize better than sharp minima, but unlike the curvature it does not depend on the model parameterization. Dinh et al. (2017) showed one can increase the Hessian eigenvalues by rescaling the parameters, but they must simultaneously rescale the regularization coefficients, otherwise the model changes. Since Occam’s factor arises from the log ratio, &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; , these two effects cancel out. Note however that while the evidence itself is invariant to model parameterization, one can find reparameterizations which change the approximate evidence after the Laplace approximation. . It is difficult to evaluate the evidence for deep networks, as we cannot compute the Hessian of millions of parameters. Additionally, neural networks exhibit many equivalent minima, since we can permute the hidden units without changing the model. To compute the evidence we must carefully account for this “degeneracy”. The authors argue these issues are not a major limitation, since the intuition they build studying the evidence in simple cases will be sufficient to explain the results of both Zhang et al. (2016) and Keskar et al. (2016).<br /> <br /> ==Bayes Theorem and Generalization==<br /> Zhang et al. (2016) showed that deep neural networks generalize well on training inputs with informative labels, but the same model can overfit on the same input images when the labels are randomized; perfectly memorizing the training set. To demonstrate that these observations are not unique to deep network, the authors use logistic regression. They form a small balanced training set comprising 800 images from MNIST, of which half have true label “0” and half true label “1”. The test set is balanced, comprising 5000 MNIST images of zeros and 5000 MNIST images of ones. There are two tasks. In the first task, the labels of both the training and test sets are randomized. In the second task, the labels are informative, matching the true MNIST labels. The model has 784 weights and 1 bias.<br /> [[File:bg1.png|800px|thumb|center|]]<br /> [[File:bg2.png|800px|thumb|center|]]<br /> [[File:bg3.png|800px|thumb|center|]]<br /> [[File:bg4.png|800px|thumb|center|]]<br /> [[File:bg5.png|800px|thumb|center|]]<br /> [[File:bg6.png|800px|thumb|center|]]<br /> [[File:bg7.png|800px|thumb|center|]]<br /> <br /> ==Critiques==<br /> The paper presents how mini-batch noises with SGD can improve. However, the usefulness of the approach can be described and analyzed in greater details, if the author coudl provide the performance for various well-known real-life datas. Note that even without those evidences, the paper's approach and methods are still very interesting. <br /> <br /> ==Conclusion==<br /> <br /> The paper showed that Mini-batch noise helps SGD to go away from sharp minima, and provided an evidence that there is an optimal optimum batch size for a maximum the test accuracy. Based on interpreting SGD as integrating stochastic differential equation, this batch size is proportional to the learning rate and the training set size. Moreover, the authors shown that Bopt ∝ 1/(1 − m), where m is the momentum coefficient. More analysis was done on the relation between the learning rate, effective learning rate, and batch size is presented in ICLR 2018, where the authors proved by experiments that all the benefits of decaying the learning rate are achieved by increasing the batch size in addition to reducing the number of parameter updates dramatically, and also were able use literature parameters without the need of any hyper parameter tuning (Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le).<br /> <br /> ==References==<br /> <br /> #Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. arXiv preprint arXiv:1612.05086, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012. <br /> #Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference converges to limit cycles for deep networks. arXiv preprint arXiv:1710.11029, 2017.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.<br /> #Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Crispin W Gardiner. Handbook of Stochastic Methods, volume 4. Springer Berlin, 1985.<br /> #Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884– 1892, 2016.<br /> #Priya Goyal, Piotr Dolla ́r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Stephen F Gull. Bayesian inductive inference and maximum entropy. In Maximum-entropy and Bayesian methods in science and engineering, pp. 53–74. Springer, 1988.<br /> #Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM,1993.<br /> #Sepp Hochreiter and Ju ̈rgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Stanisław Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017.<br /> #Robert E Kass and Adrian E Raftery. Bayes factors. Journal of the american statistical association, 90(430):773–795, 1995.<br /> #Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don’t learn via mem- orization. ICLR Workshop, 2017.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101–2110, 2017.<br /> #David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448–472, 1992.<br /> #Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810, 2017.<br /> #Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=File:bg7.png&diff=42051 File:bg7.png 2018-11-30T14:19:53Z <p>C9sharma: </p> <hr /> <div></div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=File:bg6.png&diff=42050 File:bg6.png 2018-11-30T14:19:19Z <p>C9sharma: </p> <hr /> <div></div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=File:bg5.png&diff=42049 File:bg5.png 2018-11-30T14:18:56Z <p>C9sharma: </p> <hr /> <div></div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=File:bg4.png&diff=42048 File:bg4.png 2018-11-30T14:18:25Z <p>C9sharma: </p> <hr /> <div></div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=File:bg3.png&diff=42047 File:bg3.png 2018-11-30T14:16:51Z <p>C9sharma: </p> <hr /> <div></div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=File:bg2.png&diff=42046 File:bg2.png 2018-11-30T14:16:06Z <p>C9sharma: </p> <hr /> <div></div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=File:bg1.png&diff=42045 File:bg1.png 2018-11-30T14:15:08Z <p>C9sharma: </p> <hr /> <div></div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_Bayesian_Perspective_on_Generalization_and_Stochastic_Gradient_Descent&diff=42044 A Bayesian Perspective on Generalization and Stochastic Gradient Descent 2018-11-30T14:12:31Z <p>C9sharma: </p> <hr /> <div>==Introduction==<br /> This work builds on Zhang et al.(2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. The authors consider two questions: how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? <br /> They show that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. We also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.<br /> The authors propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the “noise scale” &lt;math display=&quot;inline&quot;&gt; g \approx \epsilon N/B &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;ε&lt;/math&gt; is the learning rate, &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt; the training set size and &lt;math display=&quot;inline&quot;&gt;B&lt;/math&gt; the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, &lt;math display=&quot;inline&quot;&gt;B_{opt} \propto \epsilon N&lt;/math&gt; . They verify these predictions empirically.<br /> <br /> ==Motivation==<br /> This paper shows Bayesian principles can explain many recent observations in the deep learning literature, while also discovering practical new insights. Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite &amp; Roy, 2017). Nonetheless, their results beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? Meanwhile, Keskar et al. (2016) observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking result shows improving our estimate of the full-batch gradient can harm performance. Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Hoffer et al. (2017) proposed a square root rule on theoretical grounds.<br /> Many authors have suggested “broad minima” whose curvature is small may generalize better than “sharp minima” whose curvature is large (Chaudhari et al., 2016; Hochreiter &amp; Schmidhuber, 1997). Indeed, Dziugaite &amp; Roy (2017) argued the results of Zhang et al. (2016) can be understood using “nonvacuous” PAC-Bayes generalization bounds which penalize sharp minima, while Keskar et al. (2016) showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However, Dinh et al. (2017) challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization.<br /> <br /> ==Contribution==<br /> <br /> The main contributions of this paper are to show that:<br /> * The results of Zhang et al. (2016) are not unique to deep learning; it is observed the same phenomenon in a small “over-parameterized” linear model. Overparameterization occurs when a model is able to effectively “remember” training data. This occurs when there are enough parameters that the system of equations ends up with an infinite number of possible solutions. One can see why this over-training would lead to poor results in test cases, as this “memorization” learns noise as opposed to the inherent structure of different classes. It is demonstrated that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.<br /> * SGD integrates a stochastic differential equation whose “noise scale” &lt;math&gt;g &amp;asymp; &amp;epsilon;N/B&lt;/math&gt;, where<br /> &amp;epsilon; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is '''proportional to the learning rate and training set size'''.<br /> <br /> Zhang et al. (2016) showed high training competency of neural networks under informative labels, but drastic overfitting on improper labels. This implies weak generalizability even when a small proportion of labels are improper. The authors show that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Bayesians tend to make distributional assumptions on gradient updates by adding isotropic Gaussian noise. This paper builds upon these Bayesian principles by driving SGD away from sharp minima, and towards broad minima (the more broad, the better generalization due to less influence from small perturbations within input). The stochastic differential equation used as a component of gradient updates effectively serves as injected noise that improves a network's generalizability.<br /> <br /> ==Main Results==<br /> <br /> The weakly regularized model memorizes random labels, however, generalizes properly on informative labels. Besides, the predictions are overconfident. The authors also showed that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is postulated that the optimum represents a tradeoff between depth and breadth in the Bayesian evidence. However it is the underlying scale of random fluctuations in the SGD dynamics which controls the tradeoff, not the batch size itself. Furthermore, this test accuracy peak shifts as the training set size rises. The authors observed that the best found batch size is proportional to the learning rate. This scaling rule allowed the authors to increase the learning rate by simultaneously increasing the batch size with no loss in test accuracy and no increase in computational cost, thus parallelism across multiple GPU's can be fully leveraged to easily decrease training time. The scaling rule could also be applied to production models by consequentially increasing the batch size as new training data is introduced.<br /> <br /> ==Bayesian Model Comparison==<br /> <br /> We first consider a classification model &lt;math&gt;M &lt;/math&gt; with a single parameter &lt;math&gt;\omega &lt;/math&gt;, training inputs &lt;math&gt;x &lt;/math&gt; and training labels &lt;math&gt;y &lt;/math&gt;. We can infer a posterior probability distribution over the parameter by applying Bayes theorem :<br /> <br /> \begin{align*}P(\omega\mid y,x;M) = \frac{P(y\mid \omega,x;M)P(\omega;M) }{P(y\mid x;M)}\end{align*}<br /> <br /> The likelihood, &lt;math&gt;P(y\mid \omega,x;M) = \Pi_i P(y_i\mid \omega,x_i;M) = e^{-H(\omega;M)} &lt;/math&gt;, where &lt;math&gt;H(\omega;M) &lt;/math&gt; denotes the cross-entropy of unique categorical labels. Using a Gaussian prior, &lt;math&gt;P(\omega;M) = \sqrt{\lambda/2\pi e^{-\lambda\omega^2/2}} &lt;/math&gt;, and therefore the posterior probability density of the parameter given the training data, &lt;math&gt;P(\omega\mid y,x;M) \propto \sqrt{\lambda/2\pi e^{-C(\omega;M)}} &lt;/math&gt;, where &lt;math&gt;C(\omega;M) = H(\omega;M) + \lambda\omega^2/2 &lt;/math&gt; denotes the L2 regularized cross entropy, or “cost function”, and &lt;math&gt;\lambda &lt;/math&gt; is the regularization coefficient. <br /> <br /> The value &lt;math&gt;\omega_0 &lt;/math&gt; which minimizes the cost function lies at the maximum of this posterior. To predict an unknown label &lt;math&gt;y_t &lt;/math&gt; of a new input &lt;math&gt;x_t &lt;/math&gt;, we should compute the integral,<br /> <br /> \begin{align*} P(y_t\mid x_t,y,x;M) &amp;= \int \frac{d\omega P(y_t\mid \omega,x_t;M)}{P(\omega\mid y,x;M)}\\ &amp;= \frac{\int d \omega P(y_t \mid \omega ,x_t;M)e^{-C(\omega;M)}}{\int d \omega e^{-C(\omega;M)}} \end{align*}&lt;/math&gt;<br /> <br /> However, these integrals are dominated by the region near &lt;math&gt;\omega_0 &lt;/math&gt; . We usually approximate &lt;math&gt;P(y_t\mid x_t,x,y;M) \approx P(y_t\mid \omega_0,x_t;M) &lt;/math&gt;. Having minimized &lt;math&gt;C(\omega;M) &lt;/math&gt; to find &lt;math&gt;\omega_0 &lt;/math&gt;, we now wish to compare two different models and select the best one. We use the probability ratio<br /> <br /> \begin{align*}\frac{P(M_1\mid y,x)}{P (M_2\mid y, x)} = \frac{P(y\mid x;M_1) P(M_1)}{ P (y\mid x; M_2) P (M_2)} . \end{align*} <br /> <br /> The second factor on the right is the prior ratio, which describes which model is most plausible. To avoid unnecessary subjectivity, we usually set this to 1. Meanwhile the first factor on the right is the evidence ratio, which controls how much the training data changes our prior beliefs<br /> <br /> Germain et al. (2016) showed that maximizing the evidence (or “marginal likelihood”) minimizes a PAC-Bayes generalization bound. To compute it, we evaluate <br /> \begin{align*}P(y\mid x;M) &amp;= \int d\omega P(y\mid \omega,x;M)P(\omega;M) \\ &amp;=\sqrt{\frac{\lambda}{2\pi}}\int d \omega e^{C(\omega;M)}\end{align*}<br /> <br /> Notice that the evidence is computed by integrating out the parameters; and consequently it is invariant to the model parameterization. <br /> Since this integral is dominated by the region near the minimum &lt;math&gt;\omega_0 &lt;/math&gt;, we can estimate the evidence by Taylor expanding &lt;math&gt;C(\omega; M) \approx C(\omega_0) + C′′(\omega_0)(\omega - \omega_0)^2/2&lt;/math&gt;. This gives us<br /> <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2}\\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2}\ln(C (\omega_0)/\lambda) \big\}.\end{align*}<br /> <br /> The evidence is controlled by the value of the cost function at the minimum, and by the logarithm of the ratio of the curvature about this minimum compared to the regularization constant. In models with many parameters <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2} \\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) \big\}.\end{align*}<br /> <br /> Occam’s factor arises from the log ratio &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; The Occam factor describes the fraction of the prior parameter space consistent with the data. Occam’s factor penalizes the amount of information the model must learn about the parameters to accurately model the training data. Since the fraction is always less than one, the authors propose to approximate &lt;math&gt;P(y\mid x;M) &lt;/math&gt; away from local minima by only performing the summation over eigenvalues &lt;math&gt;\lambda_i \geq \lambda &lt;/math&gt;.<br /> <br /> The authors compare evidence against a null model which assumes the labels are entirely random. This model has no parameters, and so the evidence is controlled by the likelihood alone. &lt;math&gt;P(y\mid x;NULL) = (1/n)^N = e^{-N \ln(n)} &lt;/math&gt;, where &lt;math&gt;n &lt;/math&gt; denotes the number of model classes and &lt;/math&gt;N &lt;/math&gt; the number of training labels. The evidence ratio :<br /> \begin{equation*}\frac{P(y\mid x;M) }{P(y\mid x;NULL) } = e ^{-E(\omega_0)} \end{equation*}<br /> &lt;math&gt;E(\omega_0) = C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) - N\ln (n) &lt;/math&gt; is the log evidence ratio in favor of the null model.<br /> The authors assign confidence to the predictions of a model iff &lt;math&gt;E(\omega_0 &lt; 0 &lt;/math&gt;.<br /> <br /> The evidence supports the intuition that broad minima generalize better than sharp minima, but unlike the curvature it does not depend on the model parameterization. Dinh et al. (2017) showed one can increase the Hessian eigenvalues by rescaling the parameters, but they must simultaneously rescale the regularization coefficients, otherwise the model changes. Since Occam’s factor arises from the log ratio, &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; , these two effects cancel out. Note however that while the evidence itself is invariant to model parameterization, one can find reparameterizations which change the approximate evidence after the Laplace approximation. . It is difficult to evaluate the evidence for deep networks, as we cannot compute the Hessian of millions of parameters. Additionally, neural networks exhibit many equivalent minima, since we can permute the hidden units without changing the model. To compute the evidence we must carefully account for this “degeneracy”. The authors argue these issues are not a major limitation, since the intuition they build studying the evidence in simple cases will be sufficient to explain the results of both Zhang et al. (2016) and Keskar et al. (2016).<br /> <br /> ==Bayes Theorem and Generalization==<br /> Zhang et al. (2016) showed that deep neural networks generalize well on training inputs with informative labels, but the same model can overfit on the same input images when the labels are randomized; perfectly memorizing the training set. To demonstrate that these observations are not unique to deep network, the authors use logistic regression. They form a small balanced training set comprising 800 images from MNIST, of which half have true label “0” and half true label “1”. The test set is balanced, comprising 5000 MNIST images of zeros and 5000 MNIST images of ones. There are two tasks. In the first task, the labels of both the training and test sets are randomized. In the second task, the labels are informative, matching the true MNIST labels. The model has 784 weights and 1 bias.<br /> <br /> ==Critiques==<br /> The paper presents how mini-batch noises with SGD can improve. However, the usefulness of the approach can be described and analyzed in greater details, if the author coudl provide the performance for various well-known real-life datas. Note that even without those evidences, the paper's approach and methods are still very interesting. <br /> <br /> ==Conclusion==<br /> <br /> The paper showed that Mini-batch noise helps SGD to go away from sharp minima, and provided an evidence that there is an optimal optimum batch size for a maximum the test accuracy. Based on interpreting SGD as integrating stochastic differential equation, this batch size is proportional to the learning rate and the training set size. Moreover, the authors shown that Bopt ∝ 1/(1 − m), where m is the momentum coefficient. More analysis was done on the relation between the learning rate, effective learning rate, and batch size is presented in ICLR 2018, where the authors proved by experiments that all the benefits of decaying the learning rate are achieved by increasing the batch size in addition to reducing the number of parameter updates dramatically, and also were able use literature parameters without the need of any hyper parameter tuning (Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le).<br /> <br /> ==References==<br /> <br /> #Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. arXiv preprint arXiv:1612.05086, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012. <br /> #Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference converges to limit cycles for deep networks. arXiv preprint arXiv:1710.11029, 2017.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.<br /> #Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Crispin W Gardiner. Handbook of Stochastic Methods, volume 4. Springer Berlin, 1985.<br /> #Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884– 1892, 2016.<br /> #Priya Goyal, Piotr Dolla ́r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Stephen F Gull. Bayesian inductive inference and maximum entropy. In Maximum-entropy and Bayesian methods in science and engineering, pp. 53–74. Springer, 1988.<br /> #Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM,1993.<br /> #Sepp Hochreiter and Ju ̈rgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Stanisław Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017.<br /> #Robert E Kass and Adrian E Raftery. Bayes factors. Journal of the american statistical association, 90(430):773–795, 1995.<br /> #Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don’t learn via mem- orization. ICLR Workshop, 2017.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101–2110, 2017.<br /> #David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448–472, 1992.<br /> #Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810, 2017.<br /> #Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_Bayesian_Perspective_on_Generalization_and_Stochastic_Gradient_Descent&diff=42043 A Bayesian Perspective on Generalization and Stochastic Gradient Descent 2018-11-30T14:06:26Z <p>C9sharma: </p> <hr /> <div>==Introduction==<br /> This work builds on Zhang et al.(2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. The authors consider two questions: how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? <br /> They show that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. We also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.<br /> The authors propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the “noise scale” &lt;math display=&quot;inline&quot;&gt; g \approx \epsilon N/B &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;ε&lt;/math&gt; is the learning rate, &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt; the training set size and &lt;math display=&quot;inline&quot;&gt;B&lt;/math&gt; the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, &lt;math display=&quot;inline&quot;&gt;B_{opt} \propto \epsilon N&lt;/math&gt; . They verify these predictions empirically.<br /> <br /> ==Motivation==<br /> This paper shows Bayesian principles can explain many recent observations in the deep learning literature, while also discovering practical new insights. Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite &amp; Roy, 2017). Nonetheless, their results beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? Meanwhile, Keskar et al. (2016) observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking result shows improving our estimate of the full-batch gradient can harm performance. Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Hoffer et al. (2017) proposed a square root rule on theoretical grounds.<br /> Many authors have suggested “broad minima” whose curvature is small may generalize better than “sharp minima” whose curvature is large (Chaudhari et al., 2016; Hochreiter &amp; Schmidhuber, 1997). Indeed, Dziugaite &amp; Roy (2017) argued the results of Zhang et al. (2016) can be understood using “nonvacuous” PAC-Bayes generalization bounds which penalize sharp minima, while Keskar et al. (2016) showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However, Dinh et al. (2017) challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization.<br /> <br /> ==Contribution==<br /> <br /> The main contributions of this paper are to show that:<br /> * The results of Zhang et al. (2016) are not unique to deep learning; it is observed the same phenomenon in a small “over-parameterized” linear model. Overparameterization occurs when a model is able to effectively “remember” training data. This occurs when there are enough parameters that the system of equations ends up with an infinite number of possible solutions. One can see why this over-training would lead to poor results in test cases, as this “memorization” learns noise as opposed to the inherent structure of different classes. It is demonstrated that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.<br /> * SGD integrates a stochastic differential equation whose “noise scale” &lt;math&gt;g &amp;asymp; &amp;epsilon;N/B&lt;/math&gt;, where<br /> &amp;epsilon; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is '''proportional to the learning rate and training set size'''.<br /> <br /> Zhang et al. (2016) showed high training competency of neural networks under informative labels, but drastic overfitting on improper labels. This implies weak generalizability even when a small proportion of labels are improper. The authors show that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Bayesians tend to make distributional assumptions on gradient updates by adding isotropic Gaussian noise. This paper builds upon these Bayesian principles by driving SGD away from sharp minima, and towards broad minima (the more broad, the better generalization due to less influence from small perturbations within input). The stochastic differential equation used as a component of gradient updates effectively serves as injected noise that improves a network's generalizability.<br /> <br /> ==Main Results==<br /> <br /> The weakly regularized model memorizes random labels, however, generalizes properly on informative labels. Besides, the predictions are overconfident. The authors also showed that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is postulated that the optimum represents a tradeoff between depth and breadth in the Bayesian evidence. However it is the underlying scale of random fluctuations in the SGD dynamics which controls the tradeoff, not the batch size itself. Furthermore, this test accuracy peak shifts as the training set size rises. The authors observed that the best found batch size is proportional to the learning rate. This scaling rule allowed the authors to increase the learning rate by simultaneously increasing the batch size with no loss in test accuracy and no increase in computational cost, thus parallelism across multiple GPU's can be fully leveraged to easily decrease training time. The scaling rule could also be applied to production models by consequentially increasing the batch size as new training data is introduced.<br /> <br /> ==Bayesian Model Comparison==<br /> <br /> We first consider a classification model &lt;math&gt;M &lt;/math&gt; with a single parameter &lt;math&gt;\omega &lt;/math&gt;, training inputs &lt;math&gt;x &lt;/math&gt; and training labels &lt;math&gt;y &lt;/math&gt;. We can infer a posterior probability distribution over the parameter by applying Bayes theorem :<br /> <br /> \begin{align*}P(\omega\mid y,x;M) = \frac{P(y\mid \omega,x;M)P(\omega;M) }{P(y\mid x;M)}\end{align*}<br /> <br /> The likelihood, &lt;math&gt;P(y\mid \omega,x;M) = \Pi_i P(y_i\mid \omega,x_i;M) = e^{-H(\omega;M)} &lt;/math&gt;, where &lt;math&gt;H(\omega;M) &lt;/math&gt; denotes the cross-entropy of unique categorical labels. Using a Gaussian prior, &lt;math&gt;P(\omega;M) = \sqrt{\lambda/2\pi e^{-\lambda\omega^2/2}} &lt;/math&gt;, and therefore the posterior probability density of the parameter given the training data, &lt;math&gt;P(\omega\mid y,x;M) \propto \sqrt{\lambda/2\pi e^{-C(\omega;M)}} &lt;/math&gt;, where &lt;math&gt;C(\omega;M) = H(\omega;M) + \lambda\omega^2/2 &lt;/math&gt; denotes the L2 regularized cross entropy, or “cost function”, and &lt;math&gt;\lambda &lt;/math&gt; is the regularization coefficient. <br /> <br /> The value &lt;math&gt;\omega_0 &lt;/math&gt; which minimizes the cost function lies at the maximum of this posterior. To predict an unknown label &lt;math&gt;y_t &lt;/math&gt; of a new input &lt;math&gt;x_t &lt;/math&gt;, we should compute the integral,<br /> <br /> \begin{align*} P(y_t\mid x_t,y,x;M) &amp;= \int \frac{d\omega P(y_t\mid \omega,x_t;M)}{P(\omega\mid y,x;M)}\\ &amp;= \frac{\int d \omega P(y_t \mid \omega ,x_t;M)e^{-C(\omega;M)}}{\int d \omega e^{-C(\omega;M)}} \end{align*}&lt;/math&gt;<br /> <br /> However, these integrals are dominated by the region near &lt;math&gt;\omega_0 &lt;/math&gt; . We usually approximate &lt;math&gt;P(y_t\mid x_t,x,y;M) \approx P(y_t\mid \omega_0,x_t;M) &lt;/math&gt;. Having minimized &lt;math&gt;C(\omega;M) &lt;/math&gt; to find &lt;math&gt;\omega_0 &lt;/math&gt;, we now wish to compare two different models and select the best one. We use the probability ratio<br /> <br /> \begin{align*}\frac{P(M_1\mid y,x)}{P (M_2\mid y, x)} = \frac{P(y\mid x;M_1) P(M_1)}{ P (y\mid x; M_2) P (M_2)} . \end{align*} <br /> <br /> The second factor on the right is the prior ratio, which describes which model is most plausible. To avoid unnecessary subjectivity, we usually set this to 1. Meanwhile the first factor on the right is the evidence ratio, which controls how much the training data changes our prior beliefs<br /> <br /> Germain et al. (2016) showed that maximizing the evidence (or “marginal likelihood”) minimizes a PAC-Bayes generalization bound. To compute it, we evaluate <br /> \begin{align*}P(y\mid x;M) &amp;= \int d\omega P(y\mid \omega,x;M)P(\omega;M) \\ &amp;=\sqrt{\frac{\lambda}{2\pi}}\int d \omega e^{C(\omega;M)}\end{align*}<br /> <br /> Notice that the evidence is computed by integrating out the parameters; and consequently it is invariant to the model parameterization. <br /> Since this integral is dominated by the region near the minimum &lt;math&gt;\omega_0 &lt;/math&gt;, we can estimate the evidence by Taylor expanding &lt;math&gt;C(\omega; M) \approx C(\omega_0) + C′′(\omega_0)(\omega - \omega_0)^2/2&lt;/math&gt;. This gives us<br /> <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2}\\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2}\ln(C (\omega_0)/\lambda) \big\}.\end{align*}<br /> <br /> The evidence is controlled by the value of the cost function at the minimum, and by the logarithm of the ratio of the curvature about this minimum compared to the regularization constant. In models with many parameters <br /> \begin{align*} P(y\mid x;M) &amp;\approx e^{-C(\omega_0)}\sqrt{\frac{\lambda}{2\pi}} \int d \omega e^{-C′′(\omega_0)(\omega - \omega_0)^2/2} \\ &amp;= exp \big\{- C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) \big\}.\end{align*}<br /> <br /> Occam’s factor arises from the log ratio &lt;math&gt;\ln (\lambda_i/\lambda) &lt;/math&gt; The Occam factor describes the fraction of the prior parameter space consistent with the data. Occam’s factor penalizes the amount of information the model must learn about the parameters to accurately model the training data. Since the fraction is always less than one, the authors propose to approximate &lt;math&gt;P(y\mid x;M) &lt;/math&gt; away from local minima by only performing the summation over eigenvalues &lt;math&gt;\lambda_i \geq \lambda &lt;/math&gt;.<br /> <br /> The authors compare evidence against a null model which assumes the labels are entirely random. This model has no parameters, and so the evidence is controlled by the likelihood alone. &lt;math&gt;P(y\mid x;NULL) = (1/n)^N = e^{-N \ln(n)} &lt;/math&gt;, where &lt;math&gt;n &lt;/math&gt; denotes the number of model classes and &lt;/math&gt;N &lt;/math&gt; the number of training labels. The evidence ratio :<br /> \begin{equation*}\frac{P(y\mid x;M) }{P(y\mid x;NULL) } = e ^{-E(\omega_0)} \end{equation*}<br /> &lt;math&gt;E(\omega_0) = C(\omega_0)-\frac{1}{2} \sum_{i=1}^p \ln (\lambda_i/\lambda) - N\ln (n) &lt;/math&gt; is the log evidence ratio in favor of the null model.<br /> We assign confidence to the predictions of a model iff &lt;math&gt;E(\omega_0 &lt; 0 &lt;/math&gt;.<br /> <br /> <br /> <br /> ==Critiques==<br /> The paper presents how mini-batch noises with SGD can improve. However, the usefulness of the approach can be described and analyzed in greater details, if the author coudl provide the performance for various well-known real-life datas. Note that even without those evidences, the paper's approach and methods are still very interesting. <br /> <br /> ==Conclusion==<br /> <br /> The paper showed that Mini-batch noise helps SGD to go away from sharp minima, and provided an evidence that there is an optimal optimum batch size for a maximum the test accuracy. Based on interpreting SGD as integrating stochastic differential equation, this batch size is proportional to the learning rate and the training set size. Moreover, the authors shown that Bopt ∝ 1/(1 − m), where m is the momentum coefficient. More analysis was done on the relation between the learning rate, effective learning rate, and batch size is presented in ICLR 2018, where the authors proved by experiments that all the benefits of decaying the learning rate are achieved by increasing the batch size in addition to reducing the number of parameter updates dramatically, and also were able use literature parameters without the need of any hyper parameter tuning (Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le).<br /> <br /> ==References==<br /> <br /> #Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. arXiv preprint arXiv:1612.05086, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012. <br /> #Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference converges to limit cycles for deep networks. arXiv preprint arXiv:1710.11029, 2017.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.<br /> #Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Crispin W Gardiner. Handbook of Stochastic Methods, volume 4. Springer Berlin, 1985.<br /> #Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884– 1892, 2016.<br /> #Priya Goyal, Piotr Dolla ́r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Stephen F Gull. Bayesian inductive inference and maximum entropy. In Maximum-entropy and Bayesian methods in science and engineering, pp. 53–74. Springer, 1988.<br /> #Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM,1993.<br /> #Sepp Hochreiter and Ju ̈rgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Stanisław Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017.<br /> #Robert E Kass and Adrian E Raftery. Bayes factors. Journal of the american statistical association, 90(430):773–795, 1995.<br /> #Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don’t learn via mem- orization. ICLR Workshop, 2017.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101–2110, 2017.<br /> #David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448–472, 1992.<br /> #Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810, 2017.<br /> #Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_Bayesian_Perspective_on_Generalization_and_Stochastic_Gradient_Descent&diff=42042 A Bayesian Perspective on Generalization and Stochastic Gradient Descent 2018-11-30T13:33:42Z <p>C9sharma: /* Introduction */</p> <hr /> <div>==Introduction==<br /> This work builds on Zhang et al.(2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. The authors consider two questions: how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? <br /> They show that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. We also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.<br /> The authors propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the “noise scale” &lt;math display=&quot;inline&quot;&gt; g \approx \epsilon N/B &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;ε&lt;/math&gt; is the learning rate, &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt; the training set size and &lt;math display=&quot;inline&quot;&gt;B&lt;/math&gt; the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, &lt;math display=&quot;inline&quot;&gt;B_{opt} \propto \epsilon N&lt;/math&gt; . They verify these predictions empirically.<br /> <br /> ==Motivation==<br /> This paper shows Bayesian principles can explain many recent observations in the deep learning literature, while also discovering practical new insights. Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite &amp; Roy, 2017). Nonetheless, their results beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? Meanwhile, Keskar et al. (2016) observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking result shows improving our estimate of the full-batch gradient can harm performance. Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Hoffer et al. (2017) proposed a square root rule on theoretical grounds.<br /> Many authors have suggested “broad minima” whose curvature is small may generalize better than “sharp minima” whose curvature is large (Chaudhari et al., 2016; Hochreiter &amp; Schmidhuber, 1997). Indeed, Dziugaite &amp; Roy (2017) argued the results of Zhang et al. (2016) can be understood using “nonvacuous” PAC-Bayes generalization bounds which penalize sharp minima, while Keskar et al. (2016) showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However, Dinh et al. (2017) challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization.<br /> <br /> ==Contribution==<br /> <br /> The main contributions of this paper are to show that:<br /> * The results of Zhang et al. (2016) are not unique to deep learning; it is observed the same phenomenon in a small “over-parameterized” linear model. Overparameterization occurs when a model is able to effectively “remember” training data. This occurs when there are enough parameters that the system of equations ends up with an infinite number of possible solutions. One can see why this over-training would lead to poor results in test cases, as this “memorization” learns noise as opposed to the inherent structure of different classes. It is demonstrated that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.<br /> * SGD integrates a stochastic differential equation whose “noise scale” &lt;math&gt;g &amp;asymp; &amp;epsilon;N/B&lt;/math&gt;, where<br /> &amp;epsilon; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is '''proportional to the learning rate and training set size'''.<br /> <br /> Zhang et al. (2016) showed high training competency of neural networks under informative labels, but drastic overfitting on improper labels. This implies weak generalizability even when a small proportion of labels are improper. The authors show that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Bayesians tend to make distributional assumptions on gradient updates by adding isotropic Gaussian noise. This paper builds upon these Bayesian principles by driving SGD away from sharp minima, and towards broad minima (the more broad, the better generalization due to less influence from small perturbations within input). The stochastic differential equation used as a component of gradient updates effectively serves as injected noise that improves a network's generalizability.<br /> <br /> ==Main Results==<br /> <br /> The weakly regularized model memorizes random labels, however, generalizes properly on informative labels. Besides, the predictions are overconfident. The authors also showed that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is postulated that the optimum represents a tradeoff between depth and breadth in the Bayesian evidence. However it is the underlying scale of random fluctuations in the SGD dynamics which controls the tradeoff, not the batch size itself. Furthermore, this test accuracy peak shifts as the training set size rises. The authors observed that the best found batch size is proportional to the learning rate. This scaling rule allowed the authors to increase the learning rate by simultaneously increasing the batch size with no loss in test accuracy and no increase in computational cost, thus parallelism across multiple GPU's can be fully leveraged to easily decrease training time. The scaling rule could also be applied to production models by consequentially increasing the batch size as new training data is introduced.<br /> <br /> ==Critiques==<br /> The paper presents how mini-batch noises with SGD can improve. However, the usefulness of the approach can be described and analyzed in greater details, if the author coudl provide the performance for various well-known real-life datas. Note that even without those evidences, the paper's approach and methods are still very interesting. <br /> <br /> ==Conclusion==<br /> <br /> The paper showed that Mini-batch noise helps SGD to go away from sharp minima, and provided an evidence that there is an optimal optimum batch size for a maximum the test accuracy. Based on interpreting SGD as integrating stochastic differential equation, this batch size is proportional to the learning rate and the training set size. Moreover, the authors shown that Bopt ∝ 1/(1 − m), where m is the momentum coefficient. More analysis was done on the relation between the learning rate, effective learning rate, and batch size is presented in ICLR 2018, where the authors proved by experiments that all the benefits of decaying the learning rate are achieved by increasing the batch size in addition to reducing the number of parameter updates dramatically, and also were able use literature parameters without the need of any hyper parameter tuning (Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le).<br /> <br /> ==References==<br /> <br /> #Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. arXiv preprint arXiv:1612.05086, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012. <br /> #Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference converges to limit cycles for deep networks. arXiv preprint arXiv:1710.11029, 2017.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.<br /> #Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Crispin W Gardiner. Handbook of Stochastic Methods, volume 4. Springer Berlin, 1985.<br /> #Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884– 1892, 2016.<br /> #Priya Goyal, Piotr Dolla ́r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Stephen F Gull. Bayesian inductive inference and maximum entropy. In Maximum-entropy and Bayesian methods in science and engineering, pp. 53–74. Springer, 1988.<br /> #Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM,1993.<br /> #Sepp Hochreiter and Ju ̈rgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Stanisław Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017.<br /> #Robert E Kass and Adrian E Raftery. Bayes factors. Journal of the american statistical association, 90(430):773–795, 1995.<br /> #Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don’t learn via mem- orization. ICLR Workshop, 2017.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101–2110, 2017.<br /> #David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448–472, 1992.<br /> #Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810, 2017.<br /> #Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_Bayesian_Perspective_on_Generalization_and_Stochastic_Gradient_Descent&diff=42041 A Bayesian Perspective on Generalization and Stochastic Gradient Descent 2018-11-30T13:32:48Z <p>C9sharma: /* References */</p> <hr /> <div>==Introduction==<br /> This work builds on Zhang et al.(2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. The authors consider two questions: how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? <br /> They show that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. We also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.<br /> We propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the “noise scale” &lt;math display=&quot;inline&quot;&gt; g \approx \epsilon N/B &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;ε&lt;/math&gt; is the learning rate, &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt; the training set size and &lt;math display=&quot;inline&quot;&gt;B&lt;/math&gt; the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, &lt;math display=&quot;inline&quot;&gt;B_{opt} \propto \epsilon N&lt;/math&gt; . They verify these predictions empirically.<br /> <br /> ==Motivation==<br /> This paper shows Bayesian principles can explain many recent observations in the deep learning literature, while also discovering practical new insights. Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite &amp; Roy, 2017). Nonetheless, their results beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? Meanwhile, Keskar et al. (2016) observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking result shows improving our estimate of the full-batch gradient can harm performance. Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Hoffer et al. (2017) proposed a square root rule on theoretical grounds.<br /> Many authors have suggested “broad minima” whose curvature is small may generalize better than “sharp minima” whose curvature is large (Chaudhari et al., 2016; Hochreiter &amp; Schmidhuber, 1997). Indeed, Dziugaite &amp; Roy (2017) argued the results of Zhang et al. (2016) can be understood using “nonvacuous” PAC-Bayes generalization bounds which penalize sharp minima, while Keskar et al. (2016) showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However, Dinh et al. (2017) challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization.<br /> <br /> ==Contribution==<br /> <br /> The main contributions of this paper are to show that:<br /> * The results of Zhang et al. (2016) are not unique to deep learning; it is observed the same phenomenon in a small “over-parameterized” linear model. Overparameterization occurs when a model is able to effectively “remember” training data. This occurs when there are enough parameters that the system of equations ends up with an infinite number of possible solutions. One can see why this over-training would lead to poor results in test cases, as this “memorization” learns noise as opposed to the inherent structure of different classes. It is demonstrated that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.<br /> * SGD integrates a stochastic differential equation whose “noise scale” &lt;math&gt;g &amp;asymp; &amp;epsilon;N/B&lt;/math&gt;, where<br /> &amp;epsilon; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is '''proportional to the learning rate and training set size'''.<br /> <br /> Zhang et al. (2016) showed high training competency of neural networks under informative labels, but drastic overfitting on improper labels. This implies weak generalizability even when a small proportion of labels are improper. The authors show that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Bayesians tend to make distributional assumptions on gradient updates by adding isotropic Gaussian noise. This paper builds upon these Bayesian principles by driving SGD away from sharp minima, and towards broad minima (the more broad, the better generalization due to less influence from small perturbations within input). The stochastic differential equation used as a component of gradient updates effectively serves as injected noise that improves a network's generalizability.<br /> <br /> ==Main Results==<br /> <br /> The weakly regularized model memorizes random labels, however, generalizes properly on informative labels. Besides, the predictions are overconfident. The authors also showed that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is postulated that the optimum represents a tradeoff between depth and breadth in the Bayesian evidence. However it is the underlying scale of random fluctuations in the SGD dynamics which controls the tradeoff, not the batch size itself. Furthermore, this test accuracy peak shifts as the training set size rises. The authors observed that the best found batch size is proportional to the learning rate. This scaling rule allowed the authors to increase the learning rate by simultaneously increasing the batch size with no loss in test accuracy and no increase in computational cost, thus parallelism across multiple GPU's can be fully leveraged to easily decrease training time. The scaling rule could also be applied to production models by consequentially increasing the batch size as new training data is introduced.<br /> <br /> ==Critiques==<br /> The paper presents how mini-batch noises with SGD can improve. However, the usefulness of the approach can be described and analyzed in greater details, if the author coudl provide the performance for various well-known real-life datas. Note that even without those evidences, the paper's approach and methods are still very interesting. <br /> <br /> ==Conclusion==<br /> <br /> The paper showed that Mini-batch noise helps SGD to go away from sharp minima, and provided an evidence that there is an optimal optimum batch size for a maximum the test accuracy. Based on interpreting SGD as integrating stochastic differential equation, this batch size is proportional to the learning rate and the training set size. Moreover, the authors shown that Bopt ∝ 1/(1 − m), where m is the momentum coefficient. More analysis was done on the relation between the learning rate, effective learning rate, and batch size is presented in ICLR 2018, where the authors proved by experiments that all the benefits of decaying the learning rate are achieved by increasing the batch size in addition to reducing the number of parameter updates dramatically, and also were able use literature parameters without the need of any hyper parameter tuning (Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le).<br /> <br /> ==References==<br /> <br /> #Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 2017.<br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. arXiv preprint arXiv:1612.05086, 2016.<br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012. <br /> #Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference converges to limit cycles for deep networks. arXiv preprint arXiv:1710.11029, 2017.<br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> #Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.<br /> #Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.<br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> #Crispin W Gardiner. Handbook of Stochastic Methods, volume 4. Springer Berlin, 1985.<br /> #Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884– 1892, 2016.<br /> #Priya Goyal, Piotr Dolla ́r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> #Stephen F Gull. Bayesian inductive inference and maximum entropy. In Maximum-entropy and Bayesian methods in science and engineering, pp. 53–74. Springer, 1988.<br /> #Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM,1993.<br /> #Sepp Hochreiter and Ju ̈rgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> #Stanisław Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017.<br /> #Robert E Kass and Adrian E Raftery. Bayes factors. Journal of the american statistical association, 90(430):773–795, 1995.<br /> #Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.<br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> #David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don’t learn via mem- orization. ICLR Workshop, 2017.<br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101–2110, 2017.<br /> #David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448–472, 1992.<br /> #Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> #Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810, 2017.<br /> #Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.<br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_Bayesian_Perspective_on_Generalization_and_Stochastic_Gradient_Descent&diff=42040 A Bayesian Perspective on Generalization and Stochastic Gradient Descent 2018-11-30T13:31:23Z <p>C9sharma: /* References */</p> <hr /> <div>==Introduction==<br /> This work builds on Zhang et al.(2016), who showed deep neural networks can easily memorize randomly labeled training data, despite generalizing well on real labels of the same inputs. The authors consider two questions: how can we predict if a minimum will generalize to the test set, and why does stochastic gradient descent find minima that generalize well? <br /> They show that the same phenomenon occurs in small linear models. These observations are explained by the Bayesian evidence, which penalizes sharp minima but is invariant to model parameterization. We also demonstrate that, when one holds the learning rate fixed, there is an optimum batch size which maximizes the test set accuracy.<br /> We propose that the noise introduced by small mini-batches drives the parameters towards minima whose evidence is large. Interpreting stochastic gradient descent as a stochastic differential equation, we identify the “noise scale” &lt;math display=&quot;inline&quot;&gt; g \approx \epsilon N/B &lt;/math&gt; where &lt;math display=&quot;inline&quot;&gt;ε&lt;/math&gt; is the learning rate, &lt;math display=&quot;inline&quot;&gt;N&lt;/math&gt; the training set size and &lt;math display=&quot;inline&quot;&gt;B&lt;/math&gt; the batch size. Consequently the optimum batch size is proportional to both the learning rate and the size of the training set, &lt;math display=&quot;inline&quot;&gt;B_{opt} \propto \epsilon N&lt;/math&gt; . They verify these predictions empirically.<br /> <br /> ==Motivation==<br /> This paper shows Bayesian principles can explain many recent observations in the deep learning literature, while also discovering practical new insights. Zhang et al. (2016) trained deep convolutional networks on ImageNet and CIFAR10, achieving excellent accuracy on both training and test sets. They then took the same input images, but randomized the labels, and found that while their networks were now unable to generalize to the test set, they still memorized the training labels. They claimed these results contradict learning theory, although this claim is disputed (Kawaguchi et al., 2017; Dziugaite &amp; Roy, 2017). Nonetheless, their results beg the question; if our models can assign arbitrary labels to the training set, why do they work so well in practice? Meanwhile, Keskar et al. (2016) observed that if we hold the learning rate fixed and increase the batch size, the test accuracy usually falls. This striking result shows improving our estimate of the full-batch gradient can harm performance. Goyal et al. (2017) observed a linear scaling rule between batch size and learning rate in a deep ResNet, while Hoffer et al. (2017) proposed a square root rule on theoretical grounds.<br /> Many authors have suggested “broad minima” whose curvature is small may generalize better than “sharp minima” whose curvature is large (Chaudhari et al., 2016; Hochreiter &amp; Schmidhuber, 1997). Indeed, Dziugaite &amp; Roy (2017) argued the results of Zhang et al. (2016) can be understood using “nonvacuous” PAC-Bayes generalization bounds which penalize sharp minima, while Keskar et al. (2016) showed stochastic gradient descent (SGD) finds wider minima as the batch size is reduced. However, Dinh et al. (2017) challenged this interpretation, by arguing that the curvature of a minimum can be arbitrarily increased by changing the model parameterization.<br /> <br /> ==Contribution==<br /> <br /> The main contributions of this paper are to show that:<br /> * The results of Zhang et al. (2016) are not unique to deep learning; it is observed the same phenomenon in a small “over-parameterized” linear model. Overparameterization occurs when a model is able to effectively “remember” training data. This occurs when there are enough parameters that the system of equations ends up with an infinite number of possible solutions. One can see why this over-training would lead to poor results in test cases, as this “memorization” learns noise as opposed to the inherent structure of different classes. It is demonstrated that this phenomenon is straightforwardly understood by evaluating the Bayesian evidence in favor of each model, which penalizes sharp minima but is invariant to the model parameterization.<br /> * SGD integrates a stochastic differential equation whose “noise scale” &lt;math&gt;g &amp;asymp; &amp;epsilon;N/B&lt;/math&gt;, where<br /> &amp;epsilon; is the learning rate, &lt;math&gt;N&lt;/math&gt; training set size and &lt;math&gt;B&lt;/math&gt; batch size. Noise drives SGD away from sharp minima, and therefore there is an optimal batch size which maximizes the test set accuracy. This optimal batch size is '''proportional to the learning rate and training set size'''.<br /> <br /> Zhang et al. (2016) showed high training competency of neural networks under informative labels, but drastic overfitting on improper labels. This implies weak generalizability even when a small proportion of labels are improper. The authors show that generalization is strongly correlated with the Bayesian evidence, a weighted combination of the depth of a minimum (the cost function) and its breadth (the Occam factor). Bayesians tend to make distributional assumptions on gradient updates by adding isotropic Gaussian noise. This paper builds upon these Bayesian principles by driving SGD away from sharp minima, and towards broad minima (the more broad, the better generalization due to less influence from small perturbations within input). The stochastic differential equation used as a component of gradient updates effectively serves as injected noise that improves a network's generalizability.<br /> <br /> ==Main Results==<br /> <br /> The weakly regularized model memorizes random labels, however, generalizes properly on informative labels. Besides, the predictions are overconfident. The authors also showed that the test accuracy peaks at an optimal batch size, if one holds the other SGD hyper-parameters constant. It is postulated that the optimum represents a tradeoff between depth and breadth in the Bayesian evidence. However it is the underlying scale of random fluctuations in the SGD dynamics which controls the tradeoff, not the batch size itself. Furthermore, this test accuracy peak shifts as the training set size rises. The authors observed that the best found batch size is proportional to the learning rate. This scaling rule allowed the authors to increase the learning rate by simultaneously increasing the batch size with no loss in test accuracy and no increase in computational cost, thus parallelism across multiple GPU's can be fully leveraged to easily decrease training time. The scaling rule could also be applied to production models by consequentially increasing the batch size as new training data is introduced.<br /> <br /> ==Critiques==<br /> The paper presents how mini-batch noises with SGD can improve. However, the usefulness of the approach can be described and analyzed in greater details, if the author coudl provide the performance for various well-known real-life datas. Note that even without those evidences, the paper's approach and methods are still very interesting. <br /> <br /> ==Conclusion==<br /> <br /> The paper showed that Mini-batch noise helps SGD to go away from sharp minima, and provided an evidence that there is an optimal optimum batch size for a maximum the test accuracy. Based on interpreting SGD as integrating stochastic differential equation, this batch size is proportional to the learning rate and the training set size. Moreover, the authors shown that Bopt ∝ 1/(1 − m), where m is the momentum coefficient. More analysis was done on the relation between the learning rate, effective learning rate, and batch size is presented in ICLR 2018, where the authors proved by experiments that all the benefits of decaying the learning rate are achieved by increasing the batch size in addition to reducing the number of parameter updates dramatically, and also were able use literature parameters without the need of any hyper parameter tuning (Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le).<br /> <br /> ==References==<br /> #Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 2017.<br /> <br /> #Lukas Balles, Javier Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates. arXiv preprint arXiv:1612.05086, 2016.<br /> <br /> #Richard H Byrd, Gillian M Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathematical programming, 134(1):127–155, 2012. <br /> <br /> #Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference converges to limit cycles for deep networks. arXiv preprint arXiv:1710.11029, 2017.<br /> <br /> #Pratik Chaudhari, Anna Choromanska, Stefano Soatto, and Yann LeCun. Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.<br /> <br /> #Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. In Artificial Intelligence and Statistics, pp. 1504–1513, 2017.<br /> <br /> #Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv preprint arXiv:1703.04933, 2017.<br /> <br /> #Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.<br /> <br /> #Michael P Friedlander and Mark Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380–A1405, 2012.<br /> <br /> #Crispin W Gardiner. Handbook of Stochastic Methods, volume 4. Springer Berlin, 1985.<br /> <br /> #Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884– 1892, 2016.<br /> <br /> #Priya Goyal, Piotr Dolla ́r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.<br /> <br /> #Stephen F Gull. Bayesian inductive inference and maximum entropy. In Maximum-entropy and Bayesian methods in science and engineering, pp. 53–74. Springer, 1988.<br /> <br /> #Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM,1993.<br /> <br /> #Sepp Hochreiter and Ju ̈rgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017.<br /> <br /> #Stanisław Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623, 2017.<br /> <br /> #Robert E Kass and Adrian E Raftery. Bayes factors. Journal of the american statistical association, 90(430):773–795, 1995.<br /> <br /> #Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.<br /> <br /> #Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.<br /> <br /> #David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville. Deep nets don’t learn via mem- orization. ICLR Workshop, 2017.<br /> <br /> #Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101–2110, 2017.<br /> <br /> #David JC MacKay. A practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448–472, 1992.<br /> <br /> #Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.<br /> <br /> #Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv:1703.00810, 2017.<br /> <br /> #Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.<br /> <br /> #Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681–688, 2011.<br /> <br /> #Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Attend_and_Predict:_Understanding_Gene_Regulation_by_Selective_Attention_on_Chromatin&diff=42039 Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin 2018-11-30T13:23:41Z <p>C9sharma: /* Previous model for gene expression predictions */</p> <hr /> <div>This page contains a summary of the paper [https://arxiv.org/abs/1708.00339 &quot;Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin.&quot;] by Singh, Ritambhara, et al. It was published at the Advances in Neural Information Processing Systems (NIPS) in 2017. The code for this paper is shared here[https://qdata.github.io/deep4biomed-web/].<br /> <br /> <br /> = Background =<br /> <br /> Gene regulation is the process of controlling which genes in a cell's DNA are turned 'on' (expressed) or 'off' (not expressed). By this process, a functional product such as a protein is created. Even though all the cells of a multicellular organism (e.g., humans) contain the same DNA, different types of cells in that organism may express very different sets of genes. As a result, each cell types have distinct functionality. In other words how a cell operates depends upon the genes expressed in that cell. Many factors including ‘Chromatin modification marks’ influence which genes are abundant in that cell.<br /> <br /> The function of chromatin is to efficiently wraps DNA around bead-like structures of histones into a condensed volume to fit into the nucleus of a cell, and protect the DNA structure and sequence during cell division and replication. Different chemical modifications in the histones of the chromatin, known as histone marks, change spatial arrangement of the condensed DNA structure. Which in turn affects the gene’s expression of the histone mark’s neighboring region. Histone marks can promote (obstruct) the gene to be turned on by making the gene region accessible (restricted). This section of the DNA, where histone marks can potentially have an impact, is known as DNA flanking region or ‘gene region’ which is considered to cover 10k base pair centered at the transcription start site (TSS) (i.e., a 5k base pair in each direction). Unlike genetic mutations, histone modifications are reversible . Therefore, understanding the influence of histone marks in determining gene regulation can assist in developing drugs for genetic diseases.<br /> <br /> = Introduction = <br /> <br /> Revolution in genomic technologies now enables us to profile genome-wide chromatin mark signals. Therefore, biologists can now measure gene expressions and chromatin signals of the ‘gene region’ for different cell types covering whole human genome. The Roadmap Epigenome Project (REMC, publicly available)  recently released 2,804 genome-wide datasets of 100 separate “normal” (not diseased) human cells/tissues, among which 166 datasets are gene expression reads and the rest are signal reads of various histone marks. The goal is to understand which histone marks are the most important and how they interact together in gene regulation for each cell type.<br /> <br /> Signal reads for histone marks are high-dimensional and spatially structured. Influence of a histone modification mark can be anywhere in the gene region (covering 10k base pairs centered around the Transcription Start Site of each gene). It is important to understand how the impact of the mark on gene expression varies over the gene region. In other words, how histone signals over the gene region impacts the gene expression. There are different types of histone marks in human chromatin that can have an influence on gene regulation. Researchers have found five standard histone proteins. These five histone proteins can be altered in different combinations with different chemical modifications resulting in a large number of distinct histone modification marks. Different histone modification marks can act as a module to interact with each other and influence the gene expression.<br /> <br /> <br /> This paper proposes an attention-based deep learning model to find how this chromatin factors/ histone modification marks contributes to the gene expression of a particular cell. AttentiveChrome utilizes a hierarchy of multiple LSTM to discover interactions between signals of each histone marks, and learn dependencies among the marks on expressing a gene. The authors included two levels of soft attention mechanism, (1) to attend to the most relevant signals of a histone mark, and (2) to attend to the important marks and their interactions. In this context, ''attention'' refers to weighting the importance of different items differently.<br /> <br /> == Main Contributions ==<br /> The contributions of this work can be summarized as follows:<br /> <br /> * More accurate predictions than the state-of-the-art baselines. This is measured using datasets from REMC on 56 different cell types.<br /> * Better interpretation than the state-of-the-art methods for visualizing deep learning model. They compute the correlation of the attention scores of the model with the mark signal from REMC. <br /> * Like the application of attention models previously in indirectly hinting the parts of the input that the model deemed important, AttentiveChrome can too explain it's decisions by hinting at “what” and “where” it has focused.<br /> * This is the first time that the attention based deep learning approach is applied to a problem in molecular biology.<br /> * Ability to deal with highly modular inputs<br /> <br /> = Previous Works = <br /> <br /> Machine learning algorithms to classify gene expression from histone modification signals have been surveyed by . These algorithms vary from linear regression, support vector machine, and random forests to rule-based learning, and CNNs. To accommodate the spatially structured, high dimensional input data (histone modification signals) these studies applied different feature selection strategies. The preceding research study, DeepChrome , by the authors incorporated the best position selection strategy. The positions that are highly correlated to the gene expression are considered as the best positions. This model can learn the relationship between the histone marks. This CNN based DeepChrome model outperforms all the previous works. However, these approaches either (1) failed to model the spatial dependencies among the marks, or (2) required additional feature analysis. Only AttentiveChrome is reported to satisfy all of the eight desirable metrics of a model.<br /> <br /> = AttentiveChrome: Model Formulation =<br /> <br /> The authors proposed an end-to-end architecture which has the ability to simultaneously attend and predict. This method incorporates recurrent neural networks (RNN) composed of LSTM units to model the sequential spatial dependencies of the gene regions and predict gene expression level from The embedding vector, &lt;math&gt; h_t &lt;/math&gt;, output of an LSTM module encodes the learned representation of the feature dependencies from the time step 0 to &lt;math&gt; t &lt;/math&gt;. For this task, each bin position of the gene region is considered as a time step.<br /> <br /> The proposed AttentiveChrome framework contains following 5 important modules:<br /> <br /> * Bin-level LSTM encoder encoding the bin positions of the gene region (one for each HM mark)<br /> * Bin-level &lt;math&gt; \alpha &lt;/math&gt;-Attention across all bin positions (one for each HM mark)<br /> * HM-level LSTM encoder (one encoder encoding all HM marks)<br /> * HM-level &lt;math&gt; \beta &lt;/math&gt;-Attention among all HM marks (one)<br /> * The final classification module<br /> <br /> Figure 1 (Supplementary Figure 2) presents an overview of the proposed AttentiveChrome framework.<br /> <br /> <br /> [[File:supplemntary_figure_2.png|thumb|center| 800px |Figure 1: Overview of the all five modules of the proposed AttentiveChrome framework]]<br /> <br /> <br /> <br /> == Input and Output ==<br /> <br /> Each dataset contains the gene expression labels and the histone signal reads for one specific cell type. The authors evaluated AttentiveChrome on 56 different cell types. For each mark, we have a feature/input vector containing the signals reads surrounding the gene’s TSS position (gene region) for the histone mark. The label of this input vector denotes the gene expression of the specific gene. This study considers binary labeling where &lt;math&gt; +1 &lt;/math&gt; denotes gene is expressed (on) and &lt;math&gt; -1 &lt;/math&gt; denotes that the gene is not expressed (off). Each histone marks will have one feature vector for each gene. The authors integrates the feature inputs and outputs of their previous work DeepChrome  into this research. The input feature is represented by a matrix &lt;math&gt; \textbf{X} &lt;/math&gt; of size &lt;math&gt; M \times T &lt;/math&gt;, where &lt;math&gt; M &lt;/math&gt; is the number of HM marks considered in the input, and &lt;math&gt; T &lt;/math&gt; is the number of bin positions taken into account to represent the gene region. The &lt;math&gt; j^{th} &lt;/math&gt; row of the vector &lt;math&gt; \textbf{X} &lt;/math&gt;, &lt;math&gt; x_j&lt;/math&gt;, represents sequentially structured signals from the &lt;math&gt; j^{th} &lt;/math&gt; HM mark, where &lt;math&gt; j\in \{1, \cdots, M\} &lt;/math&gt;. Therefore, &lt;math&gt; x_j^t&lt;/math&gt;, in the matrix &lt;math&gt; \textbf{X} &lt;/math&gt; represents the value from the &lt;math&gt; t^{th}&lt;/math&gt; bin belonging to the &lt;math&gt; j^{th} &lt;/math&gt; HM mark, where &lt;math&gt; t\in \{1, \cdots, T\} &lt;/math&gt;. If the training set contains &lt;math&gt;N_{tr} &lt;/math&gt; labeled pairs, the &lt;math&gt; n^{th} &lt;/math&gt; is specified as &lt;math&gt;( X^n, y^n)&lt;/math&gt;, where &lt;math&gt; X^n &lt;/math&gt; is a matrix of size &lt;math&gt; M \times T &lt;/math&gt; and &lt;math&gt; y^n \in \{ -1, +1 \} &lt;/math&gt; is the binary label, and &lt;math&gt; n \in \{ 1, \cdots, N_{tr} \} &lt;/math&gt;.<br /> <br /> Figure 2 (also refer to Figure 1 (a), and 1(b) for better understanding) exhibits the input feature, and the output of AttentiveChrome for a particular gene (one sample).<br /> <br /> [[File:input-output-attentivechrome.png|center|thumb| 700px | Figure 2: Input and Output of the AttentiveChrome model]]<br /> <br /> == Bin-Level Encoder (one LSTM for each HM) ==<br /> The sequentially ordered elements (each element actually is a bin position) of the gene region of &lt;math&gt; n^{th} &lt;/math&gt; gene is represented by the &lt;math&gt; j_{th} &lt;/math&gt; row vector &lt;math&gt; x^j &lt;/math&gt;. The authors considered each bin position as a time step for LSTM. This study incorporates bidirectional LSTM to model the overall dependencies among a total of &lt;math&gt; T &lt;/math&gt; bin positions in the gene region. The bidirectional LSTM contains two LSTMs<br /> * A forward LSTM, &lt;math&gt; \overrightarrow{LSTM_j} &lt;/math&gt;, to model &lt;math&gt; x^j &lt;/math&gt; from &lt;math&gt; x_1^j &lt;/math&gt; to &lt;math&gt; x_T^j &lt;/math&gt;, which outputs the embedding vector &lt;math&gt; \overrightarrow{h^t_j} &lt;/math&gt;, of size &lt;math&gt; d &lt;/math&gt; for each bin &lt;math&gt; t &lt;/math&gt;<br /> * A reverse LSTM, &lt;math&gt; \overleftarrow{LSTM_j} &lt;/math&gt;, to model &lt;math&gt; x^j &lt;/math&gt; from &lt;math&gt; x_T^j &lt;/math&gt; to &lt;math&gt; x_1^j &lt;/math&gt;, which outputs the embedding vector &lt;math&gt; \overleftarrow{h^j_t} &lt;/math&gt;, of size &lt;math&gt; d &lt;/math&gt; for each bin &lt;math&gt; t &lt;/math&gt;<br /> <br /> The final output of this layer, embedding vector at &lt;math&gt; t^{th} &lt;/math&gt; bin for the &lt;math&gt; j^{th} &lt;/math&gt; HM, &lt;math&gt; h^j_t &lt;/math&gt;, of size &lt;math&gt; d &lt;/math&gt;, is obtained by concatenating the two vectors from the both directions. Therefore, &lt;math&gt; h^j_t = [ \overrightarrow{h^j_t}, \overleftarrow{h^j_t}]&lt;/math&gt;. By pairing these LSTM-based HM encoders with the final classification, embedding each HM mark by drawing out the dependencies among bins can be learned by these pairs.Figure 1 (c) illustrates the module for &lt;math&gt; j=2 &lt;/math&gt;.<br /> <br /> == Bin-Level &lt;math&gt; \alpha&lt;/math&gt;-attention ==<br /> <br /> Each bin contributes differently in the encoding of the entire &lt;math&gt; j^{th} &lt;/math&gt; mark. To automatically and adaptively highlight the most important bins for prediction, a soft attention weight vector &lt;math&gt; \alpha^j &lt;/math&gt; of size &lt;math&gt; T &lt;/math&gt; is learned for each &lt;math&gt; j &lt;/math&gt;. To calculated the soft weight &lt;math&gt; \alpha^j_t &lt;/math&gt;, for each &lt;math&gt; t &lt;/math&gt;, the embedding vectors &lt;math&gt; \{h^j_1, \cdots, h^j_t \} &lt;/math&gt; of all the bins are utilized. The following equation is used:<br /> <br /> &lt;center&gt;&lt;math&gt; \alpha^j_t = \frac{exp(\textbf{W}_b h^j_t)}{\sum_{i=1}^T{exp(\textbf{W}_b h^j_i)}} &lt;/math&gt;&lt;/center&gt;<br /> <br /> <br /> &lt;math&gt; \alpha^j_t&lt;/math&gt; is a scalar and is computed by all bins’ embedding vectors &lt;math&gt;h^j&lt;/math&gt;. The parameter &lt;math&gt; W_b &lt;/math&gt; is initialized randomly, and learned alongside during the process with the other model parameters. Therefore, once we have importance weight of each bin position, the &lt;math&gt; j^{th} &lt;/math&gt; HM mark can be represented by &lt;math&gt; m^j = \sum_{t=1}^T{\alpha^j_t \times h^j_t}&lt;/math&gt;. Here, &lt;math&gt; h^j_t&lt;/math&gt; is the embedding vector and &lt;math&gt; \alpha^t_j &lt;/math&gt; is the importance weight of the &lt;math&gt; t^{th} &lt;/math&gt; bin in the representation of the &lt;math&gt; j^{th} &lt;/math&gt; HM mark. Intuitively &lt;math&gt; \textbf{W}_b &lt;/math&gt; will learn the cell type. Figure 1(d) shows this module for &lt;math&gt; HM_2 &lt;/math&gt;.<br /> <br /> == HM-level Encoder (one LSTM) ==<br /> <br /> Studies observed that HMs work cooperatively to provoke or subdue gene expression . The HM-level encoder (not in the fFgure 1) utilizes one bidirectional LSTM to capture this relationship between the HMs. To formulate the sequential dependency a random sequence is imagined as the authors did not find influence of any specific ordering of the HMs. The representation &lt;math&gt; m_j &lt;/math&gt;of the &lt;math&gt; j^{th} &lt;/math&gt; HM, &lt;math&gt; HM_j &lt;/math&gt;, which is calculated from the bin-level attention layer, is the input of this step. This set based encoder outputs an embedding vector &lt;math&gt; s^j &lt;/math&gt; of size &lt;math&gt; d’ &lt;/math&gt;, which is the encoding for the &lt;math&gt; j^{th} &lt;/math&gt; HM.<br /> <br /> &lt;math&gt; s^j = [ \overrightarrow{LSTM_s}(m_j), \overleftarrow{LSTM_s}(m_j) ] &lt;/math&gt;<br /> <br /> The dependencies between &lt;math&gt; j^{th} &lt;/math&gt; HM and the other HM marks are encoded in &lt;math&gt; s^j &lt;/math&gt;, whereas &lt;math&gt; m^j &lt;/math&gt; from the previous step encodes the bin dependencies of the &lt;math&gt; j^{th} &lt;/math&gt; HM.<br /> <br /> <br /> == HM-Level &lt;math&gt; \beta&lt;/math&gt;-attention ==<br /> This second soft attention level (Figure 1(e)) finds the important HM marks for classifying a gene’s expression by learning the importance weights, &lt;math&gt; \beta_j &lt;/math&gt;, for each &lt;math&gt; HM_j &lt;/math&gt;, where &lt;math&gt; j \in \{ 1, \cdots, M \} &lt;/math&gt;. The equation is <br /> <br /> &lt;math&gt; \beta^j = \frac{exp(\textbf{W}_s s^j)}{\sum_{i=1}^M{exp(\textbf{W}_s s^j)}} &lt;/math&gt;<br /> <br /> The HM-level context parameter &lt;math&gt; \textbf{W}_s &lt;/math&gt; is trained jointly in the process. Intuitively &lt;math&gt; \textbf{W}_s &lt;/math&gt; learns how the HMs are significant for a cell type. Finally the entire gene region is encoded in a hidden representation &lt;math&gt; \textbf{v} &lt;/math&gt;, using the weighted sum of the embedding of all HM marks. <br /> <br /> <br /> &lt;math&gt; \textbf{v} = \sum_{j=1}^MT{\beta^j \times s^j}&lt;/math&gt;<br /> <br /> == End-to-end training ==<br /> <br /> The embedding vector &lt;math&gt; \textbf{v} &lt;/math&gt; is fed to a simple classification module, &lt;math&gt; f(\textbf{v}) = &lt;/math&gt;softmax&lt;math&gt; (\textbf{W}_c\textbf{v}+b_c) &lt;/math&gt;, where &lt;math&gt; \textbf{W}_c &lt;/math&gt;, and &lt;math&gt; b_c &lt;/math&gt; are learnable parameters. The output is the probability of gene expression being high (expressed) or low (suppressed).<br /> The whole model including the attention modules is differentiable. Thus backpropagation can perform end-to-end learning trivially. The negative log-likelihood loss function is minimized in the learning.<br /> <br /> = Experimental Settings =<br /> <br /> This work makes use of the REMC dataset. AttentiveChrome is evaluated on 56 different cell types. Similar to DeepChrome, this study considered the following five core HM marks (&lt;math&gt; M=5 &lt;/math&gt;). Because these selected marks are uniformly profiled across all 56 cell types in the REMC study.<br /> <br /> [[File:HM.png|center|thumb| 700px | Table 1: Five core HM marks and their attributes considered in this paper]]<br /> <br /> <br /> <br /> For a gene region 10k base pairs centred at the TSS site (5k bp in each direction) are taken into account. These 10k base pairs are divided into 100 bins, each bin consisting of &lt;math&gt; T=100 &lt;/math&gt; continuous bp). Therefore, for each gene in a particular cell type, the input matrix will be of size &lt;math&gt; 5 \times 100 &lt;/math&gt;. The gene expression labels are normalized and discretized to represent binary labelling. The sample dataset is divided into three equal sized folds for training, validation, and testing.<br /> <br /> == Model Variations and Two Baselines ==<br /> To evaluate the performance of the proposed model the authors considered RNN method (direct LSTM without any attention), and their prior work DeepChrome as baselines. The results obtained from multiple variations of the AttentiveChrome model are compared with the baselines. The authors considered five variant of AttentiveChrome during performance evaluation. The variants are:<br /> <br /> * LSTM-Attn: one LSTM with attention on the input matrix (does not consider the modular nature of HM marks)<br /> * CNN-Attn: DeepChrome  with one attention mechanism incorporated. <br /> * LSTM-&lt;math&gt;\alpha , \beta&lt;/math&gt;: the proposed architecture.<br /> * CNN-&lt;math&gt;\alpha , \beta&lt;/math&gt;: LSTM module of the proposed architecture replaced with CNN. This variation includes two attention mechanisms. First attention mechanism contains one &lt;math&gt;\alpha&lt;/math&gt;-attention on top of a CNN module per HM mark. And, the second -&lt;math&gt;\beta&lt;/math&gt;- attention mechanism is used to combine HMs.<br /> * LSTM-&lt;math&gt;\alpha&lt;/math&gt;: one LSTM and &lt;math&gt;\alpha&lt;/math&gt;-attention per HM mark.<br /> <br /> == Hyperparameters ==<br /> <br /> For all the variants of AttentiveChrome the bin-level LSTM embedding size &lt;math&gt; d&lt;/math&gt; is set to 32, and the HM-level LSTM embedding size &lt;math&gt;d’&lt;/math&gt; is set to 16. Because of bidirectional LSTM, the size of the embedding vector &lt;math&gt; h_t&lt;/math&gt;, and &lt;math&gt;m_j&lt;/math&gt; will be 64, and 32 respectively. Size of the context vectors are set accordingly.<br /> <br /> = Performance Evaluation =<br /> <br /> == AUC Scores ==<br /> <br /> This study summarizes AUC scores across all 56 cell types on the test set to compare the methods.<br /> <br /> [[File:AUC.JPG|center|thumb| 700px | Table 2: AUC score performances for different variations of AttentiveChrome and baselines]]<br /> <br /> Overall the LSTM-attention models perform better than the DeepChrome (CNN-based) and LSTM baselines. The authors argue that the proposed AttentiveChrome model is a good choice because of its interpretability, even though the performance improvement from DeepChrome is insignificant.<br /> <br /> == Evaluation of Attention Scores for Interpretation ==<br /> <br /> To understand if the model is focusing on the right regions, the authors make use of additional study results from REMC database. To validate the bin attention,signal data of a new histone mark, H3K27ac, referred to as &lt;math&gt;H_{active}&lt;/math&gt; in this article, from REMC database is utilized. This particular histone mark is known to mark active region when the gene is expressed (ON). Genome-wide read of this HM mark is available for three important cell types: stem cell (H1-hESC), blood cell (GM12878), and leukemia cell (K562). This particular HM mark is used to analyze the visualization results only and not applied in the learning phase. The authors discussed performance of both the attention mechanisms in this section. <br /> <br /> === Correlation of Importance Weight of &lt;math&gt;H_{prom}&lt;/math&gt; with &lt;math&gt;H_{active}&lt;/math&gt; ===<br /> <br /> Average read count of &lt;math&gt;H_{active}&lt;/math&gt; across all 100 bins for all the active genes (ON or labeled as &lt;math&gt;+1&lt;/math&gt;) in the three selected cell types is calculated. The proposed AttentiveChrome and LSTM-&lt;math&gt;\alpha&lt;/math&gt; methods are compared with two widely used visualization techniques, (1) class based, and (2) saliency map applied on the baseline DeepChrome model (CNN-based prior work). Using these visualization methods, the authors calculate the importance weights for &lt;math&gt;H_{prom}&lt;/math&gt; (promoter HM mark used in training) across the 100 bins. The Pearson Correlation score between these importance weights and the read count of the &lt;math&gt;H_{active}&lt;/math&gt; (HM mark for validation) across the same 100 bins is computed. The &lt;math&gt;H_{active}&lt;/math&gt; read counts indicates the actual active regions of those cells. <br /> <br /> [[File: pc.JPG|center|thumb| 700px | Figure 4: Pearson Correlation between a known active HM mark]]<br /> <br /> <br /> The results indicate that the proposed models consistently gained highest correlation with &lt;math&gt;H_{active}&lt;/math&gt; for all three cell types. Thus, the proposed method is successful to capture the important signals.<br /> <br /> === Visualization of Attention Weight of bins for each HM of a specific cell type GM12878===<br /> <br /> To visualize bin level attention weights, the authors plotted the average bin-level attention weights for each HM for a specific cell type GM12878 (blood cell) for expressed (ON) genes and suppressed (OFF) genes separately. <br /> <br /> [[File: figure2.png|center|thumb| 700px |]]<br /> <br /> For the “ON” genes, the attention profiles are well defined for the HM marks, &lt;math&gt;H_{prom}&lt;/math&gt;, &lt;math&gt;H_{enhc}&lt;/math&gt;, &lt;math&gt;H_{struct}&lt;/math&gt;. On the other hand, the weights are low for &lt;math&gt;H_{reprA}&lt;/math&gt; and &lt;math&gt;H_{reprB}&lt;/math&gt;. The average trend reverses for the “OFF” genes, where the repressor HM marks have more influence than the &lt;math&gt;H_{prom}&lt;/math&gt;, &lt;math&gt;H_{enhc}&lt;/math&gt;, &lt;math&gt;H_{struct}&lt;/math&gt;. This observation agrees with the biologist finding that &lt;math&gt;H_{prom}&lt;/math&gt;, &lt;math&gt;H_{enhc}&lt;/math&gt;, &lt;math&gt;H_{struct}&lt;/math&gt; marks stimulates gene activation and, &lt;math&gt;H_{reprA}&lt;/math&gt; and &lt;math&gt;H_{reprB}&lt;/math&gt; mark restrains the genes.<br /> <br /> === Attention Weight of bins with &lt;math&gt;H_{active}&lt;/math&gt;===<br /> <br /> The average read counts of &lt;math&gt;H_{active}&lt;/math&gt; for the same 100 bins across all the active (ON) genes for the cell type GM12878 is plotted (FIGURE 2(b)). Besides, for AttentiveChrome the plot of bin-level attention weights of averaged over all the genes that are PREDICTED ON for GM12878 is also provided. The plots exhibit that the &lt;math&gt;H_{prom}&lt;/math&gt; profile is similar to &lt;math&gt;H_{active}&lt;/math&gt;.<br /> <br /> === Visualization of HM-level Attention Weight for Gene PAX5 ===<br /> <br /> To visualize HM-level attention weight the authors produces a heatmap for a differentially regulated gene, PAX5, for the three aforementioned cell types. The heatmap is presented in FIGURE 2(c). PAX5 plays significant role in gene regulation when stem cells convert to blood cells. This gene is OFF in stem cells (H1-hESC), however it becomes activated when the stem cell is transformed into blood cell (GM12878). The &lt;math&gt;\beta_j&lt;/math&gt; weight for &lt;math&gt;H_{repr}&lt;/math&gt; is high when the gene is OFF in H1-hESC, and the weight decreases when the gene is ON in GM12878. On the contrary, for &lt;math&gt;H_{prom}&lt;/math&gt; mark the &lt;math&gt;\beta_j&lt;/math&gt; weight increases from H1-hESC to GM12878 as the gene becomes activated. This information extracted by the deep learning model is also supported by biological literature .<br /> <br /> = Related Works/Studies =<br /> <br /> In the last few years, deep learning models obtained models obtained unprecedented success in diverse research fields. Though as not rapidly as other fields, deep learning based algorithms are gaining popularity among bioinformaticians.<br /> <br /> == Attention-based Deep Models ==<br /> <br /> The idea of attention technique in deep learning is adapted from the human visual perception system. Humans tend to focus over some parts more than the others while perceiving a scene. This mechanism augmented with deep neural networks achieved an excellent outcome in several research topics, such as machine translation. Various types of attention models e.g., soft , or location-aware , or hard [8, 9] attentions have been proposed in the literature. In the soft attention model, a soft weight vector is calculated for the overall feature vectors. The extent of the weight is correlated with the degree of importance of the feature in the prediction. In practice, RNN is often used to help implement such models.<br /> <br /> == Visualization and Apprehension of Deep Models ==<br /> <br /> Prior studies mostly focused on interpreting convolutional neural networks (CNN) for image classification. Deconvulation approaches  attempt to map hidden layer representations back to an input space. Saliency maps [11, 12], attempt to use taylor expansion to approximate the network, and identify the most relevant input features. Class optimization  based visualization techniques attempt to find the best example member of each class. Some recent research works [13, 14] tried to understand recurrent neural networks (RNN) for text-based problems. By looking into the features the model attends to, we can interpret the output of a deep model.<br /> <br /> == Deep Learning in Bioinformatics ==<br /> Deep learning is also getting popular in bioinformatics fields because it is able to extract meaningful representations from datasets. Scholars use deep learning to model protein sequences and DNA sequences and predicting gene expressions.<br /> <br /> == Previous model for gene expression predictions ==<br /> There were multiple machine learning models had been used to predict gene expressions from histone modification data (surveyed in ), such as linear regression, random forests, rule-based learning  and CNNs  and support vector machines.These studies designed different feature selection strategies to accommodate a large amount of histone modification signals as input. The strategies included using signal averaging across all relevant positions and selecting input signals at positions where was highly correlated to target gene expression and then use CNN (called DeepChrome ) to learn combinatorial interactions among histone modification marks. DeepChrome outperformed all previous methods (see Supplementary) on this task and used a class optimization-based technique for visualizing the learned model. However, this class-level visualization lacks the necessary granularity to understand the signals from multiple chromatin marks at the individual gene level.<br /> <br /> = Conclusion = <br /> <br /> The paper has introduced an attention-based approach called &quot;AttentiveChrome&quot; that deals with both understanding and prediction with several advantages on previous architectures including higher accuracy from state-of-the-art baselines, clearer interpretation than saliency map, which allows them to view what the model ‘sees’ during prediction, and class optimization. Another advantage of this approach is that it can model modular feature inputs which are sequentially structured. Finally, according to the authors, this is the first implementation of deep attention to understand gene regulation. AttentiveChrome is claimed to be the first attention based model applied on a molecular biology dataset. The authors expect that through this deep attention mechanism, the biologists can have a better understanding of epigenomic data. This model can handle understanding and prediction of hard to interpret biological data as it grants insights<br /> to the predictions by locating ‘what’ and ‘where’ AttentiveChrome has focused.<br /> <br /> = Critiques =<br /> <br /> This paper does not give a considerable algorithmic contribution. They have only used existing methods for this application. This deep learning based method is shown to perform better than simple machine learning models like linear regression and SVMs but this is considerably harder to implement and has many more hyperparameters to tune. The training time is considerably higher, especially because all the parameters are learned together. The dataset considered in the application here also seems to have only a limited number of samples for a study of high complexity. Model hyperparameters have been chosen randomly without any explanation of intuition for them. The authors have also not cited any relevant literature to understand where these numbers came from. <br /> <br /> Discussion about attention scores for interpretation does not provide any clear definition or mention previous literature using them. Reference of literature about H3K27ac, and how its read counts represent active region of a cell should be included. No reasoning given for why only one specific cell type is used to visualize bin level attention weights. Example of some other real world problems where this model can be useful should be provided.<br /> <br /> Moreover, this paper relies heavily on the intuition. Due to complicated structures, it must be challenging to provide algorithmic/theoretical justifications. This means that there is no proper guidence of how hyperparameters should be chosen or any kinds of treatment that the author performs on other data sets.<br /> <br /> = Additional Resources =<br /> <br /> # [https://qdata.github.io/deep4biomed-web/ Official DeepChrome Website]<br /> # [http://papers.nips.cc/paper/7255-attend-and-predict-understanding-gene-regulation-by-selective-attention-on-chromatin-supplemental.zip Supplemental Resources]<br /> # [https://github.com/QData/AttentiveChrome/blob/master/NIPS%20poster.pdf Poster]<br /> # [https://www.youtube.com/watch?v=tfgmXvSgsQE&amp;feature=youtu.be Video Presentation]<br /> <br /> = Reference =<br /> <br />  Andrew J Bannister and Tony Kouzarides. Regulation of chromatin by histone modifications. Cell Research, 21(3):381–395, 2011.<br /> <br />  Anshul Kundaje, Wouter Meuleman, Jason Ernst, Misha Bilenky, Angela Yen, Alireza Heravi-Moussavi, Pouya Kheradpour, Zhizhuo Zhang, Jianrong Wang, Michael J Ziller, et al. Integrative analysis of 111 reference human epigenomes. Nature, 518(7539):317–330, 2015.<br /> <br />  Singh, Ritambhara, et al. &quot;Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin.&quot; Advances in Neural Information Processing Systems. 2017.<br /> <br />  Ritambhara Singh, Jack Lanchantin, Gabriel Robins, and Yanjun Qi. Deepchrome: deep-learning for predicting gene expression from histone modifications. Bioinformatics, 32(17):i639–i648, 2016.<br /> <br />  Joanna Boros, Nausica Arnoult, Vincent Stroobant, Jean-François Collet, and Anabelle Decottignies. Polycomb repressive complex 2 and h3k27me3 cooperate with h3k9 methylation to maintain heterochromatin protein 1α at chromatin. Molecular and cellular biology, 34(19):3662–3674, 2014.<br /> <br />  Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.<br /> <br />  Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 577–585. Curran Associates, Inc., 2015.<br /> <br />  Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1412–1421, Lisbon, Portugal, September 2015. Association for Computational Linguistics.<br /> <br />  Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In ECCV, 2016.<br /> <br />  Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014, pages 818–833. Springer, 2014.<br /> <br />  David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert MÃžller. How to explain individual classification decisions. volume 11, pages 1803–1831, 2010.<br /> <br />  Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. 2013.<br /> <br />  Andrej Karpathy, Justin Johnson, and Fei-Fei Li. Visualizing and understanding recurrent networks. 2015.<br /> <br />  Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nlp. 2015.<br /> <br />  Xianjun Dong and Zhiping Weng. The correlation between histone modifications and gene expression. Epigenomics, 5(2):113–116, 2013.<br /> <br />  Shane McManus, Anja Ebert, Giorgia Salvagiotto, Jasna Medvedovic, Qiong Sun, Ido Tamir, Markus Jaritz, Hiromi Tagoh, and Meinrad Busslinger. The transcription factor pax5 regulates its target genes by recruiting chromatin-modifying proteins in committed b cells. The EMBO journal, 30(12):2388–2404, 2011.<br /> <br />  ChaoCheng,Koon-KiuYan,KevinYYip,JoelRozowsky,RogerAlexander,ChongShou,MarkGerstein, et al. A statistical framework for modeling gene expression using chromatin features and application to modencode datasets. Genome Biol, 12(2):R15, 2011.<br /> <br />  XianjunDong,MelissaCGreven,AnshulKundaje,SarahDjebali,JamesBBrown,ChaoCheng,ThomasR Gingeras, Mark Gerstein, Roderic Guigó, Ewan Birney, et al. Modeling gene expression using chromatin features in various cellular contexts. Genome Biol, 13(9):R53, 2012.<br /> <br />  Xianjun Dong and Zhiping Weng. The correlation between histone modifications and gene expression. Epigenomics, 5(2):113–116, 2013.<br /> <br />  Bich Hai Ho, Rania Mohammed Kotb Hassen, and Ngoc Tu Le. Combinatorial roles of dna methylation and histone modifications on gene expression. In Some Current Advanced Researches on Information and Computer Science in Vietnam, pages 123–135. Springer, 2015.<br /> <br />  Rosa Karlic ́, Ho-Ryun Chung, Julia Lasserre, Kristian Vlahovicˇek, and Martin Vingron. Histone mod- ification levels are predictive for gene expression. Proceedings of the National Academy of Sciences, 107(7):2926–2931, 2010.<br /> <br />  Ritambhara Singh, Jack Lanchantin, Gabriel Robins, and Yanjun Qi. Deepchrome: deep-learning for predicting gene expression from histone modifications. Bioinformatics, 32(17):i639–i648, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Attend_and_Predict:_Understanding_Gene_Regulation_by_Selective_Attention_on_Chromatin&diff=42038 Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin 2018-11-30T13:23:36Z <p>C9sharma: /* Reference */</p> <hr /> <div>This page contains a summary of the paper [https://arxiv.org/abs/1708.00339 &quot;Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin.&quot;] by Singh, Ritambhara, et al. It was published at the Advances in Neural Information Processing Systems (NIPS) in 2017. The code for this paper is shared here[https://qdata.github.io/deep4biomed-web/].<br /> <br /> <br /> = Background =<br /> <br /> Gene regulation is the process of controlling which genes in a cell's DNA are turned 'on' (expressed) or 'off' (not expressed). By this process, a functional product such as a protein is created. Even though all the cells of a multicellular organism (e.g., humans) contain the same DNA, different types of cells in that organism may express very different sets of genes. As a result, each cell types have distinct functionality. In other words how a cell operates depends upon the genes expressed in that cell. Many factors including ‘Chromatin modification marks’ influence which genes are abundant in that cell.<br /> <br /> The function of chromatin is to efficiently wraps DNA around bead-like structures of histones into a condensed volume to fit into the nucleus of a cell, and protect the DNA structure and sequence during cell division and replication. Different chemical modifications in the histones of the chromatin, known as histone marks, change spatial arrangement of the condensed DNA structure. Which in turn affects the gene’s expression of the histone mark’s neighboring region. Histone marks can promote (obstruct) the gene to be turned on by making the gene region accessible (restricted). This section of the DNA, where histone marks can potentially have an impact, is known as DNA flanking region or ‘gene region’ which is considered to cover 10k base pair centered at the transcription start site (TSS) (i.e., a 5k base pair in each direction). Unlike genetic mutations, histone modifications are reversible . Therefore, understanding the influence of histone marks in determining gene regulation can assist in developing drugs for genetic diseases.<br /> <br /> = Introduction = <br /> <br /> Revolution in genomic technologies now enables us to profile genome-wide chromatin mark signals. Therefore, biologists can now measure gene expressions and chromatin signals of the ‘gene region’ for different cell types covering whole human genome. The Roadmap Epigenome Project (REMC, publicly available)  recently released 2,804 genome-wide datasets of 100 separate “normal” (not diseased) human cells/tissues, among which 166 datasets are gene expression reads and the rest are signal reads of various histone marks. The goal is to understand which histone marks are the most important and how they interact together in gene regulation for each cell type.<br /> <br /> Signal reads for histone marks are high-dimensional and spatially structured. Influence of a histone modification mark can be anywhere in the gene region (covering 10k base pairs centered around the Transcription Start Site of each gene). It is important to understand how the impact of the mark on gene expression varies over the gene region. In other words, how histone signals over the gene region impacts the gene expression. There are different types of histone marks in human chromatin that can have an influence on gene regulation. Researchers have found five standard histone proteins. These five histone proteins can be altered in different combinations with different chemical modifications resulting in a large number of distinct histone modification marks. Different histone modification marks can act as a module to interact with each other and influence the gene expression.<br /> <br /> <br /> This paper proposes an attention-based deep learning model to find how this chromatin factors/ histone modification marks contributes to the gene expression of a particular cell. AttentiveChrome utilizes a hierarchy of multiple LSTM to discover interactions between signals of each histone marks, and learn dependencies among the marks on expressing a gene. The authors included two levels of soft attention mechanism, (1) to attend to the most relevant signals of a histone mark, and (2) to attend to the important marks and their interactions. In this context, ''attention'' refers to weighting the importance of different items differently.<br /> <br /> == Main Contributions ==<br /> The contributions of this work can be summarized as follows:<br /> <br /> * More accurate predictions than the state-of-the-art baselines. This is measured using datasets from REMC on 56 different cell types.<br /> * Better interpretation than the state-of-the-art methods for visualizing deep learning model. They compute the correlation of the attention scores of the model with the mark signal from REMC. <br /> * Like the application of attention models previously in indirectly hinting the parts of the input that the model deemed important, AttentiveChrome can too explain it's decisions by hinting at “what” and “where” it has focused.<br /> * This is the first time that the attention based deep learning approach is applied to a problem in molecular biology.<br /> * Ability to deal with highly modular inputs<br /> <br /> = Previous Works = <br /> <br /> Machine learning algorithms to classify gene expression from histone modification signals have been surveyed by . These algorithms vary from linear regression, support vector machine, and random forests to rule-based learning, and CNNs. To accommodate the spatially structured, high dimensional input data (histone modification signals) these studies applied different feature selection strategies. The preceding research study, DeepChrome , by the authors incorporated the best position selection strategy. The positions that are highly correlated to the gene expression are considered as the best positions. This model can learn the relationship between the histone marks. This CNN based DeepChrome model outperforms all the previous works. However, these approaches either (1) failed to model the spatial dependencies among the marks, or (2) required additional feature analysis. Only AttentiveChrome is reported to satisfy all of the eight desirable metrics of a model.<br /> <br /> = AttentiveChrome: Model Formulation =<br /> <br /> The authors proposed an end-to-end architecture which has the ability to simultaneously attend and predict. This method incorporates recurrent neural networks (RNN) composed of LSTM units to model the sequential spatial dependencies of the gene regions and predict gene expression level from The embedding vector, &lt;math&gt; h_t &lt;/math&gt;, output of an LSTM module encodes the learned representation of the feature dependencies from the time step 0 to &lt;math&gt; t &lt;/math&gt;. For this task, each bin position of the gene region is considered as a time step.<br /> <br /> The proposed AttentiveChrome framework contains following 5 important modules:<br /> <br /> * Bin-level LSTM encoder encoding the bin positions of the gene region (one for each HM mark)<br /> * Bin-level &lt;math&gt; \alpha &lt;/math&gt;-Attention across all bin positions (one for each HM mark)<br /> * HM-level LSTM encoder (one encoder encoding all HM marks)<br /> * HM-level &lt;math&gt; \beta &lt;/math&gt;-Attention among all HM marks (one)<br /> * The final classification module<br /> <br /> Figure 1 (Supplementary Figure 2) presents an overview of the proposed AttentiveChrome framework.<br /> <br /> <br /> [[File:supplemntary_figure_2.png|thumb|center| 800px |Figure 1: Overview of the all five modules of the proposed AttentiveChrome framework]]<br /> <br /> <br /> <br /> == Input and Output ==<br /> <br /> Each dataset contains the gene expression labels and the histone signal reads for one specific cell type. The authors evaluated AttentiveChrome on 56 different cell types. For each mark, we have a feature/input vector containing the signals reads surrounding the gene’s TSS position (gene region) for the histone mark. The label of this input vector denotes the gene expression of the specific gene. This study considers binary labeling where &lt;math&gt; +1 &lt;/math&gt; denotes gene is expressed (on) and &lt;math&gt; -1 &lt;/math&gt; denotes that the gene is not expressed (off). Each histone marks will have one feature vector for each gene. The authors integrates the feature inputs and outputs of their previous work DeepChrome  into this research. The input feature is represented by a matrix &lt;math&gt; \textbf{X} &lt;/math&gt; of size &lt;math&gt; M \times T &lt;/math&gt;, where &lt;math&gt; M &lt;/math&gt; is the number of HM marks considered in the input, and &lt;math&gt; T &lt;/math&gt; is the number of bin positions taken into account to represent the gene region. The &lt;math&gt; j^{th} &lt;/math&gt; row of the vector &lt;math&gt; \textbf{X} &lt;/math&gt;, &lt;math&gt; x_j&lt;/math&gt;, represents sequentially structured signals from the &lt;math&gt; j^{th} &lt;/math&gt; HM mark, where &lt;math&gt; j\in \{1, \cdots, M\} &lt;/math&gt;. Therefore, &lt;math&gt; x_j^t&lt;/math&gt;, in the matrix &lt;math&gt; \textbf{X} &lt;/math&gt; represents the value from the &lt;math&gt; t^{th}&lt;/math&gt; bin belonging to the &lt;math&gt; j^{th} &lt;/math&gt; HM mark, where &lt;math&gt; t\in \{1, \cdots, T\} &lt;/math&gt;. If the training set contains &lt;math&gt;N_{tr} &lt;/math&gt; labeled pairs, the &lt;math&gt; n^{th} &lt;/math&gt; is specified as &lt;math&gt;( X^n, y^n)&lt;/math&gt;, where &lt;math&gt; X^n &lt;/math&gt; is a matrix of size &lt;math&gt; M \times T &lt;/math&gt; and &lt;math&gt; y^n \in \{ -1, +1 \} &lt;/math&gt; is the binary label, and &lt;math&gt; n \in \{ 1, \cdots, N_{tr} \} &lt;/math&gt;.<br /> <br /> Figure 2 (also refer to Figure 1 (a), and 1(b) for better understanding) exhibits the input feature, and the output of AttentiveChrome for a particular gene (one sample).<br /> <br /> [[File:input-output-attentivechrome.png|center|thumb| 700px | Figure 2: Input and Output of the AttentiveChrome model]]<br /> <br /> == Bin-Level Encoder (one LSTM for each HM) ==<br /> The sequentially ordered elements (each element actually is a bin position) of the gene region of &lt;math&gt; n^{th} &lt;/math&gt; gene is represented by the &lt;math&gt; j_{th} &lt;/math&gt; row vector &lt;math&gt; x^j &lt;/math&gt;. The authors considered each bin position as a time step for LSTM. This study incorporates bidirectional LSTM to model the overall dependencies among a total of &lt;math&gt; T &lt;/math&gt; bin positions in the gene region. The bidirectional LSTM contains two LSTMs<br /> * A forward LSTM, &lt;math&gt; \overrightarrow{LSTM_j} &lt;/math&gt;, to model &lt;math&gt; x^j &lt;/math&gt; from &lt;math&gt; x_1^j &lt;/math&gt; to &lt;math&gt; x_T^j &lt;/math&gt;, which outputs the embedding vector &lt;math&gt; \overrightarrow{h^t_j} &lt;/math&gt;, of size &lt;math&gt; d &lt;/math&gt; for each bin &lt;math&gt; t &lt;/math&gt;<br /> * A reverse LSTM, &lt;math&gt; \overleftarrow{LSTM_j} &lt;/math&gt;, to model &lt;math&gt; x^j &lt;/math&gt; from &lt;math&gt; x_T^j &lt;/math&gt; to &lt;math&gt; x_1^j &lt;/math&gt;, which outputs the embedding vector &lt;math&gt; \overleftarrow{h^j_t} &lt;/math&gt;, of size &lt;math&gt; d &lt;/math&gt; for each bin &lt;math&gt; t &lt;/math&gt;<br /> <br /> The final output of this layer, embedding vector at &lt;math&gt; t^{th} &lt;/math&gt; bin for the &lt;math&gt; j^{th} &lt;/math&gt; HM, &lt;math&gt; h^j_t &lt;/math&gt;, of size &lt;math&gt; d &lt;/math&gt;, is obtained by concatenating the two vectors from the both directions. Therefore, &lt;math&gt; h^j_t = [ \overrightarrow{h^j_t}, \overleftarrow{h^j_t}]&lt;/math&gt;. By pairing these LSTM-based HM encoders with the final classification, embedding each HM mark by drawing out the dependencies among bins can be learned by these pairs.Figure 1 (c) illustrates the module for &lt;math&gt; j=2 &lt;/math&gt;.<br /> <br /> == Bin-Level &lt;math&gt; \alpha&lt;/math&gt;-attention ==<br /> <br /> Each bin contributes differently in the encoding of the entire &lt;math&gt; j^{th} &lt;/math&gt; mark. To automatically and adaptively highlight the most important bins for prediction, a soft attention weight vector &lt;math&gt; \alpha^j &lt;/math&gt; of size &lt;math&gt; T &lt;/math&gt; is learned for each &lt;math&gt; j &lt;/math&gt;. To calculated the soft weight &lt;math&gt; \alpha^j_t &lt;/math&gt;, for each &lt;math&gt; t &lt;/math&gt;, the embedding vectors &lt;math&gt; \{h^j_1, \cdots, h^j_t \} &lt;/math&gt; of all the bins are utilized. The following equation is used:<br /> <br /> &lt;center&gt;&lt;math&gt; \alpha^j_t = \frac{exp(\textbf{W}_b h^j_t)}{\sum_{i=1}^T{exp(\textbf{W}_b h^j_i)}} &lt;/math&gt;&lt;/center&gt;<br /> <br /> <br /> &lt;math&gt; \alpha^j_t&lt;/math&gt; is a scalar and is computed by all bins’ embedding vectors &lt;math&gt;h^j&lt;/math&gt;. The parameter &lt;math&gt; W_b &lt;/math&gt; is initialized randomly, and learned alongside during the process with the other model parameters. Therefore, once we have importance weight of each bin position, the &lt;math&gt; j^{th} &lt;/math&gt; HM mark can be represented by &lt;math&gt; m^j = \sum_{t=1}^T{\alpha^j_t \times h^j_t}&lt;/math&gt;. Here, &lt;math&gt; h^j_t&lt;/math&gt; is the embedding vector and &lt;math&gt; \alpha^t_j &lt;/math&gt; is the importance weight of the &lt;math&gt; t^{th} &lt;/math&gt; bin in the representation of the &lt;math&gt; j^{th} &lt;/math&gt; HM mark. Intuitively &lt;math&gt; \textbf{W}_b &lt;/math&gt; will learn the cell type. Figure 1(d) shows this module for &lt;math&gt; HM_2 &lt;/math&gt;.<br /> <br /> == HM-level Encoder (one LSTM) ==<br /> <br /> Studies observed that HMs work cooperatively to provoke or subdue gene expression . The HM-level encoder (not in the fFgure 1) utilizes one bidirectional LSTM to capture this relationship between the HMs. To formulate the sequential dependency a random sequence is imagined as the authors did not find influence of any specific ordering of the HMs. The representation &lt;math&gt; m_j &lt;/math&gt;of the &lt;math&gt; j^{th} &lt;/math&gt; HM, &lt;math&gt; HM_j &lt;/math&gt;, which is calculated from the bin-level attention layer, is the input of this step. This set based encoder outputs an embedding vector &lt;math&gt; s^j &lt;/math&gt; of size &lt;math&gt; d’ &lt;/math&gt;, which is the encoding for the &lt;math&gt; j^{th} &lt;/math&gt; HM.<br /> <br /> &lt;math&gt; s^j = [ \overrightarrow{LSTM_s}(m_j), \overleftarrow{LSTM_s}(m_j) ] &lt;/math&gt;<br /> <br /> The dependencies between &lt;math&gt; j^{th} &lt;/math&gt; HM and the other HM marks are encoded in &lt;math&gt; s^j &lt;/math&gt;, whereas &lt;math&gt; m^j &lt;/math&gt; from the previous step encodes the bin dependencies of the &lt;math&gt; j^{th} &lt;/math&gt; HM.<br /> <br /> <br /> == HM-Level &lt;math&gt; \beta&lt;/math&gt;-attention ==<br /> This second soft attention level (Figure 1(e)) finds the important HM marks for classifying a gene’s expression by learning the importance weights, &lt;math&gt; \beta_j &lt;/math&gt;, for each &lt;math&gt; HM_j &lt;/math&gt;, where &lt;math&gt; j \in \{ 1, \cdots, M \} &lt;/math&gt;. The equation is <br /> <br /> &lt;math&gt; \beta^j = \frac{exp(\textbf{W}_s s^j)}{\sum_{i=1}^M{exp(\textbf{W}_s s^j)}} &lt;/math&gt;<br /> <br /> The HM-level context parameter &lt;math&gt; \textbf{W}_s &lt;/math&gt; is trained jointly in the process. Intuitively &lt;math&gt; \textbf{W}_s &lt;/math&gt; learns how the HMs are significant for a cell type. Finally the entire gene region is encoded in a hidden representation &lt;math&gt; \textbf{v} &lt;/math&gt;, using the weighted sum of the embedding of all HM marks. <br /> <br /> <br /> &lt;math&gt; \textbf{v} = \sum_{j=1}^MT{\beta^j \times s^j}&lt;/math&gt;<br /> <br /> == End-to-end training ==<br /> <br /> The embedding vector &lt;math&gt; \textbf{v} &lt;/math&gt; is fed to a simple classification module, &lt;math&gt; f(\textbf{v}) = &lt;/math&gt;softmax&lt;math&gt; (\textbf{W}_c\textbf{v}+b_c) &lt;/math&gt;, where &lt;math&gt; \textbf{W}_c &lt;/math&gt;, and &lt;math&gt; b_c &lt;/math&gt; are learnable parameters. The output is the probability of gene expression being high (expressed) or low (suppressed).<br /> The whole model including the attention modules is differentiable. Thus backpropagation can perform end-to-end learning trivially. The negative log-likelihood loss function is minimized in the learning.<br /> <br /> = Experimental Settings =<br /> <br /> This work makes use of the REMC dataset. AttentiveChrome is evaluated on 56 different cell types. Similar to DeepChrome, this study considered the following five core HM marks (&lt;math&gt; M=5 &lt;/math&gt;). Because these selected marks are uniformly profiled across all 56 cell types in the REMC study.<br /> <br /> [[File:HM.png|center|thumb| 700px | Table 1: Five core HM marks and their attributes considered in this paper]]<br /> <br /> <br /> <br /> For a gene region 10k base pairs centred at the TSS site (5k bp in each direction) are taken into account. These 10k base pairs are divided into 100 bins, each bin consisting of &lt;math&gt; T=100 &lt;/math&gt; continuous bp). Therefore, for each gene in a particular cell type, the input matrix will be of size &lt;math&gt; 5 \times 100 &lt;/math&gt;. The gene expression labels are normalized and discretized to represent binary labelling. The sample dataset is divided into three equal sized folds for training, validation, and testing.<br /> <br /> == Model Variations and Two Baselines ==<br /> To evaluate the performance of the proposed model the authors considered RNN method (direct LSTM without any attention), and their prior work DeepChrome as baselines. The results obtained from multiple variations of the AttentiveChrome model are compared with the baselines. The authors considered five variant of AttentiveChrome during performance evaluation. The variants are:<br /> <br /> * LSTM-Attn: one LSTM with attention on the input matrix (does not consider the modular nature of HM marks)<br /> * CNN-Attn: DeepChrome  with one attention mechanism incorporated. <br /> * LSTM-&lt;math&gt;\alpha , \beta&lt;/math&gt;: the proposed architecture.<br /> * CNN-&lt;math&gt;\alpha , \beta&lt;/math&gt;: LSTM module of the proposed architecture replaced with CNN. This variation includes two attention mechanisms. First attention mechanism contains one &lt;math&gt;\alpha&lt;/math&gt;-attention on top of a CNN module per HM mark. And, the second -&lt;math&gt;\beta&lt;/math&gt;- attention mechanism is used to combine HMs.<br /> * LSTM-&lt;math&gt;\alpha&lt;/math&gt;: one LSTM and &lt;math&gt;\alpha&lt;/math&gt;-attention per HM mark.<br /> <br /> == Hyperparameters ==<br /> <br /> For all the variants of AttentiveChrome the bin-level LSTM embedding size &lt;math&gt; d&lt;/math&gt; is set to 32, and the HM-level LSTM embedding size &lt;math&gt;d’&lt;/math&gt; is set to 16. Because of bidirectional LSTM, the size of the embedding vector &lt;math&gt; h_t&lt;/math&gt;, and &lt;math&gt;m_j&lt;/math&gt; will be 64, and 32 respectively. Size of the context vectors are set accordingly.<br /> <br /> = Performance Evaluation =<br /> <br /> == AUC Scores ==<br /> <br /> This study summarizes AUC scores across all 56 cell types on the test set to compare the methods.<br /> <br /> [[File:AUC.JPG|center|thumb| 700px | Table 2: AUC score performances for different variations of AttentiveChrome and baselines]]<br /> <br /> Overall the LSTM-attention models perform better than the DeepChrome (CNN-based) and LSTM baselines. The authors argue that the proposed AttentiveChrome model is a good choice because of its interpretability, even though the performance improvement from DeepChrome is insignificant.<br /> <br /> == Evaluation of Attention Scores for Interpretation ==<br /> <br /> To understand if the model is focusing on the right regions, the authors make use of additional study results from REMC database. To validate the bin attention,signal data of a new histone mark, H3K27ac, referred to as &lt;math&gt;H_{active}&lt;/math&gt; in this article, from REMC database is utilized. This particular histone mark is known to mark active region when the gene is expressed (ON). Genome-wide read of this HM mark is available for three important cell types: stem cell (H1-hESC), blood cell (GM12878), and leukemia cell (K562). This particular HM mark is used to analyze the visualization results only and not applied in the learning phase. The authors discussed performance of both the attention mechanisms in this section. <br /> <br /> === Correlation of Importance Weight of &lt;math&gt;H_{prom}&lt;/math&gt; with &lt;math&gt;H_{active}&lt;/math&gt; ===<br /> <br /> Average read count of &lt;math&gt;H_{active}&lt;/math&gt; across all 100 bins for all the active genes (ON or labeled as &lt;math&gt;+1&lt;/math&gt;) in the three selected cell types is calculated. The proposed AttentiveChrome and LSTM-&lt;math&gt;\alpha&lt;/math&gt; methods are compared with two widely used visualization techniques, (1) class based, and (2) saliency map applied on the baseline DeepChrome model (CNN-based prior work). Using these visualization methods, the authors calculate the importance weights for &lt;math&gt;H_{prom}&lt;/math&gt; (promoter HM mark used in training) across the 100 bins. The Pearson Correlation score between these importance weights and the read count of the &lt;math&gt;H_{active}&lt;/math&gt; (HM mark for validation) across the same 100 bins is computed. The &lt;math&gt;H_{active}&lt;/math&gt; read counts indicates the actual active regions of those cells. <br /> <br /> [[File: pc.JPG|center|thumb| 700px | Figure 4: Pearson Correlation between a known active HM mark]]<br /> <br /> <br /> The results indicate that the proposed models consistently gained highest correlation with &lt;math&gt;H_{active}&lt;/math&gt; for all three cell types. Thus, the proposed method is successful to capture the important signals.<br /> <br /> === Visualization of Attention Weight of bins for each HM of a specific cell type GM12878===<br /> <br /> To visualize bin level attention weights, the authors plotted the average bin-level attention weights for each HM for a specific cell type GM12878 (blood cell) for expressed (ON) genes and suppressed (OFF) genes separately. <br /> <br /> [[File: figure2.png|center|thumb| 700px |]]<br /> <br /> For the “ON” genes, the attention profiles are well defined for the HM marks, &lt;math&gt;H_{prom}&lt;/math&gt;, &lt;math&gt;H_{enhc}&lt;/math&gt;, &lt;math&gt;H_{struct}&lt;/math&gt;. On the other hand, the weights are low for &lt;math&gt;H_{reprA}&lt;/math&gt; and &lt;math&gt;H_{reprB}&lt;/math&gt;. The average trend reverses for the “OFF” genes, where the repressor HM marks have more influence than the &lt;math&gt;H_{prom}&lt;/math&gt;, &lt;math&gt;H_{enhc}&lt;/math&gt;, &lt;math&gt;H_{struct}&lt;/math&gt;. This observation agrees with the biologist finding that &lt;math&gt;H_{prom}&lt;/math&gt;, &lt;math&gt;H_{enhc}&lt;/math&gt;, &lt;math&gt;H_{struct}&lt;/math&gt; marks stimulates gene activation and, &lt;math&gt;H_{reprA}&lt;/math&gt; and &lt;math&gt;H_{reprB}&lt;/math&gt; mark restrains the genes.<br /> <br /> === Attention Weight of bins with &lt;math&gt;H_{active}&lt;/math&gt;===<br /> <br /> The average read counts of &lt;math&gt;H_{active}&lt;/math&gt; for the same 100 bins across all the active (ON) genes for the cell type GM12878 is plotted (FIGURE 2(b)). Besides, for AttentiveChrome the plot of bin-level attention weights of averaged over all the genes that are PREDICTED ON for GM12878 is also provided. The plots exhibit that the &lt;math&gt;H_{prom}&lt;/math&gt; profile is similar to &lt;math&gt;H_{active}&lt;/math&gt;.<br /> <br /> === Visualization of HM-level Attention Weight for Gene PAX5 ===<br /> <br /> To visualize HM-level attention weight the authors produces a heatmap for a differentially regulated gene, PAX5, for the three aforementioned cell types. The heatmap is presented in FIGURE 2(c). PAX5 plays significant role in gene regulation when stem cells convert to blood cells. This gene is OFF in stem cells (H1-hESC), however it becomes activated when the stem cell is transformed into blood cell (GM12878). The &lt;math&gt;\beta_j&lt;/math&gt; weight for &lt;math&gt;H_{repr}&lt;/math&gt; is high when the gene is OFF in H1-hESC, and the weight decreases when the gene is ON in GM12878. On the contrary, for &lt;math&gt;H_{prom}&lt;/math&gt; mark the &lt;math&gt;\beta_j&lt;/math&gt; weight increases from H1-hESC to GM12878 as the gene becomes activated. This information extracted by the deep learning model is also supported by biological literature .<br /> <br /> = Related Works/Studies =<br /> <br /> In the last few years, deep learning models obtained models obtained unprecedented success in diverse research fields. Though as not rapidly as other fields, deep learning based algorithms are gaining popularity among bioinformaticians.<br /> <br /> == Attention-based Deep Models ==<br /> <br /> The idea of attention technique in deep learning is adapted from the human visual perception system. Humans tend to focus over some parts more than the others while perceiving a scene. This mechanism augmented with deep neural networks achieved an excellent outcome in several research topics, such as machine translation. Various types of attention models e.g., soft , or location-aware , or hard [8, 9] attentions have been proposed in the literature. In the soft attention model, a soft weight vector is calculated for the overall feature vectors. The extent of the weight is correlated with the degree of importance of the feature in the prediction. In practice, RNN is often used to help implement such models.<br /> <br /> == Visualization and Apprehension of Deep Models ==<br /> <br /> Prior studies mostly focused on interpreting convolutional neural networks (CNN) for image classification. Deconvulation approaches  attempt to map hidden layer representations back to an input space. Saliency maps [11, 12], attempt to use taylor expansion to approximate the network, and identify the most relevant input features. Class optimization  based visualization techniques attempt to find the best example member of each class. Some recent research works [13, 14] tried to understand recurrent neural networks (RNN) for text-based problems. By looking into the features the model attends to, we can interpret the output of a deep model.<br /> <br /> == Deep Learning in Bioinformatics ==<br /> Deep learning is also getting popular in bioinformatics fields because it is able to extract meaningful representations from datasets. Scholars use deep learning to model protein sequences and DNA sequences and predicting gene expressions.<br /> <br /> == Previous model for gene expression predictions ==<br /> There were multiple machine learning models had been used to predict gene expressions, such as linear regression and support vector machines. The strategies included using signal averaging across all relevant positions and selecting input signals at positions where was highly correlated to target gene expression and then use CNN to learn combinatorial interactions among histone modification marks.<br /> <br /> = Conclusion = <br /> <br /> The paper has introduced an attention-based approach called &quot;AttentiveChrome&quot; that deals with both understanding and prediction with several advantages on previous architectures including higher accuracy from state-of-the-art baselines, clearer interpretation than saliency map, which allows them to view what the model ‘sees’ during prediction, and class optimization. Another advantage of this approach is that it can model modular feature inputs which are sequentially structured. Finally, according to the authors, this is the first implementation of deep attention to understand gene regulation. AttentiveChrome is claimed to be the first attention based model applied on a molecular biology dataset. The authors expect that through this deep attention mechanism, the biologists can have a better understanding of epigenomic data. This model can handle understanding and prediction of hard to interpret biological data as it grants insights<br /> to the predictions by locating ‘what’ and ‘where’ AttentiveChrome has focused.<br /> <br /> = Critiques =<br /> <br /> This paper does not give a considerable algorithmic contribution. They have only used existing methods for this application. This deep learning based method is shown to perform better than simple machine learning models like linear regression and SVMs but this is considerably harder to implement and has many more hyperparameters to tune. The training time is considerably higher, especially because all the parameters are learned together. The dataset considered in the application here also seems to have only a limited number of samples for a study of high complexity. Model hyperparameters have been chosen randomly without any explanation of intuition for them. The authors have also not cited any relevant literature to understand where these numbers came from. <br /> <br /> Discussion about attention scores for interpretation does not provide any clear definition or mention previous literature using them. Reference of literature about H3K27ac, and how its read counts represent active region of a cell should be included. No reasoning given for why only one specific cell type is used to visualize bin level attention weights. Example of some other real world problems where this model can be useful should be provided.<br /> <br /> Moreover, this paper relies heavily on the intuition. Due to complicated structures, it must be challenging to provide algorithmic/theoretical justifications. This means that there is no proper guidence of how hyperparameters should be chosen or any kinds of treatment that the author performs on other data sets.<br /> <br /> = Additional Resources =<br /> <br /> # [https://qdata.github.io/deep4biomed-web/ Official DeepChrome Website]<br /> # [http://papers.nips.cc/paper/7255-attend-and-predict-understanding-gene-regulation-by-selective-attention-on-chromatin-supplemental.zip Supplemental Resources]<br /> # [https://github.com/QData/AttentiveChrome/blob/master/NIPS%20poster.pdf Poster]<br /> # [https://www.youtube.com/watch?v=tfgmXvSgsQE&amp;feature=youtu.be Video Presentation]<br /> <br /> = Reference =<br /> <br />  Andrew J Bannister and Tony Kouzarides. Regulation of chromatin by histone modifications. Cell Research, 21(3):381–395, 2011.<br /> <br />  Anshul Kundaje, Wouter Meuleman, Jason Ernst, Misha Bilenky, Angela Yen, Alireza Heravi-Moussavi, Pouya Kheradpour, Zhizhuo Zhang, Jianrong Wang, Michael J Ziller, et al. Integrative analysis of 111 reference human epigenomes. Nature, 518(7539):317–330, 2015.<br /> <br />  Singh, Ritambhara, et al. &quot;Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin.&quot; Advances in Neural Information Processing Systems. 2017.<br /> <br />  Ritambhara Singh, Jack Lanchantin, Gabriel Robins, and Yanjun Qi. Deepchrome: deep-learning for predicting gene expression from histone modifications. Bioinformatics, 32(17):i639–i648, 2016.<br /> <br />  Joanna Boros, Nausica Arnoult, Vincent Stroobant, Jean-François Collet, and Anabelle Decottignies. Polycomb repressive complex 2 and h3k27me3 cooperate with h3k9 methylation to maintain heterochromatin protein 1α at chromatin. Molecular and cellular biology, 34(19):3662–3674, 2014.<br /> <br />  Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.<br /> <br />  Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 577–585. Curran Associates, Inc., 2015.<br /> <br />  Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1412–1421, Lisbon, Portugal, September 2015. Association for Computational Linguistics.<br /> <br />  Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In ECCV, 2016.<br /> <br />  Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014, pages 818–833. Springer, 2014.<br /> <br />  David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert MÃžller. How to explain individual classification decisions. volume 11, pages 1803–1831, 2010.<br /> <br />  Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. 2013.<br /> <br />  Andrej Karpathy, Justin Johnson, and Fei-Fei Li. Visualizing and understanding recurrent networks. 2015.<br /> <br />  Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nlp. 2015.<br /> <br />  Xianjun Dong and Zhiping Weng. The correlation between histone modifications and gene expression. Epigenomics, 5(2):113–116, 2013.<br /> <br />  Shane McManus, Anja Ebert, Giorgia Salvagiotto, Jasna Medvedovic, Qiong Sun, Ido Tamir, Markus Jaritz, Hiromi Tagoh, and Meinrad Busslinger. The transcription factor pax5 regulates its target genes by recruiting chromatin-modifying proteins in committed b cells. The EMBO journal, 30(12):2388–2404, 2011.<br /> <br />  ChaoCheng,Koon-KiuYan,KevinYYip,JoelRozowsky,RogerAlexander,ChongShou,MarkGerstein, et al. A statistical framework for modeling gene expression using chromatin features and application to modencode datasets. Genome Biol, 12(2):R15, 2011.<br /> <br />  XianjunDong,MelissaCGreven,AnshulKundaje,SarahDjebali,JamesBBrown,ChaoCheng,ThomasR Gingeras, Mark Gerstein, Roderic Guigó, Ewan Birney, et al. Modeling gene expression using chromatin features in various cellular contexts. Genome Biol, 13(9):R53, 2012.<br /> <br />  Xianjun Dong and Zhiping Weng. The correlation between histone modifications and gene expression. Epigenomics, 5(2):113–116, 2013.<br /> <br />  Bich Hai Ho, Rania Mohammed Kotb Hassen, and Ngoc Tu Le. Combinatorial roles of dna methylation and histone modifications on gene expression. In Some Current Advanced Researches on Information and Computer Science in Vietnam, pages 123–135. Springer, 2015.<br /> <br />  Rosa Karlic ́, Ho-Ryun Chung, Julia Lasserre, Kristian Vlahovicˇek, and Martin Vingron. Histone mod- ification levels are predictive for gene expression. Proceedings of the National Academy of Sciences, 107(7):2926–2931, 2010.<br /> <br />  Ritambhara Singh, Jack Lanchantin, Gabriel Robins, and Yanjun Qi. Deepchrome: deep-learning for predicting gene expression from histone modifications. Bioinformatics, 32(17):i639–i648, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Robot_Learning_in_Homes:_Improving_Generalization_and_Reducing_Dataset_Bias&diff=42037 Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias 2018-11-30T13:14:19Z <p>C9sharma: /* Conclusion */</p> <hr /> <div>==Introduction==<br /> <br /> <br /> The use of data-driven approaches in robotics has increased in the last decade. Instead of using hand-designed models, these data-driven approaches work on large-scale datasets and learn appropriate policies that map from high-dimensional observations to actions. Since collecting data using an actual robot in real-time is very expensive, most of the data-driven approaches in robotics use simulators in order to collect simulated data. The concern here is whether these approaches have the capability to be robust enough to domain shift and to be used for real-world data. It is an undeniable fact that there is a wide reality gap between simulators and the real world.<br /> <br /> This has motivated the robotics community to increase their efforts in collecting real-world physical interaction data for a variety of tasks. This effort has been accelerated by the declining costs of hardware. This approach has been quite successful at tasks such as grasping, pushing, poking and imitation learning. However, the major problem is that the performance of these learning models are not good enough and tend to plateau fast. Furthermore, robotic action data did not lead to similar gains in other areas such as computer vision and natural language processing. As the paper claimed, the solution for all of these obstacles is using “real data”. Current robotic datasets lack diversity of environment. Learning-based approaches need to move out of simulators in the labs and go to real environments such as real homes so that they can learn from real datasets. <br /> <br /> Like every other process, the process of collecting real-world data is made difficult by a number of problems. First, there is a need for cheap and compact robots to collect data in homes but current industrial robots (i.e. Sawyer and Baxter) are too expensive. Secondly, cheap robots are not accurate enough to collect reliable data. Also, there is a lack of constant supervision for data collection in homes. Finally, there is also a circular dependency problem in home-robotics: there is a lack of real-world data which are needed to improve current robots, but current robots are not good enough to collect reliable data in homes. These challenges in addition to some other external factors will likely result in noisy data collection. In this paper, a first systematic effort has been presented for collecting a dataset inside homes. In accomplishing this goal, the authors: <br /> <br /> 1. Build a cheap robot costing less than USD 3K which is appropriate for use in homes<br /> <br /> 2. Collect training data in 6 different homes and testing data in 3 homes<br /> <br /> 3. Propose a method for modelling the noise in the labelled data<br /> <br /> 4. Demonstrate that the diversity in the collected data provides superior performance and requires little-to-no domain adaptation<br /> <br /> [[File:aa1.PNG|600px|thumb|center|]]<br /> <br /> ==Overview==<br /> <br /> This paper emphasizes the importance of diversifying the data for robotic learning in order to have a greater generalization, by focusing on the task of grasping. A diverse dataset also allows for removing biases in the data. By considering these facts, the paper argues that even for simple tasks like grasping, datasets which are collected in labs suffer from strong biases such as simple backgrounds and same environment dynamics. Hence, the learning approaches cannot generalize the models and work well on real datasets.<br /> <br /> As a future possibility, there would be a need for having a low-cost robot to collect large-scale data inside a huge number of homes. For this reason, they introduced a customized mobile manipulator. They used a Dobot Magician which is a robotic arm mounted on a Kobuki which is a low-cost mobile robot base equipped with sensors such as bumper contact sensors and wheel encoders. The resulting robot arm has five degrees of freedom (DOF) (x, y, z, roll, pitch). The gripper is a two-fingered electric gripper with a 0.3kg payload. They also add an Intel R200 RGBD camera to their robot which is at a height of 1m above the ground. An Intel Core i5 processor is also used as an onboard laptop to perform all the processing. The whole system can run for 1.5 hours with a single charge.<br /> <br /> As there is always a trade-off, when we gain a low-cost robot, we are actually losing accuracy for controlling it. So, the low-cost robot which is built from cheaper components than the expensive setups such as Baxter and Sawyer suffers from higher calibration errors and execution errors. This means that the dataset collected with this approach is diverse and huge but it has noisy labels. To illustrate, consider when the robot wants to grasp at location &lt;math&gt; {(x, y)}&lt;/math&gt;. Since there is a noise in the execution, the robot may perform this action in the location &lt;math&gt; {(x + \delta_{x}, y+ \delta_{y})}&lt;/math&gt; which would assign the success or failure label of this action to a wrong place. Therefore, to solve the problem, they used an approach to learn from noisy data. They modeled noise as a latent variable and used two networks, one for predicting the noise and one for predicting the action to execute.<br /> <br /> ==Learning on low-cost robot data==<br /> <br /> This paper uses a patch grasping framework in its proposed architecture. Also, as mentioned before, there is a high tendency for noisy labels in the datasets which are collected by inaccurate and cheap robots. The cause of the noise in the labels could be due to the hardware execution error, inaccurate kinematics, camera calibration, proprioception, wear, and tear, etc. Here are more explanations about different parts of the architecture in order to disentangle the noise of the low-cost robot’s actual and commanded executions.<br /> <br /> ===Grasping Formulation===<br /> <br /> Planar grasping is the object of interest in this architecture. It means that all the objects are grasped at the same height and vertical to the ground (ie: a fixed end-effector pitch). The final goal is to find &lt;math&gt;{(x, y, \theta)}&lt;/math&gt; given an observation &lt;math&gt; {I}&lt;/math&gt; of the object, where &lt;math&gt; {x}&lt;/math&gt; and &lt;math&gt; {y}&lt;/math&gt; are the translational degrees of freedom and &lt;math&gt; {\theta}&lt;/math&gt; is the rotational degrees of freedom (roll of the end-effector). For the purpose of comparison, they used a model which does not predict the &lt;math&gt;{(x, y, \theta)}&lt;/math&gt; directly from the image &lt;math&gt; {I}&lt;/math&gt;, but samples several smaller patches &lt;math&gt; {I_{P}}&lt;/math&gt; at different locations &lt;math&gt;{(x, y)}&lt;/math&gt;. Thus, the angle of grasp &lt;math&gt; {\theta}&lt;/math&gt; is predicted from these patches. Also, in order to have multi-modal predictions, discrete steps of the angle &lt;math&gt; {\theta}&lt;/math&gt;, &lt;math&gt; {\theta_{D}}&lt;/math&gt; is used. <br /> <br /> Hence, each datapoint consists of an image &lt;math&gt; {I}&lt;/math&gt;, the executed grasp &lt;math&gt;{(x, y, \theta)}&lt;/math&gt; and the grasp success/failure label g. Then, the image &lt;math&gt; {I}&lt;/math&gt; and the angle &lt;math&gt; {\theta}&lt;/math&gt; are converted to image patch &lt;math&gt; {I_{P}}&lt;/math&gt; and angle &lt;math&gt; {\theta_{D}}&lt;/math&gt;. Then, to minimize the classification error, a binary cross entropy loss is used which minimizes the error between the predicted and ground truth label &lt;math&gt; g &lt;/math&gt;. A convolutional neural network with weight initialization from pre-training on Imagenet is used for this formulation.<br /> <br /> (Note: On Cross Entropy:<br /> <br /> If we think of a distribution as the tool we use to encode symbols, then entropy measures the number of bits we'll need if we use the correct tool. This is optimal, in that we can't encode the symbols using fewer bits on average.<br /> In contrast, cross entropy is the number of bits we'll need if we encode symbols from y using the wrong tool &lt;math&gt; {\hat h}&lt;/math&gt; . This consists of encoding the &lt;math&gt; {i_{th}}&lt;/math&gt; symbol using &lt;math&gt; {\log(\frac{1}{{\hat h_i}})}&lt;/math&gt; bits instead of &lt;math&gt; {\log(\frac{1}{{ h_i}})}&lt;/math&gt; bits. We of course still take the expected value to the true distribution y , since it's the distribution that truly generates the symbols:<br /> <br /> \begin{align}<br /> H(y,\hat y) = \sum_i{y_i\log{\frac{1}{\hat y_i}}}<br /> \end{align}<br /> <br /> Cross entropy is always larger than entropy; encoding symbols according to the wrong distribution &lt;math&gt; {\hat y}&lt;/math&gt; will always make us use more bits. The only exception is the trivial case where y and &lt;math&gt; {\hat y}&lt;/math&gt; are equal, and in this case entropy and cross entropy are equal.)<br /> <br /> ===Modeling noise as latent variable===<br /> <br /> In order to tackle the problem of inaccurate position control and calibration due to cheap robot, they found a structure in the noise which is dependent on the robot and the design. They modeled this structure of noise as a latent variable and decoupled during training. The approach is shown in figure 2: <br /> <br /> <br /> [[File:aa2.PNG|600px|thumb|center|]]<br /> <br /> The conventional approach models the grasp success probability for a given image patch at a given angle where the variables of the environment which can introduce noise in the system is generally insignificant, due to the high accuracy of expensive, commercial robots. However, in the low cost setting with multiple robots collecting data in parallel, it becomes an important consideration for learning. <br /> <br /> The grasp success probability for image patch &lt;math&gt; {I_{P}}&lt;/math&gt; at angle &lt;math&gt; {\theta_{D}}&lt;/math&gt; is represented as &lt;math&gt; {P(g|I_{P},\theta_{D}; \mathcal{R} )}&lt;/math&gt; where &lt;math&gt; \mathcal{R}&lt;/math&gt; represents environment variables that can add noise to the system.<br /> <br /> The conditional probability of grasping at a noisy image patch &lt;math&gt;I_P&lt;/math&gt; for this model is computed by:<br /> <br /> <br /> ${ P(g|I_{P},\theta_{D}, \mathcal{R} ) = ∑_{( \widehat{I_P} \in \mathcal{P})} P(g│z=\widehat{I_P},\theta_{D},\mathcal{R}) \cdot P(z=\widehat{I_P} | \theta_{D},I_P,\mathcal{R})}$<br /> <br /> <br /> Here, &lt;math&gt; {z}&lt;/math&gt; models the latent variable of the actual patch executed, and &lt;math&gt;\widehat{I_P}&lt;/math&gt; belongs to a set of possible neighboring patches &lt;math&gt; \mathcal{P}&lt;/math&gt;.&lt;math&gt; P(z=\widehat{I_P}|\theta_D,I_P,\mathcal{R})&lt;/math&gt; shows the noise which can be caused by &lt;math&gt;\mathcal{R}&lt;/math&gt; variables and is implemented as the Noise Modelling Network (NMN). &lt;math&gt; {P(g│z=\widehat{I_P},\theta_{D}, \mathcal{R} )}&lt;/math&gt; shows the grasp prediction probability given the true patch and is implemented as the Grasp Prediction Network (GPN). The overall Robust-Grasp model is computed by marginalizing GPN and NMN.<br /> <br /> ===Learning the latent noise model===<br /> <br /> This section concerns what be the inputs to the NMN network should be and how should the inputs can be trained. The authors assume that &lt;math&gt; {z}&lt;/math&gt; is conditionally independent of the local patch-specific variables &lt;math&gt; {(I_{P}, \theta_{D})}&lt;/math&gt;. To estimate the latent variable &lt;math&gt; {z}&lt;/math&gt; given the global information &lt;math&gt;\mathcal{R}&lt;/math&gt;, i.e &lt;math&gt; P(z=\widehat{I_P}|\theta_D,I_P,\mathcal{R}) \equiv P(z=\widehat{I_P}|\mathcal{R})&lt;/math&gt;. Apart from the patch &lt;math&gt; I_{P} &lt;/math&gt; and grasp information (x, y, θ), they use information like image of the entire scene, ID of the robot and the location of the raw pixel. They argue that the image of the full scene could contain some essential information about the system such as the relative location of camera to the ground which may change over the lifetime of the robot. They used direct optimization to learn both NMN and GPN with noisy labels. However, explicit labels are not available to train NMN but the latent variable &lt;math&gt;z&lt;/math&gt; can be estimated using a technique such as Expectation-Maximization. The entire image of the scene and the environment information are the inputs of the NMN, as well as robot ID and raw-pixel grasp location. The output of the NMN is the probability distribution of the actual patches where the grasps are executed. Finally, a binary cross entropy loss is applied to the marginalized output of these two networks and the true grasp label g.<br /> <br /> ===Training details===<br /> <br /> They implemented their model in PyTorch and fine tuned a pretrained ResNet-18 model. They concatenated 512 dimensional ResNet feature with a 1-hot vector of robot ID and the raw pixel location of the grasp for their NMN. This passes through a series of three fully connected layers and a SoftMax layer to convert the correct patch predictions to a probability distribution. Also, the inputs of the GPN are the original noisy patch plus 8 other equidistant patches from the original one. The angle predictions for all the patches are passed through a sigmoid activation at the end to obtain grasp success probability for a specific patch at a specific angle.<br /> The training of the network takes place in two stages. It starts with training only GPN over 5 epochs of the data. Then, the NMN and the marginalization operator are added to the model. So, they train NMN and GPN simultaneously in an end-to-end fashion for the other 25 epochs.<br /> This two-stage approach is crucial for effective training of their networks, without which NMN trivially selects the same patch irrespective of the input. The optimizer used for training is Adam .<br /> <br /> ==Results==<br /> <br /> In the results part of the paper, they show that collecting dataset in homes is essential for generalizing learning from unseen environments. They also show that modelling the noise in their Low-Cost Arm (LCA) can improve grasping performance.<br /> They collected data in parallel using multiple robots in 6 different homes, as shown in Figure 3. They used an object detector (tiny-YOLO) as the input data were unstructured due to LCA limited memory and computational capabilities. With an object location detected, class information was discarded, and a grasp was attempted. The grasp location in 3D was computed using PointCloud data. They scattered different objects in homes within 2m area to prevent collision of the robot with obstacles and let the robot move randomly and grasp objects. Finally, they collected a dataset with 28K grasp results.<br /> <br /> [[File:aa3.PNG|600px|thumb|center|]]<br /> <br /> To evaluate their approach in a more quantitative way, they used three test settings:<br /> <br /> - The first one is a binary classification or held-out data. The test set is collected by performing random grasps on objects. They measure the performance of binary classification by predicting the success or failure of grasping, given a location and the angle. Using binary classification allows for testing a lot of models without running them on real robots. They collected two held-out datasets using LCA in lab and homes and the dataset for Baxter robot.<br /> <br /> - The second one is Real Low-Cost Arm (Real-LCA). Here, they evaluate their model by running it in three unseen homes. They put 20 new objects in these three homes in different orientations. Since the objects and the environments are completely new, this tests could measure the generalization of the model.<br /> <br /> - The third one is Real Sawyer (Real-Sawyer). They evaluate the performance of their model by running the model on the Sawyer robot which is more accurate than the LCA. They tested their model in the lab environment to show that training models with the datasets collected from homes can improve the performance of models even in lab environments.<br /> <br /> They used baselines for both their data which is collected in homes and their model which is Robust-Grasp. They used two datasets for the baseline. The dataset collected by (Lab-Baxter) and the dataset collected by their LCA in the lab (Lab-LCA).<br /> They compared their Robust-Grasp model with the noise independent patch grasping model (Patch-Grasp) . They also compared their data and model with DexNet-3.0 (DexNet) for a strong real-world grasping baseline.<br /> <br /> ===Experiment 1: Performance on held-out data===<br /> <br /> Table 1 shows that the models trained on lab data cannot generalize to the Home-LCA environment (i.e. they overfit to their respective environments and attain a lower binary classification score). However, the model trained on Home-LCA has a good performance on both lab data and home environment.<br /> <br /> [[File:aa4.PNG|600px|thumb|center|]]<br /> <br /> ===Experiment 2: Performance on Real LCA Robot===<br /> <br /> In table 2, the performance of the Home-LCA is compared against a pre-trained DexNet and the model trained on the Lab-Baxter. Training on the Home-LCA dataset performs 43.7% better than training on the Lab-Baxter dataset and 33% better than DexNet. The low performance of DexNet can be described by the possible noise in the depth images that are caused by the natural light. DexNet, which requires high-quality depth sensing, cannot perform well in these scenarios. By using cheap commodity RGBD cameras in LCA, the noise in the depth images is not a matter of concern, as the model has no expectation of high-quality sensing.<br /> <br /> [[File:aa5.PNG|600px|thumb|center|]]<br /> <br /> ===Performance on Real Sawyer===<br /> <br /> To compare the performance of the Robust-Grasp model against the Patch-Grasp model without collecting noise-free data, they used Lab-Baxter for benchmarking, which is an accurate and better calibrated robot. The Sawyer robot is used for testing to ensure that the testing robot is different from both training robots. As shown in Table 3, the Robust-Grasp model trained on Home-LCA outperforms the Patch-Grasp model and achieves 77.5% accuracy. This accuracy is similar to several recent papers, however, this model was trained and tested in a different environment. The Robust-Grasp model also outperforms the Patch-Grasp by about 4% on binary classification. Furthermore, the visualizations of predicted noise corrections in Figure 4 shows that the corrections depend on both the pixel locations of the noisy grasp and the robot.<br /> <br /> [[File:aa6.PNG|600px|thumb|center|]]<br /> <br /> [[File:aa7.PNG|600px|thumb|center|]]<br /> <br /> ==Related work==<br /> <br /> Over the last few years, the interest of scaling up robot learning with large-scale datasets has been increased. Hence, many papers were published in this area. A hand annotated grasping dataset, a self-supervised grasping dataset, and grasping using reinforcement learning are some examples of using large-scale datasets for grasping. The work mentioned above used high-cost hardware and data labeling mechanisms. There were also many papers that worked on other robotic tasks like material recognition, pushing objects and manipulating a rope. However, none of these papers worked on real data in real environments like homes, they all used lab data.<br /> <br /> Furthermore, since grasping is one of the basic problems in robotics, there were some efforts to improve grasping. Classical approaches focused on physics-based issues of grasping and required 3D models of the objects. However, recent works focused on data-driven approaches which learn from visual observations to grasp objects. Simulation and real-world robots are both required for large-scale data collection. A versatile grasping model was proposed to achieve a 90% performance for a bin-picking task. The point here is that they usually require high-quality depth as input which seems to be a barrier for practical use of robots in real environments. High-quality depth sensing means a high cost to implement in hardware and thus is a barrier for practical use.<br /> <br /> Most labs use industrial robots or standard collaborative hardware for their experiments. Therefore, there is few research that used low-cost robots. One of the examples is learning using a cheap inaccurate robot for stack multiple blocks. Although mobile robots like iRobot’s Roomba have been in the home consumer electronics market for a decade, it is not clear whether learning approaches are used in it alongside mapping and planning.<br /> <br /> Learning from noisy inputs is another challenge specifically in computer vision. A controversial question which is often raised in this area is whether learning from noise can improve the performance. Some works show it could have bad effects on the performance; however, some other works find it valuable when the noise is independent or statistically dependent on the environment. In this paper, they used a model that can exploit the noise and learn a better grasping model.<br /> <br /> ==Conclusion==<br /> <br /> All in all, the paper presents an approach for collecting large-scale robot data in real home environments. They implemented their approach by using a mobile manipulator which is a lot cheaper than the existing industrial robots. They collected a dataset of 28K grasps in six different homes. In order to solve the problem of noisy labels which were caused by their inaccurate robots, they presented a framework to factor out the noise in the data. They tested their model by physically grasping 20 new objects in three new homes and in the lab. The model trained with home dataset showed 43.7% improvement over the models trained with lab data. Their framework performed 33% better than a baseline DexNet model, which struggled with the typically poor depth sensing in common household environments with a lot of natural light.. Their results also showed that their model can improve the grasping performance even in lab environments. They also demonstrated that their architecture for modeling the noise improved the performance by about 10%.<br /> <br /> ==Critiques==<br /> <br /> This paper does not contain a significant algorithmic contribution. They are just combining a large number of data engineering techniques for the robot learning problem. The authors claim that they have obtained 43.7% more accuracy than baseline models, but it does not seem to be a fair comparison as the data collection happened in simulated settings in the lab for other methods, whereas the authors use the home dataset. The authors must have also discussed safety issues when training robots in real environments as against simulated environments like labs. The authors are encouraging other researchers to look outside the labs, but are not discussing the critical safety issues in this approach.<br /> <br /> Another strange finding is that the paper mentions that they &quot;follow a model architecture similar to [Pinto and Gupta ],&quot; however, the proposed model is, in fact, a fine-tuned resnet-18 architecture. Pinto and Gupta, implement a version similar to AlexNet as shown below in Figure 5.<br /> <br /> [[File:Figure_5_PandG.JPG | 450px|thumb|center|Figure 5: AlexNet architecture implemented in Pinto and Gupta .]]<br /> <br /> <br /> The paper argues that the dataset collected by the LCA is noisy, since the robot is cheap and inaccurate. It further asserts that in order to handle the noise in the dataset, they can model the noise as a latent variable and their model can improve the performance of grasping. Although learning from noisy data and achieving a good performance is valuable, it is better that they test their noise modeling network for other robots as well. Since their noise modelling network takes robot information as an input, it would be a good idea to generalize it by testing it using different inaccurate robots to ensure that it would perform well.<br /> <br /> They did not mention other aspects of their comparison, for example they could mention their training time compared to other models or the size of other datasets.<br /> <br /> ==References==<br /> <br /> #Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. &quot;Domain randomization for transferring deep neural networks from simulation to the real world.&quot; 2017. URL https://arxiv.org/abs/1703.06907.<br /> #Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. &quot;Sim-to-real transfer of robotic control with dynamics randomization.&quot; arXiv preprint arXiv:1710.06537,2017.<br /> #Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. &quot;Asymmetric actor-critic for image-based robot learning.&quot; Robotics Science and Systems, 2018.<br /> #Lerrel Pinto and Abhinav Gupta. &quot;Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours.&quot; CoRR, abs/1509.06825, 2015. URL http://arxiv.org/abs/1509. 06825.<br /> #Adithyavairavan Murali, Lerrel Pinto, Dhiraj Gandhi, and Abhinav Gupta. &quot;CASSL: Curriculum accelerated self-supervised learning.&quot; International Conference on Robotics and Automation, 2018.<br /> # Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. &quot;End-to-end training of deep visuomotor policies.&quot; The Journal of Machine Learning Research, 17(1):1334–1373, 2016.<br /> #Sergey Levine, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen. &quot;Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.&quot; CoRR, abs/1603.02199, 2016. URL http://arxiv.org/abs/1603.02199.<br /> #Pulkit Agarwal, Ashwin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. &quot;Learning to poke by poking: Experiential learning of intuitive physics.&quot; 2016. URL http://arxiv.org/ abs/1606.07419<br /> #Chelsea Finn, Ian Goodfellow, and Sergey Levine. &quot;Unsupervised learning for physical interaction through video prediction.&quot; In Advances in neural information processing systems, 2016.<br /> #Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. &quot;Combining self-supervised learning and imitation for vision-based rope manipulation.&quot; International Conference on Robotics and Automation, 2017.<br /> #Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. &quot;Revisiting unreasonable effectiveness of data in deep learning era.&quot; ICCV, 2017.<br /> #Marc Peter Deisenroth, Carl Edward Rasmussen, and Dieter Fox. Learning to control a low-cost manipulator using data-efficient reinforcement learning. RSS, 2011.<br /> #David F Nettleton, Albert Orriols-Puig, and Albert Fornells. A study of the effect of different types of noise on the precision of supervised learning techniques. Artificial intelligence review, 33(4):275–306, 2010.<br /> #Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems, 25(5):845–869, 2014.<br /> #Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2691–2699, 2015.<br /> #Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Robot_Learning_in_Homes:_Improving_Generalization_and_Reducing_Dataset_Bias&diff=42036 Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias 2018-11-30T13:11:35Z <p>C9sharma: /* Training details */</p> <hr /> <div>==Introduction==<br /> <br /> <br /> The use of data-driven approaches in robotics has increased in the last decade. Instead of using hand-designed models, these data-driven approaches work on large-scale datasets and learn appropriate policies that map from high-dimensional observations to actions. Since collecting data using an actual robot in real-time is very expensive, most of the data-driven approaches in robotics use simulators in order to collect simulated data. The concern here is whether these approaches have the capability to be robust enough to domain shift and to be used for real-world data. It is an undeniable fact that there is a wide reality gap between simulators and the real world.<br /> <br /> This has motivated the robotics community to increase their efforts in collecting real-world physical interaction data for a variety of tasks. This effort has been accelerated by the declining costs of hardware. This approach has been quite successful at tasks such as grasping, pushing, poking and imitation learning. However, the major problem is that the performance of these learning models are not good enough and tend to plateau fast. Furthermore, robotic action data did not lead to similar gains in other areas such as computer vision and natural language processing. As the paper claimed, the solution for all of these obstacles is using “real data”. Current robotic datasets lack diversity of environment. Learning-based approaches need to move out of simulators in the labs and go to real environments such as real homes so that they can learn from real datasets. <br /> <br /> Like every other process, the process of collecting real-world data is made difficult by a number of problems. First, there is a need for cheap and compact robots to collect data in homes but current industrial robots (i.e. Sawyer and Baxter) are too expensive. Secondly, cheap robots are not accurate enough to collect reliable data. Also, there is a lack of constant supervision for data collection in homes. Finally, there is also a circular dependency problem in home-robotics: there is a lack of real-world data which are needed to improve current robots, but current robots are not good enough to collect reliable data in homes. These challenges in addition to some other external factors will likely result in noisy data collection. In this paper, a first systematic effort has been presented for collecting a dataset inside homes. In accomplishing this goal, the authors: <br /> <br /> 1. Build a cheap robot costing less than USD 3K which is appropriate for use in homes<br /> <br /> 2. Collect training data in 6 different homes and testing data in 3 homes<br /> <br /> 3. Propose a method for modelling the noise in the labelled data<br /> <br /> 4. Demonstrate that the diversity in the collected data provides superior performance and requires little-to-no domain adaptation<br /> <br /> [[File:aa1.PNG|600px|thumb|center|]]<br /> <br /> ==Overview==<br /> <br /> This paper emphasizes the importance of diversifying the data for robotic learning in order to have a greater generalization, by focusing on the task of grasping. A diverse dataset also allows for removing biases in the data. By considering these facts, the paper argues that even for simple tasks like grasping, datasets which are collected in labs suffer from strong biases such as simple backgrounds and same environment dynamics. Hence, the learning approaches cannot generalize the models and work well on real datasets.<br /> <br /> As a future possibility, there would be a need for having a low-cost robot to collect large-scale data inside a huge number of homes. For this reason, they introduced a customized mobile manipulator. They used a Dobot Magician which is a robotic arm mounted on a Kobuki which is a low-cost mobile robot base equipped with sensors such as bumper contact sensors and wheel encoders. The resulting robot arm has five degrees of freedom (DOF) (x, y, z, roll, pitch). The gripper is a two-fingered electric gripper with a 0.3kg payload. They also add an Intel R200 RGBD camera to their robot which is at a height of 1m above the ground. An Intel Core i5 processor is also used as an onboard laptop to perform all the processing. The whole system can run for 1.5 hours with a single charge.<br /> <br /> As there is always a trade-off, when we gain a low-cost robot, we are actually losing accuracy for controlling it. So, the low-cost robot which is built from cheaper components than the expensive setups such as Baxter and Sawyer suffers from higher calibration errors and execution errors. This means that the dataset collected with this approach is diverse and huge but it has noisy labels. To illustrate, consider when the robot wants to grasp at location &lt;math&gt; {(x, y)}&lt;/math&gt;. Since there is a noise in the execution, the robot may perform this action in the location &lt;math&gt; {(x + \delta_{x}, y+ \delta_{y})}&lt;/math&gt; which would assign the success or failure label of this action to a wrong place. Therefore, to solve the problem, they used an approach to learn from noisy data. They modeled noise as a latent variable and used two networks, one for predicting the noise and one for predicting the action to execute.<br /> <br /> ==Learning on low-cost robot data==<br /> <br /> This paper uses a patch grasping framework in its proposed architecture. Also, as mentioned before, there is a high tendency for noisy labels in the datasets which are collected by inaccurate and cheap robots. The cause of the noise in the labels could be due to the hardware execution error, inaccurate kinematics, camera calibration, proprioception, wear, and tear, etc. Here are more explanations about different parts of the architecture in order to disentangle the noise of the low-cost robot’s actual and commanded executions.<br /> <br /> ===Grasping Formulation===<br /> <br /> Planar grasping is the object of interest in this architecture. It means that all the objects are grasped at the same height and vertical to the ground (ie: a fixed end-effector pitch). The final goal is to find &lt;math&gt;{(x, y, \theta)}&lt;/math&gt; given an observation &lt;math&gt; {I}&lt;/math&gt; of the object, where &lt;math&gt; {x}&lt;/math&gt; and &lt;math&gt; {y}&lt;/math&gt; are the translational degrees of freedom and &lt;math&gt; {\theta}&lt;/math&gt; is the rotational degrees of freedom (roll of the end-effector). For the purpose of comparison, they used a model which does not predict the &lt;math&gt;{(x, y, \theta)}&lt;/math&gt; directly from the image &lt;math&gt; {I}&lt;/math&gt;, but samples several smaller patches &lt;math&gt; {I_{P}}&lt;/math&gt; at different locations &lt;math&gt;{(x, y)}&lt;/math&gt;. Thus, the angle of grasp &lt;math&gt; {\theta}&lt;/math&gt; is predicted from these patches. Also, in order to have multi-modal predictions, discrete steps of the angle &lt;math&gt; {\theta}&lt;/math&gt;, &lt;math&gt; {\theta_{D}}&lt;/math&gt; is used. <br /> <br /> Hence, each datapoint consists of an image &lt;math&gt; {I}&lt;/math&gt;, the executed grasp &lt;math&gt;{(x, y, \theta)}&lt;/math&gt; and the grasp success/failure label g. Then, the image &lt;math&gt; {I}&lt;/math&gt; and the angle &lt;math&gt; {\theta}&lt;/math&gt; are converted to image patch &lt;math&gt; {I_{P}}&lt;/math&gt; and angle &lt;math&gt; {\theta_{D}}&lt;/math&gt;. Then, to minimize the classification error, a binary cross entropy loss is used which minimizes the error between the predicted and ground truth label &lt;math&gt; g &lt;/math&gt;. A convolutional neural network with weight initialization from pre-training on Imagenet is used for this formulation.<br /> <br /> (Note: On Cross Entropy:<br /> <br /> If we think of a distribution as the tool we use to encode symbols, then entropy measures the number of bits we'll need if we use the correct tool. This is optimal, in that we can't encode the symbols using fewer bits on average.<br /> In contrast, cross entropy is the number of bits we'll need if we encode symbols from y using the wrong tool &lt;math&gt; {\hat h}&lt;/math&gt; . This consists of encoding the &lt;math&gt; {i_{th}}&lt;/math&gt; symbol using &lt;math&gt; {\log(\frac{1}{{\hat h_i}})}&lt;/math&gt; bits instead of &lt;math&gt; {\log(\frac{1}{{ h_i}})}&lt;/math&gt; bits. We of course still take the expected value to the true distribution y , since it's the distribution that truly generates the symbols:<br /> <br /> \begin{align}<br /> H(y,\hat y) = \sum_i{y_i\log{\frac{1}{\hat y_i}}}<br /> \end{align}<br /> <br /> Cross entropy is always larger than entropy; encoding symbols according to the wrong distribution &lt;math&gt; {\hat y}&lt;/math&gt; will always make us use more bits. The only exception is the trivial case where y and &lt;math&gt; {\hat y}&lt;/math&gt; are equal, and in this case entropy and cross entropy are equal.)<br /> <br /> ===Modeling noise as latent variable===<br /> <br /> In order to tackle the problem of inaccurate position control and calibration due to cheap robot, they found a structure in the noise which is dependent on the robot and the design. They modeled this structure of noise as a latent variable and decoupled during training. The approach is shown in figure 2: <br /> <br /> <br /> [[File:aa2.PNG|600px|thumb|center|]]<br /> <br /> The conventional approach models the grasp success probability for a given image patch at a given angle where the variables of the environment which can introduce noise in the system is generally insignificant, due to the high accuracy of expensive, commercial robots. However, in the low cost setting with multiple robots collecting data in parallel, it becomes an important consideration for learning. <br /> <br /> The grasp success probability for image patch &lt;math&gt; {I_{P}}&lt;/math&gt; at angle &lt;math&gt; {\theta_{D}}&lt;/math&gt; is represented as &lt;math&gt; {P(g|I_{P},\theta_{D}; \mathcal{R} )}&lt;/math&gt; where &lt;math&gt; \mathcal{R}&lt;/math&gt; represents environment variables that can add noise to the system.<br /> <br /> The conditional probability of grasping at a noisy image patch &lt;math&gt;I_P&lt;/math&gt; for this model is computed by:<br /> <br /> <br /> ${ P(g|I_{P},\theta_{D}, \mathcal{R} ) = ∑_{( \widehat{I_P} \in \mathcal{P})} P(g│z=\widehat{I_P},\theta_{D},\mathcal{R}) \cdot P(z=\widehat{I_P} | \theta_{D},I_P,\mathcal{R})}$<br /> <br /> <br /> Here, &lt;math&gt; {z}&lt;/math&gt; models the latent variable of the actual patch executed, and &lt;math&gt;\widehat{I_P}&lt;/math&gt; belongs to a set of possible neighboring patches &lt;math&gt; \mathcal{P}&lt;/math&gt;.&lt;math&gt; P(z=\widehat{I_P}|\theta_D,I_P,\mathcal{R})&lt;/math&gt; shows the noise which can be caused by &lt;math&gt;\mathcal{R}&lt;/math&gt; variables and is implemented as the Noise Modelling Network (NMN). &lt;math&gt; {P(g│z=\widehat{I_P},\theta_{D}, \mathcal{R} )}&lt;/math&gt; shows the grasp prediction probability given the true patch and is implemented as the Grasp Prediction Network (GPN). The overall Robust-Grasp model is computed by marginalizing GPN and NMN.<br /> <br /> ===Learning the latent noise model===<br /> <br /> This section concerns what be the inputs to the NMN network should be and how should the inputs can be trained. The authors assume that &lt;math&gt; {z}&lt;/math&gt; is conditionally independent of the local patch-specific variables &lt;math&gt; {(I_{P}, \theta_{D})}&lt;/math&gt;. To estimate the latent variable &lt;math&gt; {z}&lt;/math&gt; given the global information &lt;math&gt;\mathcal{R}&lt;/math&gt;, i.e &lt;math&gt; P(z=\widehat{I_P}|\theta_D,I_P,\mathcal{R}) \equiv P(z=\widehat{I_P}|\mathcal{R})&lt;/math&gt;. Apart from the patch &lt;math&gt; I_{P} &lt;/math&gt; and grasp information (x, y, θ), they use information like image of the entire scene, ID of the robot and the location of the raw pixel. They argue that the image of the full scene could contain some essential information about the system such as the relative location of camera to the ground which may change over the lifetime of the robot. They used direct optimization to learn both NMN and GPN with noisy labels. However, explicit labels are not available to train NMN but the latent variable &lt;math&gt;z&lt;/math&gt; can be estimated using a technique such as Expectation-Maximization. The entire image of the scene and the environment information are the inputs of the NMN, as well as robot ID and raw-pixel grasp location. The output of the NMN is the probability distribution of the actual patches where the grasps are executed. Finally, a binary cross entropy loss is applied to the marginalized output of these two networks and the true grasp label g.<br /> <br /> ===Training details===<br /> <br /> They implemented their model in PyTorch and fine tuned a pretrained ResNet-18 model. They concatenated 512 dimensional ResNet feature with a 1-hot vector of robot ID and the raw pixel location of the grasp for their NMN. This passes through a series of three fully connected layers and a SoftMax layer to convert the correct patch predictions to a probability distribution. Also, the inputs of the GPN are the original noisy patch plus 8 other equidistant patches from the original one. The angle predictions for all the patches are passed through a sigmoid activation at the end to obtain grasp success probability for a specific patch at a specific angle.<br /> The training of the network takes place in two stages. It starts with training only GPN over 5 epochs of the data. Then, the NMN and the marginalization operator are added to the model. So, they train NMN and GPN simultaneously in an end-to-end fashion for the other 25 epochs.<br /> This two-stage approach is crucial for effective training of their networks, without which NMN trivially selects the same patch irrespective of the input. The optimizer used for training is Adam .<br /> <br /> ==Results==<br /> <br /> In the results part of the paper, they show that collecting dataset in homes is essential for generalizing learning from unseen environments. They also show that modelling the noise in their Low-Cost Arm (LCA) can improve grasping performance.<br /> They collected data in parallel using multiple robots in 6 different homes, as shown in Figure 3. They used an object detector (tiny-YOLO) as the input data were unstructured due to LCA limited memory and computational capabilities. With an object location detected, class information was discarded, and a grasp was attempted. The grasp location in 3D was computed using PointCloud data. They scattered different objects in homes within 2m area to prevent collision of the robot with obstacles and let the robot move randomly and grasp objects. Finally, they collected a dataset with 28K grasp results.<br /> <br /> [[File:aa3.PNG|600px|thumb|center|]]<br /> <br /> To evaluate their approach in a more quantitative way, they used three test settings:<br /> <br /> - The first one is a binary classification or held-out data. The test set is collected by performing random grasps on objects. They measure the performance of binary classification by predicting the success or failure of grasping, given a location and the angle. Using binary classification allows for testing a lot of models without running them on real robots. They collected two held-out datasets using LCA in lab and homes and the dataset for Baxter robot.<br /> <br /> - The second one is Real Low-Cost Arm (Real-LCA). Here, they evaluate their model by running it in three unseen homes. They put 20 new objects in these three homes in different orientations. Since the objects and the environments are completely new, this tests could measure the generalization of the model.<br /> <br /> - The third one is Real Sawyer (Real-Sawyer). They evaluate the performance of their model by running the model on the Sawyer robot which is more accurate than the LCA. They tested their model in the lab environment to show that training models with the datasets collected from homes can improve the performance of models even in lab environments.<br /> <br /> They used baselines for both their data which is collected in homes and their model which is Robust-Grasp. They used two datasets for the baseline. The dataset collected by (Lab-Baxter) and the dataset collected by their LCA in the lab (Lab-LCA).<br /> They compared their Robust-Grasp model with the noise independent patch grasping model (Patch-Grasp) . They also compared their data and model with DexNet-3.0 (DexNet) for a strong real-world grasping baseline.<br /> <br /> ===Experiment 1: Performance on held-out data===<br /> <br /> Table 1 shows that the models trained on lab data cannot generalize to the Home-LCA environment (i.e. they overfit to their respective environments and attain a lower binary classification score). However, the model trained on Home-LCA has a good performance on both lab data and home environment.<br /> <br /> [[File:aa4.PNG|600px|thumb|center|]]<br /> <br /> ===Experiment 2: Performance on Real LCA Robot===<br /> <br /> In table 2, the performance of the Home-LCA is compared against a pre-trained DexNet and the model trained on the Lab-Baxter. Training on the Home-LCA dataset performs 43.7% better than training on the Lab-Baxter dataset and 33% better than DexNet. The low performance of DexNet can be described by the possible noise in the depth images that are caused by the natural light. DexNet, which requires high-quality depth sensing, cannot perform well in these scenarios. By using cheap commodity RGBD cameras in LCA, the noise in the depth images is not a matter of concern, as the model has no expectation of high-quality sensing.<br /> <br /> [[File:aa5.PNG|600px|thumb|center|]]<br /> <br /> ===Performance on Real Sawyer===<br /> <br /> To compare the performance of the Robust-Grasp model against the Patch-Grasp model without collecting noise-free data, they used Lab-Baxter for benchmarking, which is an accurate and better calibrated robot. The Sawyer robot is used for testing to ensure that the testing robot is different from both training robots. As shown in Table 3, the Robust-Grasp model trained on Home-LCA outperforms the Patch-Grasp model and achieves 77.5% accuracy. This accuracy is similar to several recent papers, however, this model was trained and tested in a different environment. The Robust-Grasp model also outperforms the Patch-Grasp by about 4% on binary classification. Furthermore, the visualizations of predicted noise corrections in Figure 4 shows that the corrections depend on both the pixel locations of the noisy grasp and the robot.<br /> <br /> [[File:aa6.PNG|600px|thumb|center|]]<br /> <br /> [[File:aa7.PNG|600px|thumb|center|]]<br /> <br /> ==Related work==<br /> <br /> Over the last few years, the interest of scaling up robot learning with large-scale datasets has been increased. Hence, many papers were published in this area. A hand annotated grasping dataset, a self-supervised grasping dataset, and grasping using reinforcement learning are some examples of using large-scale datasets for grasping. The work mentioned above used high-cost hardware and data labeling mechanisms. There were also many papers that worked on other robotic tasks like material recognition, pushing objects and manipulating a rope. However, none of these papers worked on real data in real environments like homes, they all used lab data.<br /> <br /> Furthermore, since grasping is one of the basic problems in robotics, there were some efforts to improve grasping. Classical approaches focused on physics-based issues of grasping and required 3D models of the objects. However, recent works focused on data-driven approaches which learn from visual observations to grasp objects. Simulation and real-world robots are both required for large-scale data collection. A versatile grasping model was proposed to achieve a 90% performance for a bin-picking task. The point here is that they usually require high-quality depth as input which seems to be a barrier for practical use of robots in real environments. High-quality depth sensing means a high cost to implement in hardware and thus is a barrier for practical use.<br /> <br /> Most labs use industrial robots or standard collaborative hardware for their experiments. Therefore, there is few research that used low-cost robots. One of the examples is learning using a cheap inaccurate robot for stack multiple blocks. Although mobile robots like iRobot’s Roomba have been in the home consumer electronics market for a decade, it is not clear whether learning approaches are used in it alongside mapping and planning.<br /> <br /> Learning from noisy inputs is another challenge specifically in computer vision. A controversial question which is often raised in this area is whether learning from noise can improve the performance. Some works show it could have bad effects on the performance; however, some other works find it valuable when the noise is independent or statistically dependent on the environment. In this paper, they used a model that can exploit the noise and learn a better grasping model.<br /> <br /> ==Conclusion==<br /> <br /> All in all, the paper presents an approach for collecting large-scale robot data in real home environments. They implemented their approach by using a mobile manipulator which is a lot cheaper than the existing industrial robots. They collected a dataset of 28K grasps in six different homes. In order to solve the problem of noisy labels which were caused by their inaccurate robots, they presented a framework to factor out the noise in the data. They tested their model by physically grasping 20 new objects in three new homes and in the lab. The model trained with home dataset showed 43.7% improvement over the models trained with lab data. Their results also showed that their model can improve the grasping performance even in lab environments. They also demonstrated that their architecture for modeling the noise improved the performance by about 10%.<br /> <br /> ==Critiques==<br /> <br /> This paper does not contain a significant algorithmic contribution. They are just combining a large number of data engineering techniques for the robot learning problem. The authors claim that they have obtained 43.7% more accuracy than baseline models, but it does not seem to be a fair comparison as the data collection happened in simulated settings in the lab for other methods, whereas the authors use the home dataset. The authors must have also discussed safety issues when training robots in real environments as against simulated environments like labs. The authors are encouraging other researchers to look outside the labs, but are not discussing the critical safety issues in this approach.<br /> <br /> Another strange finding is that the paper mentions that they &quot;follow a model architecture similar to [Pinto and Gupta ],&quot; however, the proposed model is, in fact, a fine-tuned resnet-18 architecture. Pinto and Gupta, implement a version similar to AlexNet as shown below in Figure 5.<br /> <br /> [[File:Figure_5_PandG.JPG | 450px|thumb|center|Figure 5: AlexNet architecture implemented in Pinto and Gupta .]]<br /> <br /> <br /> The paper argues that the dataset collected by the LCA is noisy, since the robot is cheap and inaccurate. It further asserts that in order to handle the noise in the dataset, they can model the noise as a latent variable and their model can improve the performance of grasping. Although learning from noisy data and achieving a good performance is valuable, it is better that they test their noise modeling network for other robots as well. Since their noise modelling network takes robot information as an input, it would be a good idea to generalize it by testing it using different inaccurate robots to ensure that it would perform well.<br /> <br /> They did not mention other aspects of their comparison, for example they could mention their training time compared to other models or the size of other datasets.<br /> <br /> ==References==<br /> <br /> #Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. &quot;Domain randomization for transferring deep neural networks from simulation to the real world.&quot; 2017. URL https://arxiv.org/abs/1703.06907.<br /> #Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. &quot;Sim-to-real transfer of robotic control with dynamics randomization.&quot; arXiv preprint arXiv:1710.06537,2017.<br /> #Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. &quot;Asymmetric actor-critic for image-based robot learning.&quot; Robotics Science and Systems, 2018.<br /> #Lerrel Pinto and Abhinav Gupta. &quot;Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours.&quot; CoRR, abs/1509.06825, 2015. URL http://arxiv.org/abs/1509. 06825.<br /> #Adithyavairavan Murali, Lerrel Pinto, Dhiraj Gandhi, and Abhinav Gupta. &quot;CASSL: Curriculum accelerated self-supervised learning.&quot; International Conference on Robotics and Automation, 2018.<br /> # Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. &quot;End-to-end training of deep visuomotor policies.&quot; The Journal of Machine Learning Research, 17(1):1334–1373, 2016.<br /> #Sergey Levine, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen. &quot;Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.&quot; CoRR, abs/1603.02199, 2016. URL http://arxiv.org/abs/1603.02199.<br /> #Pulkit Agarwal, Ashwin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. &quot;Learning to poke by poking: Experiential learning of intuitive physics.&quot; 2016. URL http://arxiv.org/ abs/1606.07419<br /> #Chelsea Finn, Ian Goodfellow, and Sergey Levine. &quot;Unsupervised learning for physical interaction through video prediction.&quot; In Advances in neural information processing systems, 2016.<br /> #Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. &quot;Combining self-supervised learning and imitation for vision-based rope manipulation.&quot; International Conference on Robotics and Automation, 2017.<br /> #Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. &quot;Revisiting unreasonable effectiveness of data in deep learning era.&quot; ICCV, 2017.<br /> #Marc Peter Deisenroth, Carl Edward Rasmussen, and Dieter Fox. Learning to control a low-cost manipulator using data-efficient reinforcement learning. RSS, 2011.<br /> #David F Nettleton, Albert Orriols-Puig, and Albert Fornells. A study of the effect of different types of noise on the precision of supervised learning techniques. Artificial intelligence review, 33(4):275–306, 2010.<br /> #Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems, 25(5):845–869, 2014.<br /> #Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2691–2699, 2015.<br /> #Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Robot_Learning_in_Homes:_Improving_Generalization_and_Reducing_Dataset_Bias&diff=42035 Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias 2018-11-30T13:10:59Z <p>C9sharma: /* References */</p> <hr /> <div>==Introduction==<br /> <br /> <br /> The use of data-driven approaches in robotics has increased in the last decade. Instead of using hand-designed models, these data-driven approaches work on large-scale datasets and learn appropriate policies that map from high-dimensional observations to actions. Since collecting data using an actual robot in real-time is very expensive, most of the data-driven approaches in robotics use simulators in order to collect simulated data. The concern here is whether these approaches have the capability to be robust enough to domain shift and to be used for real-world data. It is an undeniable fact that there is a wide reality gap between simulators and the real world.<br /> <br /> This has motivated the robotics community to increase their efforts in collecting real-world physical interaction data for a variety of tasks. This effort has been accelerated by the declining costs of hardware. This approach has been quite successful at tasks such as grasping, pushing, poking and imitation learning. However, the major problem is that the performance of these learning models are not good enough and tend to plateau fast. Furthermore, robotic action data did not lead to similar gains in other areas such as computer vision and natural language processing. As the paper claimed, the solution for all of these obstacles is using “real data”. Current robotic datasets lack diversity of environment. Learning-based approaches need to move out of simulators in the labs and go to real environments such as real homes so that they can learn from real datasets. <br /> <br /> Like every other process, the process of collecting real-world data is made difficult by a number of problems. First, there is a need for cheap and compact robots to collect data in homes but current industrial robots (i.e. Sawyer and Baxter) are too expensive. Secondly, cheap robots are not accurate enough to collect reliable data. Also, there is a lack of constant supervision for data collection in homes. Finally, there is also a circular dependency problem in home-robotics: there is a lack of real-world data which are needed to improve current robots, but current robots are not good enough to collect reliable data in homes. These challenges in addition to some other external factors will likely result in noisy data collection. In this paper, a first systematic effort has been presented for collecting a dataset inside homes. In accomplishing this goal, the authors: <br /> <br /> 1. Build a cheap robot costing less than USD 3K which is appropriate for use in homes<br /> <br /> 2. Collect training data in 6 different homes and testing data in 3 homes<br /> <br /> 3. Propose a method for modelling the noise in the labelled data<br /> <br /> 4. Demonstrate that the diversity in the collected data provides superior performance and requires little-to-no domain adaptation<br /> <br /> [[File:aa1.PNG|600px|thumb|center|]]<br /> <br /> ==Overview==<br /> <br /> This paper emphasizes the importance of diversifying the data for robotic learning in order to have a greater generalization, by focusing on the task of grasping. A diverse dataset also allows for removing biases in the data. By considering these facts, the paper argues that even for simple tasks like grasping, datasets which are collected in labs suffer from strong biases such as simple backgrounds and same environment dynamics. Hence, the learning approaches cannot generalize the models and work well on real datasets.<br /> <br /> As a future possibility, there would be a need for having a low-cost robot to collect large-scale data inside a huge number of homes. For this reason, they introduced a customized mobile manipulator. They used a Dobot Magician which is a robotic arm mounted on a Kobuki which is a low-cost mobile robot base equipped with sensors such as bumper contact sensors and wheel encoders. The resulting robot arm has five degrees of freedom (DOF) (x, y, z, roll, pitch). The gripper is a two-fingered electric gripper with a 0.3kg payload. They also add an Intel R200 RGBD camera to their robot which is at a height of 1m above the ground. An Intel Core i5 processor is also used as an onboard laptop to perform all the processing. The whole system can run for 1.5 hours with a single charge.<br /> <br /> As there is always a trade-off, when we gain a low-cost robot, we are actually losing accuracy for controlling it. So, the low-cost robot which is built from cheaper components than the expensive setups such as Baxter and Sawyer suffers from higher calibration errors and execution errors. This means that the dataset collected with this approach is diverse and huge but it has noisy labels. To illustrate, consider when the robot wants to grasp at location &lt;math&gt; {(x, y)}&lt;/math&gt;. Since there is a noise in the execution, the robot may perform this action in the location &lt;math&gt; {(x + \delta_{x}, y+ \delta_{y})}&lt;/math&gt; which would assign the success or failure label of this action to a wrong place. Therefore, to solve the problem, they used an approach to learn from noisy data. They modeled noise as a latent variable and used two networks, one for predicting the noise and one for predicting the action to execute.<br /> <br /> ==Learning on low-cost robot data==<br /> <br /> This paper uses a patch grasping framework in its proposed architecture. Also, as mentioned before, there is a high tendency for noisy labels in the datasets which are collected by inaccurate and cheap robots. The cause of the noise in the labels could be due to the hardware execution error, inaccurate kinematics, camera calibration, proprioception, wear, and tear, etc. Here are more explanations about different parts of the architecture in order to disentangle the noise of the low-cost robot’s actual and commanded executions.<br /> <br /> ===Grasping Formulation===<br /> <br /> Planar grasping is the object of interest in this architecture. It means that all the objects are grasped at the same height and vertical to the ground (ie: a fixed end-effector pitch). The final goal is to find &lt;math&gt;{(x, y, \theta)}&lt;/math&gt; given an observation &lt;math&gt; {I}&lt;/math&gt; of the object, where &lt;math&gt; {x}&lt;/math&gt; and &lt;math&gt; {y}&lt;/math&gt; are the translational degrees of freedom and &lt;math&gt; {\theta}&lt;/math&gt; is the rotational degrees of freedom (roll of the end-effector). For the purpose of comparison, they used a model which does not predict the &lt;math&gt;{(x, y, \theta)}&lt;/math&gt; directly from the image &lt;math&gt; {I}&lt;/math&gt;, but samples several smaller patches &lt;math&gt; {I_{P}}&lt;/math&gt; at different locations &lt;math&gt;{(x, y)}&lt;/math&gt;. Thus, the angle of grasp &lt;math&gt; {\theta}&lt;/math&gt; is predicted from these patches. Also, in order to have multi-modal predictions, discrete steps of the angle &lt;math&gt; {\theta}&lt;/math&gt;, &lt;math&gt; {\theta_{D}}&lt;/math&gt; is used. <br /> <br /> Hence, each datapoint consists of an image &lt;math&gt; {I}&lt;/math&gt;, the executed grasp &lt;math&gt;{(x, y, \theta)}&lt;/math&gt; and the grasp success/failure label g. Then, the image &lt;math&gt; {I}&lt;/math&gt; and the angle &lt;math&gt; {\theta}&lt;/math&gt; are converted to image patch &lt;math&gt; {I_{P}}&lt;/math&gt; and angle &lt;math&gt; {\theta_{D}}&lt;/math&gt;. Then, to minimize the classification error, a binary cross entropy loss is used which minimizes the error between the predicted and ground truth label &lt;math&gt; g &lt;/math&gt;. A convolutional neural network with weight initialization from pre-training on Imagenet is used for this formulation.<br /> <br /> (Note: On Cross Entropy:<br /> <br /> If we think of a distribution as the tool we use to encode symbols, then entropy measures the number of bits we'll need if we use the correct tool. This is optimal, in that we can't encode the symbols using fewer bits on average.<br /> In contrast, cross entropy is the number of bits we'll need if we encode symbols from y using the wrong tool &lt;math&gt; {\hat h}&lt;/math&gt; . This consists of encoding the &lt;math&gt; {i_{th}}&lt;/math&gt; symbol using &lt;math&gt; {\log(\frac{1}{{\hat h_i}})}&lt;/math&gt; bits instead of &lt;math&gt; {\log(\frac{1}{{ h_i}})}&lt;/math&gt; bits. We of course still take the expected value to the true distribution y , since it's the distribution that truly generates the symbols:<br /> <br /> \begin{align}<br /> H(y,\hat y) = \sum_i{y_i\log{\frac{1}{\hat y_i}}}<br /> \end{align}<br /> <br /> Cross entropy is always larger than entropy; encoding symbols according to the wrong distribution &lt;math&gt; {\hat y}&lt;/math&gt; will always make us use more bits. The only exception is the trivial case where y and &lt;math&gt; {\hat y}&lt;/math&gt; are equal, and in this case entropy and cross entropy are equal.)<br /> <br /> ===Modeling noise as latent variable===<br /> <br /> In order to tackle the problem of inaccurate position control and calibration due to cheap robot, they found a structure in the noise which is dependent on the robot and the design. They modeled this structure of noise as a latent variable and decoupled during training. The approach is shown in figure 2: <br /> <br /> <br /> [[File:aa2.PNG|600px|thumb|center|]]<br /> <br /> The conventional approach models the grasp success probability for a given image patch at a given angle where the variables of the environment which can introduce noise in the system is generally insignificant, due to the high accuracy of expensive, commercial robots. However, in the low cost setting with multiple robots collecting data in parallel, it becomes an important consideration for learning. <br /> <br /> The grasp success probability for image patch &lt;math&gt; {I_{P}}&lt;/math&gt; at angle &lt;math&gt; {\theta_{D}}&lt;/math&gt; is represented as &lt;math&gt; {P(g|I_{P},\theta_{D}; \mathcal{R} )}&lt;/math&gt; where &lt;math&gt; \mathcal{R}&lt;/math&gt; represents environment variables that can add noise to the system.<br /> <br /> The conditional probability of grasping at a noisy image patch &lt;math&gt;I_P&lt;/math&gt; for this model is computed by:<br /> <br /> <br /> ${ P(g|I_{P},\theta_{D}, \mathcal{R} ) = ∑_{( \widehat{I_P} \in \mathcal{P})} P(g│z=\widehat{I_P},\theta_{D},\mathcal{R}) \cdot P(z=\widehat{I_P} | \theta_{D},I_P,\mathcal{R})}$<br /> <br /> <br /> Here, &lt;math&gt; {z}&lt;/math&gt; models the latent variable of the actual patch executed, and &lt;math&gt;\widehat{I_P}&lt;/math&gt; belongs to a set of possible neighboring patches &lt;math&gt; \mathcal{P}&lt;/math&gt;.&lt;math&gt; P(z=\widehat{I_P}|\theta_D,I_P,\mathcal{R})&lt;/math&gt; shows the noise which can be caused by &lt;math&gt;\mathcal{R}&lt;/math&gt; variables and is implemented as the Noise Modelling Network (NMN). &lt;math&gt; {P(g│z=\widehat{I_P},\theta_{D}, \mathcal{R} )}&lt;/math&gt; shows the grasp prediction probability given the true patch and is implemented as the Grasp Prediction Network (GPN). The overall Robust-Grasp model is computed by marginalizing GPN and NMN.<br /> <br /> ===Learning the latent noise model===<br /> <br /> This section concerns what be the inputs to the NMN network should be and how should the inputs can be trained. The authors assume that &lt;math&gt; {z}&lt;/math&gt; is conditionally independent of the local patch-specific variables &lt;math&gt; {(I_{P}, \theta_{D})}&lt;/math&gt;. To estimate the latent variable &lt;math&gt; {z}&lt;/math&gt; given the global information &lt;math&gt;\mathcal{R}&lt;/math&gt;, i.e &lt;math&gt; P(z=\widehat{I_P}|\theta_D,I_P,\mathcal{R}) \equiv P(z=\widehat{I_P}|\mathcal{R})&lt;/math&gt;. Apart from the patch &lt;math&gt; I_{P} &lt;/math&gt; and grasp information (x, y, θ), they use information like image of the entire scene, ID of the robot and the location of the raw pixel. They argue that the image of the full scene could contain some essential information about the system such as the relative location of camera to the ground which may change over the lifetime of the robot. They used direct optimization to learn both NMN and GPN with noisy labels. However, explicit labels are not available to train NMN but the latent variable &lt;math&gt;z&lt;/math&gt; can be estimated using a technique such as Expectation-Maximization. The entire image of the scene and the environment information are the inputs of the NMN, as well as robot ID and raw-pixel grasp location. The output of the NMN is the probability distribution of the actual patches where the grasps are executed. Finally, a binary cross entropy loss is applied to the marginalized output of these two networks and the true grasp label g.<br /> <br /> ===Training details===<br /> <br /> They implemented their model in PyTorch using a pretrained ResNet-18 model. They concatenated 512 dimensional ResNet feature with a 1-hot vector of robot ID and the raw pixel location of the grasp for their NMN. Also, the inputs of the GPN are the original noisy patch plus 8 other equidistant patches from the original one.<br /> Their training process starts with training only GPN over 5 epochs of the data. Then, the NMN and the marginalization operator are added to the model. So, they train NMN and GPN simultaneously for the other 25 epochs.<br /> <br /> ==Results==<br /> <br /> In the results part of the paper, they show that collecting dataset in homes is essential for generalizing learning from unseen environments. They also show that modelling the noise in their Low-Cost Arm (LCA) can improve grasping performance.<br /> They collected data in parallel using multiple robots in 6 different homes, as shown in Figure 3. They used an object detector (tiny-YOLO) as the input data were unstructured due to LCA limited memory and computational capabilities. With an object location detected, class information was discarded, and a grasp was attempted. The grasp location in 3D was computed using PointCloud data. They scattered different objects in homes within 2m area to prevent collision of the robot with obstacles and let the robot move randomly and grasp objects. Finally, they collected a dataset with 28K grasp results.<br /> <br /> [[File:aa3.PNG|600px|thumb|center|]]<br /> <br /> To evaluate their approach in a more quantitative way, they used three test settings:<br /> <br /> - The first one is a binary classification or held-out data. The test set is collected by performing random grasps on objects. They measure the performance of binary classification by predicting the success or failure of grasping, given a location and the angle. Using binary classification allows for testing a lot of models without running them on real robots. They collected two held-out datasets using LCA in lab and homes and the dataset for Baxter robot.<br /> <br /> - The second one is Real Low-Cost Arm (Real-LCA). Here, they evaluate their model by running it in three unseen homes. They put 20 new objects in these three homes in different orientations. Since the objects and the environments are completely new, this tests could measure the generalization of the model.<br /> <br /> - The third one is Real Sawyer (Real-Sawyer). They evaluate the performance of their model by running the model on the Sawyer robot which is more accurate than the LCA. They tested their model in the lab environment to show that training models with the datasets collected from homes can improve the performance of models even in lab environments.<br /> <br /> They used baselines for both their data which is collected in homes and their model which is Robust-Grasp. They used two datasets for the baseline. The dataset collected by (Lab-Baxter) and the dataset collected by their LCA in the lab (Lab-LCA).<br /> They compared their Robust-Grasp model with the noise independent patch grasping model (Patch-Grasp) . They also compared their data and model with DexNet-3.0 (DexNet) for a strong real-world grasping baseline.<br /> <br /> ===Experiment 1: Performance on held-out data===<br /> <br /> Table 1 shows that the models trained on lab data cannot generalize to the Home-LCA environment (i.e. they overfit to their respective environments and attain a lower binary classification score). However, the model trained on Home-LCA has a good performance on both lab data and home environment.<br /> <br /> [[File:aa4.PNG|600px|thumb|center|]]<br /> <br /> ===Experiment 2: Performance on Real LCA Robot===<br /> <br /> In table 2, the performance of the Home-LCA is compared against a pre-trained DexNet and the model trained on the Lab-Baxter. Training on the Home-LCA dataset performs 43.7% better than training on the Lab-Baxter dataset and 33% better than DexNet. The low performance of DexNet can be described by the possible noise in the depth images that are caused by the natural light. DexNet, which requires high-quality depth sensing, cannot perform well in these scenarios. By using cheap commodity RGBD cameras in LCA, the noise in the depth images is not a matter of concern, as the model has no expectation of high-quality sensing.<br /> <br /> [[File:aa5.PNG|600px|thumb|center|]]<br /> <br /> ===Performance on Real Sawyer===<br /> <br /> To compare the performance of the Robust-Grasp model against the Patch-Grasp model without collecting noise-free data, they used Lab-Baxter for benchmarking, which is an accurate and better calibrated robot. The Sawyer robot is used for testing to ensure that the testing robot is different from both training robots. As shown in Table 3, the Robust-Grasp model trained on Home-LCA outperforms the Patch-Grasp model and achieves 77.5% accuracy. This accuracy is similar to several recent papers, however, this model was trained and tested in a different environment. The Robust-Grasp model also outperforms the Patch-Grasp by about 4% on binary classification. Furthermore, the visualizations of predicted noise corrections in Figure 4 shows that the corrections depend on both the pixel locations of the noisy grasp and the robot.<br /> <br /> [[File:aa6.PNG|600px|thumb|center|]]<br /> <br /> [[File:aa7.PNG|600px|thumb|center|]]<br /> <br /> ==Related work==<br /> <br /> Over the last few years, the interest of scaling up robot learning with large-scale datasets has been increased. Hence, many papers were published in this area. A hand annotated grasping dataset, a self-supervised grasping dataset, and grasping using reinforcement learning are some examples of using large-scale datasets for grasping. The work mentioned above used high-cost hardware and data labeling mechanisms. There were also many papers that worked on other robotic tasks like material recognition, pushing objects and manipulating a rope. However, none of these papers worked on real data in real environments like homes, they all used lab data.<br /> <br /> Furthermore, since grasping is one of the basic problems in robotics, there were some efforts to improve grasping. Classical approaches focused on physics-based issues of grasping and required 3D models of the objects. However, recent works focused on data-driven approaches which learn from visual observations to grasp objects. Simulation and real-world robots are both required for large-scale data collection. A versatile grasping model was proposed to achieve a 90% performance for a bin-picking task. The point here is that they usually require high-quality depth as input which seems to be a barrier for practical use of robots in real environments. High-quality depth sensing means a high cost to implement in hardware and thus is a barrier for practical use.<br /> <br /> Most labs use industrial robots or standard collaborative hardware for their experiments. Therefore, there is few research that used low-cost robots. One of the examples is learning using a cheap inaccurate robot for stack multiple blocks. Although mobile robots like iRobot’s Roomba have been in the home consumer electronics market for a decade, it is not clear whether learning approaches are used in it alongside mapping and planning.<br /> <br /> Learning from noisy inputs is another challenge specifically in computer vision. A controversial question which is often raised in this area is whether learning from noise can improve the performance. Some works show it could have bad effects on the performance; however, some other works find it valuable when the noise is independent or statistically dependent on the environment. In this paper, they used a model that can exploit the noise and learn a better grasping model.<br /> <br /> ==Conclusion==<br /> <br /> All in all, the paper presents an approach for collecting large-scale robot data in real home environments. They implemented their approach by using a mobile manipulator which is a lot cheaper than the existing industrial robots. They collected a dataset of 28K grasps in six different homes. In order to solve the problem of noisy labels which were caused by their inaccurate robots, they presented a framework to factor out the noise in the data. They tested their model by physically grasping 20 new objects in three new homes and in the lab. The model trained with home dataset showed 43.7% improvement over the models trained with lab data. Their results also showed that their model can improve the grasping performance even in lab environments. They also demonstrated that their architecture for modeling the noise improved the performance by about 10%.<br /> <br /> ==Critiques==<br /> <br /> This paper does not contain a significant algorithmic contribution. They are just combining a large number of data engineering techniques for the robot learning problem. The authors claim that they have obtained 43.7% more accuracy than baseline models, but it does not seem to be a fair comparison as the data collection happened in simulated settings in the lab for other methods, whereas the authors use the home dataset. The authors must have also discussed safety issues when training robots in real environments as against simulated environments like labs. The authors are encouraging other researchers to look outside the labs, but are not discussing the critical safety issues in this approach.<br /> <br /> Another strange finding is that the paper mentions that they &quot;follow a model architecture similar to [Pinto and Gupta ],&quot; however, the proposed model is, in fact, a fine-tuned resnet-18 architecture. Pinto and Gupta, implement a version similar to AlexNet as shown below in Figure 5.<br /> <br /> [[File:Figure_5_PandG.JPG | 450px|thumb|center|Figure 5: AlexNet architecture implemented in Pinto and Gupta .]]<br /> <br /> <br /> The paper argues that the dataset collected by the LCA is noisy, since the robot is cheap and inaccurate. It further asserts that in order to handle the noise in the dataset, they can model the noise as a latent variable and their model can improve the performance of grasping. Although learning from noisy data and achieving a good performance is valuable, it is better that they test their noise modeling network for other robots as well. Since their noise modelling network takes robot information as an input, it would be a good idea to generalize it by testing it using different inaccurate robots to ensure that it would perform well.<br /> <br /> They did not mention other aspects of their comparison, for example they could mention their training time compared to other models or the size of other datasets.<br /> <br /> ==References==<br /> <br /> #Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. &quot;Domain randomization for transferring deep neural networks from simulation to the real world.&quot; 2017. URL https://arxiv.org/abs/1703.06907.<br /> #Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. &quot;Sim-to-real transfer of robotic control with dynamics randomization.&quot; arXiv preprint arXiv:1710.06537,2017.<br /> #Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. &quot;Asymmetric actor-critic for image-based robot learning.&quot; Robotics Science and Systems, 2018.<br /> #Lerrel Pinto and Abhinav Gupta. &quot;Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours.&quot; CoRR, abs/1509.06825, 2015. URL http://arxiv.org/abs/1509. 06825.<br /> #Adithyavairavan Murali, Lerrel Pinto, Dhiraj Gandhi, and Abhinav Gupta. &quot;CASSL: Curriculum accelerated self-supervised learning.&quot; International Conference on Robotics and Automation, 2018.<br /> # Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. &quot;End-to-end training of deep visuomotor policies.&quot; The Journal of Machine Learning Research, 17(1):1334–1373, 2016.<br /> #Sergey Levine, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen. &quot;Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.&quot; CoRR, abs/1603.02199, 2016. URL http://arxiv.org/abs/1603.02199.<br /> #Pulkit Agarwal, Ashwin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. &quot;Learning to poke by poking: Experiential learning of intuitive physics.&quot; 2016. URL http://arxiv.org/ abs/1606.07419<br /> #Chelsea Finn, Ian Goodfellow, and Sergey Levine. &quot;Unsupervised learning for physical interaction through video prediction.&quot; In Advances in neural information processing systems, 2016.<br /> #Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. &quot;Combining self-supervised learning and imitation for vision-based rope manipulation.&quot; International Conference on Robotics and Automation, 2017.<br /> #Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. &quot;Revisiting unreasonable effectiveness of data in deep learning era.&quot; ICCV, 2017.<br /> #Marc Peter Deisenroth, Carl Edward Rasmussen, and Dieter Fox. Learning to control a low-cost manipulator using data-efficient reinforcement learning. RSS, 2011.<br /> #David F Nettleton, Albert Orriols-Puig, and Albert Fornells. A study of the effect of different types of noise on the precision of supervised learning techniques. Artificial intelligence review, 33(4):275–306, 2010.<br /> #Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems, 25(5):845–869, 2014.<br /> #Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2691–2699, 2015.<br /> #Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Countering_Adversarial_Images_Using_Input_Transformations&diff=42034 Countering Adversarial Images Using Input Transformations 2018-11-30T13:01:41Z <p>C9sharma: /* Defenses */</p> <hr /> <div>The code for this paper is available here[https://github.com/facebookresearch/adversarial_image_defenses]<br /> <br /> ==Motivation ==<br /> As the use of machine intelligence has increased, robustness has become a critical feature to guarantee the reliability of deployed machine-learning systems. However, recent research has shown that existing models are not robust to small, adversarially designed perturbations to the input. Adversarial examples are inputs to Machine Learning models so that an attacker has intentionally designed to cause the model to make a mistake. Adversarially perturbed examples have been deployed to attack image classification services (Liu et al., 2016), speech recognition systems (Cisse et al., 2017a), and robot vision (Melis et al., 2017). The existence of these adversarial examples has motivated proposals for approaches that increase the robustness of learning systems to such examples. In the example below (Goodfellow et. al) , a small perturbation is applied to the original image of a panda, changing the prediction to a gibbon.<br /> <br /> [[File:Panda.png|center]]<br /> <br /> ==Introduction==<br /> The paper studies strategies that defend against adversarial example attacks on image classification systems by transforming the images before feeding them to a Convolutional Network Classifier. <br /> Generally, defenses against adversarial examples fall into two main categories:<br /> <br /> # Model-Specific – They enforce model properties such as smoothness and invariance via the learning algorithm. <br /> # Model-Agnostic – They try to remove adversarial perturbations from the input. <br /> <br /> Model-specific defense strategies make strong assumptions about expected adversarial attacks. As a result, they violate the Kerckhoffs principle, which states that adversaries can circumvent model-specific defenses by simply changing how an attack is executed. This paper focuses on increasing the effectiveness of model-agnostic defense strategies. Specifically, they investigated the following image transformations as a means for protecting against adversarial images:<br /> <br /> # Image Cropping and Re-scaling (Graese et al, 2016). <br /> # Bit Depth Reduction (Xu et. al, 2017) <br /> # JPEG Compression (Dziugaite et al, 2016) <br /> # Total Variance Minimization (Rudin et al, 1992) <br /> # Image Quilting (Efros &amp; Freeman, 2001). <br /> <br /> These image transformations have been studied against Adversarial attacks such as the fast gradient sign method (Goodfelow et. al., 2015), its iterative extension (Kurakin et al., 2016a), Deepfool (Moosavi-Dezfooli et al., 2016), and the Carlini &amp; Wagner (2017) &lt;math&gt;L_2&lt;/math&gt;attack. <br /> <br /> The authors in this paper try to focus on increasing the effectiveness of model-agnostic defense strategies through approaches that:<br /> # remove the adversarial perturbations from input images,<br /> # maintain sufficient information in input images to correctly classify them,<br /> # and are still effective in situations where the adversary has information about the defense strategy being used.<br /> <br /> From their experiments, the strongest defenses are based on Total Variance Minimization and Image Quilting. These defenses are non-differentiable and inherently random which makes it difficult for an adversary to get around them.<br /> <br /> ==Previous Work==<br /> Recently, a lot of research has focused on countering adversarial threats. Wang et al , proposed a new adversary resistant technique that obstructs attackers from constructing impactful adversarial images. This is done by randomly nullifying features within images. Tramer et al , showed the state-of-the-art Ensemble Adversarial Training Method, which augments the training process but not only included adversarial images constructed from their model but also including adversarial images generated from an ensemble of other models. Their method implemented on an Inception V2 classifier finished 1st among 70 submissions of NIPS 2017 competition on Defenses against Adversarial Attacks. Graese, et al. , showed how input transformation such as shifting, blurring and noise can render the majority of the adversarial examples as non-adversarial. Xu et al. demonstrated, how feature squeezing methods, such as reducing the color bit depth of each pixel and spatial smoothing, defends against attacks. Dziugaite et al , studied the effect of JPG compression on adversarial images. Chen et al.  introduce an advanced denoising algorithm with GAN based noise modeling in order to improve the blind denoising performance in low level vision processing. The GAN is trained to estimate the noise distribution over the input noisy images and to generate noise samples. Although meant for image processing, this method can be generalized to target adversarial examples where the unknown noise generating algorithm can be leveraged.<br /> <br /> ==Terminology==<br /> <br /> '''Gray Box Attack''' : Model Architecture and parameters are Public<br /> <br /> '''Black Box Attack''': Adversary does not have access to the model.<br /> <br /> An interesting and important observation of adversarial examples is that they generally are not model or architecture specific. Adversarial examples generated for one neural network architecture will transfer very well to another architecture. In other words, if you wanted to trick a model you could create your own model and adversarial examples based off of it. Then these same adversarial examples will most probably trick the other model as well. This has huge implications as it means that it is possible to create adversarial examples for a completely black box model where we have no prior knowledge of the internal mechanics. [https://ml.berkeley.edu/blog/2018/01/10/adversarial-examples/ reference]<br /> <br /> '''Non Targeted Adversarial Attack''': The goal of the attack is to modify a source image in a way such that the image will be classified incorrectly by the network.<br /> <br /> This is an example on non-targeted adversarial attacks to be more clear [https://ml.berkeley.edu/blog/2018/01/10/adversarial-examples/ reference]:<br /> [[File:non-targeted O.JPG| 600px|center]]<br /> <br /> '''Targeted Adversarial Attack''': The goal of the attack is to modify a source image in way such that image will be classified as a ''target'' class by the network.<br /> <br /> This is an example on targeted adversarial attacks to be more clear [https://ml.berkeley.edu/blog/2018/01/10/adversarial-examples/ reference]:<br /> [[File:Targeted O.JPG| 600px|center]]<br /> <br /> '''Defense''': A defense is a strategy that aims make the prediction on an adversarial example h(x') equal to the prediction on the corresponding clean example h(x).<br /> <br /> == Problem Definition ==<br /> The paper discusses non-targeted adversarial attacks for image recognition systems. Given image space &lt;math&gt;\mathcal{X} = [0,1]^{H \times W \times C}&lt;/math&gt;, a source image &lt;math&gt;x \in \mathcal{X}&lt;/math&gt;, and a classifier &lt;math&gt;h(.)&lt;/math&gt;, a non-targeted adversarial example of &lt;math&gt;x&lt;/math&gt; is a perturbed image &lt;math&gt;x'&lt;/math&gt;, such that &lt;math&gt;h(x) \neq h(x')&lt;/math&gt; and &lt;math&gt;d(x, x') \leq \rho&lt;/math&gt; for some dissimilarity function &lt;math&gt;d(·, ·)&lt;/math&gt; and &lt;math&gt;\rho \geq 0&lt;/math&gt;. In the best case scenario, &lt;math&gt;d(·, ·)&lt;/math&gt; measures the perceptual difference between the original image &lt;math&gt;x&lt;/math&gt; and the perturbed image &lt;math&gt;x'&lt;/math&gt;, but usually, Euclidean distance (&lt;math&gt;||x - x'||_2&lt;/math&gt;) or the Chebyshov distance (&lt;math&gt;||x - x'||_{\infty}&lt;/math&gt;) are used.<br /> <br /> From a set of N clean images &lt;math&gt;[{x_{1}, …, x_{N}}]&lt;/math&gt;, an adversarial attack aims to generate &lt;math&gt;[{x'_{1}, …, x'_{N}}]&lt;/math&gt; images, such that (&lt;math&gt;x'_{n}&lt;/math&gt;) is an adversary of (&lt;math&gt;x_{n}&lt;/math&gt;).<br /> <br /> The success rate of an attack is given as: <br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \frac{1}{N}\sum_{n=1}^{N}I[h(x_n) &amp;ne; h({x_n}^\prime)],<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> which is the proportions of predictions that were altered by an attack.<br /> <br /> The success rate is generally measured as a function of the magnitude of perturbations performed by the attack. In this paper, L2 perturbations are used and are quantified using the normalized L2-dissimilarity metric:<br /> &lt;math&gt; \frac{1}{N} \sum_{n=1}^N{\frac{\vert \vert x_n - x'_n \vert \vert_2}{\vert \vert x_n \vert \vert_2}} &lt;/math&gt;<br /> <br /> A strong adversarial attack has a high rate, while its normalized L2-dissimilarity given by the above equation is less.<br /> <br /> ==Adversarial Attacks==<br /> <br /> Although the exact effect that adversarial examples have on network is unknown, Ian Goodfellow et. al's Deep Learning book states that adversarial examples exploit the linearity of neural networks to perturb the cost function to force incorrect classifications. Images are often high resolution, and thus have thousands of pixels (millions for HD images). An epsilon ball perturbation when dimensionality is in the magnitude of thousands/millions greatly effects the cost function (especially if it increases loss at every pixel). Hence, although the following methods such as FGSM and Iterative FGSM are very straightforward, they greatly influence the network under a white box attack. <br /> <br /> For the experimental purposes, below 4 attacks have been studied in the paper:<br /> <br /> 1. '''Fast Gradient Sign Method (FGSM; Goodfellow et al. (2015)) ''': Given a source input &lt;math&gt;x&lt;/math&gt;, and true label &lt;math&gt;y&lt;/math&gt;, and let &lt;math&gt;l(.,.)&lt;/math&gt; be the differentiable loss function used to train the classifier &lt;math&gt;h(.)&lt;/math&gt;. Then the corresponding adversarial example is given by:<br /> <br /> &lt;math&gt;x' = x + \epsilon \cdot sign(\nabla_x l(x, y))&lt;/math&gt;<br /> <br /> for some &lt;math&gt;\epsilon \gt 0&lt;/math&gt; which controls the perturbation magnitude.<br /> <br /> 2. '''Iterative FGSM ((I-FGSM; Kurakin et al. (2016b)) ''': iteratively applies the FGSM update, where M is the number of iterations. It is given as:<br /> <br /> &lt;math&gt;x^{(m)} = x^{(m-1)} + \epsilon \cdot sign(\nabla_{x^{m-1}} l(x^{m-1}, y))&lt;/math&gt;<br /> <br /> where &lt;math&gt;m = 1,...,M; x^{(0)} = x;&lt;/math&gt; and &lt;math&gt;x' = x^{(M)}&lt;/math&gt;. M is set such that &lt;math&gt;h(x) \neq h(x')&lt;/math&gt;.<br /> <br /> Both FGSM and I-FGSM work by minimizing the Chebyshev distance between the inputs and the generated adversarial examples.<br /> <br /> 3. '''DeepFool ((Moosavi-Dezfooliet al., 2016) ''': projects x onto a linearization of the decision boundary defined by binary classifier h(.) for M iterations. This can be particularly effictive when a network uses ReLU activation functions. It is given as:<br /> <br /> [[File:DeepFool.PNG|400px |]]<br /> <br /> 4. '''Carlini-Wagner's L2 attack (CW-L2; Carlini &amp; Wagner (2017)) ''': propose an optimization-based attack that combines a differentiable surrogate for the model’s classification accuracy with an L2-penalty term which encourages the adversary image to be close to the original image. Let &lt;math&gt;Z(x)&lt;/math&gt; be the operation that computes the logit vector (i.e., the output before the softmax layer) for an input &lt;math&gt;x&lt;/math&gt;, and &lt;math&gt;Z(x)_k&lt;/math&gt; be the logit value corresponding to class &lt;math&gt;k&lt;/math&gt;. The untargeted variant<br /> of CW-L2 finds a solution to the unconstrained optimization problem. It is given as:<br /> <br /> [[File:Carlini.PNG|500px |]]<br /> <br /> As mentioned earlier, the first two attacks minimize the Chebyshev distance whereas the last two attacks minimize the Euclidean distance between the inputs and the adversarial examples.<br /> <br /> All the methods described above maintain &lt;math&gt;x' \in \mathcal{X}&lt;/math&gt; by performing value clipping. <br /> <br /> Below figure shows adversarial images and corresponding perturbations at five levels of normalized L2-dissimilarity for all four attacks, mentioned above.<br /> <br /> [[File:Strength.PNG|thumb|center| 600px |Figure 1: Adversarial images and corresponding perturbations at five levels of normalized L2- dissimilarity for all four attacks.]]<br /> <br /> ==Defenses==<br /> Defense is a strategy that aims to make the prediction on an adversarial example equal to the prediction on the corresponding clean example, and the particular structure of adversarial perturbations &lt;math&gt; x-x' &lt;/math&gt; have been shown in Figure 1.<br /> Five image transformations that alter the structure of these perturbations have been studied:<br /> # Image Cropping and Re-scaling, <br /> # Bit Depth Reduction, <br /> # JPEG Compression, <br /> # Total Variance Minimization, <br /> # Image Quilting.<br /> <br /> '''Image cropping and Rescaling''' has the effect of altering the spatial positioning of the adversarial perturbation. In this study, images are cropped and re-scaled during training time as part of data-augmentation. At test time, the predictions of randomly cropped are averaged.<br /> <br /> '''Bit Depth Reduction (Xu et. al) ''' performs a simple type of quantization that can remove small (adversarial) variations in pixel values from an image. Images are reduced to 3 bits in the experiment.<br /> <br /> '''JPEG Compression and Decompression (Dziugaite etal., 2016)''' removes small perturbations by performing simple quantization. The authors use a quality level of 75/100 in their experiments<br /> <br /> '''Total Variance Minimization (Rudin et. al) ''' :<br /> This combines pixel dropout with total variance minimization. This approach randomly selects a small set of pixels, and reconstructs the “simplest” image that is consistent with the selected pixels. The reconstructed image does not contain the adversarial perturbations because these perturbations tend to be small and localized.Specifically, we first select a random set of pixels by sampling a Bernoulli random variable &lt;math&gt;X(i; j; k)&lt;/math&gt; for each pixel location &lt;math&gt;(i; j; k)&lt;/math&gt;;we maintain a pixel when &lt;math&gt;(i; j; k)&lt;/math&gt;= 1. Next, we use total variation, minimization to constructs an image z that is similar to the (perturbed) input image x for the selected<br /> set of pixels, whilst also being “simple” in terms of total variation by solving:<br /> <br /> [[File:TV!.png|300px|]] , <br /> <br /> where &lt;math&gt;TV_{p}(z)&lt;/math&gt; represents &lt;math&gt;L_{p}&lt;/math&gt; total variation of '''z''' :<br /> <br /> [[File:TV2.png|500px|]]<br /> <br /> The total variation (TV) measures the amount of fine-scale variation in the image z, as a result of which TV minimization encourages removal of small (adversarial) perturbations in the image. The objective function is convex in &lt;math&gt;z&lt;/math&gt;, which makes solving for z straightforward. In the paper, p = 2 and a special-purpose solver based on the split Bregman method (Goldstein &amp; Osher, 2009) to perform total variance minimization efficiently is employed.<br /> The effectiveness of TV minimization is illustrated by the images in the middle column of the figure below: in particular, note that the adversarial perturbations that were present in the background for the non- transformed image (see bottom-left image) have nearly completely disappeared in the TV-minimized adversarial image (bottom-center image). As expected, TV minimization also changes image structure in non-homogeneous regions of the image, but as these perturbations were not adversarially designed we expect the negative effect of these changes to be limited.<br /> <br /> [[File:tvx.png]]<br /> <br /> The figure above represents illustration of total variance minimiza- tion and image quilting applied to an original and an adversarial image (produced using I-FGSM with ε = 0.03, corresponding to a normalized L2 - dissimilarity of 0.075). From left to right, the columns correspond to: (1) no transformation, (2) total variance minimization, and (3) image quilting. From top to bottom, rows correspond to: (1) the original image, (2) the corresponding adversarial image produced by I-FGSM, and (3) the absolute difference between the two images above. Difference images were multiplied by a constant scaling factor to increase visibility.<br /> <br /> <br /> '''Image Quilting (Efros &amp; Freeman, 2001) '''<br /> Image Quilting is a non-parametric technique that synthesizes images by piecing together small patches that are taken from a database of image patches. The algorithm places appropriate patches in the database for a predefined set of grid points and computes minimum graph cuts in all overlapping boundary regions to remove edge artifacts. Image Quilting can be used to remove adversarial perturbations by constructing a patch database that only contains patches from &quot;clean&quot; images ( without adversarial perturbations); the patches used to create the synthesized image are selected by finding the K nearest neighbors ( in pixel space) of the corresponding patch from the adversarial image in the patch database, and picking one of these neighbors uniformly at random. The motivation for this defense is that resulting image only contains pixels that were not modified by the adversary - the database of real patches is unlikely to contain the structures that appear in adversarial images.<br /> <br /> =Experiments=<br /> <br /> Five experiments were performed to test the efficacy of defences. The first four experiments consider gray and black box attacks. The gray-box attack applies defenses on input adversarial images for the convolutional networks. The adversary is able to read model architecture and parameters but not the defence strategy. The black-box attack replaces convolutional network by a trained network with image-transformations. The final experiment compares the authors' defenses with prior work. <br /> <br /> '''Set up:'''<br /> Experiments are performed on the ImageNet image classification dataset. The dataset comprises 1.2 million training images and 50,000 test images that correspond to one of 1000 classes. The adversarial images are produced by attacking a ResNet-50 model, with different kinds of attacks mentioned in Section5. The strength of an adversary is measured in terms of its normalized L2-dissimilarity. To produce the adversarial images, L2 dissimilarity for each of the attack was set as below:<br /> <br /> - FGSM. Increasing the step size &lt;math&gt;\epsilon&lt;/math&gt;, increases the normalized L2-dissimilarity.<br /> <br /> - I-FGSM. We fix M=10, and increase &lt;math&gt;\epsilon&lt;/math&gt; to increase the normalized L2-dissimilarity.<br /> <br /> - DeepFool. We fix M=5, and increase &lt;math&gt;\epsilon&lt;/math&gt; to increase the normalized L2-dissimilarity.<br /> <br /> - CW-L2. We fix &lt;math&gt;k&lt;/math&gt;=0 and &lt;math&gt;\lambda_{f}&lt;/math&gt; =10, and multiply the resulting perturbation <br /> <br /> The hyperparameters of the defenses have been fixed in all the experiments. Specifically the pixel dropout probability was set to &lt;math&gt;p&lt;/math&gt;=0.5 and regularization parameter of total variation minimizer &lt;math&gt;\lambda_{TV}&lt;/math&gt;=0.03.<br /> <br /> Below figure shows the difference between the set up in different experiments below. The network is either trained on a) regular images or b) transformed images. The different settings are marked by 8.1, 8.2 and 8.3 <br /> [[File:models3.png |center]] <br /> <br /> ==GrayBox - Image Transformation at Test Time== <br /> This experiment applies a transformation on adversarial images at test time before feeding them to a ResNet -50 which was trained to classify clean images. Below figure shows the results for five different transformations applied and their corresponding Top-1 accuracy. Few of the interesting observations from the plot are: All of the image transformations partly eliminate the effects of the attack, Crop ensemble gives the best accuracy around 40-60 percent, with an ensemble size of 30. The accuracy of Image Quilting Defense hardly deteriorates as the strength of the adversary increases. However, it does impact accuracy on non-adversarial examples.<br /> <br /> [[File:sFig4.png|center|600px |]]<br /> <br /> ==BlackBox - Image Transformation at Training and Test Time==<br /> ResNet-50 model was trained on transformed ImageNet Training images. Before feeding the images to the network for training, standard data augmentation (from He et al) along with bit depth reduction, JPEG Compression, TV Minimization, or Image Quilting were applied on the images. The classification accuracy on the same adversarial images as in the previous case is shown Figure below. (Adversary cannot get this trained model to generate new images - Hence this is assumed as a Black Box setting!). Below figure concludes that training Convolutional Neural Networks on images that are transformed in the same way at test time, dramatically improves the effectiveness of all transformation defenses. Nearly 80 -90 % of the attacks are defended successfully, even when the L2- dissimilarity is high.<br /> <br /> <br /> [[File:sFig5.png|center|600px |]]<br /> <br /> <br /> ==Blackbox - Ensembling==<br /> Four networks ResNet-50, ResNet-10, DenseNet-169, and Inception-v4 along with an ensemble of defenses were studied, as shown in Table 1. The adversarial images are produced by attacking a ResNet-50 model. The results in the table conclude that Inception-v4 performs best. This could be due to that network having a higher accuracy even in non-adversarial settings. The best ensemble of defenses achieves an accuracy of about 71% against all the other attacks. The attacks deteriorate the accuracy of the best defenses (a combination of cropping, TVM, image quilting, and model transfer) by at most 6%. Gains of 1-2% in classification accuracy could be found from ensembling different defenses, while gains of 2-3% were found from transferring attacks to different network architectures.<br /> <br /> <br /> [[File:sTab1.png|600px|thumb|center|Table 1. Top-1 classification accuracy of ensemble and model transfer defenses (columns) against four black-box attacks (rows). The four networks we use to classify images are ResNet-50 (RN50), ResNet-101 (RN101), DenseNet-169 (DN169), and Inception-v4 (Iv4). Adversarial images are generated by running attacks against the ResNet-50 model, aiming for an average normalized &lt;math&gt;L_2&lt;/math&gt;-dissimilarity of 0.06. Higher is better. The best defense against each attack is typeset in boldface.]]<br /> <br /> ==GrayBox - Image Transformation at Training and Test Time ==<br /> In this experiment, the adversary has access to the network and the related parameters (but does not have access to the input transformations applied at test time). From the network trained in-(BlackBox: Image Transformation at Training and Test Time), novel adversarial images were generated by the four attack methods. The results show that Bit-Depth Reduction and JPEG Compression are weak defenses in such a gray box setting. In contrast, image cropping, rescaling, variation minimization, and image quilting are more robust against adversarial images in this setting.<br /> The results for this experiment are shown in below figure. Networks using these defenses classify up to 50 % of images correctly.<br /> <br /> [[File:sFig6.png|center| 600px |]]<br /> <br /> ==Comparison With Ensemble Adversarial Training==<br /> The results of the experiment are compared with the state of the art ensemble adversarial training approach proposed by Tramer et al. . Ensemble Training fits the parameters of a Convolutional Neural Network on adversarial examples that were generated to attack an ensemble of pre-trained models. The model release by Tramer et al : an Inception-Resnet-v2, trained on adversarial examples generated by FGSM against Inception-Resnet-v2 and Inception-v3 models. The authors compared their ResNet-50 models with image cropping, total variance minimization and image quilting defenses. Two assumption differences need to be noticed. Their defenses assume the input transformation is unknown to the adversary and no prior knowledge of the attacks is being used. The results of ensemble training and the pre-processing techniques mentioned in this paper are shown in Table 2. The results show that ensemble adversarial training works better on FGSM attacks (which it uses at training time), but is outperformed by each of the transformation-based defenses all other attacks.<br /> <br /> <br /> <br /> [[File:sTab2.png|600px|thumb|center|Table 2. Top-1 classification accuracy on images perturbed using attacks against ResNet-50 models trained on input-transformed images and an Inception-v4 model trained using ensemble adversarial. Adversarial images are generated by running attacks against the models, aiming for an average normalized &lt;math&gt;L_2&lt;/math&gt;-dissimilarity of 0.06. The best defense against each attack is typeset in boldface.]]<br /> <br /> =Discussion/Conclusions=<br /> The paper proposed reasonable approaches to countering adversarial images. The authors evaluated Total Variance Minimization and Image Quilting and compared it with already proposed ideas like Image Cropping - Rescaling, Bit Depth Reduction, JPEG Compression, and Decompression on the challenging ImageNet dataset.<br /> Previous work by Wang et al.  shows that a strong input defense should be nondifferentiable and randomized. Two of the defenses - namely Total Variation Minimization and Image Quilting, both possess this property.<br /> <br /> Image quilting involves a discrete variable that conducts selection of a patch from the database, which is a non-differentiable operation.<br /> Additionally, total variation minimization randomly conducts pixels selection from the pixels it uses to measure reconstruction<br /> error during creation of the de-noised image. Image quilting conducts random selection of a particular K<br /> nearest neighbor uniformly, but in a random manner. This inherent randomness makes it difficult to attack the model. <br /> <br /> Future work suggests applying the same techniques to other domains such as speech recognition and image segmentation. For example, in speech recognition, total variance minimization can be used to remove perturbations from waveforms and &quot;spectrogram quilting&quot; techniques that reconstruct a spectrogram could be developed. The proposed input-transformation defenses can also be combined with ensemble adversarial training by Tramèr et al. to study new attack methods.<br /> <br /> =Critiques=<br /> 1. The terminology of Black Box, White Box, and Grey Box attack is not exactly given and clear.<br /> <br /> 2. White Box attacks could have been considered where the adversary has a full access to the model as well as the pre-processing techniques.<br /> <br /> 3. Though the authors did a considerable work in showing the effect of four attacks on ImageNet database, much stronger attacks (Madry et al) , could have been evaluated.<br /> <br /> 4. Authors claim that the success rate is generally measured as a function of the magnitude of perturbations, performed by the attack using the L2- dissimilarity, but the claim is not supported by any references. None of the previous work has used these metrics.<br /> <br /> 5. ([https://openreview.net/forum?id=SyJ7ClWCb])In the new draft of the paper, the authors add the sentence &quot;our defenses assume that part of the defense strategy (viz., the input transformation) is unknown to the adversary&quot;.<br /> <br /> This is a completely unreasonable assumption. Any algorithm which hopes to be secure must allow the adversary to, at the very least, understand what the defense is that's being used. Consider a world where the defense here is implemented in practice: any attacker in the world could just go look up the paper, read the description of the algorithm, and know how it works.<br /> <br /> =References=<br /> <br /> 1. Chuan Guo , Mayank Rana &amp; Moustapha Ciss´e &amp; Laurens van der Maaten , Countering Adversarial Images Using Input Transformations<br /> <br /> 2. Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, Patrick McDaniel, Ensemble Adversarial Training: Attacks and defenses.<br /> <br /> 3. Abigail Graese, Andras Rozsa, and Terrance E. Boult. Assessing threat of adversarial examples of deep neural networks. CoRR, abs/1610.04256, 2016. <br /> <br /> 4. Qinglong Wang, Wenbo Guo, Kaixuan Zhang, Alexander G. Ororbia II, Xinyu Xing, C. Lee Giles, and Xue Liu. Adversary resistant deep neural networks with an application to malware detection. CoRR, abs/1610.01239, 2016a.<br /> <br /> 5. Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. CoRR, abs/1704.01155, 2017. <br /> <br /> 6. Gintare Karolina Dziugaite, Zoubin Ghahramani, and Daniel Roy. A study of the effect of JPG compression on adversarial images. CoRR, abs/1608.00853, 2016.<br /> <br /> 7. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, Adrian Vladu .Towards Deep Learning Models Resistant to Adversarial Attacks, arXiv:1706.06083v3<br /> <br /> 8. Alexei Efros and William Freeman. Image quilting for texture synthesis and transfer. In Proc. SIGGRAPH, pp. 341–346, 2001.<br /> <br /> 9. Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D, 60:259–268, 1992.<br /> <br /> 10. Qinglong Wang, Wenbo Guo, Kaixuan Zhang, Alexander G. Ororbia II, Xinyu Xing, C. Lee Giles, and Xue Liu. Learning adversary-resistant deep neural networks. CoRR, abs/1612.01401, 2016b.<br /> <br /> 11. Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. CoRR, abs/1611.02770, 2016.<br /> <br /> 12. Moustapha Cisse, Yossi Adi, Natalia Neverova, and Joseph Keshet. Houdini: Fooling deep structured prediction models. CoRR, abs/1707.05373, 2017 <br /> <br /> 13. Marco Melis, Ambra Demontis, Battista Biggio, Gavin Brown, Giorgio Fumera, and Fabio Roli. Is deep learning safe for robot vision? adversarial examples against the icub humanoid. CoRR,abs/1708.06939, 2017.<br /> <br /> 14. Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. CoRR, abs/1607.02533, 2016b.<br /> <br /> 15. Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In Proc. CVPR, pp. 2574–2582, 2016.<br /> <br /> 16. Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, pp. 39–57, 2017.<br /> <br /> 17. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Proc. ICLR, 2015.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=File:tvx.png&diff=42033 File:tvx.png 2018-11-30T12:56:31Z <p>C9sharma: Illustration of total variance minimization and image quilting applied to an original and an adversarial image (produced using I-FGSM with ε = 0.03, corresponding to a normalized L2 - dissimilarity of 0.075). From left to right, the columns correspond...</p> <hr /> <div>Illustration of total variance minimization and image quilting applied to an original and an adversarial image (produced using I-FGSM with ε = 0.03, corresponding to a normalized L2 - dissimilarity of 0.075). From left to right, the columns correspond to: (1) no transformation, (2) total variance minimization, and (3) image quilting. From top to bottom, rows correspond to: (1) the original image, (2) the corresponding adversarial image produced by I-FGSM, and (3) the ab- solute difference between the two images above. Difference images were multiplied by a constant scaling factor to increase visibility</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=CapsuleNets&diff=41896 CapsuleNets 2018-11-29T17:54:03Z <p>C9sharma: /* Motivation */</p> <hr /> <div>The paper &quot;Dynamic Routing Between Capsules&quot; was written by three researchers at Google Brain: Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. This paper was published and presented at the 31st Conference on Neural Information Processing Systems (NIPS 2017) in Long Beach, California. The same three researchers recently published a highly related paper &quot;[https://openreview.net/pdf?id=HJWLfGWRb Matrix Capsules with EM Routing]&quot; for ICLR 2018.<br /> <br /> =Motivation=<br /> <br /> Ever since AlexNet eclipsed the performance of competing architectures in the 2012 ImageNet challenge, convolutional neural networks have maintained their dominance in computer vision applications. Despite the recent successes and innovations brought about by convolutional neural networks, some assumptions made in these networks are perhaps unwarranted and deficient. Using a novel neural network architecture, the authors create CapsuleNets, a network that they claim is able to learn image representations in a more robust, human-like manner. With only a 3 layer capsule network, they achieved near state-of-the-art results on MNIST.<br /> <br /> The activities of the neurons within an active capsule represent the various properties of a particular entity that is present in the image. These properties can include many different types of instantiation parameter such as pose (position, size, orientation), deformation, velocity, albedo, hue, texture, etc. One very special property is the existence of the instantiated entity in the image. An obvious way to represent existence is by using a separate logistic unit whose output is the probability that the entity exists. This paper explores an interesting alternative which is to use the overall length of the vector of instantiation parameters to represent the existence of the entity and to force the orientation of the vector to represent the properties of the entity. The length of the vector output of a capsule cannot exceed 1 because of an application of a non-linearity that leaves the orientation of the vector unchanged but scales down its magnitude.<br /> <br /> The fact that the output of a capsule is a vector makes it possible to use a powerful dynamic routing mechanism to ensure that the output of the capsule gets sent to an appropriate parent in the layer above. Initially, the output is routed to all possible parents but is scaled down by coupling coefficients that sum to 1. For each possible parent, the capsule computes a “prediction vector” by multiplying its own output by a weight matrix. If this prediction vector has a large scalar product with the output of a possible parent, there is top-down feedback which increases the coupling coefficient for that parent and decreasing it for other parents. This increases the contribution that the capsule makes to that parent thus further increasing the scalar product of the capsule’s prediction with the parent’s output. This type of “routing-by-agreement” should be far more effective than the very primitive form of routing implemented by max-pooling, which allows neurons in one layer to ignore all but the most active feature detector in a local pool in the layer below. The authors demonstrate that our dynamic routing mechanism is an effective way to implement the “explaining away” that is needed for segmenting highly overlapping objects<br /> <br /> ==Adversarial Examples==<br /> <br /> First discussed by Christian Szegedy et. al. in late 2013, adversarial examples have been heavily discussed by the deep learning community as a potential security threat to AI learning. Adversarial examples are defined as inputs that an attacker creates intentionally fool a machine learning model. An example of an adversarial example is shown below: <br /> <br /> [[File:adversarial_img_1.png ‎|center]]<br /> To the human eye, the image appears to be a panda both before and after noise is injected into the image, whereas the trained ConvNet model discerns the noisy image as a Gibbon with almost 100% certainty. The fact that the network is unable to classify the above image as a panda after the epsilon perturbation leads to many potential security risks in AI dependent systems such as self-driving vehicles. Although various methods have been suggested to combat adversarial examples, robust defences are hard to construct due to the inherent difficulties in constructing theoretical models for the adversarial example crafting process. However, beyond the fact that these examples may serve as a security threat, it emphasizes that these convolutional neural networks do not learn image classification/object detection patterns the same way that a human would. Rather than identifying the core features of a panda such as: its eyes, mouth, nose, and the gradient changes in its black/white fur, the convolutional neural network seems to be learning image representations in a completely different manner. Deep learning researchers often attempt to model neural networks after human learning, and it is clear that further steps must be taken to robustify ConvNets against targeted noise perturbations.<br /> <br /> ==Drawbacks of CNNs==<br /> Hinton claims that the key fault with traditional CNNs lies within the pooling function. Although pooling builds translational invariance into the network, it fails to preserve spatial relationships between objects. When we pool, we effectively reduce a kxk kernel of convolved cells into a scalar input. This results in a desired local invariance without inhibiting the network's ability to detect features, but causes valuable spatial information to be lost.<br /> <br /> In the example below, the network is able to detect the similar features (eyes, mouth, nose, etc) within both images, but fails to recognize that one image is a human face, while the other is a Picasso-esque due to the CNN's inability to encode spatial relationships after multiple pooling layers.<br /> In deep learning, the activation level of a neuron is often interpreted as the likelihood of detecting a specific feature. CNNs are good at detecting features but less effective at exploring the spatial relationships among features (perspective, size, orientation). <br /> <br /> [[File:Equivariance Face.png ‎|center]]<br /> <br /> Here, the CNN could wrongly activate the neuron for the face detection. Without realize the mis-match in spatial orientation and size, the activation for the face detection will be too high.<br /> <br /> Conversely, we hope that a CNN can recognize that both of the following pictures contain a kitten. Unfortunately, when we feed the two images into a ResNet50 architecture, only the first image is correctly classified, while the second image is predicted to be a guinea pig.<br /> <br /> <br /> [[File:kitten.jpeg ‎|center]]<br /> <br /> <br /> [[File:kitten-rotated-180.jpg ‎|center]]<br /> <br /> For a more in depth discussion on the problems with ConvNets, please listen to Geoffrey Hinton's talk &quot;What is wrong with convolutional neural nets?&quot; given at MIT during the Brain &amp; Cognitive Sciences - Fall Colloquium Series (December 4, 2014).<br /> <br /> ==Intuition for Capsules==<br /> Human vision ignores irrelevant details by using a carefully determined sequence of fixation points to ensure that only a tiny fraction of the optic array is ever processed at the highest resolution. Hinton argues that our brains reason visual information by deconstructing it into a hierarchical representation which we then match to familiar patterns and relationships from memory. The key difference between this understanding and the functionality of CNNs is that recognition of an object should not depend on the angle from which it is viewed. <br /> <br /> To enforce rotational and translational equivariance, Capsule Networks store and preserve hierarchical pose relationships between objects. The core idea behind capsule theory is the explicit numerical representations of relative relationships between different objects within an image. Building these relationships into the Capsule Networks model, the network is able to recognize newly seen objects as a rotated view of a previously seen object. For example, the below image shows the Statue of Liberty under five different angles. If a person had only seen the Statue of Liberty from one angle, they would be able to ascertain that all five pictures below contain the same object (just from a different angle).<br /> <br /> [[File:Rotational Invariance.jpeg ‎|center]]<br /> <br /> Building on this idea of hierarchical representation of spatial relationships between key entities within an image, the authors introduce Capsule Networks. Unlike traditional CNNs, Capsule Networks are better equipped to classify correctly under rotational invariance. Furthermore, the authors managed to achieve state of the art results on MNIST using a fraction of the training samples that alternative state of the art networks require.<br /> <br /> =Background, Notation, and Definitions=<br /> <br /> ==What is a Capsule==<br /> &quot;Each capsule learns to recognize an implicitly defined visual entity over a limited domain of viewing conditions and deformations and it outputs both the probability that the entity is present within its limited domain and a set of “instantiation parameters” that may include the precise pose, lighting and deformation of the visual entity relative to an implicitly defined canonical version of that entity. When the capsule is working properly, the probability of the visual entity being present is locally invariant — it does not change as the entity moves over the manifold of possible appearances within the limited domain covered by the capsule. The instantiation parameters, however, are “equivariant” — as the viewing conditions change and the entity moves over the appearance manifold, the instantiation parameters change by a corresponding amount because they are representing the intrinsic coordinates of the entity on the appearance manifold.&quot;<br /> <br /> In essence, capsules store object properties in a vector form; probability of detection is encoded as the vector's length, while spatial properties are encoded as the individual vector components. Thus, when a feature is present but the image captures it under a different angle, the probability of detection remains unchanged.<br /> <br /> A brief overview/understanding of capsules can be found in other papers from the author. To quote from [https://openreview.net/pdf?id=HJWLfGWRb this paper]:<br /> <br /> &lt;blockquote&gt;<br /> A capsule network consists of several layers of capsules. The set of capsules in layer L is denoted<br /> as &lt;math&gt;\Omega_L&lt;/math&gt;. Each capsule has a 4x4 pose matrix, &lt;math&gt;M&lt;/math&gt;, and an activation probability, &lt;math&gt;a&lt;/math&gt;. These are like the<br /> activities in a standard neural net: they depend on the current input and are not stored. In between<br /> each capsule i in layer L and each capsule j in layer L + 1 is a 4x4 trainable transformation matrix,<br /> &lt;math&gt;W_{ij}&lt;/math&gt; . These &lt;math&gt;W_{ij}&lt;/math&gt;'s (and two learned biases per capsule) are the only stored parameters and they<br /> are learned discriminatively. The pose matrix of capsule i is transformed by &lt;math&gt;W_{ij}&lt;/math&gt; to cast a vote<br /> &lt;math&gt;V_{ij} = M_iW_{ij}&lt;/math&gt; for the pose matrix of capsule j. The poses and activations of all the capsules in layer<br /> L + 1 are calculated by using a non-linear routing procedure which gets as input &lt;math&gt;V_{ij}&lt;/math&gt; and &lt;math&gt;a_i&lt;/math&gt; for all<br /> &lt;math&gt;i \in \Omega_L, j \in \Omega_{L+1}&lt;/math&gt;<br /> &lt;/blockquote&gt;<br /> &lt;math&gt;&lt;/math&gt;<br /> <br /> ==Notation==<br /> <br /> We want the length of the output vector of a capsule to represent the probability that the entity represented by the capsule is present in the current input. The paper performs a non-linear squashing operation to ensure that vector length falls between 0 and 1, with shorter vectors (less likely to exist entities) being shrunk towards 0. <br /> <br /> \begin{align} \mathbf{v}_j &amp;= \frac{||\mathbf{s}_j||^2}{1+ ||\mathbf{s}_j||^2} \frac{\mathbf{s}_j}{||\mathbf{s}_j||} \end{align}<br /> <br /> where &lt;math&gt;\mathbf{v}_j&lt;/math&gt; is the vector output of capsule &lt;math&gt;j&lt;/math&gt; and &lt;math&gt;s_j&lt;/math&gt; is its total input.<br /> <br /> For all but the first layer of capsules, the total input to a capsule &lt;math&gt;s_j&lt;/math&gt; is a weighted sum over all “prediction vectors” &lt;math&gt;\hat{\mathbf{u}}_{j|i}&lt;/math&gt; from the capsules in the layer below and is produced by multiplying the output &lt;math&gt;\mathbf{u}i&lt;/math&gt; of a capsule in the layer below by a weight matrix &lt;math&gt;\mathbf{W}ij&lt;/math&gt;<br /> <br /> \begin{align}<br /> \mathbf{s}_j = \sum_i c_{ij}\hat{\mathbf{u}}_{j|i}, ~\hspace{0.5em} \hat{\mathbf{u}}_{j|i}= \mathbf{W}_{ij}\mathbf{u}_i<br /> \end{align}<br /> where the &lt;math&gt;c_{ij}&lt;/math&gt; are coupling coefficients that are determined by the iterative dynamic routing process.<br /> <br /> The coupling coefficients between capsule &lt;math&gt;i&lt;/math&gt; and all the capsules in the layer above sum to 1 and are determined by a “routing softmax” whose initial logits &lt;math&gt;b_{ij}&lt;/math&gt; are the log prior probabilities that capsule &lt;math&gt;i&lt;/math&gt; should be coupled to capsule &lt;math&gt;j&lt;/math&gt;.<br /> <br /> \begin{align}<br /> c_{ij} = \frac{\exp(b_{ij})}{\sum_k \exp(b_{ik})}<br /> \end{align}<br /> <br /> =Network Training and Dynamic Routing=<br /> <br /> ==Understanding Capsules==<br /> The notation can get somewhat confusing, so I will provide intuition behind the computational steps within a capsule. The following image is taken from naturomic's talk on Capsule Networks.<br /> <br /> [[File:CapsuleNets.jpeg|center|800px]]<br /> <br /> The above image illustrates the key mathematical operations happening within a capsule (and compares them to the structure of a neuron). Although the operations are rather straightforward, it's crucial to note that the capsule computes an affine transformation onto each input vector. The length of the input vectors &lt;math&gt;\mathbf{u}_{i}&lt;/math&gt; represent the probability of entity &lt;math&gt;i&lt;/math&gt; existing in a lower level. This vector is then reoriented with an affine transform using &lt;math&gt;\mathbf{W}_{ij}&lt;/math&gt; matrices that encode spatial relationships between entity &lt;math&gt;\mathbf{u}_{i}&lt;/math&gt; and other lower level features.<br /> <br /> We illustrate the intuition behind vector-to-vector matrix multiplication within capsules using the following example: if vectors &lt;math&gt;\mathbf{u}_{1}&lt;/math&gt;, &lt;math&gt;\mathbf{u}_{2}&lt;/math&gt;, and &lt;math&gt;\mathbf{u}_{3}&lt;/math&gt; represent detection of eyes, nose, and mouth respectively, then after multiplication with trained weight matrices &lt;math&gt;\mathbf{W}_{ij}&lt;/math&gt; (where j denotes existence of a face), we should get a general idea of the general location of the higher level feature (face), similar to the image below.<br /> <br /> [[File:Predictions.jpeg ‎|center]]<br /> <br /> ==Dynamic Routing==<br /> A capsule &lt;math&gt;i&lt;/math&gt; in a lower-level layer needs to decide how to send its output vector to higher-level capsules &lt;math&gt;j&lt;/math&gt;. This decision is made with probability proportional to &lt;math&gt;c_{ij}&lt;/math&gt;. If there are &lt;math&gt;K&lt;/math&gt; capsules in the level that capsule &lt;math&gt;i&lt;/math&gt; routes to, then we know the following properties about &lt;math&gt;c_{ij}&lt;/math&gt;: &lt;math&gt;\sum_{j=1}^M c_{ij} = 1, c_{ij} \geq 0&lt;/math&gt;<br /> <br /> In essence, the &lt;math&gt;\{c_{ij}\}_{j=1}^M&lt;/math&gt; denotes a discrete probability distribution with respect to capsule &lt;math&gt;i&lt;/math&gt;'s output location. Lower level capsules decide which higher level capsules to send vectors into by adjusting the corresponding routing weights &lt;math&gt;\{c_{ij}\}_{j=1}^M&lt;/math&gt;. After a few iterations in training, numerous vectors will have already been sent to all higher level capsules. Based on the similarity between the current vector being routed and all vectors already sent into the higher level capsules, we decide which capsule to send the current vector into.<br /> [[File:Dynamic Routing.png|center|900px]]<br /> <br /> From the image above, we notice that a cluster of points similar to the current vector has already been routed into capsule K, while most points in capsule J are highly dissimilar. It thus makes more sense to route the current observations into capsule K; we adjust the corresponding weights upward during training.<br /> <br /> These weights are determined through the dynamic routing procedure:<br /> [[File:Routing Algo.png‎|900px]]<br /> <br /> <br /> Although dynamic routing is not the only manner in which we can encode relationships between capsules, the premise of the paper is to demonstrate the capabilities of capsules under a simple implementation. Since the paper was released in 2017, numerous alternative routing implementations have been released including an EM matrix routing algorithm by the same authors (ICLR 2018).<br /> <br /> =Architecture=<br /> The capsule network architecture given by the authors has 11.36 million trainable parameters. The paper itself is not very detailed on exact implementation of each architectural layer, and hence it leaves some degree of ambiguity on coding various aspects of the original network. The capsule network has 6 overall layers, with the first three layers denoting components of the encoder, and the last 3 denoting components of the decoder.<br /> <br /> ==Loss Function==<br /> [[File:Loss Function.png‎|900px]]<br /> <br /> The cost function looks very complicated, but can be broken down into intuitive components. Before diving into the equation, remember that the length of the vector denotes the probability of object existence. The left side of the equation denotes loss when the network classifies an observation correctly; the term becomes zero when the classification is incorrect. To compute loss when the network correctly classifies the label, we subtract the vector norm from a fixed quantity &lt;math&gt;m^+ := 0.9&lt;/math&gt;. On the other hand, when the network classifies a label incorrectly, we penalize the loss based on the network's confidence in the incorrect label; we compute the loss by subtracting &lt;math&gt;m^- := 0.1&lt;/math&gt; from the vector norm.<br /> <br /> A graphical representation of loss function values under varying vector norms is given below.<br /> [[File:Loss function chart.png|900px]]<br /> <br /> ==Encoder Layers==<br /> All experiments within this paper were conducted on the MNIST dataset, and thus the architecture is built to classify the corresponding dataset. For more complex datasets, the experiments were less promising. <br /> <br /> [[File:Architecture.png|center|900px]]<br /> <br /> The encoder layer takes in a 28x28 MNIST image and learns a 16 dimensional representation of instantiation parameters.<br /> <br /> '''Layer 1: Convolution''': <br /> This layer is a standard convolution layer. Using kernels with size 9x9x1, a stride of 1, and a ReLU activation function, we detect the 2D features within the network.<br /> <br /> '''Layer 2: PrimaryCaps''': <br /> We represent the low level features detected during convolution as 32 primary capsules. Each capsule applies eight convolutional kernels with stride 2 to the output of the convolution layer and feeds the corresponding transformed tensors into the DigiCaps layer.<br /> <br /> '''Layer 3: DigiCaps''': <br /> This layer contains 10 digit capsules, one for each digit. As explained in the dynamic routing procedure, each input vector from the PrimaryCaps layer has its own corresponding weight matrix &lt;math&gt;W_{ij}&lt;/math&gt;. Using the routing coefficients &lt;math&gt;c_{ij}&lt;/math&gt; and temporary coefficients &lt;math&gt;b_{ij}&lt;/math&gt;, we train the DigiCaps layer to output a ten 16 dimensional vectors. The length of the &lt;math&gt;i^{th}&lt;/math&gt; vector in this layer corresponds to the probability of detection of digit &lt;math&gt;i&lt;/math&gt;.<br /> <br /> ==Decoder Layers==<br /> The decoder layer aims to train the capsules to extract meaningful features for image detection/classification. During training, it takes the 16 layer instantiation vector of the correct (not predicted) DigiCaps layer, and attempts to recreate the 28x28 MNIST image as best as possible. Setting the loss function as reconstruction error (Euclidean distance between the reconstructed image and original image), we tune the capsules to encode features that are meaningful within the actual image.<br /> <br /> [[File:Decoder.png|center|900px]]<br /> <br /> The layer consists of three fully connected layers, and transforms a 16x1 vector from the encoder layer into a 28x28 image.<br /> <br /> In addition to the digicaps loss function, we add reconstruction error as a form of regularization. We minimize the Euclidean distance between the outputs of the logistic units and the pixel intensities of the original and reconstructed images. We scale down this reconstruction loss by 0.0005 so that it does not dominate the margin loss during training. As illustrated below, reconstructions from the 16D output of the CapsNet are robust while keeping only important details.<br /> <br /> [[File:Reconstruction.png|center|900px]]<br /> <br /> =MNIST Experimental Results=<br /> <br /> ==Accuracy==<br /> The paper tests on the MNIST dataset with 60K training examples, and 10K testing. Wan et al.  achieves 0.21% test error with ensembling and augmenting the data with rotation and scaling. They achieve 0.39% without them. As shown in Table 1, the authors manage to achieve 0.25% test error with only a 3 layer network; the previous state of the art only beat this number with very deep networks. This example shows the importance of routing and reconstruction regularizer, which boosts the performance. On the other hand, while the accuracies are very high, the number of parameters is much smaller compared to the baseline model.<br /> <br /> [[File:Accuracies.png|center|900px]]<br /> <br /> ==What Capsules Represent for MNIST==<br /> The following figure shows the digit representation under capsules. Each row shows the reconstruction when one of the 16 dimensions in the DigitCaps representation is tweaked by intervals of 0.05 in the range [−0.25, 0.25]. By tweaking the values, we notice how the reconstruction changes, and thus get a sense for what each dimension is representing. The authors found that some dimensions represent global properties of the digits, while other represent localized properties. <br /> [[File:CapsuleReps.png|center|900px]]<br /> <br /> One example the authors provide is: different dimensions are used for the length of the ascender of a 6 and the size of the loop. The variations include stroke thickness, skew and width, as well as digit-specific variations. The authors are able to show dimension representations using a decoder network by feeding a perturbed vector.<br /> <br /> ==Robustness of CapsNet==<br /> The authors conclude that DigitCaps capsules learn more robust representations for each digit class than traditional CNNs. The trained CapsNet becomes moderately robust to small affine transformations in the test data.<br /> <br /> To compare the robustness of CapsNet to affine transformations against traditional CNNs, both models (CapsNet and a traditional CNN with MaxPooling and DropOut) were trained on a padded and translated MNIST training set, in which each example is an MNIST digit placed randomly on a black background of 40 × 40 pixels. The networks were then tested on the [http://www.cs.toronto.edu/~tijmen/affNIST/ affNIST] dataset (MNIST digits with random affine transformation). An under-trained CapsNet which achieved 99.23% accuracy on the MNIST test set achieved a corresponding 79% accuracy on the affnist test set. A traditional CNN achieved similar accuracy (99.22%) on the mnist test set, but only 66% on the affnist test set.<br /> <br /> =MultiMNIST &amp; Other Experiments=<br /> <br /> ==MultiMNIST==<br /> To evaluate the performance of the model on highly overlapping digits, the authors generate a 'MultiMNIST' dataset. In MultiMNIST, images are two overlaid MNIST digits of the same set(train or test) but different classes. The results indicate a classification error rate of 5%. Additionally, CapsNet can be used to segment the image into the two digits that compose it. Moreover, the model is able to deal with the overlaps and reconstruct digits correctly since each digit capsule can learn the style from the votes of PrimaryCapsules layer (Figure 5).<br /> <br /> There are some additional steps to generating the MultiMNIST dataset.<br /> <br /> 1. Both images are shifted by up to 4 pixels in each direction resulting in a 36 × 36 image. Bounding boxes of digits in MNIST overlap by approximately 80%, so this is used to make both digits identifiable (since there is no RGB difference learnable by the network to separate the digits)<br /> <br /> 2. The label becomes a vector of two numbers, representing the original digit and the randomly generated (and overlaid) digit.<br /> <br /> <br /> <br /> [[File:CapsuleNets MultiMNIST.PNG|600px|thumb|center|Figure 5: Sample reconstructions of a CapsNet with 3 routing iterations on MultiMNIST test dataset.<br /> The two reconstructed digits are overlayed in green and red as the lower image. The upper image<br /> shows the input image. L:(l1; l2) represents the label for the two digits in the image and R:(r1; r2)<br /> represents the two digits used for reconstruction. The two right most columns show two examples<br /> with wrong classification reconstructed from the label and from the prediction (P). In the (2; 8)<br /> example the model confuses 8 with a 7 and in (4; 9) it confuses 9 with 0. The other columns have<br /> correct classifications and show that the model accounts for all the pixels while being able to assign<br /> one pixel to two digits in extremely difficult scenarios (column 1 − 4). Note that in dataset generation<br /> the pixel values are clipped at 1. The two columns with the (*) mark show reconstructions from a<br /> digit that is neither the label nor the prediction. These columns suggest that the model is not just<br /> finding the best fit for all the digits in the image including the ones that do not exist. Therefore in case<br /> of (5; 0) it cannot reconstruct a 7 because it knows that there is a 5 and 0 that fit best and account for<br /> all the pixels. Also, in the case of (8; 1) the loop of 8 has not triggered 0 because it is already accounted<br /> for by 8. Therefore it will not assign one pixel to two digits if one of them does not have any other<br /> support.]]<br /> <br /> ==Other datasets==<br /> The authors also tested the proposed capsule model on CIFAR10 dataset and achieved an error rate of 10.6%. The model tested was an ensemble of 7 models. Each of the models in the ensemble had the same architecture as the model used for MNIST (apart from 3 additional channels and 64 different types of primary capsules being used). These 7 models were trained on 24x24 patches of the training images for 3 iterations. During experimentation, the authors also found out that adding an additional none-of-the-above category helped improved the overall performance. The error rate achieved is comparable to the error rate achieved by a standard CNN model. According to the authors, one of the reasons for low performance is the fact that background in CIFAR-10 images are too varied for it to be adequately modeled by reasonably sized capsule net.<br /> <br /> The proposed model was also evaluated using a small subset of SVHN dataset. The network trained was much smaller and trained using only 73257 training images. The network still managed to achieve an error rate of 4.3% on the test set.<br /> <br /> =Critique=<br /> Although the network performs incredibly favorable in the author's experiments, it has a long way to go on more complex datasets. On CIFAR 10, the network achieved subpar results, and the experimental results seem to be worse when the problem becomes more complex. This is anticipated, since these networks are still in their early stage; later innovations might come in the upcoming decades/years. It could also be wise to apply the model to other datasets with larger sizes to make the functionality more acceptable. MNIST dataset has simple patterns and even if the model wanted to be presented with only one dataset, it was better not to be MNIST dataset especially in this case that the focus is on human-eye detection and numbers are not that regular in real-life experiences.<br /> <br /> Hinton talks about CapsuleNets revolutionizing areas such as self-driving, but such groundbreaking innovations are far away from CIFAR10, and even further from MNIST. Only time can tell if CapsNets will live up to their hype.<br /> <br /> Capsules inherently segment images and learn a lower dimensional embedding in a new manner, which makes them likely to perform well on segmentation and computer vision tasks once further research is done. <br /> <br /> Additionally, these networks are more interpretable than CNNs, and have strong theoretical reasoning for why they could work. Naturally, it would be hard for a new architecture to beat the heavily researched/modified CNNs.<br /> <br /> * ([https://openreview.net/forum?id=HJWLfGWRb]) it's not fully clear how effective it can be performed / how scalable it is. Evaluation is performed on a small dataset for shape recognition. The approach will need to be tested on larger, more challenging datasets.<br /> <br /> =Future Work=<br /> The same authors [N. F. Geoffrey E Hinton, Sara Sabour] presented another paper &quot;MATRIX CAPSULES WITH EM ROUTING&quot; in ICLR 2018, which achieved better results than the work presented in this paper. They presented a new multi-layered capsule network architecture, implemented an EM routing procedure, and introduced &quot;Coordinate Addition&quot;. This new type reduced number of errors by 45%, and performed better than standard CNN on white box adversarial attacks. Capsule architectures are gaining interest because of their ability to achieve equivariance of parts, and employ a new form of pooling called &quot;routing&quot; (as opposed to max pooling) which groups parts that make similar predictions of the whole to which they belong, rather than relying on spatial co-locality.<br /> Moreover, the authors hint towards trying to change the curvature and sensitivities to various factors by introducing new form of loss function. It may improve the performance of the model for more complicated data set which is one of the model's drawback.<br /> <br /> Moreover, as mentioned in critiques, a good future work for this group would be making the model more robust to the dataset and achieve acceptable performance on datasets with more regularly seen images in real life experiences.<br /> <br /> =References=<br /> #N. F. Geoffrey E Hinton, Sara Sabour. Matrix capsules with em routing. In International Conference on Learning Representations, 2018.<br /> #S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between capsules,” arXiv preprint arXiv:1710.09829v2, 2017<br /> # Hinton, G. E., Krizhevsky, A. and Wang, S. D. (2011), Transforming Auto-encoders <br /> #Geoffrey Hinton's talk: What is wrong with convolutional neural nets? - Talk given at MIT. Brain &amp; Cognitive Sciences - Fall Colloquium Series. [https://www.youtube.com/watch?v=rTawFwUvnLE ]<br /> #Understanding Hinton’s Capsule Networks - Max Pechyonkin's series [https://medium.com/ai%C2%B3-theory-practice-business/understanding-hintons-capsule-networks-part-i-intuition-b4b559d1159b]<br /> #Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg SCorrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machinelearning on heterogeneous distributed systems.arXiv preprint arXiv:1603.04467, 2016.<br /> #Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visualattention.arXiv preprint arXiv:1412.7755, 2014.<br /> #Jia-Ren Chang and Yong-Sheng Chen. Batch-normalized maxout network in network.arXiv preprintarXiv:1511.02583, 2015.<br /> #Dan C Cire ̧san, Ueli Meier, Jonathan Masci, Luca M Gambardella, and Jürgen Schmidhuber. High-performance neural networks for visual object classification.arXiv preprint arXiv:1102.0183,2011.<br /> #Ian J Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi-digit numberrecognition from street view imagery using deep convolutional neural networks.arXiv preprintarXiv:1312.6082, 2013.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=CapsuleNets&diff=41894 CapsuleNets 2018-11-29T17:49:12Z <p>C9sharma: /* Drawbacks of CNNs */</p> <hr /> <div>The paper &quot;Dynamic Routing Between Capsules&quot; was written by three researchers at Google Brain: Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. This paper was published and presented at the 31st Conference on Neural Information Processing Systems (NIPS 2017) in Long Beach, California. The same three researchers recently published a highly related paper &quot;[https://openreview.net/pdf?id=HJWLfGWRb Matrix Capsules with EM Routing]&quot; for ICLR 2018.<br /> <br /> =Motivation=<br /> <br /> Ever since AlexNet eclipsed the performance of competing architectures in the 2012 ImageNet challenge, convolutional neural networks have maintained their dominance in computer vision applications. Despite the recent successes and innovations brought about by convolutional neural networks, some assumptions made in these networks are perhaps unwarranted and deficient. Using a novel neural network architecture, the authors create CapsuleNets, a network that they claim is able to learn image representations in a more robust, human-like manner. With only a 3 layer capsule network, they achieved near state-of-the-art results on MNIST.<br /> ==Adversarial Examples==<br /> <br /> First discussed by Christian Szegedy et. al. in late 2013, adversarial examples have been heavily discussed by the deep learning community as a potential security threat to AI learning. Adversarial examples are defined as inputs that an attacker creates intentionally fool a machine learning model. An example of an adversarial example is shown below: <br /> <br /> [[File:adversarial_img_1.png ‎|center]]<br /> To the human eye, the image appears to be a panda both before and after noise is injected into the image, whereas the trained ConvNet model discerns the noisy image as a Gibbon with almost 100% certainty. The fact that the network is unable to classify the above image as a panda after the epsilon perturbation leads to many potential security risks in AI dependent systems such as self-driving vehicles. Although various methods have been suggested to combat adversarial examples, robust defences are hard to construct due to the inherent difficulties in constructing theoretical models for the adversarial example crafting process. However, beyond the fact that these examples may serve as a security threat, it emphasizes that these convolutional neural networks do not learn image classification/object detection patterns the same way that a human would. Rather than identifying the core features of a panda such as: its eyes, mouth, nose, and the gradient changes in its black/white fur, the convolutional neural network seems to be learning image representations in a completely different manner. Deep learning researchers often attempt to model neural networks after human learning, and it is clear that further steps must be taken to robustify ConvNets against targeted noise perturbations.<br /> <br /> ==Drawbacks of CNNs==<br /> Hinton claims that the key fault with traditional CNNs lies within the pooling function. Although pooling builds translational invariance into the network, it fails to preserve spatial relationships between objects. When we pool, we effectively reduce a kxk kernel of convolved cells into a scalar input. This results in a desired local invariance without inhibiting the network's ability to detect features, but causes valuable spatial information to be lost.<br /> <br /> In the example below, the network is able to detect the similar features (eyes, mouth, nose, etc) within both images, but fails to recognize that one image is a human face, while the other is a Picasso-esque due to the CNN's inability to encode spatial relationships after multiple pooling layers.<br /> In deep learning, the activation level of a neuron is often interpreted as the likelihood of detecting a specific feature. CNNs are good at detecting features but less effective at exploring the spatial relationships among features (perspective, size, orientation). <br /> <br /> [[File:Equivariance Face.png ‎|center]]<br /> <br /> Here, the CNN could wrongly activate the neuron for the face detection. Without realize the mis-match in spatial orientation and size, the activation for the face detection will be too high.<br /> <br /> Conversely, we hope that a CNN can recognize that both of the following pictures contain a kitten. Unfortunately, when we feed the two images into a ResNet50 architecture, only the first image is correctly classified, while the second image is predicted to be a guinea pig.<br /> <br /> <br /> [[File:kitten.jpeg ‎|center]]<br /> <br /> <br /> [[File:kitten-rotated-180.jpg ‎|center]]<br /> <br /> For a more in depth discussion on the problems with ConvNets, please listen to Geoffrey Hinton's talk &quot;What is wrong with convolutional neural nets?&quot; given at MIT during the Brain &amp; Cognitive Sciences - Fall Colloquium Series (December 4, 2014).<br /> <br /> ==Intuition for Capsules==<br /> Human vision ignores irrelevant details by using a carefully determined sequence of fixation points to ensure that only a tiny fraction of the optic array is ever processed at the highest resolution. Hinton argues that our brains reason visual information by deconstructing it into a hierarchical representation which we then match to familiar patterns and relationships from memory. The key difference between this understanding and the functionality of CNNs is that recognition of an object should not depend on the angle from which it is viewed. <br /> <br /> To enforce rotational and translational equivariance, Capsule Networks store and preserve hierarchical pose relationships between objects. The core idea behind capsule theory is the explicit numerical representations of relative relationships between different objects within an image. Building these relationships into the Capsule Networks model, the network is able to recognize newly seen objects as a rotated view of a previously seen object. For example, the below image shows the Statue of Liberty under five different angles. If a person had only seen the Statue of Liberty from one angle, they would be able to ascertain that all five pictures below contain the same object (just from a different angle).<br /> <br /> [[File:Rotational Invariance.jpeg ‎|center]]<br /> <br /> Building on this idea of hierarchical representation of spatial relationships between key entities within an image, the authors introduce Capsule Networks. Unlike traditional CNNs, Capsule Networks are better equipped to classify correctly under rotational invariance. Furthermore, the authors managed to achieve state of the art results on MNIST using a fraction of the training samples that alternative state of the art networks require.<br /> <br /> <br /> =Background, Notation, and Definitions=<br /> <br /> ==What is a Capsule==<br /> &quot;Each capsule learns to recognize an implicitly defined visual entity over a limited domain of viewing conditions and deformations and it outputs both the probability that the entity is present within its limited domain and a set of “instantiation parameters” that may include the precise pose, lighting and deformation of the visual entity relative to an implicitly defined canonical version of that entity. When the capsule is working properly, the probability of the visual entity being present is locally invariant — it does not change as the entity moves over the manifold of possible appearances within the limited domain covered by the capsule. The instantiation parameters, however, are “equivariant” — as the viewing conditions change and the entity moves over the appearance manifold, the instantiation parameters change by a corresponding amount because they are representing the intrinsic coordinates of the entity on the appearance manifold.&quot;<br /> <br /> In essence, capsules store object properties in a vector form; probability of detection is encoded as the vector's length, while spatial properties are encoded as the individual vector components. Thus, when a feature is present but the image captures it under a different angle, the probability of detection remains unchanged.<br /> <br /> A brief overview/understanding of capsules can be found in other papers from the author. To quote from [https://openreview.net/pdf?id=HJWLfGWRb this paper]:<br /> <br /> &lt;blockquote&gt;<br /> A capsule network consists of several layers of capsules. The set of capsules in layer L is denoted<br /> as &lt;math&gt;\Omega_L&lt;/math&gt;. Each capsule has a 4x4 pose matrix, &lt;math&gt;M&lt;/math&gt;, and an activation probability, &lt;math&gt;a&lt;/math&gt;. These are like the<br /> activities in a standard neural net: they depend on the current input and are not stored. In between<br /> each capsule i in layer L and each capsule j in layer L + 1 is a 4x4 trainable transformation matrix,<br /> &lt;math&gt;W_{ij}&lt;/math&gt; . These &lt;math&gt;W_{ij}&lt;/math&gt;'s (and two learned biases per capsule) are the only stored parameters and they<br /> are learned discriminatively. The pose matrix of capsule i is transformed by &lt;math&gt;W_{ij}&lt;/math&gt; to cast a vote<br /> &lt;math&gt;V_{ij} = M_iW_{ij}&lt;/math&gt; for the pose matrix of capsule j. The poses and activations of all the capsules in layer<br /> L + 1 are calculated by using a non-linear routing procedure which gets as input &lt;math&gt;V_{ij}&lt;/math&gt; and &lt;math&gt;a_i&lt;/math&gt; for all<br /> &lt;math&gt;i \in \Omega_L, j \in \Omega_{L+1}&lt;/math&gt;<br /> &lt;/blockquote&gt;<br /> &lt;math&gt;&lt;/math&gt;<br /> <br /> ==Notation==<br /> <br /> We want the length of the output vector of a capsule to represent the probability that the entity represented by the capsule is present in the current input. The paper performs a non-linear squashing operation to ensure that vector length falls between 0 and 1, with shorter vectors (less likely to exist entities) being shrunk towards 0. <br /> <br /> \begin{align} \mathbf{v}_j &amp;= \frac{||\mathbf{s}_j||^2}{1+ ||\mathbf{s}_j||^2} \frac{\mathbf{s}_j}{||\mathbf{s}_j||} \end{align}<br /> <br /> where &lt;math&gt;\mathbf{v}_j&lt;/math&gt; is the vector output of capsule &lt;math&gt;j&lt;/math&gt; and &lt;math&gt;s_j&lt;/math&gt; is its total input.<br /> <br /> For all but the first layer of capsules, the total input to a capsule &lt;math&gt;s_j&lt;/math&gt; is a weighted sum over all “prediction vectors” &lt;math&gt;\hat{\mathbf{u}}_{j|i}&lt;/math&gt; from the capsules in the layer below and is produced by multiplying the output &lt;math&gt;\mathbf{u}i&lt;/math&gt; of a capsule in the layer below by a weight matrix &lt;math&gt;\mathbf{W}ij&lt;/math&gt;<br /> <br /> \begin{align}<br /> \mathbf{s}_j = \sum_i c_{ij}\hat{\mathbf{u}}_{j|i}, ~\hspace{0.5em} \hat{\mathbf{u}}_{j|i}= \mathbf{W}_{ij}\mathbf{u}_i<br /> \end{align}<br /> where the &lt;math&gt;c_{ij}&lt;/math&gt; are coupling coefficients that are determined by the iterative dynamic routing process.<br /> <br /> The coupling coefficients between capsule &lt;math&gt;i&lt;/math&gt; and all the capsules in the layer above sum to 1 and are determined by a “routing softmax” whose initial logits &lt;math&gt;b_{ij}&lt;/math&gt; are the log prior probabilities that capsule &lt;math&gt;i&lt;/math&gt; should be coupled to capsule &lt;math&gt;j&lt;/math&gt;.<br /> <br /> \begin{align}<br /> c_{ij} = \frac{\exp(b_{ij})}{\sum_k \exp(b_{ik})}<br /> \end{align}<br /> <br /> =Network Training and Dynamic Routing=<br /> <br /> ==Understanding Capsules==<br /> The notation can get somewhat confusing, so I will provide intuition behind the computational steps within a capsule. The following image is taken from naturomic's talk on Capsule Networks.<br /> <br /> [[File:CapsuleNets.jpeg|center|800px]]<br /> <br /> The above image illustrates the key mathematical operations happening within a capsule (and compares them to the structure of a neuron). Although the operations are rather straightforward, it's crucial to note that the capsule computes an affine transformation onto each input vector. The length of the input vectors &lt;math&gt;\mathbf{u}_{i}&lt;/math&gt; represent the probability of entity &lt;math&gt;i&lt;/math&gt; existing in a lower level. This vector is then reoriented with an affine transform using &lt;math&gt;\mathbf{W}_{ij}&lt;/math&gt; matrices that encode spatial relationships between entity &lt;math&gt;\mathbf{u}_{i}&lt;/math&gt; and other lower level features.<br /> <br /> We illustrate the intuition behind vector-to-vector matrix multiplication within capsules using the following example: if vectors &lt;math&gt;\mathbf{u}_{1}&lt;/math&gt;, &lt;math&gt;\mathbf{u}_{2}&lt;/math&gt;, and &lt;math&gt;\mathbf{u}_{3}&lt;/math&gt; represent detection of eyes, nose, and mouth respectively, then after multiplication with trained weight matrices &lt;math&gt;\mathbf{W}_{ij}&lt;/math&gt; (where j denotes existence of a face), we should get a general idea of the general location of the higher level feature (face), similar to the image below.<br /> <br /> [[File:Predictions.jpeg ‎|center]]<br /> <br /> ==Dynamic Routing==<br /> A capsule &lt;math&gt;i&lt;/math&gt; in a lower-level layer needs to decide how to send its output vector to higher-level capsules &lt;math&gt;j&lt;/math&gt;. This decision is made with probability proportional to &lt;math&gt;c_{ij}&lt;/math&gt;. If there are &lt;math&gt;K&lt;/math&gt; capsules in the level that capsule &lt;math&gt;i&lt;/math&gt; routes to, then we know the following properties about &lt;math&gt;c_{ij}&lt;/math&gt;: &lt;math&gt;\sum_{j=1}^M c_{ij} = 1, c_{ij} \geq 0&lt;/math&gt;<br /> <br /> In essence, the &lt;math&gt;\{c_{ij}\}_{j=1}^M&lt;/math&gt; denotes a discrete probability distribution with respect to capsule &lt;math&gt;i&lt;/math&gt;'s output location. Lower level capsules decide which higher level capsules to send vectors into by adjusting the corresponding routing weights &lt;math&gt;\{c_{ij}\}_{j=1}^M&lt;/math&gt;. After a few iterations in training, numerous vectors will have already been sent to all higher level capsules. Based on the similarity between the current vector being routed and all vectors already sent into the higher level capsules, we decide which capsule to send the current vector into.<br /> [[File:Dynamic Routing.png|center|900px]]<br /> <br /> From the image above, we notice that a cluster of points similar to the current vector has already been routed into capsule K, while most points in capsule J are highly dissimilar. It thus makes more sense to route the current observations into capsule K; we adjust the corresponding weights upward during training.<br /> <br /> These weights are determined through the dynamic routing procedure:<br /> [[File:Routing Algo.png‎|900px]]<br /> <br /> <br /> Although dynamic routing is not the only manner in which we can encode relationships between capsules, the premise of the paper is to demonstrate the capabilities of capsules under a simple implementation. Since the paper was released in 2017, numerous alternative routing implementations have been released including an EM matrix routing algorithm by the same authors (ICLR 2018).<br /> <br /> =Architecture=<br /> The capsule network architecture given by the authors has 11.36 million trainable parameters. The paper itself is not very detailed on exact implementation of each architectural layer, and hence it leaves some degree of ambiguity on coding various aspects of the original network. The capsule network has 6 overall layers, with the first three layers denoting components of the encoder, and the last 3 denoting components of the decoder.<br /> <br /> ==Loss Function==<br /> [[File:Loss Function.png‎|900px]]<br /> <br /> The cost function looks very complicated, but can be broken down into intuitive components. Before diving into the equation, remember that the length of the vector denotes the probability of object existence. The left side of the equation denotes loss when the network classifies an observation correctly; the term becomes zero when the classification is incorrect. To compute loss when the network correctly classifies the label, we subtract the vector norm from a fixed quantity &lt;math&gt;m^+ := 0.9&lt;/math&gt;. On the other hand, when the network classifies a label incorrectly, we penalize the loss based on the network's confidence in the incorrect label; we compute the loss by subtracting &lt;math&gt;m^- := 0.1&lt;/math&gt; from the vector norm.<br /> <br /> A graphical representation of loss function values under varying vector norms is given below.<br /> [[File:Loss function chart.png|900px]]<br /> <br /> ==Encoder Layers==<br /> All experiments within this paper were conducted on the MNIST dataset, and thus the architecture is built to classify the corresponding dataset. For more complex datasets, the experiments were less promising. <br /> <br /> [[File:Architecture.png|center|900px]]<br /> <br /> The encoder layer takes in a 28x28 MNIST image and learns a 16 dimensional representation of instantiation parameters.<br /> <br /> '''Layer 1: Convolution''': <br /> This layer is a standard convolution layer. Using kernels with size 9x9x1, a stride of 1, and a ReLU activation function, we detect the 2D features within the network.<br /> <br /> '''Layer 2: PrimaryCaps''': <br /> We represent the low level features detected during convolution as 32 primary capsules. Each capsule applies eight convolutional kernels with stride 2 to the output of the convolution layer and feeds the corresponding transformed tensors into the DigiCaps layer.<br /> <br /> '''Layer 3: DigiCaps''': <br /> This layer contains 10 digit capsules, one for each digit. As explained in the dynamic routing procedure, each input vector from the PrimaryCaps layer has its own corresponding weight matrix &lt;math&gt;W_{ij}&lt;/math&gt;. Using the routing coefficients &lt;math&gt;c_{ij}&lt;/math&gt; and temporary coefficients &lt;math&gt;b_{ij}&lt;/math&gt;, we train the DigiCaps layer to output a ten 16 dimensional vectors. The length of the &lt;math&gt;i^{th}&lt;/math&gt; vector in this layer corresponds to the probability of detection of digit &lt;math&gt;i&lt;/math&gt;.<br /> <br /> ==Decoder Layers==<br /> The decoder layer aims to train the capsules to extract meaningful features for image detection/classification. During training, it takes the 16 layer instantiation vector of the correct (not predicted) DigiCaps layer, and attempts to recreate the 28x28 MNIST image as best as possible. Setting the loss function as reconstruction error (Euclidean distance between the reconstructed image and original image), we tune the capsules to encode features that are meaningful within the actual image.<br /> <br /> [[File:Decoder.png|center|900px]]<br /> <br /> The layer consists of three fully connected layers, and transforms a 16x1 vector from the encoder layer into a 28x28 image.<br /> <br /> In addition to the digicaps loss function, we add reconstruction error as a form of regularization. We minimize the Euclidean distance between the outputs of the logistic units and the pixel intensities of the original and reconstructed images. We scale down this reconstruction loss by 0.0005 so that it does not dominate the margin loss during training. As illustrated below, reconstructions from the 16D output of the CapsNet are robust while keeping only important details.<br /> <br /> [[File:Reconstruction.png|center|900px]]<br /> <br /> =MNIST Experimental Results=<br /> <br /> ==Accuracy==<br /> The paper tests on the MNIST dataset with 60K training examples, and 10K testing. Wan et al.  achieves 0.21% test error with ensembling and augmenting the data with rotation and scaling. They achieve 0.39% without them. As shown in Table 1, the authors manage to achieve 0.25% test error with only a 3 layer network; the previous state of the art only beat this number with very deep networks. This example shows the importance of routing and reconstruction regularizer, which boosts the performance. On the other hand, while the accuracies are very high, the number of parameters is much smaller compared to the baseline model.<br /> <br /> [[File:Accuracies.png|center|900px]]<br /> <br /> ==What Capsules Represent for MNIST==<br /> The following figure shows the digit representation under capsules. Each row shows the reconstruction when one of the 16 dimensions in the DigitCaps representation is tweaked by intervals of 0.05 in the range [−0.25, 0.25]. By tweaking the values, we notice how the reconstruction changes, and thus get a sense for what each dimension is representing. The authors found that some dimensions represent global properties of the digits, while other represent localized properties. <br /> [[File:CapsuleReps.png|center|900px]]<br /> <br /> One example the authors provide is: different dimensions are used for the length of the ascender of a 6 and the size of the loop. The variations include stroke thickness, skew and width, as well as digit-specific variations. The authors are able to show dimension representations using a decoder network by feeding a perturbed vector.<br /> <br /> ==Robustness of CapsNet==<br /> The authors conclude that DigitCaps capsules learn more robust representations for each digit class than traditional CNNs. The trained CapsNet becomes moderately robust to small affine transformations in the test data.<br /> <br /> To compare the robustness of CapsNet to affine transformations against traditional CNNs, both models (CapsNet and a traditional CNN with MaxPooling and DropOut) were trained on a padded and translated MNIST training set, in which each example is an MNIST digit placed randomly on a black background of 40 × 40 pixels. The networks were then tested on the [http://www.cs.toronto.edu/~tijmen/affNIST/ affNIST] dataset (MNIST digits with random affine transformation). An under-trained CapsNet which achieved 99.23% accuracy on the MNIST test set achieved a corresponding 79% accuracy on the affnist test set. A traditional CNN achieved similar accuracy (99.22%) on the mnist test set, but only 66% on the affnist test set.<br /> <br /> =MultiMNIST &amp; Other Experiments=<br /> <br /> ==MultiMNIST==<br /> To evaluate the performance of the model on highly overlapping digits, the authors generate a 'MultiMNIST' dataset. In MultiMNIST, images are two overlaid MNIST digits of the same set(train or test) but different classes. The results indicate a classification error rate of 5%. Additionally, CapsNet can be used to segment the image into the two digits that compose it. Moreover, the model is able to deal with the overlaps and reconstruct digits correctly since each digit capsule can learn the style from the votes of PrimaryCapsules layer (Figure 5).<br /> <br /> There are some additional steps to generating the MultiMNIST dataset.<br /> <br /> 1. Both images are shifted by up to 4 pixels in each direction resulting in a 36 × 36 image. Bounding boxes of digits in MNIST overlap by approximately 80%, so this is used to make both digits identifiable (since there is no RGB difference learnable by the network to separate the digits)<br /> <br /> 2. The label becomes a vector of two numbers, representing the original digit and the randomly generated (and overlaid) digit.<br /> <br /> <br /> <br /> [[File:CapsuleNets MultiMNIST.PNG|600px|thumb|center|Figure 5: Sample reconstructions of a CapsNet with 3 routing iterations on MultiMNIST test dataset.<br /> The two reconstructed digits are overlayed in green and red as the lower image. The upper image<br /> shows the input image. L:(l1; l2) represents the label for the two digits in the image and R:(r1; r2)<br /> represents the two digits used for reconstruction. The two right most columns show two examples<br /> with wrong classification reconstructed from the label and from the prediction (P). In the (2; 8)<br /> example the model confuses 8 with a 7 and in (4; 9) it confuses 9 with 0. The other columns have<br /> correct classifications and show that the model accounts for all the pixels while being able to assign<br /> one pixel to two digits in extremely difficult scenarios (column 1 − 4). Note that in dataset generation<br /> the pixel values are clipped at 1. The two columns with the (*) mark show reconstructions from a<br /> digit that is neither the label nor the prediction. These columns suggest that the model is not just<br /> finding the best fit for all the digits in the image including the ones that do not exist. Therefore in case<br /> of (5; 0) it cannot reconstruct a 7 because it knows that there is a 5 and 0 that fit best and account for<br /> all the pixels. Also, in the case of (8; 1) the loop of 8 has not triggered 0 because it is already accounted<br /> for by 8. Therefore it will not assign one pixel to two digits if one of them does not have any other<br /> support.]]<br /> <br /> ==Other datasets==<br /> The authors also tested the proposed capsule model on CIFAR10 dataset and achieved an error rate of 10.6%. The model tested was an ensemble of 7 models. Each of the models in the ensemble had the same architecture as the model used for MNIST (apart from 3 additional channels and 64 different types of primary capsules being used). These 7 models were trained on 24x24 patches of the training images for 3 iterations. During experimentation, the authors also found out that adding an additional none-of-the-above category helped improved the overall performance. The error rate achieved is comparable to the error rate achieved by a standard CNN model. According to the authors, one of the reasons for low performance is the fact that background in CIFAR-10 images are too varied for it to be adequately modeled by reasonably sized capsule net.<br /> <br /> The proposed model was also evaluated using a small subset of SVHN dataset. The network trained was much smaller and trained using only 73257 training images. The network still managed to achieve an error rate of 4.3% on the test set.<br /> <br /> =Critique=<br /> Although the network performs incredibly favorable in the author's experiments, it has a long way to go on more complex datasets. On CIFAR 10, the network achieved subpar results, and the experimental results seem to be worse when the problem becomes more complex. This is anticipated, since these networks are still in their early stage; later innovations might come in the upcoming decades/years. It could also be wise to apply the model to other datasets with larger sizes to make the functionality more acceptable. MNIST dataset has simple patterns and even if the model wanted to be presented with only one dataset, it was better not to be MNIST dataset especially in this case that the focus is on human-eye detection and numbers are not that regular in real-life experiences.<br /> <br /> Hinton talks about CapsuleNets revolutionizing areas such as self-driving, but such groundbreaking innovations are far away from CIFAR10, and even further from MNIST. Only time can tell if CapsNets will live up to their hype.<br /> <br /> Capsules inherently segment images and learn a lower dimensional embedding in a new manner, which makes them likely to perform well on segmentation and computer vision tasks once further research is done. <br /> <br /> Additionally, these networks are more interpretable than CNNs, and have strong theoretical reasoning for why they could work. Naturally, it would be hard for a new architecture to beat the heavily researched/modified CNNs.<br /> <br /> * ([https://openreview.net/forum?id=HJWLfGWRb]) it's not fully clear how effective it can be performed / how scalable it is. Evaluation is performed on a small dataset for shape recognition. The approach will need to be tested on larger, more challenging datasets.<br /> <br /> =Future Work=<br /> The same authors [N. F. Geoffrey E Hinton, Sara Sabour] presented another paper &quot;MATRIX CAPSULES WITH EM ROUTING&quot; in ICLR 2018, which achieved better results than the work presented in this paper. They presented a new multi-layered capsule network architecture, implemented an EM routing procedure, and introduced &quot;Coordinate Addition&quot;. This new type reduced number of errors by 45%, and performed better than standard CNN on white box adversarial attacks. Capsule architectures are gaining interest because of their ability to achieve equivariance of parts, and employ a new form of pooling called &quot;routing&quot; (as opposed to max pooling) which groups parts that make similar predictions of the whole to which they belong, rather than relying on spatial co-locality.<br /> Moreover, the authors hint towards trying to change the curvature and sensitivities to various factors by introducing new form of loss function. It may improve the performance of the model for more complicated data set which is one of the model's drawback.<br /> <br /> Moreover, as mentioned in critiques, a good future work for this group would be making the model more robust to the dataset and achieve acceptable performance on datasets with more regularly seen images in real life experiences.<br /> <br /> =References=<br /> #N. F. Geoffrey E Hinton, Sara Sabour. Matrix capsules with em routing. In International Conference on Learning Representations, 2018.<br /> #S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between capsules,” arXiv preprint arXiv:1710.09829v2, 2017<br /> # Hinton, G. E., Krizhevsky, A. and Wang, S. D. (2011), Transforming Auto-encoders <br /> #Geoffrey Hinton's talk: What is wrong with convolutional neural nets? - Talk given at MIT. Brain &amp; Cognitive Sciences - Fall Colloquium Series. [https://www.youtube.com/watch?v=rTawFwUvnLE ]<br /> #Understanding Hinton’s Capsule Networks - Max Pechyonkin's series [https://medium.com/ai%C2%B3-theory-practice-business/understanding-hintons-capsule-networks-part-i-intuition-b4b559d1159b]<br /> #Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg SCorrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machinelearning on heterogeneous distributed systems.arXiv preprint arXiv:1603.04467, 2016.<br /> #Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visualattention.arXiv preprint arXiv:1412.7755, 2014.<br /> #Jia-Ren Chang and Yong-Sheng Chen. Batch-normalized maxout network in network.arXiv preprintarXiv:1511.02583, 2015.<br /> #Dan C Cire ̧san, Ueli Meier, Jonathan Masci, Luca M Gambardella, and Jürgen Schmidhuber. High-performance neural networks for visual object classification.arXiv preprint arXiv:1102.0183,2011.<br /> #Ian J Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi-digit numberrecognition from street view imagery using deep convolutional neural networks.arXiv preprintarXiv:1312.6082, 2013.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=ShakeDrop_Regularization&diff=41893 ShakeDrop Regularization 2018-11-29T17:41:29Z <p>C9sharma: /* Critique */</p> <hr /> <div>=Introduction=<br /> Current state of the art techniques for object classification are deep neural networks based on the residual block, first published by (He et al., 2016). This technique has been the foundation of several improved networks, including Wide ResNet (Zagoruyko &amp; Komodakis, 2016), PyramdNet (Han et al., 2017) and ResNeXt (Xie et al., 2017). They have been further improved by regularization, such as Stochastic Depth (ResDrop) (Huang et al., 2016) and Shake-Shake (Gastaldi, 2017), which can avoid some problem like vanishing gradients. Shake-Shake applied to ResNeXt has achieved one of the lowest error rates on the CIFAR-10 and CIFAR-100 datasets. However, it is only applicable to multi-branch architectures and is not memory efficient since it requires two branches of residual blocks to apply. To address this problem, ShakeDrop regularization that can realize a similar disturbance to Shake-Shake on a single residual block is proposed. Moreover, they use ResDrop to stabilize the learning process. This paper seeks to formulate a general expansion of Shake-Shake that can be applied to any residual block based network.<br /> <br /> =Existing Methods=<br /> <br /> '''Deep Approaches'''<br /> <br /> '''ResNet''', was the first use of residual blocks, a foundational feature in many modern state of the art convolution neural networks. They can be formulated as &lt;math&gt;G(x) = x + F(x)&lt;/math&gt; where &lt;math&gt;x&lt;/math&gt; and &lt;math&gt;G(x)&lt;/math&gt; are the input and output of the residual block, and &lt;math&gt;F(x)&lt;/math&gt; is the output of the residual branch on the residual block. A residual block typically performs a convolution operation and then passes the result plus its input onto the next block.<br /> <br /> Intuition behind Residual blocks:<br /> If the identity mapping is optimal, We can easily push the residuals to zero (F(x) = 0) than to fit an identity mapping (x, input=output) by a stack of non-linear layers. In simple language it is very easy to come up with a solution like F(x) =0 rather than F(x)=x using stack of non-linear cnn layers as function (Think about it). So, this function F(x) is what the authors called Residual function ([https://medium.com/@14prakash/understanding-and-implementing-architectures-of-resnet-and-resnext-for-state-of-the-art-image-cf51669e1624 Reference]).<br /> <br /> <br /> [[File:ResidualBlock.png|580px|centre|thumb|An example of a simple residual block from Deep Residual Learning for Image Recognition by He et al., 2016]]<br /> <br /> ResNet is constructed out of a large number of these residual blocks sequentially stacked. It is interesting to note that having too many layers can cause overfitting, as pointed out by He et al. (2016) with the high error rates for the 1,202-layer ResNet on CIFAR datasets. Another paper (Veit et al., 2016) empirically showed that the cause of the high error rates can be mostly attributed to specific residual blocks whose channels increase greatly.<br /> <br /> '''PyramidNet''' is an important iteration that built on ResNet and WideResNet by gradually increasing channels on each residual block. The residual block is similar to those used in ResNet. It has been used to generate some of the first successful convolution neural networks with very large depth, at 272 layers. Amongst unmodified residual network architectures, it performs the best on the CIFAR datasets.<br /> <br /> [[File:ResidualBlockComparison.png|980px|centre|thumb|A simple illustration of different residual blocks from Deep Pyramidal Residual Networks by Han et al., 2017. The width of a block reflects the number of channels used in that layer.]]<br /> <br /> <br /> '''Non-Deep Approaches'''<br /> <br /> '''Wide ResNet''' modified ResNet by increasing channels in each layer, having a wider and shallower structure. Similarly to PyramidNet, this architecture avoids some of the pitfalls in the original formulation of ResNet.<br /> <br /> '''ResNeXt''' achieved performance beyond that of Wide ResNet with only a small increase in the number of parameters. It can be formulated as &lt;math&gt;G(x) = x + F_1(x)+F_2(x)&lt;/math&gt;. In this case, &lt;math&gt;F_1(x)&lt;/math&gt; and &lt;math&gt;F_2(x)&lt;/math&gt; are the outputs of two paired convolution operations in a single residual block. The number of branches is not limited to 2, and will control the result of this network.<br /> <br /> <br /> [[File:SimplifiedResNeXt.png|600px|centre|thumb|Simplified ResNeXt Convolution Block. Yamada et al., 2018]]<br /> <br /> <br /> '''Regularization Methods'''<br /> <br /> '''Stochastic Depth''' helped address the issue of vanishing gradients in ResNet. It works by randomly dropping residual blocks. On the &lt;math&gt;l^{th}&lt;/math&gt; residual block the Stochastic Depth process is given as &lt;math&gt;G(x)=x+b_lF(x)&lt;/math&gt; where &lt;math&gt;b_l \in \{0,1\}&lt;/math&gt; is a Bernoulli random variable with probability &lt;math&gt;p_l&lt;/math&gt;. Using a constant value for &lt;math&gt;p_l&lt;/math&gt; didn't work well, so instead a linear decay rule &lt;math&gt;p_l = 1 - \frac{l}{L}(1-p_L)&lt;/math&gt; was used. In this equation, &lt;math&gt;L&lt;/math&gt; is the number of layers, and &lt;math&gt;p_L&lt;/math&gt; is the initial parameter. <br /> <br /> '''Shake-Shake''' is a regularization method that specifically improves the ResNeXt architecture. It can be given as &lt;math&gt;G(x)=x+\alpha F_1(x)+(1-\alpha)F_2(x)&lt;/math&gt;, where &lt;math&gt;\alpha \in [0,1]&lt;/math&gt; is a random coefficient. &lt;math&gt;\alpha&lt;/math&gt; is used during the forward pass, and another identically distributed random parameter &lt;math&gt;\beta&lt;/math&gt; is used in the backward pass. This caused one of the two paired convolution operations to be dropped, and further improved ResNeXt.<br /> <br /> [[File:Paper 32.jpg|600px|centre|thumb| Shake-Shake (ResNeXt + Shake-Shake) (Gastaldi, 2017), in which some processing layers omitted for conciseness.]]<br /> <br /> =Proposed Method=<br /> We give an intuitive interpretation of the forward pass of Shake-Shake regularization. To the best of our knowledge, it has not been given yet, while the phenomenon in the backward pass is experimentally investigated by Gastaldi (2017). In the forward pass, Shake-Shake interpolates the outputs of two residual branches with a random variable α that controls the degree of interpolation. As DeVries &amp; Taylor (2017a) demonstrated that interpolation of two data in the feature space can synthesize reasonable augmented data, the interpolation of two residual blocks of Shake-Shake in the forward pass can be interpreted as synthesizing data. Use of a random variable α generates many different augmented data. On the other hand, in the backward pass, a different random variable β is used to disturb learning to make the network learnable long time. Gastaldi (2017) demonstrated how the difference between &lt;math&gt;\alpha&lt;/math&gt; and &lt;math&gt;\beta&lt;/math&gt; affects.<br /> <br /> The regularization mechanism of Shake-Shake relies on two or more residual branches, so that it can be applied only to 2-branch networks architectures. In addition, 2-branch network architectures consume more memory than 1-branch network architectures. One may think the number of learnable parameters of ResNeXt can be kept in 1-branch and 2-branch network architectures by controlling its cardinality and the number of channels (filters). For example, a 1-branch network (e.g., ResNeXt 1-64d) and its corresponding 2-branch network (e.g., ResNeXt 2-40d) have almost same number of learnable parameters. However, even so, it increases memory consumption due to the overhead to keep the inputs of residual blocks and so on. By comparing ResNeXt 1-64d and 2-40d, the latter requires more memory than the former by 8% in theory (for one layer) and by 11% in measured values (for 152 layers).<br /> <br /> This paper seeks to generalize the method proposed in Shake-Shake to be applied to any residual structure network. Shake-Shake. The initial formulation of 1-branch shake is &lt;math&gt;G(x) = x + \alpha F(x)&lt;/math&gt;. In this case, &lt;math&gt;\alpha&lt;/math&gt; is a coefficient that disturbs the forward pass, but is not necessarily constrained to be [0,1]. Another corresponding coefficient &lt;math&gt;\beta&lt;/math&gt; is used in the backwards pass. Applying this simple adaptation of Shake-Shake on a 110-layer version of PyramidNet with &lt;math&gt;\alpha \in [0,1]&lt;/math&gt; and &lt;math&gt;\beta \in [0,1]&lt;/math&gt; performs abysmally, with an error rate of 77.99%.<br /> <br /> This failure is a result of the setup causing too much perturbation. A trick is needed to promote learning with large perturbations, to preserve the regularization effect. The idea of the authors is to borrow from ResDrop and combine that with Shake-Shake. This works by randomly deciding whether to apply 1-branch shake. This creates in effect two networks, the original network without a regularization component, and a regularized network. When mixing up two networks, we expected the following effects: When the non regularized network is selected, learning is promoted; when the perturbed network is selected, learning is disturbed. Achieving good performance requires a balance between the two. <br /> <br /> '''ShakeDrop''' is given as <br /> <br /> &lt;div align=&quot;center&quot;&gt;<br /> &lt;math&gt;G(x) = x + (b_l + \alpha - b_l \alpha)F(x)&lt;/math&gt;,<br /> &lt;/div&gt;<br /> <br /> where &lt;math&gt;b_l&lt;/math&gt; is a Bernoulli random variable following the linear decay rule used in Stochastic Depth. An alternative presentation is <br /> <br /> &lt;div align=&quot;center&quot;&gt;<br /> &lt;math&gt;<br /> G(x) = \begin{cases}<br /> x + F(x) ~~ \text{if } b_l = 1 \\<br /> x + \alpha F(x) ~~ \text{otherwise}<br /> \end{cases}<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> If &lt;math&gt;b_l = 1&lt;/math&gt; then ShakeDrop is equivalent to the original network, otherwise it is the network + 1-branch Shake. The authors also found that the linear decay rule of ResDrop works well, compared with the uniform rule. Regardless of the value of &lt;math&gt;\beta&lt;/math&gt; on the backwards pass, network weights will be updated.<br /> <br /> =Experiments=<br /> <br /> '''Parameter Search'''<br /> <br /> The authors experiments began with a hyperparameter search utilizing ShakeDrop on pyramidal networks. The PyramidNet used was made up of a total of 110 layers which included a convolutional layer and a final fully connected layer. It had 54 additive pyramidal residual blocks and the final residual block had 286 channels. The results of this search are presented below. <br /> <br /> [[File:ShakeDropHyperParameterSearch.png|600px|centre|thumb|Average Top-1 errors (%) of “PyramidNet + ShakeDrop” with several ranges of parameters of 4 runs at the final (300th) epoch on CIFAR-100 dataset in the “Batch” level. In some settings, it is equivalent to PyramidNet and PyramidDrop. Borrowed from ShakeDrop Regularization by Yamada et al., 2018.]]<br /> <br /> The setting that are used throughout the rest of the experiments are then &lt;math&gt;\alpha \in [-1,1]&lt;/math&gt; and &lt;math&gt;\beta \in [0,1]&lt;/math&gt;. Cases H and F outperform PyramidNet, suggesting that the strong perturbations imposed by ShakeDrop are functioning as intended. However, fully applying the perturbations in the backwards pass appears to destabilize the network, resulting in performance that is worse than standard PyramidNet.<br /> <br /> [[File:ParameterUpdateShakeDrop.png|400px|centre]]<br /> <br /> Following this initial parameter decision, the authors tested 4 different strategies for parameter update among &quot;Batch&quot; (same coefficients for all images in minibatch for each residual block), &quot;Image&quot; (same scaling coefficients for each image for each residual block), &quot;Channel&quot; (same scaling coefficients for each element for each residual block), and &quot;Pixel&quot; (same scaling coefficients for each element for each residual block). While Pixel was the best in terms of error rate, it is not very memory efficient, so Image was selected as it had the second best performance without the memory drawback.<br /> <br /> '''Comparison with Regularization Methods'''<br /> <br /> For these experiments, there are a few modifications that were made to assist with training. For ResNeXt, the EraseRelu formulation has each residual block ends in batch normalization. The Wide ResNet also is compared between vanilla with batch normalization and without. Batch normalization keeps the outputs of residual blocks in a certain range, as otherwise &lt;math&gt;\alpha&lt;/math&gt; and &lt;math&gt;\beta&lt;/math&gt; could cause perturbations that are too large, causing divergent learning. There is also a comparison of ResDrop/ShakeDrop Type A (where the regularization unit is inserted before the add unit for a residual branch) and after (where the regularization unit is inserted after the add unit for a residual branch). <br /> <br /> These experiments are performed on the CIFAR-100 dataset.<br /> <br /> [[File:ShakeDropArchitectureComparison1.png|800px|centre|thumb|]]<br /> <br /> [[File:ShakeDropArchitectureComparison2.png|800px|centre|thumb|]]<br /> <br /> [[File:ShakeDropArchitectureComparison3.png|800px|centre|thumb|]]<br /> <br /> For a final round of testing, the training setup was modified to incorporate other techniques used in state of the art methods. For most of the tests, the learning rate for the 300 epoch version started at 0.1 and decayed by a factor of 0.1 1/2 &amp; 3/4 of the way through training. The alternative was cosine annealing, based on the presentation by Loshchilov and Hutter in their paper SGDR: Stochastic Gradient Descent with Warm Restarts. This is indicated in the Cos column, with a check indicating cosine annealing. <br /> <br /> [[File:CosineAnnealing.png|400px|centre|thumb|]]<br /> <br /> The Reg column indicates the regularization method used, either none, ResDrop (RD), Shake-Shake (SS), or ShakeDrop (SD). Fianlly, the Fil Column determines the type of data augmentation used, either none, cutout (CO) (DeVries &amp; Taylor, 2017b), or Random Erasing (RE) (Zhong et al., 2017). <br /> <br /> [[File:ShakeDropComparison.png|800px|centre|thumb|Top-1 Errors (%) at final epoch on CIFAR-10/100 datasets]]<br /> <br /> '''State-of-the-Art Comparisons'''<br /> <br /> A direct comparison with state of the art methods is favorable for this new method. <br /> <br /> # Fair comparison of ResNeXt + Shake-Shake with PyramidNet + ShakeDrop gives an improvement of 0.19% on CIFAR-10 and 1.86% on CIFAR-100. Under these conditions, the final error rate is then 2.67% for CIFAR-10 and 13.99% for CIFAR-100.<br /> # Fair comparison of ResNeXt + Shake-Shake + Cutout with PyramidNet + ShakeDrop + Random Erasing gives an improvement of 0.25% on CIFAR-10 and 3.01% on CIFAR 100. Under these conditions, the final error rate is then 2.31% for CIFAR-10 and 12.19% for CIFAR-100.<br /> # Comparison with the state-of-the-arts, PyramidNet + ShakeDrop gives an improvement of 0.25% on CIFAR-10 than ResNeXt + Shake-Shake + Cutout, PyramidNet + ShakeDrop gives an improvement of 2.85% on CIFAR-100 than Coupled Ensemble.<br /> <br /> =Implementation details=<br /> <br /> '''CIFAR-10/100 datasets'''<br /> <br /> All the images in these datasets were color normalized and then horizontally flipped with a probability of 50%. All of then then were zero padded to have a dimentionality of 40 by 40 pixels.<br /> <br /> <br /> =Conclusion=<br /> The paper proposes a new form of regularization that is an extension of &quot;Shake-Shake&quot; regularization [Gastaldi, 2017]. The original &quot;shake-shake&quot; proposes using two residual paths adding to the same output, and during training, considering different randomly selected convex combinations of the two paths (while using an equally weighted combination at test time). This paper contends that this requires additional memory, and attempts to achieve similar regularization with a single path. To do so, they train a network with a single residual path, where the residual is included without attenuation in some cases with some fixed probability, and attenuated randomly (or even inverted) in others. The paper contends that this achieves superior performance than choosing simply a random attenuation for every sample (although, this can be seen as choosing an attenuation under a distribution with some fixed probability mass.<br /> <br /> Their stochastic regularization method, ShakeDrop, which outperforms previous state of the art methods while maintaining similar memory efficiency. It demonstrates that heavily perturbing a network can help to overcome issues with overfitting. It is also an effective way to regularize residual networks for image classification. The method was tested by CIFAR-10/100 and Tiny ImageNet datasets and showed great performance.<br /> <br /> =Critique=<br /> <br /> The novelty of this paper is low as pointed out by the reviewers. Also, there is a confusion whether or not the results could be replicated as &lt;math&gt;\alpha&lt;/math&gt; and &lt;math&gt;\beta&lt;/math&gt; are choosen randomly. The proposed ShakeDrop regularization is essentially a combination of the PyramidDrop and Shake-Shake regularization. The most surprising part is that the forward weight can be negative thus inverting the output of a convolution. The mathematical justification for ShakeDrop regularization is limited, relying on intuition and empirical evidence instead.<br /> As pointed out from the above, the method basically relies heavily on the intuition. This means that the performance of the algorithm can not been extended beyond the CIFAR dataset and can vary a lot depending on the characteristics of data sets that users are performing, with some exaggeration. However, the performance is still impressive since it performs better than known algorithms. It is not clear as to how the proposed technique would work with a non-residual architecture.<br /> It lacks conclusive proof that &quot;shake-drop&quot; is a generically useful regularization technique. For one, the method is evaluated only on small toy-datasets: CIFAR-10 and CIFAR-100. Evaluation on Imagenet perhaps would have been valuable.<br /> <br /> =References=<br /> [Yamada et al., 2018] Yamada Y, Iwamura M, Kise K. ShakeDrop regularization. arXiv preprint arXiv:1802.02375. 2018 Feb 7.<br /> <br /> [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, 2016.<br /> <br /> [Zagoruyko &amp; Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proc. BMVC, 2016.<br /> <br /> [Han et al., 2017] Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In Proc. CVPR, 2017a.<br /> <br /> [Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proc. CVPR, 2017.<br /> <br /> [Huang et al., 2016] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. arXiv preprint arXiv:1603.09382v3, 2016.<br /> <br /> [Gastaldi, 2017] Xavier Gastaldi. Shake-shake regularization. arXiv preprint arXiv:1705.07485v2, 2017.<br /> <br /> [Loshilov &amp; Hutter, 2016] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.<br /> <br /> [DeVries &amp; Taylor, 2017b] Terrance DeVries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017b.<br /> <br /> [Zhong et al., 2017] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. arXiv preprint arXiv:1708.04896, 2017.<br /> <br /> [Dutt et al., 2017] Anuvabh Dutt, Denis Pellerin, and Georges Qunot. Coupled ensembles of neural networks. arXiv preprint 1709.06053v1, 2017.<br /> <br /> [Veit et al., 2016] Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in Neural Information Processing Systems 29, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=ShakeDrop_Regularization&diff=41891 ShakeDrop Regularization 2018-11-29T17:38:57Z <p>C9sharma: /* Conclusion */</p> <hr /> <div>=Introduction=<br /> Current state of the art techniques for object classification are deep neural networks based on the residual block, first published by (He et al., 2016). This technique has been the foundation of several improved networks, including Wide ResNet (Zagoruyko &amp; Komodakis, 2016), PyramdNet (Han et al., 2017) and ResNeXt (Xie et al., 2017). They have been further improved by regularization, such as Stochastic Depth (ResDrop) (Huang et al., 2016) and Shake-Shake (Gastaldi, 2017), which can avoid some problem like vanishing gradients. Shake-Shake applied to ResNeXt has achieved one of the lowest error rates on the CIFAR-10 and CIFAR-100 datasets. However, it is only applicable to multi-branch architectures and is not memory efficient since it requires two branches of residual blocks to apply. To address this problem, ShakeDrop regularization that can realize a similar disturbance to Shake-Shake on a single residual block is proposed. Moreover, they use ResDrop to stabilize the learning process. This paper seeks to formulate a general expansion of Shake-Shake that can be applied to any residual block based network.<br /> <br /> =Existing Methods=<br /> <br /> '''Deep Approaches'''<br /> <br /> '''ResNet''', was the first use of residual blocks, a foundational feature in many modern state of the art convolution neural networks. They can be formulated as &lt;math&gt;G(x) = x + F(x)&lt;/math&gt; where &lt;math&gt;x&lt;/math&gt; and &lt;math&gt;G(x)&lt;/math&gt; are the input and output of the residual block, and &lt;math&gt;F(x)&lt;/math&gt; is the output of the residual branch on the residual block. A residual block typically performs a convolution operation and then passes the result plus its input onto the next block.<br /> <br /> Intuition behind Residual blocks:<br /> If the identity mapping is optimal, We can easily push the residuals to zero (F(x) = 0) than to fit an identity mapping (x, input=output) by a stack of non-linear layers. In simple language it is very easy to come up with a solution like F(x) =0 rather than F(x)=x using stack of non-linear cnn layers as function (Think about it). So, this function F(x) is what the authors called Residual function ([https://medium.com/@14prakash/understanding-and-implementing-architectures-of-resnet-and-resnext-for-state-of-the-art-image-cf51669e1624 Reference]).<br /> <br /> <br /> [[File:ResidualBlock.png|580px|centre|thumb|An example of a simple residual block from Deep Residual Learning for Image Recognition by He et al., 2016]]<br /> <br /> ResNet is constructed out of a large number of these residual blocks sequentially stacked. It is interesting to note that having too many layers can cause overfitting, as pointed out by He et al. (2016) with the high error rates for the 1,202-layer ResNet on CIFAR datasets. Another paper (Veit et al., 2016) empirically showed that the cause of the high error rates can be mostly attributed to specific residual blocks whose channels increase greatly.<br /> <br /> '''PyramidNet''' is an important iteration that built on ResNet and WideResNet by gradually increasing channels on each residual block. The residual block is similar to those used in ResNet. It has been used to generate some of the first successful convolution neural networks with very large depth, at 272 layers. Amongst unmodified residual network architectures, it performs the best on the CIFAR datasets.<br /> <br /> [[File:ResidualBlockComparison.png|980px|centre|thumb|A simple illustration of different residual blocks from Deep Pyramidal Residual Networks by Han et al., 2017. The width of a block reflects the number of channels used in that layer.]]<br /> <br /> <br /> '''Non-Deep Approaches'''<br /> <br /> '''Wide ResNet''' modified ResNet by increasing channels in each layer, having a wider and shallower structure. Similarly to PyramidNet, this architecture avoids some of the pitfalls in the original formulation of ResNet.<br /> <br /> '''ResNeXt''' achieved performance beyond that of Wide ResNet with only a small increase in the number of parameters. It can be formulated as &lt;math&gt;G(x) = x + F_1(x)+F_2(x)&lt;/math&gt;. In this case, &lt;math&gt;F_1(x)&lt;/math&gt; and &lt;math&gt;F_2(x)&lt;/math&gt; are the outputs of two paired convolution operations in a single residual block. The number of branches is not limited to 2, and will control the result of this network.<br /> <br /> <br /> [[File:SimplifiedResNeXt.png|600px|centre|thumb|Simplified ResNeXt Convolution Block. Yamada et al., 2018]]<br /> <br /> <br /> '''Regularization Methods'''<br /> <br /> '''Stochastic Depth''' helped address the issue of vanishing gradients in ResNet. It works by randomly dropping residual blocks. On the &lt;math&gt;l^{th}&lt;/math&gt; residual block the Stochastic Depth process is given as &lt;math&gt;G(x)=x+b_lF(x)&lt;/math&gt; where &lt;math&gt;b_l \in \{0,1\}&lt;/math&gt; is a Bernoulli random variable with probability &lt;math&gt;p_l&lt;/math&gt;. Using a constant value for &lt;math&gt;p_l&lt;/math&gt; didn't work well, so instead a linear decay rule &lt;math&gt;p_l = 1 - \frac{l}{L}(1-p_L)&lt;/math&gt; was used. In this equation, &lt;math&gt;L&lt;/math&gt; is the number of layers, and &lt;math&gt;p_L&lt;/math&gt; is the initial parameter. <br /> <br /> '''Shake-Shake''' is a regularization method that specifically improves the ResNeXt architecture. It can be given as &lt;math&gt;G(x)=x+\alpha F_1(x)+(1-\alpha)F_2(x)&lt;/math&gt;, where &lt;math&gt;\alpha \in [0,1]&lt;/math&gt; is a random coefficient. &lt;math&gt;\alpha&lt;/math&gt; is used during the forward pass, and another identically distributed random parameter &lt;math&gt;\beta&lt;/math&gt; is used in the backward pass. This caused one of the two paired convolution operations to be dropped, and further improved ResNeXt.<br /> <br /> [[File:Paper 32.jpg|600px|centre|thumb| Shake-Shake (ResNeXt + Shake-Shake) (Gastaldi, 2017), in which some processing layers omitted for conciseness.]]<br /> <br /> =Proposed Method=<br /> We give an intuitive interpretation of the forward pass of Shake-Shake regularization. To the best of our knowledge, it has not been given yet, while the phenomenon in the backward pass is experimentally investigated by Gastaldi (2017). In the forward pass, Shake-Shake interpolates the outputs of two residual branches with a random variable α that controls the degree of interpolation. As DeVries &amp; Taylor (2017a) demonstrated that interpolation of two data in the feature space can synthesize reasonable augmented data, the interpolation of two residual blocks of Shake-Shake in the forward pass can be interpreted as synthesizing data. Use of a random variable α generates many different augmented data. On the other hand, in the backward pass, a different random variable β is used to disturb learning to make the network learnable long time. Gastaldi (2017) demonstrated how the difference between &lt;math&gt;\alpha&lt;/math&gt; and &lt;math&gt;\beta&lt;/math&gt; affects.<br /> <br /> The regularization mechanism of Shake-Shake relies on two or more residual branches, so that it can be applied only to 2-branch networks architectures. In addition, 2-branch network architectures consume more memory than 1-branch network architectures. One may think the number of learnable parameters of ResNeXt can be kept in 1-branch and 2-branch network architectures by controlling its cardinality and the number of channels (filters). For example, a 1-branch network (e.g., ResNeXt 1-64d) and its corresponding 2-branch network (e.g., ResNeXt 2-40d) have almost same number of learnable parameters. However, even so, it increases memory consumption due to the overhead to keep the inputs of residual blocks and so on. By comparing ResNeXt 1-64d and 2-40d, the latter requires more memory than the former by 8% in theory (for one layer) and by 11% in measured values (for 152 layers).<br /> <br /> This paper seeks to generalize the method proposed in Shake-Shake to be applied to any residual structure network. Shake-Shake. The initial formulation of 1-branch shake is &lt;math&gt;G(x) = x + \alpha F(x)&lt;/math&gt;. In this case, &lt;math&gt;\alpha&lt;/math&gt; is a coefficient that disturbs the forward pass, but is not necessarily constrained to be [0,1]. Another corresponding coefficient &lt;math&gt;\beta&lt;/math&gt; is used in the backwards pass. Applying this simple adaptation of Shake-Shake on a 110-layer version of PyramidNet with &lt;math&gt;\alpha \in [0,1]&lt;/math&gt; and &lt;math&gt;\beta \in [0,1]&lt;/math&gt; performs abysmally, with an error rate of 77.99%.<br /> <br /> This failure is a result of the setup causing too much perturbation. A trick is needed to promote learning with large perturbations, to preserve the regularization effect. The idea of the authors is to borrow from ResDrop and combine that with Shake-Shake. This works by randomly deciding whether to apply 1-branch shake. This creates in effect two networks, the original network without a regularization component, and a regularized network. When mixing up two networks, we expected the following effects: When the non regularized network is selected, learning is promoted; when the perturbed network is selected, learning is disturbed. Achieving good performance requires a balance between the two. <br /> <br /> '''ShakeDrop''' is given as <br /> <br /> &lt;div align=&quot;center&quot;&gt;<br /> &lt;math&gt;G(x) = x + (b_l + \alpha - b_l \alpha)F(x)&lt;/math&gt;,<br /> &lt;/div&gt;<br /> <br /> where &lt;math&gt;b_l&lt;/math&gt; is a Bernoulli random variable following the linear decay rule used in Stochastic Depth. An alternative presentation is <br /> <br /> &lt;div align=&quot;center&quot;&gt;<br /> &lt;math&gt;<br /> G(x) = \begin{cases}<br /> x + F(x) ~~ \text{if } b_l = 1 \\<br /> x + \alpha F(x) ~~ \text{otherwise}<br /> \end{cases}<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> If &lt;math&gt;b_l = 1&lt;/math&gt; then ShakeDrop is equivalent to the original network, otherwise it is the network + 1-branch Shake. The authors also found that the linear decay rule of ResDrop works well, compared with the uniform rule. Regardless of the value of &lt;math&gt;\beta&lt;/math&gt; on the backwards pass, network weights will be updated.<br /> <br /> =Experiments=<br /> <br /> '''Parameter Search'''<br /> <br /> The authors experiments began with a hyperparameter search utilizing ShakeDrop on pyramidal networks. The PyramidNet used was made up of a total of 110 layers which included a convolutional layer and a final fully connected layer. It had 54 additive pyramidal residual blocks and the final residual block had 286 channels. The results of this search are presented below. <br /> <br /> [[File:ShakeDropHyperParameterSearch.png|600px|centre|thumb|Average Top-1 errors (%) of “PyramidNet + ShakeDrop” with several ranges of parameters of 4 runs at the final (300th) epoch on CIFAR-100 dataset in the “Batch” level. In some settings, it is equivalent to PyramidNet and PyramidDrop. Borrowed from ShakeDrop Regularization by Yamada et al., 2018.]]<br /> <br /> The setting that are used throughout the rest of the experiments are then &lt;math&gt;\alpha \in [-1,1]&lt;/math&gt; and &lt;math&gt;\beta \in [0,1]&lt;/math&gt;. Cases H and F outperform PyramidNet, suggesting that the strong perturbations imposed by ShakeDrop are functioning as intended. However, fully applying the perturbations in the backwards pass appears to destabilize the network, resulting in performance that is worse than standard PyramidNet.<br /> <br /> [[File:ParameterUpdateShakeDrop.png|400px|centre]]<br /> <br /> Following this initial parameter decision, the authors tested 4 different strategies for parameter update among &quot;Batch&quot; (same coefficients for all images in minibatch for each residual block), &quot;Image&quot; (same scaling coefficients for each image for each residual block), &quot;Channel&quot; (same scaling coefficients for each element for each residual block), and &quot;Pixel&quot; (same scaling coefficients for each element for each residual block). While Pixel was the best in terms of error rate, it is not very memory efficient, so Image was selected as it had the second best performance without the memory drawback.<br /> <br /> '''Comparison with Regularization Methods'''<br /> <br /> For these experiments, there are a few modifications that were made to assist with training. For ResNeXt, the EraseRelu formulation has each residual block ends in batch normalization. The Wide ResNet also is compared between vanilla with batch normalization and without. Batch normalization keeps the outputs of residual blocks in a certain range, as otherwise &lt;math&gt;\alpha&lt;/math&gt; and &lt;math&gt;\beta&lt;/math&gt; could cause perturbations that are too large, causing divergent learning. There is also a comparison of ResDrop/ShakeDrop Type A (where the regularization unit is inserted before the add unit for a residual branch) and after (where the regularization unit is inserted after the add unit for a residual branch). <br /> <br /> These experiments are performed on the CIFAR-100 dataset.<br /> <br /> [[File:ShakeDropArchitectureComparison1.png|800px|centre|thumb|]]<br /> <br /> [[File:ShakeDropArchitectureComparison2.png|800px|centre|thumb|]]<br /> <br /> [[File:ShakeDropArchitectureComparison3.png|800px|centre|thumb|]]<br /> <br /> For a final round of testing, the training setup was modified to incorporate other techniques used in state of the art methods. For most of the tests, the learning rate for the 300 epoch version started at 0.1 and decayed by a factor of 0.1 1/2 &amp; 3/4 of the way through training. The alternative was cosine annealing, based on the presentation by Loshchilov and Hutter in their paper SGDR: Stochastic Gradient Descent with Warm Restarts. This is indicated in the Cos column, with a check indicating cosine annealing. <br /> <br /> [[File:CosineAnnealing.png|400px|centre|thumb|]]<br /> <br /> The Reg column indicates the regularization method used, either none, ResDrop (RD), Shake-Shake (SS), or ShakeDrop (SD). Fianlly, the Fil Column determines the type of data augmentation used, either none, cutout (CO) (DeVries &amp; Taylor, 2017b), or Random Erasing (RE) (Zhong et al., 2017). <br /> <br /> [[File:ShakeDropComparison.png|800px|centre|thumb|Top-1 Errors (%) at final epoch on CIFAR-10/100 datasets]]<br /> <br /> '''State-of-the-Art Comparisons'''<br /> <br /> A direct comparison with state of the art methods is favorable for this new method. <br /> <br /> # Fair comparison of ResNeXt + Shake-Shake with PyramidNet + ShakeDrop gives an improvement of 0.19% on CIFAR-10 and 1.86% on CIFAR-100. Under these conditions, the final error rate is then 2.67% for CIFAR-10 and 13.99% for CIFAR-100.<br /> # Fair comparison of ResNeXt + Shake-Shake + Cutout with PyramidNet + ShakeDrop + Random Erasing gives an improvement of 0.25% on CIFAR-10 and 3.01% on CIFAR 100. Under these conditions, the final error rate is then 2.31% for CIFAR-10 and 12.19% for CIFAR-100.<br /> # Comparison with the state-of-the-arts, PyramidNet + ShakeDrop gives an improvement of 0.25% on CIFAR-10 than ResNeXt + Shake-Shake + Cutout, PyramidNet + ShakeDrop gives an improvement of 2.85% on CIFAR-100 than Coupled Ensemble.<br /> <br /> =Implementation details=<br /> <br /> '''CIFAR-10/100 datasets'''<br /> <br /> All the images in these datasets were color normalized and then horizontally flipped with a probability of 50%. All of then then were zero padded to have a dimentionality of 40 by 40 pixels.<br /> <br /> <br /> =Conclusion=<br /> The paper proposes a new form of regularization that is an extension of &quot;Shake-Shake&quot; regularization [Gastaldi, 2017]. The original &quot;shake-shake&quot; proposes using two residual paths adding to the same output, and during training, considering different randomly selected convex combinations of the two paths (while using an equally weighted combination at test time). This paper contends that this requires additional memory, and attempts to achieve similar regularization with a single path. To do so, they train a network with a single residual path, where the residual is included without attenuation in some cases with some fixed probability, and attenuated randomly (or even inverted) in others. The paper contends that this achieves superior performance than choosing simply a random attenuation for every sample (although, this can be seen as choosing an attenuation under a distribution with some fixed probability mass.<br /> <br /> Their stochastic regularization method, ShakeDrop, which outperforms previous state of the art methods while maintaining similar memory efficiency. It demonstrates that heavily perturbing a network can help to overcome issues with overfitting. It is also an effective way to regularize residual networks for image classification. The method was tested by CIFAR-10/100 and Tiny ImageNet datasets and showed great performance.<br /> <br /> =Critique=<br /> <br /> The novelty of this paper is low as pointed out by the reviewers. Also, there is a confusion whether or not the results could be replicated as &lt;math&gt;\alpha&lt;/math&gt; and &lt;math&gt;\beta&lt;/math&gt; are choosen randomly. The proposed ShakeDrop regularization is essentially a combination of the PyramidDrop and Shake-Shake regularization. The most surprising part is that the forward weight can be negative thus inverting the output of a convolution. The mathematical justification for ShakeDrop regularization is limited, relying on intuition and empirical evidence instead.<br /> As pointed out from the above, the method basically relies heavily on the intuition. This means that the performance of the algorithm can not been extended beyond the CIFAR dataset and can vary a lot depending on the characteristics of data sets that users are performing, with some exaggeration. However, the performance is still impressive since it performs better than known algorithms. It is not clear as to how the proposed technique would work with a non-residual architecture.<br /> <br /> =References=<br /> [Yamada et al., 2018] Yamada Y, Iwamura M, Kise K. ShakeDrop regularization. arXiv preprint arXiv:1802.02375. 2018 Feb 7.<br /> <br /> [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, 2016.<br /> <br /> [Zagoruyko &amp; Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proc. BMVC, 2016.<br /> <br /> [Han et al., 2017] Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In Proc. CVPR, 2017a.<br /> <br /> [Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proc. CVPR, 2017.<br /> <br /> [Huang et al., 2016] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. arXiv preprint arXiv:1603.09382v3, 2016.<br /> <br /> [Gastaldi, 2017] Xavier Gastaldi. Shake-shake regularization. arXiv preprint arXiv:1705.07485v2, 2017.<br /> <br /> [Loshilov &amp; Hutter, 2016] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.<br /> <br /> [DeVries &amp; Taylor, 2017b] Terrance DeVries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017b.<br /> <br /> [Zhong et al., 2017] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. arXiv preprint arXiv:1708.04896, 2017.<br /> <br /> [Dutt et al., 2017] Anuvabh Dutt, Denis Pellerin, and Georges Qunot. Coupled ensembles of neural networks. arXiv preprint 1709.06053v1, 2017.<br /> <br /> [Veit et al., 2016] Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in Neural Information Processing Systems 29, 2016.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=stat946F18/Autoregressive_Convolutional_Neural_Networks_for_Asynchronous_Time_Series&diff=41881 stat946F18/Autoregressive Convolutional Neural Networks for Asynchronous Time Series 2018-11-29T17:21:26Z <p>C9sharma: /* Datasets */</p> <hr /> <div>This page is a summary of the paper &quot;[http://proceedings.mlr.press/v80/binkowski18a/binkowski18a.pdf Autoregressive Convolutional Neural Networks for Asynchronous Time Series]&quot; by Mikołaj Binkowski, Gautier Marti, Philippe Donnat. It was published at ICML in 2018. The code for this paper is provided [https://github.com/mbinkowski/nntimeseries here].<br /> <br /> =Introduction=<br /> In this paper, the authors propose a deep convolutional network architecture called Significance-Offset Convolutional Neural Network for regression of multivariate asynchronous time series. The model is inspired by standard autoregressive (AR) models and gating systems used in recurrent neural networks. The model is evaluated on various time series data including:<br /> # Hedge fund proprietary dataset of over 2 million quotes for a credit derivative index, <br /> # An artificially generated noisy auto-regressive series, <br /> # A UCI household electricity consumption dataset. <br /> <br /> This paper focused on time series that have multivariate and noisy signals, especially financial data. Financial time series is challenging to predict due to their low signal-to-noise ratio and heavy-tailed distributions. For example, the same signal (e.g. price of a stock) is obtained from different sources (e.g. financial news, an investment bank, financial analyst etc.) asynchronously. Each source may have a different bias or noise. ([[Media: Junyi1.png|Figure 1]]) The investment bank with more clients can update their information more precisely than the investment bank with fewer clients, which means the significance of each past observations may depend on other factors that change in time. Therefore, the traditional econometric models such as AR, VAR (Vector Autoregressive Model), VARMA (Vector Autoregressive Moving Average Model)  might not be sufficient. However, their relatively good performance could allow us to combine such linear econometric models with deep neural networks that can learn highly nonlinear relationships. This model is inspired by the gating mechanism which is successful in RNNs and Highway Networks.<br /> <br /> Time series forecasting is focused on modeling the predictors of future values of time series given their past. As in many cases the relationship between past and future observations is not deterministic, this amounts to expressing the conditional probability distribution as a function of the past observations: The time series forecasting problem can be expressed as a conditional probability distribution below,<br /> &lt;div style=&quot;text-align: center;&quot;&gt;&lt;math&gt;p(X_{t+d}|X_t,X_{t-1},...) = f(X_t,X_{t-1},...)&lt;/math&gt;&lt;/div&gt;<br /> This forecasting problem has been approached almost independently by econometrics and machine learning communities. In this paper, the authors focus on modeling the predictors of future values of time series given their past values. <br /> <br /> The reasons that financial time series are particularly challenging:<br /> * Low signal-to-noise ratio and heavy-tailed distributions.<br /> * Being observed different sources (e.g. financial news, analysts, portfolio managers in hedge funds, market-makers in investment banks) in asynchronous moments of time. Each of these sources may have a different bias and noise with respect to the original signal that needs to be recovered.<br /> * Data sources are usually strongly correlated and lead-lag relationships are possible (e.g. a market-maker with more clients can update its view more frequently and precisely than one with fewer clients). <br /> * The significance of each of the available past observations might be dependent on some other factors that can change in time. Hence, the traditional econometric models such as AR, VAR, VARMA might not be sufficient.<br /> <br /> The predictability of financial dataset still remains an open problem and is discussed in various publications .<br /> <br /> [[File:Junyi1.png | 500px|thumb|center|Figure 1: Quotes from four different market participants (sources) for the same credit default swaps (CDS) throughout one day. Each trader displays from time to time the prices for which he offers to buy (bid) and sell (ask) the underlying CDS. The filled area marks the difference between the best sell and buy offers (spread) at each time.]]<br /> <br /> The paper also provides empirical evidence that their model which combines linear models with deep learning models could perform better than just DL models like CNN, LSTMs and Phased LSTMs.<br /> <br /> =Related Work=<br /> ===Time series forecasting===<br /> From recent proceedings in main machine learning venues i.e. ICML, NIPS, AISTATS, UAI, we can notice that time series are often forecast using Gaussian processes[3,4], especially for irregularly sampled time series. Though still largely independent, combined models have started to appear, for example, the Gaussian Copula Process Volatility model. For this paper, the authors use coupling AR models and neural networks to achieve such combined models.<br /> <br /> Although deep neural networks have been applied into many fields and produced satisfactory results, there still is little literature on deep learning for time series forecasting. More recently, the papers include Sirignano (2016) that used 4-layer perceptrons in modeling price change distributions in Limit Order Books, and Borovykh et al. (2017) who applied more recent WaveNet architecture to several short univariate and bivariate time-series (including financial ones). Heaton et al. (2016) claimed to use autoencoders with a single hidden layer to compress multivariate financial data. Neil et al. (2016) presented augmentation of LSTM architecture suitable for asynchronous series, which stimulates learning dependencies of different frequencies through time gate. <br /> <br /> In this paper, the authors examine the capabilities of several architectures (CNN, residual network, multi-layer LSTM, and phase LSTM) on AR-like artificial asynchronous and noisy time series, household electricity consumption dataset, and on real financial data from the credit default swap market with some inefficiencies.<br /> <br /> ====AR Model====<br /> <br /> An autoregressive (AR) model describes the next value in a time-series as a combination of previous values, scaling factors, a bias, and noise [https://onlinecourses.science.psu.edu/stat501/node/358/ (source)]. For a p-th order (relating the current state to the p last states), the equation of the model is:<br /> <br /> &lt;math&gt; X_t = c + \sum_{i=1}^p \varphi_i X_{t-i}+ \varepsilon_t \,&lt;/math&gt; [https://en.wikipedia.org/wiki/Autoregressive_model#Definition (equation source)]<br /> <br /> With parameters/coefficients &lt;math&gt;\varphi_i&lt;/math&gt;, constant &lt;math&gt;c&lt;/math&gt;, and noise &lt;math&gt;\varepsilon_t&lt;/math&gt; This can be extended to vector form to create the VAR model mentioned in the paper.<br /> <br /> ===Gating and weighting mechanisms===<br /> Gating mechanisms for neural networks has ability to overcome the problem of vanishing gradient, and can be expressed as &lt;math display=&quot;inline&quot;&gt;f(x)=c(x) \otimes \sigma(x)&lt;/math&gt;, where &lt;math&gt;f&lt;/math&gt; is the output function, &lt;math&gt;c&lt;/math&gt; is a &quot;candidate output&quot; (a nonlinear function of &lt;math&gt;x&lt;/math&gt;), &lt;math&gt;\otimes&lt;/math&gt; is an element-wise matrix product, and &lt;math&gt;\sigma : \mathbb{R} \rightarrow [0,1] &lt;/math&gt; is a sigmoid nonlinearity that controls the amount of output passed to the next layer. Different composition of functions of the same type as described above have proven to be an essential ingredient in popular recurrent architecture such as LSTM and GRU.<br /> <br /> The main purpose of the proposed gating system is to weight the outputs of the intermediate layers within neural networks, and is most closely related to softmax gating used in MuFuRu(Multi-Function Recurrent Unit), i.e.<br /> &lt;math display=&quot;inline&quot;&gt; f(x) = \sum_{l=1}^L p^l(x) \otimes f^l(x)\text{,}\ p(x)=\text{softmax}(\widehat{p}(x)), &lt;/math&gt;, where &lt;math&gt;(f^l)_{l=1}^L &lt;/math&gt;are candidate outputs (composition operators in MuFuRu), &lt;math&gt;(\widehat{p}^l)_{l=1}^L &lt;/math&gt;are linear functions of inputs. <br /> <br /> This idea is also successfully used in attention networks such as image captioning and machine translation. In this paper, the proposed method is similar as the separate inputs (time series steps in this case) are weighted in accordance with learned functions of these inputs. The difference is that the functions are being modeled using multi-layer CNNs. Another difference is that the proposed method is not using recurrent layers, which enables the network to remember parts of the sentence/image already translated/described.<br /> <br /> =Motivation=<br /> There are mainly five motivations that are stated in the paper by the authors:<br /> #The forecasting problem in this paper has been done almost independently by econometrics and machine learning communities. Unlike in machine learning, research in econometrics is more likely to explain variables rather than improving out-of-sample prediction power. These models tend to 'over-fit' on financial time series, their parameters are unstable and have poor performance on out-of-sample prediction.<br /> #It is difficult for the learning algorithms to deal with time series data where the observations have been made irregularly. Although Gaussian processes provide a useful theoretical framework that is able to handle asynchronous data, they are not suitable for financial datasets, which often follow heavy-tailed distribution .<br /> #Predictions of autoregressive time series may involve highly nonlinear functions if sampled irregularly. For AR time series with higher order and have more past observations, the expectation of it &lt;math display=&quot;inline&quot;&gt;\mathbb{E}[X(t)|{X(t-m), m=1,...,M}]&lt;/math&gt; may involve more complicated functions that in general may not allow closed-form expression.<br /> #In practice, the dimensions of multivariate time series are often observed separately and asynchronously, such series at fixed frequency may lead to lose information or enlarge the dataset, which is shown in Figure 2(a). Therefore, the core of the proposed architecture SOCNN represents separate dimensions as a single one with dimension and duration indicators as additional features(Figure 2(b)).<br /> #Given a series of pairs of consecutive input values and corresponding durations, &lt;math display=&quot;inline&quot;&gt; x_n = (X(t_n),t_n-t_{n-1}) &lt;/math&gt;. One may expect that LSTM may memorize the input values in each step and weight them at the output according to the duration, but this approach may lead to an imbalance between the needs for memory and for linearity. The weights that are assigned to the memorized observations potentially require several layers of nonlinearity to be computed properly, while past observations might just need to be memorized as they are.<br /> <br /> [[File:Junyi2.png | 550px|thumb|center|Figure 2: (a) Fixed sampling frequency and its drawbacks; keep- ing all available information leads to much more datapoints. (b) Proposed data representation for the asynchronous series. Consecutive observations are stored together as a single value series, regardless of which series they belong to; this information, however, is stored in indicator features, alongside durations between observations.]]<br /> <br /> =Model Architecture=<br /> Suppose there exists a multivariate time series &lt;math display=&quot;inline&quot;&gt;(x_n)_{n=0}^{\infty} \subset \mathbb{R}^d &lt;/math&gt;, we want to predict the conditional future values of a subset of elements of &lt;math&gt;x_n&lt;/math&gt;<br /> &lt;div style=&quot;text-align: center;&quot;&gt;&lt;math&gt;y_n = \mathbb{E} [x_n^I | \{x_{n-m}, m=1,2,...\}], &lt;/math&gt;&lt;/div&gt;<br /> where &lt;math&gt; I=\{i_1,i_2,...i_{d_I}\} \subset \{1,2,...,d\} &lt;/math&gt; is a subset of features of &lt;math&gt;x_n&lt;/math&gt;.<br /> <br /> Let &lt;math&gt; \textbf{x}_n^{-M} = (x_{n-m})_{m=1}^M &lt;/math&gt;. <br /> <br /> The estimator of &lt;math&gt;y_n&lt;/math&gt; can be expressed as:<br /> &lt;div style=&quot;text-align: center;&quot;&gt;&lt;math&gt;\tilde{y}_n = \sum_{m=1}^M [F(\textbf{x}_n^{-M}) \otimes \sigma(S(\textbf{x}_n^{-M}))].,_m ,&lt;/math&gt;&lt;/div&gt;<br /> The estimate is the summation of the columns of the matrix in bracket. Here<br /> #&lt;math&gt;F,S : \mathbb{R}^{d \times M} \rightarrow \mathbb{R}^{d_I \times M}&lt;/math&gt; are neural networks. <br /> #* &lt;math&gt;S&lt;/math&gt; is a fully convolutional network which is composed of convolutional layers only. <br /> #* &lt;math display=&quot;inline&quot;&gt;F(\textbf{x}_n^{-M}) = W \otimes [\text{off}(x_{n-m}) + x_{n-m}^I)]_{m=1}^M &lt;/math&gt; <br /> #** &lt;math&gt; W \in \mathbb{R}^{d_I \times M}&lt;/math&gt; <br /> #** &lt;math&gt; \text{off}: \mathbb{R}^d \rightarrow \mathbb{R}^{d_I} &lt;/math&gt; is a multilayer perceptron.<br /> <br /> #&lt;math&gt;\sigma&lt;/math&gt; is a normalized activation function independent at each row, i.e. &lt;math display=&quot;inline&quot;&gt; \sigma ((a_1^T, ..., a_{d_I}^T)^T)=(\sigma(a_1)^T,..., \sigma(a_{d_I})^T)^T &lt;/math&gt;<br /> #* for any &lt;math&gt;a_{i} \in \mathbb{R}^{M}&lt;/math&gt;<br /> #* and &lt;math&gt;\sigma &lt;/math&gt; is defined such that &lt;math&gt;\sigma(a)^{T} \mathbf{1}_{M}=1&lt;/math&gt; for any &lt;math&gt;a \in \mathbb{R}^M&lt;/math&gt;.<br /> # &lt;math&gt;\otimes&lt;/math&gt; is element-wise matrix multiplication (also known as Hadamard matrix multiplication).<br /> #&lt;math&gt;A.,_m&lt;/math&gt; denotes the m-th column of a matrix A.<br /> <br /> Since &lt;math&gt;\sum_{m=1}^M W.,_m=W\cdot(1,1,...,1)^T&lt;/math&gt; and &lt;math&gt;\sum_{m=1}^M S.,_m=S\cdot(1,1,...,1)^T&lt;/math&gt;, we can express &lt;math&gt;\hat{y}_n&lt;/math&gt; as:<br /> &lt;div style=&quot;text-align: center;&quot;&gt;&lt;math&gt;\hat{y}_n = \sum_{m=1}^M W.,_m \otimes (off(x_{n-m}) + x_{n-m}^I) \otimes \sigma(S.,_m(\textbf{x}_n^{-M}))&lt;/math&gt;&lt;/div&gt;<br /> This is the proposed network, Significance-Offset Convolutional Neural Network, &lt;math&gt;\text{off}&lt;/math&gt; and &lt;math&gt;S&lt;/math&gt; in the equation are corresponding to Offset and Significance in the name respectively.<br /> Figure 3 shows the scheme of network.<br /> <br /> [[File:Junyi3.png | 600px|thumb|center|Figure 3: A scheme of the proposed SOCNN architecture. The network preserves the time-dimension up to the top layer, while the number of features per timestep (filters) in the hidden layers is custom. The last convolutional layer, however, has the number of filters equal to dimension of the output. The Weighting frame shows how outputs from offset and significance networks are combined in accordance with Eq. of &lt;math&gt;\hat{y}_n&lt;/math&gt;.]]<br /> <br /> The form of &lt;math&gt;\tilde{y}_n&lt;/math&gt; ensures the separation of the temporal dependence (obtained in weights &lt;math&gt;W_m&lt;/math&gt;). &lt;math&gt;S&lt;/math&gt;, which represents the local significance of observations, is determined by its filters which capture local dependencies and are independent of the relative position in time, and the predictors &lt;math&gt;\text{off}(x_{n-m})&lt;/math&gt; are completely independent of position in time. An adjusted single regressor for the target variable is provided by each past observation through the offset network. Since in asynchronous sampling procedure, consecutive values of x come from different signals and might be heterogeneous, therefore adjustment of offset network is important. In addition, significance network provides data-dependent weight for each regressor and sums them up in an autoregressive manner.<br /> <br /> ===Relation to asynchronous data===<br /> One common problem of time series is that durations are varying between consecutive observations, the paper states two ways to solve this problem<br /> #Data preprocessing: aligning the observations at some fixed frequency e.g. duplicating and interpolating observations as shown in Figure 2(a). However, as mentioned in the figure, this approach will tend to loss of information and enlarge the size of the dataset and model complexity.<br /> #Add additional features: Treating the duration or time of the observations as additional features, it is the core of SOCNN, which is shown in Figure 2(b).<br /> <br /> ===Loss function===<br /> The L2 error is a natural loss function for the estimators of expected value: &lt;math&gt;L^2(y,y')=||y-y'||^2&lt;/math&gt;<br /> <br /> The output of the offset network is series of separate predictors of changes between corresponding observations &lt;math&gt;x_{n-m}^I&lt;/math&gt; and the target value&lt;math&gt;y_n&lt;/math&gt;, this is the reason why we use auxiliary loss function, which equals to mean squared error of such intermediate predictions:<br /> &lt;div style=&quot;text-align: center;&quot;&gt;&lt;math&gt;L^{aux}(\textbf{x}_n^{-M}, y_n)=\frac{1}{M} \sum_{m=1}^M ||off(x_{n-m}) + x_{n-m}^I -y_n||^2 &lt;/math&gt;&lt;/div&gt;<br /> The total loss for the sample &lt;math&gt; \textbf{x}_n^{-M},y_n) &lt;/math&gt; is then given by:<br /> &lt;div style=&quot;text-align: center;&quot;&gt;&lt;math&gt;L^{tot}(\textbf{x}_n^{-M}, y_n)=L^2(\widehat{y}_n, y_n)+\alpha L^{aux}(\textbf{x}_n^{-M}, y_n)&lt;/math&gt;&lt;/div&gt;<br /> where &lt;math&gt;\widehat{y}_n&lt;/math&gt; was mentioned before, &lt;math&gt;\alpha \geq 0&lt;/math&gt; is a constant.<br /> <br /> =Experiments=<br /> The paper evaluated SOCNN architecture on three datasets: artificially generated datasets, [https://archive.ics.uci.edu/ml/datasets/Individual+household+electric+power+consumption household electric power consumption dataset], and the financial dataset of bid/ask quotes provided by several market participants active in the credit derivatives market. Comparing its performance with simple CNN, single and multiplayer LSTM and 25-layer ResNet. Apart from the evaluation of the SOCNN architecture, the paper also discussed the impact of network components such as auxiliary<br /> loss and the depth of the offset sub-network. The code and datasets are available at [https://github.com/mbinkowski/nntimeseries here]<br /> <br /> ==Datasets==<br /> Artificial data: They generated 4 artificial series, &lt;math&gt; X_{K \times N}&lt;/math&gt;, where &lt;math&gt;K \in \{16,64\} &lt;/math&gt;. Therefore there is a synchronous and an asynchronous series for each K value. Note that a series with K sources is K + 1-dimensional in synchronous case and K + 2-dimensional in asynchronous case. The base series in all processes was a stationary AR(10) series. Although that series has the true order of 10, in the experimental setting the input data included past 60 observations. The rationale behind that is twofold: not only is the data observed in irregular random times but also in real–life problems the order of the model is unknown.<br /> <br /> Electricity data: This UCI dataset contains 7 different features excluding date and time. The features include global active power, global reactive power, voltage, global intensity, sub-metering 1, sub-metering 2 and sub-metering 3, recorded every minute for 47 months. The data has been altered so that one observation contains only one value of 7 features, while durations between consecutive observations are ranged from 1 to 7 minutes. The goal is to predict all 7 features for the next time step.<br /> <br /> Non-anonymous quotes: The dataset contains 2.1 million quotes from 28 different sources from different market participants such as analysts, banks etc. Each quote is characterized by 31 features: the offered price, 28 indicators of the quoting source, the direction indicator (the quote refers to either a buy or a sell offer) and duration from the previous quote. For each source and direction, we want to predict the next quoted price from this given source and direction considering the last 60 quotes.<br /> <br /> [[File:async.png | 520px|center|]]<br /> <br /> ==Training details==<br /> They applied grid search on some hyperparameters in order to get the significance of its components. The hyperparameters include the offset sub-network's depth and the auxiliary weight &lt;math&gt;\alpha&lt;/math&gt;. For offset sub-network's depth, they use 1, 10,1 for artificial, electricity and quotes dataset respectively; and they compared the values of &lt;math&gt;\alpha&lt;/math&gt; in {0,0.1,0.01}.<br /> <br /> They chose LeakyReLU as activation function for all networks:<br /> &lt;div style=&quot;text-align: center;&quot;&gt;&lt;math&gt;\sigma^{LeakyReLU}(x) = x&lt;/math&gt; if &lt;math&gt;x\geq 0&lt;/math&gt;, and &lt;math&gt;0.1x&lt;/math&gt; otherwise &lt;/div&gt;<br /> They use the same number of layers, same stride and similar kernel size structure in CNN. In each trained CNN, they applied max pooling with the pool size of 2 every 2 convolutional layers.<br /> <br /> Table 1 presents the configuration of network hyperparameters used in comparison<br /> <br /> [[File:Junyi4.png | 520px|center|]]<br /> <br /> ===Network Training===<br /> The training and validation data were sampled randomly from the first 80% of timesteps in each series, with ratio of 3 to 1. The remaining 20% of data was used as a test set.<br /> <br /> All models were trained using Adam optimizer because the authors found that its rate of convergence was much faster than standard Stochastic Gradient Descent in early tests.<br /> <br /> They used a batch size of 128 for artificial and electricity data, and 256 for quotes dataset, and applied batch normalization between each convolution and the following activation. <br /> <br /> At the beginning of each epoch, the training samples were randomly sampled. To prevent overfitting, they applied dropout and early stopping.<br /> <br /> Weights were initialized using the normalized uniform procedure proposed by Glorot &amp; Bengio (2010).<br /> <br /> The authors carried out the experiments on Tensorflow and Keras and used different GPU to optimize the model for different datasets.<br /> <br /> ==Results==<br /> Table 2 shows all results performed from all datasets.<br /> [[File:Junyi5.png | 800px|center|]]<br /> We can see that SOCNN outperforms in all asynchronous artificial, electricity and quotes datasets. For synchronous data, LSTM might be slightly better, but SOCNN almost has the same results with LSTM. Phased LSTM and ResNet have performed really bad on artificial asynchronous dataset and quotes dataset respectively. Notice that having more than one layer of offset network would have negative impact on results. Also, the higher weights of auxiliary loss(&lt;math&gt;\alpha&lt;/math&gt;considerably improved the test error on asynchronous dataset, see Table 3. However, for other datasets, its impact was negligible.<br /> [[File:Junyi6.png | 480px|center|]]<br /> In general, SOCNN has significantly lower variance of the test and validation errors, especially in the early stage of the training process and for quotes dataset. This effect can be seen in the learning curves for Asynchronous 64 artificial dataset presented in Figure 5.<br /> [[File:Junyi7.png | 500px|thumb|center|Figure 5: Learning curves with different auxiliary weights for SOCNN model trained on Asynchronous 64 dataset. The solid lines indicate the test error while the dashed lines indicate the training error.]]<br /> <br /> Finally, we want to test the robustness of the proposed model SOCNN, adding noise terms to asynchronous 16 dataset and check how these networks perform. The result is shown in Figure 6.<br /> [[File:Junyi8.png | 600px|thumb|center|Figure 6: Experiment comparing robustness of the considered networks for Asynchronous 16 dataset. The plots show how the error would change if an additional noise term was added to the input series. The dotted curves show the total significance and average absolute offset (not to scale) outputs for the noisy observations. Interestingly, the significance of the noisy observations increases with the magnitude of noise; i.e. noisy observations are far from being discarded by SOCNN.]]<br /> From Figure 6, the purple lines and green lines seem to stay at the same position in training and testing process. SOCNN and single-layer LSTM are most robust and least prone to overfitting comparing to other networks.<br /> <br /> =Conclusion and Discussion=<br /> In this paper, the authors have proposed a new architecture called Significance-Offset Convolutional Neural Network, which combines AR-like weighting mechanism and convolutional neural network. This new architecture is designed for high-noise asynchronous time series and achieves outperformance in forecasting several asynchronous time series compared to popular convolutional and recurrent networks. <br /> <br /> The SOCNN can be extended further by adding intermediate weighting layers of the same type in the network structure. Another possible extension but needs further empirical studies is that we consider not just &lt;math&gt;1 \times 1&lt;/math&gt; convolutional kernels on the offset sub-network. Also, this new architecture might be tested on other real-life datasets with relevant characteristics in the future, especially on econometric datasets and more generally for time series (stochastic processes) regression.<br /> <br /> =Critiques=<br /> #The paper is most likely an application paper, and the proposed new architecture shows improved performance over baselines in the asynchronous time series.<br /> #The quote data cannot be reached as they are proprietary. Also, only two datasets available.<br /> #The 'Significance' network was described as critical to the model in paper, but they did not show how the performance of SOCNN with respect to the significance network.<br /> #The transform of the original data to asynchronous data is not clear.<br /> #The experiments on the main application are not reproducible because the data is proprietary.<br /> #The way that train and test data were split is unclear. This could be important in the case of the financial data set.<br /> #Although the auxiliary loss function was mentioned as an important part, the advantages of it was not too clear in the paper. Maybe it is better that the paper describes a little more about its effectiveness.<br /> #It was not mentioned clearly in the paper whether the model training was done on a rolling basis for time series forecasting.<br /> #The noise term used in section 5's model robustness analysis uses evenly distributed noise (see Appendix B). While the analysis is a good start, analysis with different noise distributions would make the findings more generalizable.<br /> #The paper uses financial/economic data as one of its testing data set. Instead of comparing neural network models such as CNN which is known to work badly on time series data, it would be much better if the author compared to well-known econometric time series models such as GARCH and VAR.<br /> <br /> =References=<br />  Hamilton, J. D. Time series analysis, volume 2. Princeton university press Princeton, 1994. <br /> <br />  Fama, E. F. Efficient capital markets: A review of theory and empirical work. The journal of Finance, 25(2):383–417, 1970.<br /> <br />  Petelin, D., Sˇindela ́ˇr, J., Pˇrikryl, J., and Kocijan, J. Financial modeling using gaussian process models. In Intelligent Data Acquisition and Advanced Computing Systems (IDAACS), 2011 IEEE 6th International Conference on, volume 2, pp. 672–677. IEEE, 2011.<br /> <br />  Tobar, F., Bui, T. D., and Turner, R. E. Learning stationary time series using Gaussian processes with nonparametric kernels. In Advances in Neural Information Processing Systems, pp. 3501–3509, 2015.<br /> <br />  Hwang, Y., Tong, A., and Choi, J. Automatic construction of nonparametric relational regression models for multiple time series. In Proceedings of the 33rd International Conference on Machine Learning, 2016.<br /> <br />  Wilson, A. and Ghahramani, Z. Copula processes. In Advances in Neural Information Processing Systems, pp. 2460–2468, 2010.<br /> <br />  Sirignano, J. Extended abstract: Neural networks for limit order books, February 2016.<br /> <br />  Borovykh, A., Bohte, S., and Oosterlee, C. W. Conditional time series forecasting with convolutional neural networks, March 2017.<br /> <br />  Heaton, J. B., Polson, N. G., and Witte, J. H. Deep learning in finance, February 2016.<br /> <br />  Neil, D., Pfeiffer, M., and Liu, S.-C. Phased lstm: Accelerating recurrent network training for long or event-based sequences. In Advances In Neural Information Process- ing Systems, pp. 3882–3890, 2016.<br /> <br />  Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling, December 2014.<br /> <br />  Weissenborn, D. and Rockta ̈schel, T. MuFuRU: The Multi-Function recurrent unit, June 2016.<br /> <br />  Cho, K., Courville, A., and Bengio, Y. Describing multi- media content using attention-based Encoder–Decoder networks. IEEE Transactions on Multimedia, 17(11): 1875–1886, July 2015. ISSN 1520-9210.<br /> <br />  Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural net- works. In In Proceedings of the International Con- ference on Artificial Intelligence and Statistics (AIS- TATSaˆ10). Society for Artificial Intelligence and Statistics, 2010.</div> C9sharma http://wiki.math.uwaterloo.ca/statwiki/index.php?title=stat946F18/Autoregressive_Convolutional_Neural_Networks_for_Asynchronous_Time_Series&diff=41880 stat946F18/Autoregressive Convolutional Neural Networks for Asynchronous Time Series 2018-11-29T17:19:32Z <p>C9sharma: /* Introduction */</p> <hr /> <div>This page is a summary of the paper &quot;[http://proceedings.mlr.press/v80/binkowski18a/binkowski18a.pdf Autoregressive Convolutional Neural Networks for Asynchronous Time Series]&quot; by Mikołaj Binkowski, Gautier Marti, Philippe Donnat. It was published at ICML in 2018. The code for this paper is provided [https://github.com/mbinkowski/nntimeseries here].<br /> <br /> =Introduction=<br /> In this paper, the authors propose a deep convolutional network architecture called Significance-Offset Convolutional Neural Network for regression of multivariate asynchronous time series. The model is inspired by standard autoregressive (AR) models and gating systems used in recurrent neural networks. The model is evaluated on various time series data including:<br /> # Hedge fund proprietary dataset of over 2 million quotes for a credit derivative index, <br /> # An artificially generated noisy auto-regressive series, <br /> # A UCI household electricity consumption dataset. <br /> <br /> This paper focused on time series that have multivariate and noisy signals, especially financial data. Financial time series is challenging to predict due to their low signal-to-noise ratio and heavy-tailed distributions. For example, the same signal (e.g. price of a stock) is obtained from different sources (e.g. financial news, an investment bank, financial analyst etc.) asynchronously. Each source may have a different bias or noise. ([[Media: Junyi1.png|Figure 1]]) The investment bank with more clients can update their information more precisely than the investment bank with fewer clients, which means the significance of each past observations may depend on other factors that change in time. Therefore, the traditional econometric models such as AR, VAR (Vector Autoregressive Model), VARMA (Vector Autoregressive Moving Average Model)  might not be sufficient. However, their relatively good performance could allow us to combine such linear econometric models with deep neural networks that can learn highly nonlinear relationships. This model is inspired by the gating mechanism which is successful in RNNs and Highway Networks.<br /> <br /> Time series forecasting is focused on modeling the predictors of future values of time series given their past. As in many cases the relationship between past and future observations is not deterministic, this amounts to expressing the conditional probability distribution as a function of the past observations: The time series forecasting problem can be expressed as a conditional probability distribution below,<br /> &lt;div style=&quot;text-align: center;&quot;&gt;&lt;math&gt;p(X_{t+d}|X_t,X_{t-1},...) = f(X_t,X_{t-1},...)&lt;/math&gt;&lt;/div&gt;<br /> This forecasting problem has been approached almost independently by econometrics and machine learning communities. In this paper, the authors focus on modeling the predictors of future values of time series given their past values. <br /> <br /> The reasons that financial time series are particularly challenging:<br /> * Low signal-to-noise ratio and heavy-tailed distributions.<br /> * Being observed different sources (e.g. financial news, analysts, portfolio managers in hedge funds, market-makers in investment banks) in asynchronous moments of time. Each of these sources may have a different bias and noise with respect to the original signal that needs to be recovered.<br /> * Data sources are usually strongly correlated and lead-lag relationships are possible (e.g. a market-maker with more clients can update its view more frequently and precisely than one with fewer clients). <br /> * The significance of each of the available past observations might be dependent on some other factors that can change in time. Hence, the traditional econometric models such as AR, VAR, VARMA might not be sufficient.<br /> <br /> The predictability of financial dataset still remains an open problem and is discussed in various publications .<br /> <br /> [[File:Junyi1.png | 500px|thumb|center|Figure 1: Quotes from four different market participants (sources) for the same credit default swaps (CDS) throughout one day. Each trader displays from time to time the prices for which he offers to buy (bid) and sell (ask) the underlying CDS. The filled area marks the difference between the best sell and buy offers (spread) at each time.]]<br /> <br /> The paper also provides empirical evidence that their model which combines linear models with deep learning models could perform better than just DL models like CNN, LSTMs and Phased LSTMs.<br /> <br /> =Related Work=<br /> ===Time series forecasting===<br /> From recent proceedings in main machine learning venues i.e. ICML, NIPS, AISTATS, UAI, we can notice that time series are often forecast using Gaussian processes[3,4], especially for irregularly sampled time series. Though still largely independent, combined models have started to appear, for example, the Gaussian Copula Process Volatility model. For this paper, the authors use coupling AR models and neural networks to achieve such combined models.<br /> <br /> Although deep neural networks have been applied into many fields and produced satisfactory results, there still is little literature on deep learning for time series forecasting. More recently, the papers include Sirignano (2016) that used 4-layer perceptrons in modeling price change distributions in Limit Order Books, and Borovykh et al. (2017) who applied more recent WaveNet architecture to several short univariate and bivariate time-series (including financial ones). Heaton et al. (2016) claimed to use autoencoders with a single hidden layer to compress multivariate financial data. Neil et al. (2016) presented augmentation of LSTM architecture suitable for asynchronous series, which stimulates learning dependencies of different frequencies through time gate. <br /> <br /> In this paper, the authors examine the capabilities of several architectures (CNN, residual network, multi-layer LSTM, and phase LSTM) on AR-like artificial asynchronous and noisy time series, household electricity consumption dataset, and on real financial data from the credit default swap market with some inefficiencies.<br /> <br /> ====AR Model====<br /> <br /> An autoregressive (AR) model describes the next value in a time-series as a combination of previous values, scaling factors, a bias, and noise [https://onlinecourses.science.psu.edu/stat501/node/358/ (source)]. For a p-th order (relating the current state to the p last states), the equation of the model is:<br /> <br /> &lt;math&gt; X_t = c + \sum_{i=1}^p \varphi_i X_{t-i}+ \varepsilon_t \,&lt;/math&gt; [https://en.wikipedia.org/wiki/Autoregressive_model#Definition (equation source)]<br /> <br /> With parameters/coefficients &lt;math&gt;\varphi_i&lt;/math&gt;, constant &lt;math&gt;c&lt;/math&gt;, and noise &lt;math&gt;\varepsilon_t&lt;/math&gt; This can be extended to vector form to create the VAR model mentioned in the paper.<br /> <br /> ===Gating and weighting mechanisms===<br /> Gating mechanisms for neural networks has ability to overcome the problem of vanishing gradient, and can be expressed as &lt;math display=&quot;inline&quot;&gt;f(x)=c(x) \otimes \sigma(x)&lt;/math&gt;, where &lt;math&gt;f&lt;/math&gt; is the output function, &lt;math&gt;c&lt;/math&gt; is a &quot;candidate output&quot; (a nonlinear function of &lt;math&gt;x&lt;/math&gt;), &lt;math&gt;\otimes&lt;/math&gt; is an element-wise matrix product, and &lt;math&gt;\sigma : \mathbb{R} \rightarrow [0,1] &lt;/math&gt; is a sigmoid nonlinearity that controls the amount of output passed to the next layer. Different composition of functions of the same type as described above have proven to be an essential ingredient in popular recurrent architecture such as LSTM and GRU.<br /> <br /> The main purpose of the proposed gating system is to weight the outputs of the intermediate layers within neural networks, and is most closely related to softmax gating used in MuFuRu(Multi-Function Recurrent Unit), i.e.<br /> &lt;math display=&quot;inline&quot;&gt; f(x) = \sum_{l=1}^L p^l(x) \otimes f^l(x)\text{,}\ p(x)=\text{softmax}(\widehat{p}(x)), &lt;/math&gt;, where &lt;math&gt;(f^l)_{l=1}^L &lt;/math&gt;are candidate outputs (composition operators in MuFuRu), &lt;math&gt;(\widehat{p}^l)_{l=1}^L &lt;/math&gt;are linear functions of inputs. <br /> <br /> This idea is also successfully used in attention networks such as image captioning and machine translation. In this paper, the proposed method is similar as the separate inputs (time series steps in this case) are weighted in accordance with learned functions of these inputs. The difference is that the functions are being modeled using multi-layer CNNs. Another difference is that the proposed method is not using recurrent layers, which