http://wiki.math.uwaterloo.ca/statwiki/api.php?action=feedcontributions&user=Z42qin&feedformat=atom statwiki - User contributions [US] 2023-02-05T20:21:14Z User contributions MediaWiki 1.28.3 http://wiki.math.uwaterloo.ca/statwiki/index.php?title=IPBoost&diff=49113 IPBoost 2020-12-04T06:53:32Z <p>Z42qin: /* Critique */</p> <hr /> <div>== Presented by == <br /> Casey De Vera, Solaiman Jawad<br /> <br /> == Introduction == <br /> Boosting is an important and a fairly standard technique in a classification that combines several base learners (learners which have a “low accuracy”), into one boosted learner (learner with a “high accuracy”). Pioneered by the AdaBoost approach of Freund &amp; Schapire, in recent decades there has been extensive work on boosting procedures and analyses of their limitations.<br /> <br /> In a nutshell, boosting procedures are (typically) iterative schemes that roughly work as follows:<br /> <br /> Let &lt;math&gt;T&lt;/math&gt; represents the number of iteration and &lt;math&gt;t&lt;/math&gt; represents the iterator. <br /> <br /> for &lt;math&gt; t= 1, \cdots, T &lt;/math&gt; do the following:<br /> <br /> ::1. Train a learner &lt;math&gt; \mu_t&lt;/math&gt; from a given class of base learners on the data distribution &lt;math&gt; \mathcal D_t&lt;/math&gt;<br /> <br /> ::2. Evaluate the performance of &lt;math&gt; \mu_t&lt;/math&gt; by computing its loss.<br /> <br /> ::3. Push weight of the data distribution &lt;math&gt; \mathcal D_t&lt;/math&gt; towards the misclassified examples leading to &lt;math&gt; \mathcal D_{t+1}&lt;/math&gt;, usually a relatively good model will have a higher weight on those data that misclassified.<br /> <br /> Finally, the learners are combined with some form of voting (e.g., soft or hard voting, averaging, thresholding).<br /> <br /> <br /> [[File:boosting.gif|200px|thumb|right]] A close inspection of most boosting procedures reveals that they solve an underlying convex optimization problem over a convex loss function using coordinate gradient descent. Boosting schemes of this type are often referred to as convex potential boosters. These procedures can achieve exceptional performance on many data sets if the data is correctly labeled. However, they can be trounced by a small number of incorrect labels, which can be quite challenging to fix. We try to solve misclassified examples by moving the weights around, which results in bad performance on unseen data (Shown by zoomed example to the right). In fact, in theory, providing the class of base learners is rich enough, a perfect strong learner can be constructed with accuracy 1; however, clearly, such a learner might not necessarily generalize well. Boosted learners can generate some quite complicated decision boundaries, much more complicated than the base learners. Here is an example from Paul van der Laken’s blog / Extreme gradient boosting gif by Ryan Holbrook. Here data is generated online according to some process with optimal decision boundary represented by the dotted line, and XGBoost was used to learn a classifier:<br /> <br /> <br /> Recently non-convex optimization approaches for solving machine learning problems have gained significant attention. In this paper, the non-convex boosting in classification using integer programming has been explored, and the real-world practicability of the approach while circumventing shortcomings of convex boosting approaches is also shown. The paper reports results that are comparable to or better than current state-of-the-art approaches.<br /> <br /> == Motivation ==<br /> <br /> In reality, we usually face unclean data and so-called label noise, where some percentage of the classification labels might be corrupted. We would also like to construct strong learners for such data. Noisy labels are an issue because when the writer trained a model with excessive amounts of noisy labels, the performance and accuracy deteriorate significantly. However, if we revisit the general boosting template from above, then we might suspect that we run into trouble as soon as a certain fraction of training examples is misclassified: in this case, the model cannot correctly classify these examples, and the procedure shifts more and more weight towards these bad examples. It eventually leads to a strong learner that accurately predicts the (flawed) training data; however, that does not generalize well anymore. This intuition has been formalized by [LS], who construct a &quot;hard&quot; training data distribution, where a small percentage of labels is randomly flipped. This label noise then leads to a significant reduction in these boosted learners' performance; see the tables below. The more technical reason for this problem is the convexity of the loss function minimized by the boosting procedure. One can use all types of &quot;tricks,&quot; such as early stopping, but this is not solving the fundamental problem at the end of the day.<br /> <br /> == IPBoost: Boosting via Integer Programming ==<br /> <br /> <br /> ===Integer Program Formulation===<br /> Let &lt;math&gt;(x_1,y_1),\cdots, (x_N,y_N) &lt;/math&gt; be the training set with points &lt;math&gt;x_i \in \mathbb{R}^d&lt;/math&gt; and two-class labels &lt;math&gt;y_i \in \{\pm 1\}&lt;/math&gt; <br /> * class of base learners: &lt;math&gt; \Omega :=\{h_1, \cdots, h_L: \mathbb{R}^d \rightarrow \{\pm 1\}\} &lt;/math&gt; and &lt;math&gt;\rho \ge 0&lt;/math&gt; be given. <br /> * error function &lt;math&gt; \eta &lt;/math&gt;<br /> Our boosting model is captured by the integer programming problem. We can call this our primal problem: <br /> <br /> \begin{align*} \min &amp;\sum_{i=1}^N z_i \\ s.t. &amp;\sum_{j=1}^L \eta_{ij}\lambda_k+(1+\rho)z_i \ge \rho \ \ \ <br /> \forall i=1,\cdots, N \\ <br /> &amp;\sum_{j=1}^L \lambda_j=1, \lambda \ge 0,\\ &amp;z\in \{0,1\}^N. \end{align*}<br /> <br /> For the error function &lt;math&gt;\eta&lt;/math&gt;, three options were considered:<br /> <br /> (i) &lt;math&gt; \pm 1 &lt;/math&gt; classification from learners<br /> $$\eta_{ij} := 2\mathbb{I}[h_j(x_i) = y_i] - 1 = y_i \cdot h_j(x_i)$$<br /> (ii) class probabilities of learners<br /> $$\eta_{ij} := 2\mathbb{P}[h_j(x_i) = y_i] - 1$$<br /> (iii) SAMME.R error function for learners<br /> $$\frac{1}{2}y_i\log\left(\frac{\mathbb{P}[h_j(x_i) = 1]}{\mathbb{P}[h_j(x_i) = -1]}\right)$$<br /> <br /> ===Solution of the IP using Column Generation===<br /> <br /> The goal of column generation is to provide an efficient way to solve the primal's linear programming relaxation by allowing the zi variables to assume fractional values. Moreover, columns, i.e., the base learners, &lt;math&gt; \mathcal L \subseteq [L]. &lt;/math&gt; . are left out because there are too many to handle efficiently, and most of them will have their associated weight equal to zero in the optimal solution. A branch and bound framework is used To generate columns. Columns are created within a branch-and-bound framework leading effectively to a branch-and-bound-and-price algorithm being used; this is significantly more involved compared to column generation in linear programming. To check the optimality of an LP solution, a subproblem, called the pricing problem, is solved to identify columns with a profitable reduced cost. If such columns are found, the LP is re-optimized. Branching occurs when no profitable columns are found, but the LP solution does not satisfy the integrality conditions. Branch and price apply column generation at every node of the branch and bound tree.<br /> <br /> In reality, we usually face unclean data and so-called label noise, where some percentage of the classification labels might be corrupted. We would also like to construct strong learners for such data. Noisy labels are an issue because when the writer trained a model with excessive amounts of noisy labels, the performance and accuracy deteriorate significantly. However, if we revisit the general boosting template from above, then we might suspect that we run into trouble as soon as a certain fraction of training examples is misclassified: in this case, the model cannot correctly classify these examples, and the procedure shifts more and more weight towards these bad examples. It eventually leads to a strong learner that accurately predicts the (flawed) training data; however, that does not generalize well anymore. This intuition has been formalized by [LS], who construct a &quot;hard&quot; training data distribution, where a small percentage of labels is randomly flipped. This label noise then leads to a significant reduction in these boosted learners' performance; see the tables below. The more technical reason for this problem is the convexity of the loss function minimized by the boosting procedure. One can use all types of &quot;tricks,&quot; such as early stopping, but this is not solving the fundamental problem at the end of the day.<br /> <br /> <br /> <br /> The restricted master primal problem is <br /> <br /> \begin{align*} \min &amp;\sum_{i=1}^N z_i \\ s.t. &amp;\sum_{j\in \mathcal L} \eta_{ij}\lambda_j+(1+\rho)z_i \ge \rho \ \ \ <br /> \forall i \in [N]\\ <br /> &amp;\sum_{j\in \mathcal L}\lambda_j=1, \lambda \ge 0,\\ &amp;z\in \{0,1\}^N. \end{align*}<br /> <br /> <br /> Its restricted dual problem is:<br /> <br /> \begin{align*}\max \rho &amp;\sum^{N}_{i=1}w_i + v - \sum^{N}_{i=1}u_i<br /> \\ s.t. &amp;\sum_{i=1}^N \eta_{ij}w_k+ v \le 0 \ \ \ \forall j \in L \\ <br /> &amp;(1+\rho)w_i - u_i \le 1 \ \ \ \forall i \in [N] \\ &amp;w \ge 0, u \ge 0, v\ free\end{align*}<br /> <br /> Furthermore, there is a pricing problem used to determine, for every supposed optimal solution of the dual, whether the solution is actually optimal, or whether further constraints need to be added into the primal solution. With this pricing problem, we check whether the solution to the restricted dual is feasible. This pricing problem can be expressed as follows:<br /> <br /> $$\sum_{i=1}^N \eta_{ij}w_k^* + v^* &gt; 0$$<br /> <br /> The optimal misclassification values are determined by a branch-and-price process that branches on the variables &lt;math&gt; z_i &lt;/math&gt; and solves the intermediate LPs using column generation.<br /> <br /> ===Algorithm===<br /> &lt;div style=&quot;margin-left: 3em;&quot;&gt;<br /> &lt;math&gt; D = \{(x_i, y_i) | i ∈ I\} ⊆ R^d × \{±1\} &lt;/math&gt;, class of base learners &lt;math&gt;Ω &lt;/math&gt;, margin &lt;math&gt; \rho &lt;/math&gt; &lt;br&gt;<br /> '''Output:''' Boosted learner &lt;math&gt; \sum_{j∈L^∗}h_jλ_j^* &lt;/math&gt; with base learners &lt;math&gt; h_j &lt;/math&gt; and weights &lt;math&gt; λ_j^* &lt;/math&gt; &lt;br&gt;<br /> <br /> &lt;ol&gt;<br /> <br /> &lt;li margin-left:30px&gt; &lt;math&gt; T ← \{([0, 1]^N, \emptyset)\} &lt;/math&gt; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // set of local bounds and learners for open subproblems &lt;/li&gt;<br /> &lt;li&gt; &lt;math&gt; U ← \infty, L^∗ ← \emptyset &lt;/math&gt; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Upper bound on optimal objective &lt;/li&gt;<br /> &lt;li&gt; '''while''' &lt;math&gt;\ T \neq \emptyset &lt;/math&gt; '''do''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; Choose and remove &lt;math&gt;(B,L) &lt;/math&gt; from &lt;math&gt;T &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''repeat''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Solve the primal IP using the local bounds on &lt;math&gt; z &lt;/math&gt; in &lt;math&gt;B&lt;/math&gt; with optimal dual solution &lt;math&gt; (w^∗, v^∗, u^∗) &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Find learner &lt;math&gt; h_j ∈ Ω &lt;/math&gt; satisfying the pricing problem. &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Solve pricing problem. &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''until''' &lt;math&gt; h_j &lt;/math&gt; is not found &lt;/li&gt; <br /> &lt;li&gt; &amp;emsp; Let &lt;math&gt; (\widetilde{λ} , \widetilde{z}) &lt;/math&gt; be the final solution of the primal IP with base learners &lt;math&gt; \widetilde{L} = \{j | \widetilde{λ}_j &gt; 0\} &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''if''' &lt;math&gt; \widetilde{z} ∈ \mathbb{Z}^N &lt;/math&gt; and &lt;math&gt; \sum^{N}_{i=1}\widetilde{z}_i &lt; U &lt;/math&gt; '''then''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; &lt;math&gt; U ← \sum^{N}_{i=1}\widetilde{z}_i, L^∗ ← \widetilde{L}, λ^∗ ← \widetilde{\lambda} &lt;/math&gt; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Update best solution. &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''else''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Choose &lt;math&gt; i ∈ [N] &lt;/math&gt; with &lt;math&gt; \widetilde{z}_i \notin Z &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Set &lt;math&gt; B_0 ← B ∩ \{z_i ≤ 0\}, B_1 ← B ∩ \{z_i ≥ 1\} &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Add &lt;math&gt; (B_0,\widetilde{L}), (B_1,\widetilde{L}) &lt;/math&gt; to &lt;math&gt; T &lt;/math&gt;. &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Create new branching nodes. &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''end''' if &lt;/li&gt;<br /> &lt;li&gt; '''end''' while &lt;/li&gt;<br /> &lt;li&gt; ''Optionally sparsify final solution &lt;math&gt;L^*&lt;/math&gt;'' &lt;/li&gt;<br /> <br /> &lt;/ol&gt;<br /> &lt;/div&gt;<br /> <br /> ===Sparsification===<br /> <br /> When building the boosting model, one of the challenges is to balance model accuracy and generalization. A sparse model is generally easier to interpret. In order to control complexity, overfitting, and generalization of the model, typically some sparsity is applied, which is easier to interpret for some applications. The goal of sparsification is to minimize the following: \begin{equation}\sum_{i=1}^{N}z_i+\sum_{i=1}^{L}\alpha_iy_i\end{equation} There are two techniques commonly used. One is early stopping, another is regularization by adding a complexity term for the learners in the objective function. Previous approaches in the context of LP-based boosting have promoted sparsity by means of cutting planes. Sparsification can be handled in the approach introduced in this paper by solving a delayed integer program using additional cutting planes.<br /> <br /> == Results and Performance ==<br /> <br /> ''All tests were run on identical Linux clusters with Intel Xeon Quad Core CPUs, with 3.50GHz, 10 MB cache, and 32 GB of main memory.''<br /> <br /> <br /> The following results reflect IPBoost's performance in hard instances. Note that by hard instances, we mean a binary classification problem with predefined labels. These examples are tailored to using the ±1 classification from learners. On every hard instance sample, IPBoost significantly outperforms both LPBoost and AdaBoost (although implementations depending on the libraries used have often caused results to differ slightly). For the considered instances the best value for the margin ρ was 0.05 for LPBoost and IPBoost; AdaBoost has no margin parameter. The accuracy reported is test accuracy recorded across various different walkthroughs of the algorithm, while &lt;math&gt;L &lt;/math&gt; denotes the aggregate number of learners required to find the optimal learner, N is the number of points and &lt;math&gt; \gamma &lt;/math&gt; refers to the noise level.<br /> <br /> [[File:ipboostres.png|center]]<br /> <br /> <br /> <br /> For the next table, the classification instances from LIBSVM data sets available at [https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/]. We report accuracies on the test set and train set, respectively. In each case, we report the averages of the accuracies over 10 runs with a different random seed and their standard deviations. We can see IPboost again outperforming LPBoost and AdaBoost significantly. Solving Integer Programming problems is no doubt more computationally expensive than traditional boosting methods like AdaBoost. The average run time of IPBoost (for ρ = 0.05) being 1367.78 seconds, as opposed to LPBoost's 164.35 seconds and AdaBoost's 3.59 seconds reflects exactly that. However, on the flip side, we gain much better stability in our results, as well as higher scores across the board for both training and test sets.<br /> <br /> [[file:svmlibres.png|center]]<br /> <br /> <br /> == Conclusion ==<br /> <br /> IP-boosting avoids the bad performance on well-known hard classes and improves upon LP-boosting and AdaBoost on the LIBSVM instances where even a few percent improvements is valuable. The major drawback is that the running time with the current implementation is much longer. Nevertheless, the algorithm can be improved in the future by solving the intermediate LPs only approximately and deriving tailored heuristics that generate decent primal solutions to save on time.<br /> <br /> The approach is suited very well to an offline setting in which training may take time and where even a small improvement is beneficial or when convex boosters have egregious behaviour. It can also be served as a tool to investigate the general performance of methods like this.<br /> <br /> The IPBoost algorithm added extra complexity into basic boosting models to gain slight accuracy gain while greatly increased the time spent. It is 381 times slower compared to an AdaBoost model on a small dataset which makes the actual usage of this model doubtable. If we are supplied with a larger dataset with millions of records this model would take too long to complete. The base classifier choice was XGBoost which is too complicated for a base classifier, maybe try some weaker learners such as tree stumps to compare the result with other models. In addition, this model might not be accurate compared to the model-ensembling technique where each model utilizes a different algorithm.<br /> <br /> I think IP Boost can be helpful in cases where a small improvement in accuracy is worth the additional computational effort involved. For example, there are applications in Finance where predicting the movement of stock prices slightly more accurately can help traders make millions more. In such cases, investing in expensive computing systems to implement such boosting techniques can perhaps be justified.<br /> <br /> Although the IPBoost can generate better results compared to the base boosting models such as AdaBoost. The time complexity of IPBoost is much higher than other models. Hence, it will be problem to apply IPBoost to the real-world training data since those training data will be large and it's hard to deal with. Thus, we are left with the question that this IPBoost is only a little bit more accurate than the base boosting models but a lot more time complex. So is the model really worth it?<br /> <br /> == Critique ==<br /> <br /> The introduction section should change the sentence&quot;3. Push weight of the data ... will have a higher weight on those data that misclassified.&quot; to &quot;... will have a higher weight on those misclassified data.<br /> <br /> The column generation method is used when there are many variables compared to the number of constraints. If there are only a few variables compared to the number of limitations, using this method will not be very efficient.It would be better if the efficiency of different boosting models is tested.<br /> <br /> In practice, AdaBoost &amp; LP Boost are quite robust to overfitting. In a way, if you use simple weak learns stumps (1-level decision trees), then the algorithms are much less prone to overfitting. However, the noise level in the data, particularly for AdaBoost, is prone to overfitting noisy datasets. To avoid this use, regularized models (AdaBoostReg, LPBoost). Further, when in higher dimensional spaces, AdaBoost can suffer since it's just a linear combination of classifiers. You can use k-fold cross-validation to set the stopping parameter to improve this.<br /> <br /> == References ==<br /> <br /> * Pfetsch, M. E., &amp; Pokutta, S. (2020). IPBoost--Non-Convex Boosting via Integer Programming. arXiv preprint arXiv:2002.04679.<br /> <br /> * Freund, Y., &amp; Schapire, R. E. (1995, March). A desicion-theoretic generalization of on-line learning and an application to boosting. In European conference on computational learning theory (pp. 23-37). Springer, Berlin, Heidelberg. pdf</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=IPBoost&diff=49111 IPBoost 2020-12-04T06:50:32Z <p>Z42qin: /* Solution of the IP using Column Generation */</p> <hr /> <div>== Presented by == <br /> Casey De Vera, Solaiman Jawad<br /> <br /> == Introduction == <br /> Boosting is an important and a fairly standard technique in a classification that combines several base learners (learners which have a “low accuracy”), into one boosted learner (learner with a “high accuracy”). Pioneered by the AdaBoost approach of Freund &amp; Schapire, in recent decades there has been extensive work on boosting procedures and analyses of their limitations.<br /> <br /> In a nutshell, boosting procedures are (typically) iterative schemes that roughly work as follows:<br /> <br /> Let &lt;math&gt;T&lt;/math&gt; represents the number of iteration and &lt;math&gt;t&lt;/math&gt; represents the iterator. <br /> <br /> for &lt;math&gt; t= 1, \cdots, T &lt;/math&gt; do the following:<br /> <br /> ::1. Train a learner &lt;math&gt; \mu_t&lt;/math&gt; from a given class of base learners on the data distribution &lt;math&gt; \mathcal D_t&lt;/math&gt;<br /> <br /> ::2. Evaluate the performance of &lt;math&gt; \mu_t&lt;/math&gt; by computing its loss.<br /> <br /> ::3. Push weight of the data distribution &lt;math&gt; \mathcal D_t&lt;/math&gt; towards the misclassified examples leading to &lt;math&gt; \mathcal D_{t+1}&lt;/math&gt;, usually a relatively good model will have a higher weight on those data that misclassified.<br /> <br /> Finally, the learners are combined with some form of voting (e.g., soft or hard voting, averaging, thresholding).<br /> <br /> <br /> [[File:boosting.gif|200px|thumb|right]] A close inspection of most boosting procedures reveals that they solve an underlying convex optimization problem over a convex loss function using coordinate gradient descent. Boosting schemes of this type are often referred to as convex potential boosters. These procedures can achieve exceptional performance on many data sets if the data is correctly labeled. However, they can be trounced by a small number of incorrect labels, which can be quite challenging to fix. We try to solve misclassified examples by moving the weights around, which results in bad performance on unseen data (Shown by zoomed example to the right). In fact, in theory, providing the class of base learners is rich enough, a perfect strong learner can be constructed with accuracy 1; however, clearly, such a learner might not necessarily generalize well. Boosted learners can generate some quite complicated decision boundaries, much more complicated than the base learners. Here is an example from Paul van der Laken’s blog / Extreme gradient boosting gif by Ryan Holbrook. Here data is generated online according to some process with optimal decision boundary represented by the dotted line, and XGBoost was used to learn a classifier:<br /> <br /> <br /> Recently non-convex optimization approaches for solving machine learning problems have gained significant attention. In this paper, the non-convex boosting in classification using integer programming has been explored, and the real-world practicability of the approach while circumventing shortcomings of convex boosting approaches is also shown. The paper reports results that are comparable to or better than current state-of-the-art approaches.<br /> <br /> == Motivation ==<br /> <br /> In reality, we usually face unclean data and so-called label noise, where some percentage of the classification labels might be corrupted. We would also like to construct strong learners for such data. Noisy labels are an issue because when the writer trained a model with excessive amounts of noisy labels, the performance and accuracy deteriorate significantly. However, if we revisit the general boosting template from above, then we might suspect that we run into trouble as soon as a certain fraction of training examples is misclassified: in this case, the model cannot correctly classify these examples, and the procedure shifts more and more weight towards these bad examples. It eventually leads to a strong learner that accurately predicts the (flawed) training data; however, that does not generalize well anymore. This intuition has been formalized by [LS], who construct a &quot;hard&quot; training data distribution, where a small percentage of labels is randomly flipped. This label noise then leads to a significant reduction in these boosted learners' performance; see the tables below. The more technical reason for this problem is the convexity of the loss function minimized by the boosting procedure. One can use all types of &quot;tricks,&quot; such as early stopping, but this is not solving the fundamental problem at the end of the day.<br /> <br /> == IPBoost: Boosting via Integer Programming ==<br /> <br /> <br /> ===Integer Program Formulation===<br /> Let &lt;math&gt;(x_1,y_1),\cdots, (x_N,y_N) &lt;/math&gt; be the training set with points &lt;math&gt;x_i \in \mathbb{R}^d&lt;/math&gt; and two-class labels &lt;math&gt;y_i \in \{\pm 1\}&lt;/math&gt; <br /> * class of base learners: &lt;math&gt; \Omega :=\{h_1, \cdots, h_L: \mathbb{R}^d \rightarrow \{\pm 1\}\} &lt;/math&gt; and &lt;math&gt;\rho \ge 0&lt;/math&gt; be given. <br /> * error function &lt;math&gt; \eta &lt;/math&gt;<br /> Our boosting model is captured by the integer programming problem. We can call this our primal problem: <br /> <br /> \begin{align*} \min &amp;\sum_{i=1}^N z_i \\ s.t. &amp;\sum_{j=1}^L \eta_{ij}\lambda_k+(1+\rho)z_i \ge \rho \ \ \ <br /> \forall i=1,\cdots, N \\ <br /> &amp;\sum_{j=1}^L \lambda_j=1, \lambda \ge 0,\\ &amp;z\in \{0,1\}^N. \end{align*}<br /> <br /> For the error function &lt;math&gt;\eta&lt;/math&gt;, three options were considered:<br /> <br /> (i) &lt;math&gt; \pm 1 &lt;/math&gt; classification from learners<br /> $$\eta_{ij} := 2\mathbb{I}[h_j(x_i) = y_i] - 1 = y_i \cdot h_j(x_i)$$<br /> (ii) class probabilities of learners<br /> $$\eta_{ij} := 2\mathbb{P}[h_j(x_i) = y_i] - 1$$<br /> (iii) SAMME.R error function for learners<br /> $$\frac{1}{2}y_i\log\left(\frac{\mathbb{P}[h_j(x_i) = 1]}{\mathbb{P}[h_j(x_i) = -1]}\right)$$<br /> <br /> ===Solution of the IP using Column Generation===<br /> <br /> The goal of column generation is to provide an efficient way to solve the primal's linear programming relaxation by allowing the zi variables to assume fractional values. Moreover, columns, i.e., the base learners, &lt;math&gt; \mathcal L \subseteq [L]. &lt;/math&gt; . are left out because there are too many to handle efficiently, and most of them will have their associated weight equal to zero in the optimal solution. A branch and bound framework is used To generate columns. Columns are created within a branch-and-bound framework leading effectively to a branch-and-bound-and-price algorithm being used; this is significantly more involved compared to column generation in linear programming. To check the optimality of an LP solution, a subproblem, called the pricing problem, is solved to identify columns with a profitable reduced cost. If such columns are found, the LP is re-optimized. Branching occurs when no profitable columns are found, but the LP solution does not satisfy the integrality conditions. Branch and price apply column generation at every node of the branch and bound tree.<br /> <br /> In reality, we usually face unclean data and so-called label noise, where some percentage of the classification labels might be corrupted. We would also like to construct strong learners for such data. Noisy labels are an issue because when the writer trained a model with excessive amounts of noisy labels, the performance and accuracy deteriorate significantly. However, if we revisit the general boosting template from above, then we might suspect that we run into trouble as soon as a certain fraction of training examples is misclassified: in this case, the model cannot correctly classify these examples, and the procedure shifts more and more weight towards these bad examples. It eventually leads to a strong learner that accurately predicts the (flawed) training data; however, that does not generalize well anymore. This intuition has been formalized by [LS], who construct a &quot;hard&quot; training data distribution, where a small percentage of labels is randomly flipped. This label noise then leads to a significant reduction in these boosted learners' performance; see the tables below. The more technical reason for this problem is the convexity of the loss function minimized by the boosting procedure. One can use all types of &quot;tricks,&quot; such as early stopping, but this is not solving the fundamental problem at the end of the day.<br /> <br /> <br /> <br /> The restricted master primal problem is <br /> <br /> \begin{align*} \min &amp;\sum_{i=1}^N z_i \\ s.t. &amp;\sum_{j\in \mathcal L} \eta_{ij}\lambda_j+(1+\rho)z_i \ge \rho \ \ \ <br /> \forall i \in [N]\\ <br /> &amp;\sum_{j\in \mathcal L}\lambda_j=1, \lambda \ge 0,\\ &amp;z\in \{0,1\}^N. \end{align*}<br /> <br /> <br /> Its restricted dual problem is:<br /> <br /> \begin{align*}\max \rho &amp;\sum^{N}_{i=1}w_i + v - \sum^{N}_{i=1}u_i<br /> \\ s.t. &amp;\sum_{i=1}^N \eta_{ij}w_k+ v \le 0 \ \ \ \forall j \in L \\ <br /> &amp;(1+\rho)w_i - u_i \le 1 \ \ \ \forall i \in [N] \\ &amp;w \ge 0, u \ge 0, v\ free\end{align*}<br /> <br /> Furthermore, there is a pricing problem used to determine, for every supposed optimal solution of the dual, whether the solution is actually optimal, or whether further constraints need to be added into the primal solution. With this pricing problem, we check whether the solution to the restricted dual is feasible. This pricing problem can be expressed as follows:<br /> <br /> $$\sum_{i=1}^N \eta_{ij}w_k^* + v^* &gt; 0$$<br /> <br /> The optimal misclassification values are determined by a branch-and-price process that branches on the variables &lt;math&gt; z_i &lt;/math&gt; and solves the intermediate LPs using column generation.<br /> <br /> ===Algorithm===<br /> &lt;div style=&quot;margin-left: 3em;&quot;&gt;<br /> &lt;math&gt; D = \{(x_i, y_i) | i ∈ I\} ⊆ R^d × \{±1\} &lt;/math&gt;, class of base learners &lt;math&gt;Ω &lt;/math&gt;, margin &lt;math&gt; \rho &lt;/math&gt; &lt;br&gt;<br /> '''Output:''' Boosted learner &lt;math&gt; \sum_{j∈L^∗}h_jλ_j^* &lt;/math&gt; with base learners &lt;math&gt; h_j &lt;/math&gt; and weights &lt;math&gt; λ_j^* &lt;/math&gt; &lt;br&gt;<br /> <br /> &lt;ol&gt;<br /> <br /> &lt;li margin-left:30px&gt; &lt;math&gt; T ← \{([0, 1]^N, \emptyset)\} &lt;/math&gt; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // set of local bounds and learners for open subproblems &lt;/li&gt;<br /> &lt;li&gt; &lt;math&gt; U ← \infty, L^∗ ← \emptyset &lt;/math&gt; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Upper bound on optimal objective &lt;/li&gt;<br /> &lt;li&gt; '''while''' &lt;math&gt;\ T \neq \emptyset &lt;/math&gt; '''do''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; Choose and remove &lt;math&gt;(B,L) &lt;/math&gt; from &lt;math&gt;T &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''repeat''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Solve the primal IP using the local bounds on &lt;math&gt; z &lt;/math&gt; in &lt;math&gt;B&lt;/math&gt; with optimal dual solution &lt;math&gt; (w^∗, v^∗, u^∗) &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Find learner &lt;math&gt; h_j ∈ Ω &lt;/math&gt; satisfying the pricing problem. &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Solve pricing problem. &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''until''' &lt;math&gt; h_j &lt;/math&gt; is not found &lt;/li&gt; <br /> &lt;li&gt; &amp;emsp; Let &lt;math&gt; (\widetilde{λ} , \widetilde{z}) &lt;/math&gt; be the final solution of the primal IP with base learners &lt;math&gt; \widetilde{L} = \{j | \widetilde{λ}_j &gt; 0\} &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''if''' &lt;math&gt; \widetilde{z} ∈ \mathbb{Z}^N &lt;/math&gt; and &lt;math&gt; \sum^{N}_{i=1}\widetilde{z}_i &lt; U &lt;/math&gt; '''then''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; &lt;math&gt; U ← \sum^{N}_{i=1}\widetilde{z}_i, L^∗ ← \widetilde{L}, λ^∗ ← \widetilde{\lambda} &lt;/math&gt; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Update best solution. &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''else''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Choose &lt;math&gt; i ∈ [N] &lt;/math&gt; with &lt;math&gt; \widetilde{z}_i \notin Z &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Set &lt;math&gt; B_0 ← B ∩ \{z_i ≤ 0\}, B_1 ← B ∩ \{z_i ≥ 1\} &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Add &lt;math&gt; (B_0,\widetilde{L}), (B_1,\widetilde{L}) &lt;/math&gt; to &lt;math&gt; T &lt;/math&gt;. &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Create new branching nodes. &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''end''' if &lt;/li&gt;<br /> &lt;li&gt; '''end''' while &lt;/li&gt;<br /> &lt;li&gt; ''Optionally sparsify final solution &lt;math&gt;L^*&lt;/math&gt;'' &lt;/li&gt;<br /> <br /> &lt;/ol&gt;<br /> &lt;/div&gt;<br /> <br /> ===Sparsification===<br /> <br /> When building the boosting model, one of the challenges is to balance model accuracy and generalization. A sparse model is generally easier to interpret. In order to control complexity, overfitting, and generalization of the model, typically some sparsity is applied, which is easier to interpret for some applications. The goal of sparsification is to minimize the following: \begin{equation}\sum_{i=1}^{N}z_i+\sum_{i=1}^{L}\alpha_iy_i\end{equation} There are two techniques commonly used. One is early stopping, another is regularization by adding a complexity term for the learners in the objective function. Previous approaches in the context of LP-based boosting have promoted sparsity by means of cutting planes. Sparsification can be handled in the approach introduced in this paper by solving a delayed integer program using additional cutting planes.<br /> <br /> == Results and Performance ==<br /> <br /> ''All tests were run on identical Linux clusters with Intel Xeon Quad Core CPUs, with 3.50GHz, 10 MB cache, and 32 GB of main memory.''<br /> <br /> <br /> The following results reflect IPBoost's performance in hard instances. Note that by hard instances, we mean a binary classification problem with predefined labels. These examples are tailored to using the ±1 classification from learners. On every hard instance sample, IPBoost significantly outperforms both LPBoost and AdaBoost (although implementations depending on the libraries used have often caused results to differ slightly). For the considered instances the best value for the margin ρ was 0.05 for LPBoost and IPBoost; AdaBoost has no margin parameter. The accuracy reported is test accuracy recorded across various different walkthroughs of the algorithm, while &lt;math&gt;L &lt;/math&gt; denotes the aggregate number of learners required to find the optimal learner, N is the number of points and &lt;math&gt; \gamma &lt;/math&gt; refers to the noise level.<br /> <br /> [[File:ipboostres.png|center]]<br /> <br /> <br /> <br /> For the next table, the classification instances from LIBSVM data sets available at [https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/]. We report accuracies on the test set and train set, respectively. In each case, we report the averages of the accuracies over 10 runs with a different random seed and their standard deviations. We can see IPboost again outperforming LPBoost and AdaBoost significantly. Solving Integer Programming problems is no doubt more computationally expensive than traditional boosting methods like AdaBoost. The average run time of IPBoost (for ρ = 0.05) being 1367.78 seconds, as opposed to LPBoost's 164.35 seconds and AdaBoost's 3.59 seconds reflects exactly that. However, on the flip side, we gain much better stability in our results, as well as higher scores across the board for both training and test sets.<br /> <br /> [[file:svmlibres.png|center]]<br /> <br /> <br /> == Conclusion ==<br /> <br /> IP-boosting avoids the bad performance on well-known hard classes and improves upon LP-boosting and AdaBoost on the LIBSVM instances where even a few percent improvements is valuable. The major drawback is that the running time with the current implementation is much longer. Nevertheless, the algorithm can be improved in the future by solving the intermediate LPs only approximately and deriving tailored heuristics that generate decent primal solutions to save on time.<br /> <br /> The approach is suited very well to an offline setting in which training may take time and where even a small improvement is beneficial or when convex boosters have egregious behaviour. It can also be served as a tool to investigate the general performance of methods like this.<br /> <br /> The IPBoost algorithm added extra complexity into basic boosting models to gain slight accuracy gain while greatly increased the time spent. It is 381 times slower compared to an AdaBoost model on a small dataset which makes the actual usage of this model doubtable. If we are supplied with a larger dataset with millions of records this model would take too long to complete. The base classifier choice was XGBoost which is too complicated for a base classifier, maybe try some weaker learners such as tree stumps to compare the result with other models. In addition, this model might not be accurate compared to the model-ensembling technique where each model utilizes a different algorithm.<br /> <br /> I think IP Boost can be helpful in cases where a small improvement in accuracy is worth the additional computational effort involved. For example, there are applications in Finance where predicting the movement of stock prices slightly more accurately can help traders make millions more. In such cases, investing in expensive computing systems to implement such boosting techniques can perhaps be justified.<br /> <br /> Although the IPBoost can generate better results compared to the base boosting models such as AdaBoost. The time complexity of IPBoost is much higher than other models. Hence, it will be problem to apply IPBoost to the real-world training data since those training data will be large and it's hard to deal with. Thus, we are left with the question that this IPBoost is only a little bit more accurate than the base boosting models but a lot more time complex. So is the model really worth it?<br /> <br /> == Critique ==<br /> <br /> In the introduction section, should change the sentence&quot;3. Push weight of the data ... will have a higher weight on those data that misclassified.&quot; to &quot;... will have a higher weight on those data that are misclassified.<br /> <br /> The column generation method is used when there are a huge number of variables compared to the number of constraints. If there are only a few variables compared to the number of constraints, using this method will not be very efficient.<br /> <br /> It would be better if the efficiency of different boosting models are tested.<br /> <br /> In practice AdaBoost &amp; LP Boost are quite robust to over fitting. In practice if you use simple weak learns stumps (1-level decision trees) then the algorithms are much less prone to over fitting. However the noise level in the data, particularly for AdaBoost is prone to overfitting on noisy datasets. To avoid this use regularized models (AdaBoostReg, LPBoost). Further, when in higher dimensional spaces, AdaBoost can suffer, since its just a linear combination of classifiers. You can use k-fold cross validation to set the stopping parameter to improve this.<br /> <br /> == References ==<br /> <br /> * Pfetsch, M. E., &amp; Pokutta, S. (2020). IPBoost--Non-Convex Boosting via Integer Programming. arXiv preprint arXiv:2002.04679.<br /> <br /> * Freund, Y., &amp; Schapire, R. E. (1995, March). A desicion-theoretic generalization of on-line learning and an application to boosting. In European conference on computational learning theory (pp. 23-37). Springer, Berlin, Heidelberg. pdf</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=IPBoost&diff=49110 IPBoost 2020-12-04T06:45:54Z <p>Z42qin: /* Motivation */</p> <hr /> <div>== Presented by == <br /> Casey De Vera, Solaiman Jawad<br /> <br /> == Introduction == <br /> Boosting is an important and a fairly standard technique in a classification that combines several base learners (learners which have a “low accuracy”), into one boosted learner (learner with a “high accuracy”). Pioneered by the AdaBoost approach of Freund &amp; Schapire, in recent decades there has been extensive work on boosting procedures and analyses of their limitations.<br /> <br /> In a nutshell, boosting procedures are (typically) iterative schemes that roughly work as follows:<br /> <br /> Let &lt;math&gt;T&lt;/math&gt; represents the number of iteration and &lt;math&gt;t&lt;/math&gt; represents the iterator. <br /> <br /> for &lt;math&gt; t= 1, \cdots, T &lt;/math&gt; do the following:<br /> <br /> ::1. Train a learner &lt;math&gt; \mu_t&lt;/math&gt; from a given class of base learners on the data distribution &lt;math&gt; \mathcal D_t&lt;/math&gt;<br /> <br /> ::2. Evaluate the performance of &lt;math&gt; \mu_t&lt;/math&gt; by computing its loss.<br /> <br /> ::3. Push weight of the data distribution &lt;math&gt; \mathcal D_t&lt;/math&gt; towards the misclassified examples leading to &lt;math&gt; \mathcal D_{t+1}&lt;/math&gt;, usually a relatively good model will have a higher weight on those data that misclassified.<br /> <br /> Finally, the learners are combined with some form of voting (e.g., soft or hard voting, averaging, thresholding).<br /> <br /> <br /> [[File:boosting.gif|200px|thumb|right]] A close inspection of most boosting procedures reveals that they solve an underlying convex optimization problem over a convex loss function using coordinate gradient descent. Boosting schemes of this type are often referred to as convex potential boosters. These procedures can achieve exceptional performance on many data sets if the data is correctly labeled. However, they can be trounced by a small number of incorrect labels, which can be quite challenging to fix. We try to solve misclassified examples by moving the weights around, which results in bad performance on unseen data (Shown by zoomed example to the right). In fact, in theory, providing the class of base learners is rich enough, a perfect strong learner can be constructed with accuracy 1; however, clearly, such a learner might not necessarily generalize well. Boosted learners can generate some quite complicated decision boundaries, much more complicated than the base learners. Here is an example from Paul van der Laken’s blog / Extreme gradient boosting gif by Ryan Holbrook. Here data is generated online according to some process with optimal decision boundary represented by the dotted line, and XGBoost was used to learn a classifier:<br /> <br /> <br /> Recently non-convex optimization approaches for solving machine learning problems have gained significant attention. In this paper, the non-convex boosting in classification using integer programming has been explored, and the real-world practicability of the approach while circumventing shortcomings of convex boosting approaches is also shown. The paper reports results that are comparable to or better than current state-of-the-art approaches.<br /> <br /> == Motivation ==<br /> <br /> In reality, we usually face unclean data and so-called label noise, where some percentage of the classification labels might be corrupted. We would also like to construct strong learners for such data. Noisy labels are an issue because when the writer trained a model with excessive amounts of noisy labels, the performance and accuracy deteriorate significantly. However, if we revisit the general boosting template from above, then we might suspect that we run into trouble as soon as a certain fraction of training examples is misclassified: in this case, the model cannot correctly classify these examples, and the procedure shifts more and more weight towards these bad examples. It eventually leads to a strong learner that accurately predicts the (flawed) training data; however, that does not generalize well anymore. This intuition has been formalized by [LS], who construct a &quot;hard&quot; training data distribution, where a small percentage of labels is randomly flipped. This label noise then leads to a significant reduction in these boosted learners' performance; see the tables below. The more technical reason for this problem is the convexity of the loss function minimized by the boosting procedure. One can use all types of &quot;tricks,&quot; such as early stopping, but this is not solving the fundamental problem at the end of the day.<br /> <br /> == IPBoost: Boosting via Integer Programming ==<br /> <br /> <br /> ===Integer Program Formulation===<br /> Let &lt;math&gt;(x_1,y_1),\cdots, (x_N,y_N) &lt;/math&gt; be the training set with points &lt;math&gt;x_i \in \mathbb{R}^d&lt;/math&gt; and two-class labels &lt;math&gt;y_i \in \{\pm 1\}&lt;/math&gt; <br /> * class of base learners: &lt;math&gt; \Omega :=\{h_1, \cdots, h_L: \mathbb{R}^d \rightarrow \{\pm 1\}\} &lt;/math&gt; and &lt;math&gt;\rho \ge 0&lt;/math&gt; be given. <br /> * error function &lt;math&gt; \eta &lt;/math&gt;<br /> Our boosting model is captured by the integer programming problem. We can call this our primal problem: <br /> <br /> \begin{align*} \min &amp;\sum_{i=1}^N z_i \\ s.t. &amp;\sum_{j=1}^L \eta_{ij}\lambda_k+(1+\rho)z_i \ge \rho \ \ \ <br /> \forall i=1,\cdots, N \\ <br /> &amp;\sum_{j=1}^L \lambda_j=1, \lambda \ge 0,\\ &amp;z\in \{0,1\}^N. \end{align*}<br /> <br /> For the error function &lt;math&gt;\eta&lt;/math&gt;, three options were considered:<br /> <br /> (i) &lt;math&gt; \pm 1 &lt;/math&gt; classification from learners<br /> $$\eta_{ij} := 2\mathbb{I}[h_j(x_i) = y_i] - 1 = y_i \cdot h_j(x_i)$$<br /> (ii) class probabilities of learners<br /> $$\eta_{ij} := 2\mathbb{P}[h_j(x_i) = y_i] - 1$$<br /> (iii) SAMME.R error function for learners<br /> $$\frac{1}{2}y_i\log\left(\frac{\mathbb{P}[h_j(x_i) = 1]}{\mathbb{P}[h_j(x_i) = -1]}\right)$$<br /> <br /> ===Solution of the IP using Column Generation===<br /> <br /> The goal of column generation is to provide an efficient way to solve the linear programming relaxation of the primal by allowing the &lt;math&gt;z_i &lt;/math&gt; variables to assume fractional values. Moreover, columns, i.e., the base learners, &lt;math&gt; \mathcal L \subseteq [L]. &lt;/math&gt; are left out because there are too many to handle efficiently and most of them will have their associated weight equal to zero in the optimal solution anyway. To generate columns, a &lt;i&gt;branch and bound&lt;/i&gt; framework is used. Columns are generated within a<br /> branch-and-bound framework leading effectively to a branch-and-bound-and-price algorithm being used; this is significantly more involved compared to column generation in linear programming. To check the optimality of an LP solution, a subproblem, called the pricing problem, is solved to try to identify columns with a profitable reduced cost. If such columns are found, the LP is re-optimized. Branching occurs when no profitable columns are found, but the LP solution does not satisfy the integrality conditions. Branch and price apply column generation at every node of the branch and bound tree.<br /> <br /> The restricted master primal problem is <br /> <br /> \begin{align*} \min &amp;\sum_{i=1}^N z_i \\ s.t. &amp;\sum_{j\in \mathcal L} \eta_{ij}\lambda_j+(1+\rho)z_i \ge \rho \ \ \ <br /> \forall i \in [N]\\ <br /> &amp;\sum_{j\in \mathcal L}\lambda_j=1, \lambda \ge 0,\\ &amp;z\in \{0,1\}^N. \end{align*}<br /> <br /> <br /> Its restricted dual problem is:<br /> <br /> \begin{align*}\max \rho &amp;\sum^{N}_{i=1}w_i + v - \sum^{N}_{i=1}u_i<br /> \\ s.t. &amp;\sum_{i=1}^N \eta_{ij}w_k+ v \le 0 \ \ \ \forall j \in L \\ <br /> &amp;(1+\rho)w_i - u_i \le 1 \ \ \ \forall i \in [N] \\ &amp;w \ge 0, u \ge 0, v\ free\end{align*}<br /> <br /> Furthermore, there is a pricing problem used to determine, for every supposed optimal solution of the dual, whether the solution is actually optimal, or whether further constraints need to be added into the primal solution. With this pricing problem, we check whether the solution to the restricted dual is feasible. This pricing problem can be expressed as follows:<br /> <br /> $$\sum_{i=1}^N \eta_{ij}w_k^* + v^* &gt; 0$$<br /> <br /> The optimal misclassification values are determined by a branch-and-price process that branches on the variables &lt;math&gt; z_i &lt;/math&gt; and solves the intermediate LPs using column generation.<br /> <br /> ===Algorithm===<br /> &lt;div style=&quot;margin-left: 3em;&quot;&gt;<br /> &lt;math&gt; D = \{(x_i, y_i) | i ∈ I\} ⊆ R^d × \{±1\} &lt;/math&gt;, class of base learners &lt;math&gt;Ω &lt;/math&gt;, margin &lt;math&gt; \rho &lt;/math&gt; &lt;br&gt;<br /> '''Output:''' Boosted learner &lt;math&gt; \sum_{j∈L^∗}h_jλ_j^* &lt;/math&gt; with base learners &lt;math&gt; h_j &lt;/math&gt; and weights &lt;math&gt; λ_j^* &lt;/math&gt; &lt;br&gt;<br /> <br /> &lt;ol&gt;<br /> <br /> &lt;li margin-left:30px&gt; &lt;math&gt; T ← \{([0, 1]^N, \emptyset)\} &lt;/math&gt; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // set of local bounds and learners for open subproblems &lt;/li&gt;<br /> &lt;li&gt; &lt;math&gt; U ← \infty, L^∗ ← \emptyset &lt;/math&gt; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Upper bound on optimal objective &lt;/li&gt;<br /> &lt;li&gt; '''while''' &lt;math&gt;\ T \neq \emptyset &lt;/math&gt; '''do''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; Choose and remove &lt;math&gt;(B,L) &lt;/math&gt; from &lt;math&gt;T &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''repeat''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Solve the primal IP using the local bounds on &lt;math&gt; z &lt;/math&gt; in &lt;math&gt;B&lt;/math&gt; with optimal dual solution &lt;math&gt; (w^∗, v^∗, u^∗) &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Find learner &lt;math&gt; h_j ∈ Ω &lt;/math&gt; satisfying the pricing problem. &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Solve pricing problem. &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''until''' &lt;math&gt; h_j &lt;/math&gt; is not found &lt;/li&gt; <br /> &lt;li&gt; &amp;emsp; Let &lt;math&gt; (\widetilde{λ} , \widetilde{z}) &lt;/math&gt; be the final solution of the primal IP with base learners &lt;math&gt; \widetilde{L} = \{j | \widetilde{λ}_j &gt; 0\} &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''if''' &lt;math&gt; \widetilde{z} ∈ \mathbb{Z}^N &lt;/math&gt; and &lt;math&gt; \sum^{N}_{i=1}\widetilde{z}_i &lt; U &lt;/math&gt; '''then''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; &lt;math&gt; U ← \sum^{N}_{i=1}\widetilde{z}_i, L^∗ ← \widetilde{L}, λ^∗ ← \widetilde{\lambda} &lt;/math&gt; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Update best solution. &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''else''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Choose &lt;math&gt; i ∈ [N] &lt;/math&gt; with &lt;math&gt; \widetilde{z}_i \notin Z &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Set &lt;math&gt; B_0 ← B ∩ \{z_i ≤ 0\}, B_1 ← B ∩ \{z_i ≥ 1\} &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Add &lt;math&gt; (B_0,\widetilde{L}), (B_1,\widetilde{L}) &lt;/math&gt; to &lt;math&gt; T &lt;/math&gt;. &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Create new branching nodes. &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''end''' if &lt;/li&gt;<br /> &lt;li&gt; '''end''' while &lt;/li&gt;<br /> &lt;li&gt; ''Optionally sparsify final solution &lt;math&gt;L^*&lt;/math&gt;'' &lt;/li&gt;<br /> <br /> &lt;/ol&gt;<br /> &lt;/div&gt;<br /> <br /> ===Sparsification===<br /> <br /> When building the boosting model, one of the challenges is to balance model accuracy and generalization. A sparse model is generally easier to interpret. In order to control complexity, overfitting, and generalization of the model, typically some sparsity is applied, which is easier to interpret for some applications. The goal of sparsification is to minimize the following: \begin{equation}\sum_{i=1}^{N}z_i+\sum_{i=1}^{L}\alpha_iy_i\end{equation} There are two techniques commonly used. One is early stopping, another is regularization by adding a complexity term for the learners in the objective function. Previous approaches in the context of LP-based boosting have promoted sparsity by means of cutting planes. Sparsification can be handled in the approach introduced in this paper by solving a delayed integer program using additional cutting planes.<br /> <br /> == Results and Performance ==<br /> <br /> ''All tests were run on identical Linux clusters with Intel Xeon Quad Core CPUs, with 3.50GHz, 10 MB cache, and 32 GB of main memory.''<br /> <br /> <br /> The following results reflect IPBoost's performance in hard instances. Note that by hard instances, we mean a binary classification problem with predefined labels. These examples are tailored to using the ±1 classification from learners. On every hard instance sample, IPBoost significantly outperforms both LPBoost and AdaBoost (although implementations depending on the libraries used have often caused results to differ slightly). For the considered instances the best value for the margin ρ was 0.05 for LPBoost and IPBoost; AdaBoost has no margin parameter. The accuracy reported is test accuracy recorded across various different walkthroughs of the algorithm, while &lt;math&gt;L &lt;/math&gt; denotes the aggregate number of learners required to find the optimal learner, N is the number of points and &lt;math&gt; \gamma &lt;/math&gt; refers to the noise level.<br /> <br /> [[File:ipboostres.png|center]]<br /> <br /> <br /> <br /> For the next table, the classification instances from LIBSVM data sets available at [https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/]. We report accuracies on the test set and train set, respectively. In each case, we report the averages of the accuracies over 10 runs with a different random seed and their standard deviations. We can see IPboost again outperforming LPBoost and AdaBoost significantly. Solving Integer Programming problems is no doubt more computationally expensive than traditional boosting methods like AdaBoost. The average run time of IPBoost (for ρ = 0.05) being 1367.78 seconds, as opposed to LPBoost's 164.35 seconds and AdaBoost's 3.59 seconds reflects exactly that. However, on the flip side, we gain much better stability in our results, as well as higher scores across the board for both training and test sets.<br /> <br /> [[file:svmlibres.png|center]]<br /> <br /> <br /> == Conclusion ==<br /> <br /> IP-boosting avoids the bad performance on well-known hard classes and improves upon LP-boosting and AdaBoost on the LIBSVM instances where even a few percent improvements is valuable. The major drawback is that the running time with the current implementation is much longer. Nevertheless, the algorithm can be improved in the future by solving the intermediate LPs only approximately and deriving tailored heuristics that generate decent primal solutions to save on time.<br /> <br /> The approach is suited very well to an offline setting in which training may take time and where even a small improvement is beneficial or when convex boosters have egregious behaviour. It can also be served as a tool to investigate the general performance of methods like this.<br /> <br /> The IPBoost algorithm added extra complexity into basic boosting models to gain slight accuracy gain while greatly increased the time spent. It is 381 times slower compared to an AdaBoost model on a small dataset which makes the actual usage of this model doubtable. If we are supplied with a larger dataset with millions of records this model would take too long to complete. The base classifier choice was XGBoost which is too complicated for a base classifier, maybe try some weaker learners such as tree stumps to compare the result with other models. In addition, this model might not be accurate compared to the model-ensembling technique where each model utilizes a different algorithm.<br /> <br /> I think IP Boost can be helpful in cases where a small improvement in accuracy is worth the additional computational effort involved. For example, there are applications in Finance where predicting the movement of stock prices slightly more accurately can help traders make millions more. In such cases, investing in expensive computing systems to implement such boosting techniques can perhaps be justified.<br /> <br /> Although the IPBoost can generate better results compared to the base boosting models such as AdaBoost. The time complexity of IPBoost is much higher than other models. Hence, it will be problem to apply IPBoost to the real-world training data since those training data will be large and it's hard to deal with. Thus, we are left with the question that this IPBoost is only a little bit more accurate than the base boosting models but a lot more time complex. So is the model really worth it?<br /> <br /> == Critique ==<br /> <br /> In the introduction section, should change the sentence&quot;3. Push weight of the data ... will have a higher weight on those data that misclassified.&quot; to &quot;... will have a higher weight on those data that are misclassified.<br /> <br /> The column generation method is used when there are a huge number of variables compared to the number of constraints. If there are only a few variables compared to the number of constraints, using this method will not be very efficient.<br /> <br /> It would be better if the efficiency of different boosting models are tested.<br /> <br /> In practice AdaBoost &amp; LP Boost are quite robust to over fitting. In practice if you use simple weak learns stumps (1-level decision trees) then the algorithms are much less prone to over fitting. However the noise level in the data, particularly for AdaBoost is prone to overfitting on noisy datasets. To avoid this use regularized models (AdaBoostReg, LPBoost). Further, when in higher dimensional spaces, AdaBoost can suffer, since its just a linear combination of classifiers. You can use k-fold cross validation to set the stopping parameter to improve this.<br /> <br /> == References ==<br /> <br /> * Pfetsch, M. E., &amp; Pokutta, S. (2020). IPBoost--Non-Convex Boosting via Integer Programming. arXiv preprint arXiv:2002.04679.<br /> <br /> * Freund, Y., &amp; Schapire, R. E. (1995, March). A desicion-theoretic generalization of on-line learning and an application to boosting. In European conference on computational learning theory (pp. 23-37). Springer, Berlin, Heidelberg. pdf</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=IPBoost&diff=49109 IPBoost 2020-12-04T06:42:04Z <p>Z42qin: /* Introduction */</p> <hr /> <div>== Presented by == <br /> Casey De Vera, Solaiman Jawad<br /> <br /> == Introduction == <br /> Boosting is an important and a fairly standard technique in a classification that combines several base learners (learners which have a “low accuracy”), into one boosted learner (learner with a “high accuracy”). Pioneered by the AdaBoost approach of Freund &amp; Schapire, in recent decades there has been extensive work on boosting procedures and analyses of their limitations.<br /> <br /> In a nutshell, boosting procedures are (typically) iterative schemes that roughly work as follows:<br /> <br /> Let &lt;math&gt;T&lt;/math&gt; represents the number of iteration and &lt;math&gt;t&lt;/math&gt; represents the iterator. <br /> <br /> for &lt;math&gt; t= 1, \cdots, T &lt;/math&gt; do the following:<br /> <br /> ::1. Train a learner &lt;math&gt; \mu_t&lt;/math&gt; from a given class of base learners on the data distribution &lt;math&gt; \mathcal D_t&lt;/math&gt;<br /> <br /> ::2. Evaluate the performance of &lt;math&gt; \mu_t&lt;/math&gt; by computing its loss.<br /> <br /> ::3. Push weight of the data distribution &lt;math&gt; \mathcal D_t&lt;/math&gt; towards the misclassified examples leading to &lt;math&gt; \mathcal D_{t+1}&lt;/math&gt;, usually a relatively good model will have a higher weight on those data that misclassified.<br /> <br /> Finally, the learners are combined with some form of voting (e.g., soft or hard voting, averaging, thresholding).<br /> <br /> <br /> [[File:boosting.gif|200px|thumb|right]] A close inspection of most boosting procedures reveals that they solve an underlying convex optimization problem over a convex loss function using coordinate gradient descent. Boosting schemes of this type are often referred to as convex potential boosters. These procedures can achieve exceptional performance on many data sets if the data is correctly labeled. However, they can be trounced by a small number of incorrect labels, which can be quite challenging to fix. We try to solve misclassified examples by moving the weights around, which results in bad performance on unseen data (Shown by zoomed example to the right). In fact, in theory, providing the class of base learners is rich enough, a perfect strong learner can be constructed with accuracy 1; however, clearly, such a learner might not necessarily generalize well. Boosted learners can generate some quite complicated decision boundaries, much more complicated than the base learners. Here is an example from Paul van der Laken’s blog / Extreme gradient boosting gif by Ryan Holbrook. Here data is generated online according to some process with optimal decision boundary represented by the dotted line, and XGBoost was used to learn a classifier:<br /> <br /> <br /> Recently non-convex optimization approaches for solving machine learning problems have gained significant attention. In this paper, the non-convex boosting in classification using integer programming has been explored, and the real-world practicability of the approach while circumventing shortcomings of convex boosting approaches is also shown. The paper reports results that are comparable to or better than current state-of-the-art approaches.<br /> <br /> == Motivation ==<br /> <br /> In reality, we usually face unclean data and so-called label noise, where some percentage of the classification labels might be corrupted. We would also like to construct strong learners for such data. Noisy labels are an issue because when a model is trained with excessive amounts of noisy labels, the performance and accuracy deteriorate greatly. However, if we revisit the general boosting template from above, then we might suspect that we run into trouble as soon as a certain fraction of training examples is misclassified: in this case, these examples cannot be correctly classified and the procedure shifts more and more weight towards these bad examples. This eventually leads to a strong learner, that perfectly predicts the (flawed) training data; however, that does not generalize well anymore. This intuition has been formalized by [LS] who construct a “hard” training data distribution, where a small percentage of labels is randomly flipped. This label noise then leads to a significant reduction in performance of these boosted learners; see tables below. The more technical reason for this problem is actually the convexity of the loss function that is minimized by the boosting procedure. Clearly, one can use all types of “tricks” such as early stopping but at the end of the day, this is not solving the fundamental problem.<br /> <br /> == IPBoost: Boosting via Integer Programming ==<br /> <br /> <br /> ===Integer Program Formulation===<br /> Let &lt;math&gt;(x_1,y_1),\cdots, (x_N,y_N) &lt;/math&gt; be the training set with points &lt;math&gt;x_i \in \mathbb{R}^d&lt;/math&gt; and two-class labels &lt;math&gt;y_i \in \{\pm 1\}&lt;/math&gt; <br /> * class of base learners: &lt;math&gt; \Omega :=\{h_1, \cdots, h_L: \mathbb{R}^d \rightarrow \{\pm 1\}\} &lt;/math&gt; and &lt;math&gt;\rho \ge 0&lt;/math&gt; be given. <br /> * error function &lt;math&gt; \eta &lt;/math&gt;<br /> Our boosting model is captured by the integer programming problem. We can call this our primal problem: <br /> <br /> \begin{align*} \min &amp;\sum_{i=1}^N z_i \\ s.t. &amp;\sum_{j=1}^L \eta_{ij}\lambda_k+(1+\rho)z_i \ge \rho \ \ \ <br /> \forall i=1,\cdots, N \\ <br /> &amp;\sum_{j=1}^L \lambda_j=1, \lambda \ge 0,\\ &amp;z\in \{0,1\}^N. \end{align*}<br /> <br /> For the error function &lt;math&gt;\eta&lt;/math&gt;, three options were considered:<br /> <br /> (i) &lt;math&gt; \pm 1 &lt;/math&gt; classification from learners<br /> $$\eta_{ij} := 2\mathbb{I}[h_j(x_i) = y_i] - 1 = y_i \cdot h_j(x_i)$$<br /> (ii) class probabilities of learners<br /> $$\eta_{ij} := 2\mathbb{P}[h_j(x_i) = y_i] - 1$$<br /> (iii) SAMME.R error function for learners<br /> $$\frac{1}{2}y_i\log\left(\frac{\mathbb{P}[h_j(x_i) = 1]}{\mathbb{P}[h_j(x_i) = -1]}\right)$$<br /> <br /> ===Solution of the IP using Column Generation===<br /> <br /> The goal of column generation is to provide an efficient way to solve the linear programming relaxation of the primal by allowing the &lt;math&gt;z_i &lt;/math&gt; variables to assume fractional values. Moreover, columns, i.e., the base learners, &lt;math&gt; \mathcal L \subseteq [L]. &lt;/math&gt; are left out because there are too many to handle efficiently and most of them will have their associated weight equal to zero in the optimal solution anyway. To generate columns, a &lt;i&gt;branch and bound&lt;/i&gt; framework is used. Columns are generated within a<br /> branch-and-bound framework leading effectively to a branch-and-bound-and-price algorithm being used; this is significantly more involved compared to column generation in linear programming. To check the optimality of an LP solution, a subproblem, called the pricing problem, is solved to try to identify columns with a profitable reduced cost. If such columns are found, the LP is re-optimized. Branching occurs when no profitable columns are found, but the LP solution does not satisfy the integrality conditions. Branch and price apply column generation at every node of the branch and bound tree.<br /> <br /> The restricted master primal problem is <br /> <br /> \begin{align*} \min &amp;\sum_{i=1}^N z_i \\ s.t. &amp;\sum_{j\in \mathcal L} \eta_{ij}\lambda_j+(1+\rho)z_i \ge \rho \ \ \ <br /> \forall i \in [N]\\ <br /> &amp;\sum_{j\in \mathcal L}\lambda_j=1, \lambda \ge 0,\\ &amp;z\in \{0,1\}^N. \end{align*}<br /> <br /> <br /> Its restricted dual problem is:<br /> <br /> \begin{align*}\max \rho &amp;\sum^{N}_{i=1}w_i + v - \sum^{N}_{i=1}u_i<br /> \\ s.t. &amp;\sum_{i=1}^N \eta_{ij}w_k+ v \le 0 \ \ \ \forall j \in L \\ <br /> &amp;(1+\rho)w_i - u_i \le 1 \ \ \ \forall i \in [N] \\ &amp;w \ge 0, u \ge 0, v\ free\end{align*}<br /> <br /> Furthermore, there is a pricing problem used to determine, for every supposed optimal solution of the dual, whether the solution is actually optimal, or whether further constraints need to be added into the primal solution. With this pricing problem, we check whether the solution to the restricted dual is feasible. This pricing problem can be expressed as follows:<br /> <br /> $$\sum_{i=1}^N \eta_{ij}w_k^* + v^* &gt; 0$$<br /> <br /> The optimal misclassification values are determined by a branch-and-price process that branches on the variables &lt;math&gt; z_i &lt;/math&gt; and solves the intermediate LPs using column generation.<br /> <br /> ===Algorithm===<br /> &lt;div style=&quot;margin-left: 3em;&quot;&gt;<br /> &lt;math&gt; D = \{(x_i, y_i) | i ∈ I\} ⊆ R^d × \{±1\} &lt;/math&gt;, class of base learners &lt;math&gt;Ω &lt;/math&gt;, margin &lt;math&gt; \rho &lt;/math&gt; &lt;br&gt;<br /> '''Output:''' Boosted learner &lt;math&gt; \sum_{j∈L^∗}h_jλ_j^* &lt;/math&gt; with base learners &lt;math&gt; h_j &lt;/math&gt; and weights &lt;math&gt; λ_j^* &lt;/math&gt; &lt;br&gt;<br /> <br /> &lt;ol&gt;<br /> <br /> &lt;li margin-left:30px&gt; &lt;math&gt; T ← \{([0, 1]^N, \emptyset)\} &lt;/math&gt; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // set of local bounds and learners for open subproblems &lt;/li&gt;<br /> &lt;li&gt; &lt;math&gt; U ← \infty, L^∗ ← \emptyset &lt;/math&gt; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Upper bound on optimal objective &lt;/li&gt;<br /> &lt;li&gt; '''while''' &lt;math&gt;\ T \neq \emptyset &lt;/math&gt; '''do''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; Choose and remove &lt;math&gt;(B,L) &lt;/math&gt; from &lt;math&gt;T &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''repeat''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Solve the primal IP using the local bounds on &lt;math&gt; z &lt;/math&gt; in &lt;math&gt;B&lt;/math&gt; with optimal dual solution &lt;math&gt; (w^∗, v^∗, u^∗) &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Find learner &lt;math&gt; h_j ∈ Ω &lt;/math&gt; satisfying the pricing problem. &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Solve pricing problem. &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''until''' &lt;math&gt; h_j &lt;/math&gt; is not found &lt;/li&gt; <br /> &lt;li&gt; &amp;emsp; Let &lt;math&gt; (\widetilde{λ} , \widetilde{z}) &lt;/math&gt; be the final solution of the primal IP with base learners &lt;math&gt; \widetilde{L} = \{j | \widetilde{λ}_j &gt; 0\} &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''if''' &lt;math&gt; \widetilde{z} ∈ \mathbb{Z}^N &lt;/math&gt; and &lt;math&gt; \sum^{N}_{i=1}\widetilde{z}_i &lt; U &lt;/math&gt; '''then''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; &lt;math&gt; U ← \sum^{N}_{i=1}\widetilde{z}_i, L^∗ ← \widetilde{L}, λ^∗ ← \widetilde{\lambda} &lt;/math&gt; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Update best solution. &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''else''' &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Choose &lt;math&gt; i ∈ [N] &lt;/math&gt; with &lt;math&gt; \widetilde{z}_i \notin Z &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Set &lt;math&gt; B_0 ← B ∩ \{z_i ≤ 0\}, B_1 ← B ∩ \{z_i ≥ 1\} &lt;/math&gt; &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; &amp;emsp; Add &lt;math&gt; (B_0,\widetilde{L}), (B_1,\widetilde{L}) &lt;/math&gt; to &lt;math&gt; T &lt;/math&gt;. &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; &amp;emsp; // Create new branching nodes. &lt;/li&gt;<br /> &lt;li&gt; &amp;emsp; '''end''' if &lt;/li&gt;<br /> &lt;li&gt; '''end''' while &lt;/li&gt;<br /> &lt;li&gt; ''Optionally sparsify final solution &lt;math&gt;L^*&lt;/math&gt;'' &lt;/li&gt;<br /> <br /> &lt;/ol&gt;<br /> &lt;/div&gt;<br /> <br /> ===Sparsification===<br /> <br /> When building the boosting model, one of the challenges is to balance model accuracy and generalization. A sparse model is generally easier to interpret. In order to control complexity, overfitting, and generalization of the model, typically some sparsity is applied, which is easier to interpret for some applications. The goal of sparsification is to minimize the following: \begin{equation}\sum_{i=1}^{N}z_i+\sum_{i=1}^{L}\alpha_iy_i\end{equation} There are two techniques commonly used. One is early stopping, another is regularization by adding a complexity term for the learners in the objective function. Previous approaches in the context of LP-based boosting have promoted sparsity by means of cutting planes. Sparsification can be handled in the approach introduced in this paper by solving a delayed integer program using additional cutting planes.<br /> <br /> == Results and Performance ==<br /> <br /> ''All tests were run on identical Linux clusters with Intel Xeon Quad Core CPUs, with 3.50GHz, 10 MB cache, and 32 GB of main memory.''<br /> <br /> <br /> The following results reflect IPBoost's performance in hard instances. Note that by hard instances, we mean a binary classification problem with predefined labels. These examples are tailored to using the ±1 classification from learners. On every hard instance sample, IPBoost significantly outperforms both LPBoost and AdaBoost (although implementations depending on the libraries used have often caused results to differ slightly). For the considered instances the best value for the margin ρ was 0.05 for LPBoost and IPBoost; AdaBoost has no margin parameter. The accuracy reported is test accuracy recorded across various different walkthroughs of the algorithm, while &lt;math&gt;L &lt;/math&gt; denotes the aggregate number of learners required to find the optimal learner, N is the number of points and &lt;math&gt; \gamma &lt;/math&gt; refers to the noise level.<br /> <br /> [[File:ipboostres.png|center]]<br /> <br /> <br /> <br /> For the next table, the classification instances from LIBSVM data sets available at [https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/]. We report accuracies on the test set and train set, respectively. In each case, we report the averages of the accuracies over 10 runs with a different random seed and their standard deviations. We can see IPboost again outperforming LPBoost and AdaBoost significantly. Solving Integer Programming problems is no doubt more computationally expensive than traditional boosting methods like AdaBoost. The average run time of IPBoost (for ρ = 0.05) being 1367.78 seconds, as opposed to LPBoost's 164.35 seconds and AdaBoost's 3.59 seconds reflects exactly that. However, on the flip side, we gain much better stability in our results, as well as higher scores across the board for both training and test sets.<br /> <br /> [[file:svmlibres.png|center]]<br /> <br /> <br /> == Conclusion ==<br /> <br /> IP-boosting avoids the bad performance on well-known hard classes and improves upon LP-boosting and AdaBoost on the LIBSVM instances where even a few percent improvements is valuable. The major drawback is that the running time with the current implementation is much longer. Nevertheless, the algorithm can be improved in the future by solving the intermediate LPs only approximately and deriving tailored heuristics that generate decent primal solutions to save on time.<br /> <br /> The approach is suited very well to an offline setting in which training may take time and where even a small improvement is beneficial or when convex boosters have egregious behaviour. It can also be served as a tool to investigate the general performance of methods like this.<br /> <br /> The IPBoost algorithm added extra complexity into basic boosting models to gain slight accuracy gain while greatly increased the time spent. It is 381 times slower compared to an AdaBoost model on a small dataset which makes the actual usage of this model doubtable. If we are supplied with a larger dataset with millions of records this model would take too long to complete. The base classifier choice was XGBoost which is too complicated for a base classifier, maybe try some weaker learners such as tree stumps to compare the result with other models. In addition, this model might not be accurate compared to the model-ensembling technique where each model utilizes a different algorithm.<br /> <br /> I think IP Boost can be helpful in cases where a small improvement in accuracy is worth the additional computational effort involved. For example, there are applications in Finance where predicting the movement of stock prices slightly more accurately can help traders make millions more. In such cases, investing in expensive computing systems to implement such boosting techniques can perhaps be justified.<br /> <br /> Although the IPBoost can generate better results compared to the base boosting models such as AdaBoost. The time complexity of IPBoost is much higher than other models. Hence, it will be problem to apply IPBoost to the real-world training data since those training data will be large and it's hard to deal with. Thus, we are left with the question that this IPBoost is only a little bit more accurate than the base boosting models but a lot more time complex. So is the model really worth it?<br /> <br /> == Critique ==<br /> <br /> In the introduction section, should change the sentence&quot;3. Push weight of the data ... will have a higher weight on those data that misclassified.&quot; to &quot;... will have a higher weight on those data that are misclassified.<br /> <br /> The column generation method is used when there are a huge number of variables compared to the number of constraints. If there are only a few variables compared to the number of constraints, using this method will not be very efficient.<br /> <br /> It would be better if the efficiency of different boosting models are tested.<br /> <br /> In practice AdaBoost &amp; LP Boost are quite robust to over fitting. In practice if you use simple weak learns stumps (1-level decision trees) then the algorithms are much less prone to over fitting. However the noise level in the data, particularly for AdaBoost is prone to overfitting on noisy datasets. To avoid this use regularized models (AdaBoostReg, LPBoost). Further, when in higher dimensional spaces, AdaBoost can suffer, since its just a linear combination of classifiers. You can use k-fold cross validation to set the stopping parameter to improve this.<br /> <br /> == References ==<br /> <br /> * Pfetsch, M. E., &amp; Pokutta, S. (2020). IPBoost--Non-Convex Boosting via Integer Programming. arXiv preprint arXiv:2002.04679.<br /> <br /> * Freund, Y., &amp; Schapire, R. E. (1995, March). A desicion-theoretic generalization of on-line learning and an application to boosting. In European conference on computational learning theory (pp. 23-37). Springer, Berlin, Heidelberg. pdf</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Neural_ODEs&diff=49108 Neural ODEs 2020-12-04T06:27:53Z <p>Z42qin: /* Scope and Limitations */</p> <hr /> <div>== Introduction ==<br /> Chen et al. propose a new class of neural networks called neural ordinary differential equations (ODEs) in their 2018 paper under the same title. Neural network models, such as residual or recurrent networks, can be generalized as a set of transformations through hidden states (a.k.a layers) &lt;math&gt;\mathbf{h}&lt;/math&gt;, given by the equation <br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;&lt;math&gt; \mathbf{h}_{t+1} = \mathbf{h}_t + f(\mathbf{h}_t,\theta_t) &lt;/math&gt; (1) &lt;/div&gt;<br /> <br /> where &lt;math&gt;t \in \{0,...,T\}&lt;/math&gt; and &lt;math&gt;\theta_t&lt;/math&gt; corresponds to the set of parameters or weights in state &lt;math&gt;t&lt;/math&gt;. It is important to note that it has been shown (Lu et al., 2017)(Haber<br /> and Ruthotto, 2017)(Ruthotto and Haber, 2018) that Equation 1 can be viewed as an Euler discretization. Given this Euler description, if the number of layers and step size between layers are taken to their limits, then Equation 1 can instead be described continuously in the form of the ODE, <br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;&lt;math&gt; \frac{d\mathbf{h}(t)}{dt} = f(\mathbf{h}(t),t,\theta) &lt;/math&gt; (2). &lt;/div&gt;<br /> <br /> Equation 2 now describes a network where the output layer &lt;math&gt;\mathbf{h}(T)&lt;/math&gt; is generated by solving for the ODE at time &lt;math&gt;T&lt;/math&gt;, given the initial value at &lt;math&gt;t=0&lt;/math&gt;, where &lt;math&gt;\mathbf{h}(0)&lt;/math&gt; is the input layer of the network. <br /> <br /> With a vast amount of theory and research in the field of solving ODEs numerically, there are a number of benefits to formulating the hidden state dynamics this way. One major advantage is that a continuous description of the network allows for the calculation of &lt;math&gt;f&lt;/math&gt; at arbitrary intervals and locations. The authors provide an example in section five of how the neural ODE network outperforms the discretized version i.e. residual networks, by taking advantage of the continuity of &lt;math&gt;f&lt;/math&gt;. A depiction of this distinction is shown in the figure below. <br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt; [[File:NeuralODEs_Fig1.png|350px]] &lt;/div&gt;<br /> <br /> In section four the authors show that the single-unit bottleneck of normalizing flows can be overcome by constructing a new class of density models that incorporates the neural ODE network formulation.<br /> The next section on automatic differentiation will describe how utilizing ODE solvers allows for the calculation of gradients of the loss function without storing any of the hidden state information. This results in a very low memory requirement for neural ODE networks in comparison to traditional networks that rely on intermediate hidden state quantities for backpropagation.<br /> <br /> == Reverse-mode Automatic Differentiation of ODE Solutions ==<br /> Like most neural networks, optimizing the weight parameters &lt;math&gt;\theta&lt;/math&gt; for a neural ODE network involves finding the gradient of a loss function with respect to those parameters. Differentiating in the forward direction is a simple task, however, this method is very computationally expensive and unstable, as it introduces additional numerical error. Instead, the authors suggest that the gradients can be calculated in the reverse-mode with the adjoint sensitivity method (Pontryagin et al., 1962). This &quot;backpropagation&quot; method solves an augmented version of the forward ODE problem but in reverse, which is something that all ODE solvers are capable of. Section 3 provides results showing that this method gives very desirable memory costs and numerical stability. <br /> <br /> The authors provide an example of the adjoint method by considering the minimization of the scalar-valued loss function &lt;math&gt;L&lt;/math&gt;, which takes the solution of the ODE solver as its argument.<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;[[File:NeuralODEs_Eq1.png|700px]],&lt;/div&gt; <br /> This minimization problem requires the calculation of &lt;math&gt;\frac{\partial L}{\partial \mathbf{z}(t_0)}&lt;/math&gt; and &lt;math&gt;\frac{\partial L}{\partial \theta}&lt;/math&gt;.<br /> <br /> The adjoint itself is defined as &lt;math&gt;\mathbf{a}(t) = \frac{\partial L}{\partial \mathbf{z}(t)}&lt;/math&gt;, which describes the gradient of the loss with respect to the hidden state &lt;math&gt;\mathbf{z}(t)&lt;/math&gt;. By taking the first derivative of the adjoint, another ODE arises in the form of,<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;&lt;math&gt;\frac{d \mathbf{a}(t)}{dt} = -\mathbf{a}(t)^T \frac{\partial f(\mathbf{z}(t),t,\theta)}{\partial \mathbf{z}}&lt;/math&gt; (3).&lt;/div&gt; <br /> <br /> Since the value &lt;math&gt;\mathbf{a}(t_0)&lt;/math&gt; is required to minimize the loss, the ODE in equation 3 must be solved backwards in time from &lt;math&gt;\mathbf{a}(t_1)&lt;/math&gt;. Solving this problem is dependent on the knowledge of the hidden state &lt;math&gt;\mathbf{z}(t)&lt;/math&gt; for all &lt;math&gt;t&lt;/math&gt;, which an neural ODE does not save on the forward pass. Luckily, both &lt;math&gt;\mathbf{a}(t)&lt;/math&gt; and &lt;math&gt;\mathbf{z}(t)&lt;/math&gt; can be calculated in reverse, at the same time, by setting up an augmented version of the dynamics and is shown in the final algorithm. Finally, the derivative &lt;math&gt;dL/d\theta&lt;/math&gt; can be expressed in terms of the adjoint and the hidden state as, <br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;&lt;math&gt; \frac{dL}{d\theta} -\int_{t_1}^{t_0} \mathbf{a}(t)^T\frac{\partial f(\mathbf{z}(t),t,\theta)}{\partial \theta}dt&lt;/math&gt; (4).&lt;/div&gt;<br /> <br /> To obtain very inexpensive calculations of &lt;math&gt;\frac{\partial f}{\partial z}&lt;/math&gt; and &lt;math&gt;\frac{\partial f}{\partial \theta}&lt;/math&gt; in equation 3 and 4, automatic differentiation can be utilized. The authors present an algorithm to calculate the gradients of &lt;math&gt;L&lt;/math&gt; and their dependent quantities with only one call to an ODE solver and is shown below. <br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;[[File:NeuralODEs Algorithm1.png|850px]]&lt;/div&gt;<br /> <br /> If the loss function has a stronger dependence on the hidden states for &lt;math&gt;t \neq t_0,t_1&lt;/math&gt;, then Algorithm 1 can be modified to handle multiple calls to the ODESolve step since most ODE solvers have the capability to provide &lt;math&gt;z(t)&lt;/math&gt; at arbitrary times. A visual depiction of this scenario is shown below. <br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;[[File:NeuralODES Fig2.png|350px]]&lt;/div&gt;<br /> <br /> Please see the [https://arxiv.org/pdf/1806.07366.pdf#page=13 appendix] for extended versions of Algorithm 1 and detailed derivations of each equation in this section.<br /> <br /> == Replacing Residual Networks with ODEs for Supervised Learning ==<br /> Section three of the paper investigates an application of the reverse-mode differentiation described in section two, for the training of neural ODE networks on the MNIST digit data set. To solve for the forward pass in the neural ODE network, the following experiment used Adams-Moulton (AM) method, which is an implicit ODE solver. Although it has a marked improvement over explicit ODE solvers in numerical accuracy, integrating backward through the network for backpropagation is still not preferred and the adjoint sensitivity method is used to perform efficient weight optimization. The network with this &quot;backpropagation&quot; technique is referred to as ODE-Net in this section. <br /> <br /> === Implementation ===<br /> A residual network (ResNet), studied by He et al. (2016), with six standard residual blocks was used as a comparative model for this experiment. The competing model, ODE-net, replaces the residual blocks of the ResNet with the AM solver. As a hybrid of the two models ResNet and ODE-net, a third network was created called RK-Net, which solves the weight optimization of the neural ODE network explicitly through backward Runge-Kutta integration. The following table shows the training and performance results of each network. <br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;[[File:NeuralODEs Table1.png|400px]]&lt;/div&gt;<br /> <br /> Note that &lt;math&gt;L&lt;/math&gt; and &lt;math&gt;\tilde{L}&lt;/math&gt; are the number of layers in ResNet and the number of function calls that the AM method makes for the two ODE networks and are effectively analogous quantities. As shown in Table 1, both of the ODE networks achieve comparable performance to that of the ResNet with a notable decrease in memory cost for ODE-net.<br /> <br /> <br /> Another interesting component of ODE networks is the ability to control the tolerance in the ODE solver used and subsequently the numerical error in the solution. <br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;[[File:NeuralODEs Fig3.png|700px]]&lt;/div&gt;<br /> <br /> The tolerance of the ODE solver is represented by the color bar in Figure 3 above and notice that a variety of effects arise from adjusting this parameter. Primarily, if one was to treat the tolerance as a hyperparameter of sorts, you could tune it such that you find a balance between accuracy (Figure 3a) and computational complexity (Figure 3b). Figure 3c also provides further evidence for the benefits of the adjoint method for the backward pass in ODE-nets since there is a nearly 1:0.5 ratio of forward to backward function calls. In the ResNet and RK-Net examples, this ratio is 1:1.<br /> <br /> Additionally, the authors loosely define the concept of depth in a neural ODE network by referring to Figure 3d. Here it's evident that as you continue to train an ODE network, the number of function evaluations the ODE solver performs increases. As previously mentioned, this quantity is comparable to the network depth of a discretized network. However, as the authors note, this result should be seen as the progression of the network's complexity over training epochs, which is something we expect to increase over time.<br /> <br /> == Continuous Normalizing Flows ==<br /> <br /> Section four tackles the implementation of continuous-depth Neural Networks, but to do so, in the first part of section four the authors discuss theoretically how to establish this kind of network through the use of normalizing flows. The authors use a change of variables method presented in other works (Rezende and Mohamed, 2015), (Dinh et al., 2014), to compute the change of a probability distribution if sample points are transformed through a bijective function, &lt;math&gt;f&lt;/math&gt;.<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;&lt;math&gt;z_1=f(z_0) \Rightarrow \log⁡(p(z_1))=\log⁡(p(z_0))-\log⁡|\det⁡\frac{\partial f}{\partial z_0}|&lt;/math&gt;&lt;/div&gt;<br /> <br /> Where p(z) is the probability distribution of the samples and &lt;math&gt;det⁡\frac{\partial f}{\partial z_0}&lt;/math&gt; is the determinant of the Jacobian which has a cubic cost in the dimension of '''z''' or the number of hidden units in the network. The authors discovered however that transforming the discrete set of hidden layers in the normalizing flow network to continuous transformations simplifies the computations significantly, due primarily to the following theorem:<br /> <br /> '''''Theorem 1:''' (Instantaneous Change of Variables). Let z(t) be a finite continuous random variable with probability p(z(t)) dependent on time. Let dz/dt=f(z(t),t) be a differential equation describing a continuous-in-time transformation of z(t). Assuming that f is uniformly Lipschitz continuous in z and continuous in t, then the change in log probability also follows a differential equation:''<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;&lt;math&gt;\frac{\partial \log(p(z(t)))}{\partial t}=-tr\left(\frac{df}{dz(t)}\right)&lt;/math&gt;&lt;/div&gt;<br /> <br /> The biggest advantage to using this theorem is that the trace function is a linear function, so if the dynamics of the problem, f, is represented by a sum of functions, then so is the log density. This essentially means that you can now compute flow models with only a linear cost with respect to the number of hidden units, &lt;math&gt;M&lt;/math&gt;. In standard normalizing flow models, the cost is &lt;math&gt;O(M^3)&lt;/math&gt;, so they will generally fit many layers with a single hidden unit in each layer.<br /> <br /> Finally the authors use these realizations to construct Continuous Normalizing Flow networks (CNFs) by specifying the parameters of the flow as a function of ''t'', ie, &lt;math&gt;f(z(t),t)&lt;/math&gt;. They also use a gating mechanism for each hidden unit, &lt;math&gt;\frac{dz}{dt}=\sum_n \sigma_n(t)f_n(z)&lt;/math&gt; where &lt;math&gt;\sigma_n(t)\in (0,1)&lt;/math&gt; is a separate neural network which learns when to apply each dynamic &lt;math&gt;f_n&lt;/math&gt;.<br /> <br /> ===Implementation===<br /> <br /> The authors construct two separate types of neural networks to compare against each other, the first is the standard planar Normalizing Flow network (NF) using 64 layers of single hidden units, and the second is their new CNF with 64 hidden units. The NF model is trained over 500,000 iterations using RMSprop, and the CNF network is trained over 10,000 iterations using Adam(algorithm for first-order gradient-based optimization of stochastic objective functions). The loss function is &lt;math&gt;KL(q(x)||p(x))&lt;/math&gt; where &lt;math&gt;q(x)&lt;/math&gt; is the flow model and &lt;math&gt;p(x)&lt;/math&gt; is the target probability density.<br /> <br /> One of the biggest advantages when implementing CNF is that you can train the flow parameters just by performing maximum likelihood estimation on &lt;math&gt;\log(q(x))&lt;/math&gt; given &lt;math&gt;p(x)&lt;/math&gt;, where &lt;math&gt;q(x)&lt;/math&gt; is found via the theorem above, and then reversing the CNF to generate random samples from &lt;math&gt;q(x)&lt;/math&gt;. This reversal of the CNF is done with about the same cost of the forward pass which is not able to be done in an NF network. The following two figures demonstrate the ability of CNF to generate more expressive and accurate output data as compared to standard NF networks.<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;<br /> [[Image:CNFcomparisons.png]]<br /> <br /> [[Image:CNFtransitions.png]]<br /> &lt;/div&gt;<br /> <br /> Figure 4 shows clearly that the CNF structure exhibits significantly lower loss functions than NF. In figure 5 both networks were tasked with transforming a standard Gaussian distribution into a target distribution, not only was the CNF network more accurate on the two moons target, but also the steps it took along the way are much more intuitive than the output from NF.<br /> <br /> == A Generative Latent Function Time-Series Model ==<br /> <br /> One of the largest issues at play in terms of Neural ODE networks is the fact that in many instances, data points are either very sparsely distributed, or irregularly-sampled. The latent dynamics are discretized and the observations are in the bins of fixed duration. This creates issues with missing data and ill-defined latent variables. An example of this is medical records which are only updated when a patient visits a doctor or the hospital. To solve this issue the authors had to create a generative time-series model which would be able to fill in the gaps of missing data. The authors consider each time series as a latent trajectory stemming from the initial local state &lt;math&gt;z_{t_0 }&lt;/math&gt; and determined from a global set of latent parameters. Given a set of observation times and initial state, the generative model constructs points via the following sample procedure:<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;<br /> &lt;math&gt;<br /> z_{t_0}∼p(z_{t_0}) <br /> &lt;/math&gt;<br /> &lt;/div&gt; <br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;<br /> &lt;math&gt;<br /> z_{t_1},z_{t_2},\dots,z_{t_N}={\rm ODESolve}(z_{t_0},f,θ_f,t_0,...,t_N)<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;<br /> each <br /> &lt;math&gt;<br /> x_{t_i}∼p(x│z_{t_i},θ_x)<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> &lt;math&gt;f&lt;/math&gt; is a function which outputs the gradient &lt;math&gt;\frac{\partial z(t)}{\partial t}=f(z(t),θ_f)&lt;/math&gt; which is parameterized via a neural net. In order to train this latent variable model, the authors had to first encode their given data and observation times using an RNN encoder, construct the new points using the trained parameters, then decode the points back into the original space. The following figure describes this process:<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;<br /> [[Image:EncodingFigure.png]]<br /> &lt;/div&gt;<br /> <br /> Another variable which could affect the latent state of a time-series model is how often an event actually occurs. The authors solved this by parameterizing the rate of events in terms of a Poisson process. They described the set of independent observation times in an interval &lt;math&gt;\left[t_{start},t_{end}\right]&lt;/math&gt; as:<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt; <br /> &lt;math&gt;<br /> {\rm log}⁡(p(t_1,t_2,\dots,t_N ))=\sum_{i=1}^N{\rm log}⁡(\lambda(z(t_i)))-\int_{t_{start}}^{t_{end}}λ(z(t))dt<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> where &lt;math&gt;\lambda(*)&lt;/math&gt; is parameterized via another neural network.<br /> <br /> ===Implementation===<br /> <br /> To test the effectiveness of the Latent time-series ODE model (LODE), they fit the encoder with 25 hidden units, parametrize function f with a one-layer 20 hidden unit network, and the decoder as another neural network with 20 hidden units. They compare this against a standard recurrent neural net (RNN) with 25 hidden units trained to minimize Gaussian log-likelihood. The authors tested both of these network systems on a dataset of 2-dimensional spirals which either rotated clockwise or counter-clockwise and sampled the positions of each spiral at 100 equally spaced time steps. They can then simulate irregularly timed data by taking random amounts of points without replacement from each spiral. The next two figures show the outcome of these experiments:<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt;<br /> [[Image:LODEtestresults.png]] [[Image:SpiralFigure.png|The blue lines represent the test data learned curves and the red lines represent the extrapolated curves predicted by each model]]<br /> &lt;/div&gt;<br /> <br /> In the figure on the right the blue lines represent the test data learned curves and the red lines represent the extrapolated curves predicted by each model. It is noted that the LODE performs significantly better than the standard RNN model, especially on smaller sets of data points.<br /> <br /> == Scope and Limitations ==<br /> <br /> This part mainly discusses the scope and limitations of the paper. Firstly, while &quot;batching&quot; the training data is a useful step in standard neural nets and can still be applied here by combining the ODEs associated with each batch, the authors found that controlling the error, in this case, may increase the number of calculations required. In practice, however, the number of calculations did not increase significantly.<br /> <br /> So long as the model proposed in this paper uses finite weights and Lipschitz nonlinearities, Picard's existence theorem (Coddington and Levinson, 1955) applies, which guarantees that the solution to the IVP exists and is unique. This theorem holds for the model presented above when the network has finite weights and uses nonlinearities in the Lipshitz class.<br /> <br /> In controlling the error amount in the model, the authors could only reduce tolerances to approximately 10−3 and 10−5 in classification and density estimation, respectively, without also degrading the computational performance.<br /> <br /> The authors believe that reconstructing state trajectories by running the dynamics backward can introduce extra numerical error. They address a possible solution to this problem by checkpointing specific time steps and storing intermediate values of z on the forward pass. Then while reconstructing, it does each part individually between checkpoints. The authors acknowledged that they informally checked this method's validity since they do not consider it a practical problem.<br /> <br /> There remain, however, areas where standard neural networks may perform better than Neural ODEs. Firstly, conventional nets can fit non-homeomorphic functions, such as functions whose output has a smaller dimension than their input or that change the input space's topology. However, this could be handled by composing ODE nets with standard network layers. Another point is that conventional nets can be evaluated precisely with a fixed amount of computation, are typically faster to train, and do not require an error tolerance for a solver.<br /> <br /> == Conclusions and Critiques ==<br /> <br /> We covered the use of black-box ODE solvers as a model component and their application to initial value problems constructed from real applications​. Neural ODE Networks show promising gains in computational cost without large sacrifices in accuracy when applied to certain problems​. A drawback of some of these implementations is that the ODE Neural Networks are limited by the underlying distributions of the problems they are trying to solve (requirement of Lipschitz continuity, etc.)​. There are plenty of further advances to be made in this field as hundreds of years of ODE theory and literature is available, so this is currently an important area of research.<br /> <br /> ODEs indeed represent an important area of applied mathematics where neural networks can be used to solve them numerically. Perhaps, a parallel area of investigation can be PDEs (Partial Differential Equations). PDEs are also widely encountered in many areas of applied mathematics, physics, social sciences, and many other fields. It will be interesting to see how neural networks can be used to solve PDEs.<br /> <br /> == More Critiques ==<br /> Table 1 shows a comparison between different implementations which is very helpful. We can see from the table that the 1-Layer MLP has the largest test error and the one with the best performance should be ODE-Net. Although it doesn't have the lowest test error (the test error of ODE-Net is 0.42% and the lowest test error is 0.41% for ResNet), it still has the least number of parameters, memory, and time. This convinced us that it can be widely used in other applications. <br /> <br /> For the last paragraph in the scope and limitations section, I guess the author wants to use the word &quot;than&quot; instead of using &quot;that&quot; in the sentence &quot;for example, functions whose output has a smaller dimension that their input, or that change the topology of the input space.&quot;<br /> <br /> This paper covers the memory efficiency of Neural ODE Networks, but does not address runtime. In practice, most systems are bound by latency requirements more-so than memory requirements (except in edge device cases). Though it may be unreasonable to expect the authors to produce a performance-optimized implementation, it would be insightful to understand the computational bottlenecks so existing frameworks can take steps to address them. This model looks promising and practical performance is the key to enabling future research in this.<br /> <br /> The above critique also questions the need for a neural network for such a problem. This problem was studied by Brunel et al. and they presented their solution in their paper ''Parametric Estimation of Ordinary Differential Equations with Orthogonality Conditions''. While this solution also requires iteratively solving a complex optimization problem, they did not require the massive memory and runtime overhead of a neural network. For the neural network solution to demonstrate its potential, it should be including experimental comparisons with specialized ordinary differential equation algorithms instead of simply comparing with a general recurrent neural network.<br /> <br /> Table 2 shows that potential ODEs have lower predicted RMSE, and more relevant information should be provided. For example, the reason of setting the n to 30/50/100 can be simply described. And it is good to talk more about the performance of Latent ODE since the change of n does not have much impact on its RMSE.<br /> <br /> == References ==<br /> Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. ''arXiv preprint arXiv'':1710.10121, 2017.<br /> <br /> Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. ''Inverse Problems'', 34 (1):014004, 2017.<br /> <br /> Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. ''arXiv preprint arXiv'':1804.04272, 2018.<br /> <br /> Lev Semenovich Pontryagin, EF Mishchenko, VG Boltyanskii, and RV Gamkrelidze. ''The mathematical theory of optimal processes''. 1962.<br /> <br /> Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ''European conference on computer vision'', pages 630–645. Springer, 2016b.<br /> <br /> Earl A Coddington and Norman Levinson. ''Theory of ordinary differential equations''. Tata McGrawHill Education, 1955.<br /> <br /> Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. ''arXiv preprint arXiv:1505.05770'', 2015.<br /> <br /> Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. ''arXiv preprint arXiv:1410.8516'', 2014.<br /> <br /> Brunel, N. J., Clairon, Q., &amp; d’Alché-Buc, F. (2014). Parametric estimation of ordinary differential equations with orthogonality conditions. ''Journal of the American Statistical Association'', 109(505), 173-185.<br /> <br /> A. Iserles. A first course in the numerical analysis of differential equations. Cambridge Texts in Applied Mathematics, second edition, 2009</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_universal_SNP_and_small-indel_variant_caller_using_deep_neural_networks&diff=49107 A universal SNP and small-indel variant caller using deep neural networks 2020-12-04T06:24:31Z <p>Z42qin: /* Critique and Discussion */</p> <hr /> <div>== Background ==<br /> <br /> Genes determine Biological functions, and mutants or alleles(one of two or more alternative forms of a gene that arise by mutation and are found at the same place on a chromosome) of those genes determine differences within a function. Determining novel alleles is very important in understanding the genetic variation within a species. For example, different alleles of the gene OCA2 determine eyes' color. All animals receive one copy of each gene from each of their parents. Mutations of a gene are classified as either homozygous (both copies are the same) or heterozygous (the two copies are different).<br /> <br /> Next-generation sequencing is a prevalent technique for sequencing or reading DNA. Since all genes are encoded as DNA, sequencing is an essential tool for understanding genes. Next-generation sequencing works by reading short sections of DNA of length k, called k-means, and then piecing them together or aligning them to a reference genome. Next-generation sequencing is relatively fast and inexpensive, although it can randomly misidentify some nucleotides, introducing errors. However, NGS reading is errorful and arises from a complex error process depending on various factors.<br /> <br /> The process of variant calling is determining novel alleles from sequencing data (typically next-generation sequencing data). Some significant alleles only differ from the &quot;standard&quot; version of a gene by only a single base pair, such as the mutation which causes multiple sclerosis. Therefore it is crucial to accurately call single nucleotide swaps/polymorphisms (SNPs), insertions, and deletions (indels). Calling SNPs and small indels are technically challenging since it requires a program to distinguish between genuinely novel mutations and errors in the sequencing data.<br /> <br /> Previous approaches usually involved using various statistical techniques. A widely used one is GATK. GATK uses a combination of logistic regression, hidden Markov models, naive Bayes classification, and Gaussian mixture models to perform the process . However, these methods have their weaknesses as some assumptions do not hold (i.e., independence assumptions), and it's hard to generalize them to other sequencing technologies.<br /> <br /> This paper aims to solve the problem of calling SNPs and small indels using a convolutional neural net by casting the reads as images and classifying whether they contain a mutation. It introduces a variant caller called &quot;DeepVariant&quot;, which requires no specialized knowledge, but performs better than previous state-of-art methods.<br /> <br /> == Overview ==<br /> <br /> In Figure 1, the DeepVariant workflow overview is illustrated.<br /> <br /> [[File:figure 111.JPG|Figure 1. In all panels, blue boxes represent data and red boxes are processes]]<br /> <br /> <br /> Initially, the NGS reads aligned to a reference genome are scanned for candidate variants which are different sites from the reference genome. The read and reference data are encoded as an image for each candidate variant site. Then, trained CNN can compute the genotype likelihoods, (heterozygous or homozygous) for each of the candidate variants (figure1, left box). <br /> To train the CNN for image classification purposes, the DeepVariant machinery makes pileup images for a labeled sample with known genotypes. These labeled images and known genotypes are provided to CNN for training, and a stochastic gradient descent algorithm is used to optimize the CNN parameters to maximize genotype prediction accuracy. After the convergence of the model, the final model is frozen to use for calling mutations for other image classification tests (figure1, middle box).<br /> For example, in figure 1 (right box), the reference and read bases are encoded into a pileup image at a candidate variant site. CNN using this encoded image computes the genotype likelihoods for the three diploid genotype states of homozygous reference (hom-ref), heterozygous (het) or homozygous alternate (hom-alt). In this example, a heterozygous variant call is emitted, as the most probable genotype here is “het”.<br /> <br /> == Preprocessing ==<br /> <br /> Before the sequencing reads can be fed into the classifier, they must be pre-processed. There are many pre-processing steps that are necessary for this algorithm. These steps represent the real novelty in this technique by transforming the data to allow us to use more common neural network architectures for classification. The pre-processing of the data can be broken into three phases: the realignment of reads, finding candidate variants and creating the candidate variants' images.<br /> <br /> The realignment of the pre-processing reads phase is essential to ensure the sequences can be adequately compared to the reference sequences. First, the sequences are aligned to a reference sequence. Reads that align poorly are grouped with other reads around them to build that section, or haplotype, from scratch. If there is strong evidence that the new version of the haplotype fits the reads well, the reads are re-aligned. This process updates the CIGAR (Compact Idiosyncratic Gapped Alignment Report) string to represent a sequence's alignment to a reference for each read.<br /> <br /> Once the reads are correctly aligned, the algorithm then proceeds to find candidate variants, regions in the DNA sequence containing variants. It is these candidate variants that will eventually be passed as input to the neural network. To find these, we need to consider each position in the reference sequence independently. Any unusable reads are filtered at this point. This includes reads that are not appropriately aligned, marked as duplicates, those that fail vendor quality checks, or whose mapping quality is less than ten. For each site in the genome, we collect all the remaining reads that overlap that site. The corresponding allele aligned to that site is then determined by decoding the CIGAR string, which was updated in each read's realignment phase. The alleles are then classified into one of four categories: reference-matching base, reference-mismatching base, insertion with a specific sequence, or deletion with a specific length, and the number of occurrences of each distinct allele across all reads is counted. Read bases are only included as potential alleles if each base in the allele has a quality score of at least 10.<br /> <br /> The last phase of pre-processing is to convert these candidate variants into images representing the data with candidate variants identified. This allows for the use of well established convolutional neural networks for image classification for this technical problem. Each color channel is used to store a different piece of information about a candidate variant. The red channel encodes which base we have (A, G, C, or T) by mapping each base to a particular value. The quality of the read is mapped to the green color channel.<br /> <br /> Moreover, the blue channel encodes whether or not the reference is on the positive strand of the DNA. Each row of the image represents a read, and each column represents a particular base in that read. The reference strand is repeated for the first five rows of the encoded image to maintain its information after a 5x5 convolution is applied. With the data pre-processing complete, the images can then be passed into the neural network for classification.<br /> <br /> == Neural Network ==<br /> <br /> The neural network used is a convolutional neural network. Although the full network architecture is not revealed in the paper, there are several details which we can discuss. The architecture of the network is an input layer attached to an adapted Inception v2 ImageNet model with nine partitions. The inception v2 model in particular uses a series of CNNs. One interesting aspect about the Inception model is that rather than optimizing a series of hyperparameters in order to determine the most optimal parameter configuration, Inception instead concatenates a series of different sizes of filters on the same layer, which acts to learn the best architecture out of these concatenated filters. The input layer takes as input the images representing the candidate variants and rescales them to 299x299 pixels. The output layer is a three-class Softmax layer initialized with Gaussian random weights with a standard deviation of 0.001. This final layer is fully connected to the previous layer. The three classes are the homozygous reference (meaning it is not a variant), heterozygous variant, and homozygous variant. The candidate variant is classified into the class with the highest probability. The model is trained using stochastic gradient descent with a weight decay of 0.00004. The training was done in mini-batches, each with 32 images, using a root mean squared (RMS) decay of 0.9. For the multiple sequencing technologies experiments, a single model was trained with a learning rate of 0.0015 and momentum 0.8 for 250,000 update steps. For all other experiments, multiple models were trained, and the one with the highest accuracy on the training set was chosen as the final model. The multiple models stem from using each combination of the possible parameter values for the learning rate (0.00095, 0.001, 0.0015) and momentum (0.8, 0.85, 0.9). These models were trained for 80 hours, or until the training accuracy converged.<br /> <br /> == Results ==<br /> <br /> DeepVariant was trained using data available from the CEPH (Centre d’Etude du Polymorphism Humain) female sample NA12878 and was evaluated on the unseen Ashkenazi male sample NA24385. The results were compared with other most commonly used bioinformatics methods, such as the GATK, FreeBayes22, SAMtools23, 16GT24 and Strelka25 (Table 1). For better comparison, the overall accuracy (F1), recall, precision, and numbers of true positives (TP), false negatives (FN) and false positives (FP) are illustrated over the whole genome.<br /> <br /> [[File:table 11.JPG]]<br /> <br /> DeepVariant showed the highest accuracy and more than 50% fewer errors per genome compared to the next best algorithm. <br /> <br /> They also evaluated the same set of algorithms using the synthetic diploid sample CHM1-CHM1326 (Table 2).<br /> <br /> [[File:Table 333.JPG]]<br /> <br /> Results illustrated that the DeepVariant method outperformed all other algorithms for variant calling (SNP and indel) and showed the highest accuracy in terms of F1, Recall, precision and TP.<br /> <br /> == Conclusion ==<br /> <br /> This endeavor to further advance a data-centric approach to understanding the gene sequence illustrates the advantages of deep learning over humans. With billions of DNA base pairs, no humans can digest that amount of gene expressions. In the past, computational techniques are unfeasible due to the lack of computing power, but in the 21st century, it seems that machine learning is the way to go for molecular biology.<br /> <br /> DeepVariant’s strong performance on human data proves that deep learning is a promising technique for variant calling. Perhaps the most exciting feature of DeepVariant is its simplicity. Unlike other states of the art variant callers, DeepVariant does not know the sequencing technologies that create the reads or even the biological processes that introduce mutations. It simplifies the problem of variant calling to preprocessing the reads and training a generic deep learning model. It also suggests that DeepVariant could be significantly improved by tailoring the preprocessing to specific sequencing technologies and developing a dedicated CNN architecture for the reads, rather than casting them as images.<br /> <br /> == Critique and Discussion==<br /> <br /> The paper presents an attractive method for solving a significant problem. Building &quot;images&quot; of reads and running them through a generic image classification CNN seems like a strange approach, and, interestingly, it works well. The most significant issues with the paper are the lack of specific information about how the methods. Some extra information is included in the supplementary material, but there are still some significant gaps. In particular:<br /> <br /> 1. What is the structure of the neural net? How many layers, and what sizes? The paper for ConvNet does not have this information. We suspect that this might be a trade secret that Google is protecting.<br /> <br /> 2. How is the realignment step implemented? The paper mentions that it uses a &quot;De-Bruijn-graph-based read assembly procedure&quot; to realign reads to a new haplotype. It is a non-standard step in most genomics workflows, yet the paper does not describe how they do the realignment or build the haplotypes.<br /> <br /> 3. How did they settle on the image construction algorithm? The authors provide pseudocode for the construction of pileup images, but they do not describe how to make decisions. For instance, the color values for different base pairs are not evenly spaced. Also, the image begins with five rows of the reference genome.<br /> <br /> One thing we appreciated about the paper was their commentary on future developments. The authors clarify that this approach can be improved on and provide specific ideas for the next steps.<br /> <br /> Overall, the paper presents an interesting idea with strong results but lacks detail in some vital implementation pieces.<br /> <br /> The topic of this project is good, but we need more details on the algorithm. In the neural network part, the details are not enough; authors should provide a figure to explain better how the model works and the model's structure. Otherwise, we cannot understand how the model works. When we preprocess the data if different data have different lengths, shall we add more information or drop some information to match?<br /> <br /> 5 Particularly, which package did the researchers use to perform this analysis? Different packages of deep learning can have different accuracy and efficiency while making predictions on this data set. <br /> <br /> Further studies on DeepVariant [https://www.nature.com/articles/s41598-018-36177-7 have shown] that it is a framework with great potential and sets the medical standard genetics field.<br /> <br /> 6. It was mentioned that part of the network used is an &quot;adapted Inception v2 ImageNet model&quot; - does this mean that it used an Inception v2 model that was trained on ImageNet? This is not clear, but if this is the case, then why is this useful? Why would the features that are extracted for an image be useful for genomics? Did they try using a model that was not trained? Also, they describe the preprocessing method used but were there any alternatives that they considered?<br /> <br /> 7. A more extensive discussion on the &quot;Neural Network&quot; section can be given. For example, the paragraph's last sentence says that &quot;Models are trained for 80 hours, or until the training accuracy converged.&quot; This sentence implies that if the training accuracy does converge, it usually takes less than 80 hours. More exciting data can thus be presented about the training accuracy converging time. How often do the models converge? How long does it take for the models to converge on average? Moreover, why is the number 80 chosen here? Is it a random upper bound set by people to bound the runtime of the model training, or is the number 80 carefully chosen so that it is twice/three times/ten times the average training accuracy converging time?<br /> <br /> == References ==<br />  Hartwell, L.H. ''et. al.'' ''Genetics: From Genes to Genomes''. (McGraw-Hill Ryerson, 2014).<br /> <br />  Poplin, R. ''et. al''. A universal SNP and small-indel variant caller using deep neural networks. ''Nature Biotechnology'' '''36''', 983-987 (2018).</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_universal_SNP_and_small-indel_variant_caller_using_deep_neural_networks&diff=49106 A universal SNP and small-indel variant caller using deep neural networks 2020-12-04T06:20:40Z <p>Z42qin: /* Conclusion */</p> <hr /> <div>== Background ==<br /> <br /> Genes determine Biological functions, and mutants or alleles(one of two or more alternative forms of a gene that arise by mutation and are found at the same place on a chromosome) of those genes determine differences within a function. Determining novel alleles is very important in understanding the genetic variation within a species. For example, different alleles of the gene OCA2 determine eyes' color. All animals receive one copy of each gene from each of their parents. Mutations of a gene are classified as either homozygous (both copies are the same) or heterozygous (the two copies are different).<br /> <br /> Next-generation sequencing is a prevalent technique for sequencing or reading DNA. Since all genes are encoded as DNA, sequencing is an essential tool for understanding genes. Next-generation sequencing works by reading short sections of DNA of length k, called k-means, and then piecing them together or aligning them to a reference genome. Next-generation sequencing is relatively fast and inexpensive, although it can randomly misidentify some nucleotides, introducing errors. However, NGS reading is errorful and arises from a complex error process depending on various factors.<br /> <br /> The process of variant calling is determining novel alleles from sequencing data (typically next-generation sequencing data). Some significant alleles only differ from the &quot;standard&quot; version of a gene by only a single base pair, such as the mutation which causes multiple sclerosis. Therefore it is crucial to accurately call single nucleotide swaps/polymorphisms (SNPs), insertions, and deletions (indels). Calling SNPs and small indels are technically challenging since it requires a program to distinguish between genuinely novel mutations and errors in the sequencing data.<br /> <br /> Previous approaches usually involved using various statistical techniques. A widely used one is GATK. GATK uses a combination of logistic regression, hidden Markov models, naive Bayes classification, and Gaussian mixture models to perform the process . However, these methods have their weaknesses as some assumptions do not hold (i.e., independence assumptions), and it's hard to generalize them to other sequencing technologies.<br /> <br /> This paper aims to solve the problem of calling SNPs and small indels using a convolutional neural net by casting the reads as images and classifying whether they contain a mutation. It introduces a variant caller called &quot;DeepVariant&quot;, which requires no specialized knowledge, but performs better than previous state-of-art methods.<br /> <br /> == Overview ==<br /> <br /> In Figure 1, the DeepVariant workflow overview is illustrated.<br /> <br /> [[File:figure 111.JPG|Figure 1. In all panels, blue boxes represent data and red boxes are processes]]<br /> <br /> <br /> Initially, the NGS reads aligned to a reference genome are scanned for candidate variants which are different sites from the reference genome. The read and reference data are encoded as an image for each candidate variant site. Then, trained CNN can compute the genotype likelihoods, (heterozygous or homozygous) for each of the candidate variants (figure1, left box). <br /> To train the CNN for image classification purposes, the DeepVariant machinery makes pileup images for a labeled sample with known genotypes. These labeled images and known genotypes are provided to CNN for training, and a stochastic gradient descent algorithm is used to optimize the CNN parameters to maximize genotype prediction accuracy. After the convergence of the model, the final model is frozen to use for calling mutations for other image classification tests (figure1, middle box).<br /> For example, in figure 1 (right box), the reference and read bases are encoded into a pileup image at a candidate variant site. CNN using this encoded image computes the genotype likelihoods for the three diploid genotype states of homozygous reference (hom-ref), heterozygous (het) or homozygous alternate (hom-alt). In this example, a heterozygous variant call is emitted, as the most probable genotype here is “het”.<br /> <br /> == Preprocessing ==<br /> <br /> Before the sequencing reads can be fed into the classifier, they must be pre-processed. There are many pre-processing steps that are necessary for this algorithm. These steps represent the real novelty in this technique by transforming the data to allow us to use more common neural network architectures for classification. The pre-processing of the data can be broken into three phases: the realignment of reads, finding candidate variants and creating the candidate variants' images.<br /> <br /> The realignment of the pre-processing reads phase is essential to ensure the sequences can be adequately compared to the reference sequences. First, the sequences are aligned to a reference sequence. Reads that align poorly are grouped with other reads around them to build that section, or haplotype, from scratch. If there is strong evidence that the new version of the haplotype fits the reads well, the reads are re-aligned. This process updates the CIGAR (Compact Idiosyncratic Gapped Alignment Report) string to represent a sequence's alignment to a reference for each read.<br /> <br /> Once the reads are correctly aligned, the algorithm then proceeds to find candidate variants, regions in the DNA sequence containing variants. It is these candidate variants that will eventually be passed as input to the neural network. To find these, we need to consider each position in the reference sequence independently. Any unusable reads are filtered at this point. This includes reads that are not appropriately aligned, marked as duplicates, those that fail vendor quality checks, or whose mapping quality is less than ten. For each site in the genome, we collect all the remaining reads that overlap that site. The corresponding allele aligned to that site is then determined by decoding the CIGAR string, which was updated in each read's realignment phase. The alleles are then classified into one of four categories: reference-matching base, reference-mismatching base, insertion with a specific sequence, or deletion with a specific length, and the number of occurrences of each distinct allele across all reads is counted. Read bases are only included as potential alleles if each base in the allele has a quality score of at least 10.<br /> <br /> The last phase of pre-processing is to convert these candidate variants into images representing the data with candidate variants identified. This allows for the use of well established convolutional neural networks for image classification for this technical problem. Each color channel is used to store a different piece of information about a candidate variant. The red channel encodes which base we have (A, G, C, or T) by mapping each base to a particular value. The quality of the read is mapped to the green color channel.<br /> <br /> Moreover, the blue channel encodes whether or not the reference is on the positive strand of the DNA. Each row of the image represents a read, and each column represents a particular base in that read. The reference strand is repeated for the first five rows of the encoded image to maintain its information after a 5x5 convolution is applied. With the data pre-processing complete, the images can then be passed into the neural network for classification.<br /> <br /> == Neural Network ==<br /> <br /> The neural network used is a convolutional neural network. Although the full network architecture is not revealed in the paper, there are several details which we can discuss. The architecture of the network is an input layer attached to an adapted Inception v2 ImageNet model with nine partitions. The inception v2 model in particular uses a series of CNNs. One interesting aspect about the Inception model is that rather than optimizing a series of hyperparameters in order to determine the most optimal parameter configuration, Inception instead concatenates a series of different sizes of filters on the same layer, which acts to learn the best architecture out of these concatenated filters. The input layer takes as input the images representing the candidate variants and rescales them to 299x299 pixels. The output layer is a three-class Softmax layer initialized with Gaussian random weights with a standard deviation of 0.001. This final layer is fully connected to the previous layer. The three classes are the homozygous reference (meaning it is not a variant), heterozygous variant, and homozygous variant. The candidate variant is classified into the class with the highest probability. The model is trained using stochastic gradient descent with a weight decay of 0.00004. The training was done in mini-batches, each with 32 images, using a root mean squared (RMS) decay of 0.9. For the multiple sequencing technologies experiments, a single model was trained with a learning rate of 0.0015 and momentum 0.8 for 250,000 update steps. For all other experiments, multiple models were trained, and the one with the highest accuracy on the training set was chosen as the final model. The multiple models stem from using each combination of the possible parameter values for the learning rate (0.00095, 0.001, 0.0015) and momentum (0.8, 0.85, 0.9). These models were trained for 80 hours, or until the training accuracy converged.<br /> <br /> == Results ==<br /> <br /> DeepVariant was trained using data available from the CEPH (Centre d’Etude du Polymorphism Humain) female sample NA12878 and was evaluated on the unseen Ashkenazi male sample NA24385. The results were compared with other most commonly used bioinformatics methods, such as the GATK, FreeBayes22, SAMtools23, 16GT24 and Strelka25 (Table 1). For better comparison, the overall accuracy (F1), recall, precision, and numbers of true positives (TP), false negatives (FN) and false positives (FP) are illustrated over the whole genome.<br /> <br /> [[File:table 11.JPG]]<br /> <br /> DeepVariant showed the highest accuracy and more than 50% fewer errors per genome compared to the next best algorithm. <br /> <br /> They also evaluated the same set of algorithms using the synthetic diploid sample CHM1-CHM1326 (Table 2).<br /> <br /> [[File:Table 333.JPG]]<br /> <br /> Results illustrated that the DeepVariant method outperformed all other algorithms for variant calling (SNP and indel) and showed the highest accuracy in terms of F1, Recall, precision and TP.<br /> <br /> == Conclusion ==<br /> <br /> This endeavor to further advance a data-centric approach to understanding the gene sequence illustrates the advantages of deep learning over humans. With billions of DNA base pairs, no humans can digest that amount of gene expressions. In the past, computational techniques are unfeasible due to the lack of computing power, but in the 21st century, it seems that machine learning is the way to go for molecular biology.<br /> <br /> DeepVariant’s strong performance on human data proves that deep learning is a promising technique for variant calling. Perhaps the most exciting feature of DeepVariant is its simplicity. Unlike other states of the art variant callers, DeepVariant does not know the sequencing technologies that create the reads or even the biological processes that introduce mutations. It simplifies the problem of variant calling to preprocessing the reads and training a generic deep learning model. It also suggests that DeepVariant could be significantly improved by tailoring the preprocessing to specific sequencing technologies and developing a dedicated CNN architecture for the reads, rather than casting them as images.<br /> <br /> == Critique and Discussion==<br /> <br /> The paper presents an interesting method for solving an important problem. Building &quot;images&quot; of reads and running them through a generic image classification CNN seems like a strange approach, and it is interesting that it works well. The biggest issues with the paper are the lack of specific information about how the methods. Some extra information is included in the supplementary material, but there are still some big gaps. In particular:<br /> <br /> 1. What is the structure of the neural net? How many layers, and what sizes? The paper for ConvNet which is cited does not have this information. We suspect that this might be a trade secret that Google is protecting.<br /> <br /> 2. How is the realignment step implemented? The paper mentions that it uses a &quot;De-Bruijn-graph-based read assembly procedure&quot; to realign reads to a new haplotype. This is a non-standard step in most genomics workflows yet the paper does not describe how they do the realignment or how they build the haplotypes.<br /> <br /> 3. How did they settle on the image construction algorithm? The authors provide pseudocode for the construction of pileup images but they do not describe how the decisions for made. For instance, the colour values for different base pairs are not evenly spaced. Also, the image begins with 5 rows of the reference genome.<br /> <br /> One thing we appreciated about the paper was their commentary on future developments. The authors make it very clear that this approach can be improved on and provide specific ideas for next steps.<br /> <br /> Overall, the paper presents an interesting idea with strong results, but lacks detail in some key pieces of the implementation.<br /> <br /> The topic of this project is good but we need to more details of the algorithm. In the neural network part, the details are not enough, Authors should provide a figure to better explain how the model works and the structure of the model. Otherwise we cannot understand how the model works. Also, when we are preprocessing the data, if different data have different lengths, shall we add more information or drop some information so they match?<br /> <br /> 5 Particularly, which package did the researchers use to perform this analysis? Different packages of deep learning can have different accuracy and efficiency while making predictions on this data set. <br /> <br /> Further studies on DeepVariant [https://www.nature.com/articles/s41598-018-36177-7 have shown] that it is a framework with great potential and sets the standard in the medical genetics field.<br /> <br /> 6. It was mentioned that part of the network used is an &quot;adapted Inception v2 ImageNet model&quot; - does this mean that it used an Inception v2 model that was pretrained on ImageNet? This is not clear, but if this is the case then why is this useful? Why would the features that are extracted for an image be useful for genomics? Did they try using a model that wasn't pretrained? Also, they describe the preprocessing method used but were there any alternatives that they considered?<br /> <br /> 7. A more extensive discussion on the &quot;Neural Network&quot; section can be give. For example, the last sentence of the paragraph says that &quot;Models are trained for 80 hours, or until the training accuracy converged.&quot; This sentence implies that if the training accuracy does converge, it usually takes less than 80 hours. More interesting data can thus be presented about the training accuracy converging time. How often do the models converge? How long does it take for the models to converge on average? Moreover, why is the number 80 chosen here? Is it a random upper bound set by people to bound the runtime of the model training, or is the number 80 carefully chosen so that it's twice/three times/ten times of the average training accuracy converging time? I believe that these are interesting facts to look at.<br /> <br /> == References ==<br />  Hartwell, L.H. ''et. al.'' ''Genetics: From Genes to Genomes''. (McGraw-Hill Ryerson, 2014).<br /> <br />  Poplin, R. ''et. al''. A universal SNP and small-indel variant caller using deep neural networks. ''Nature Biotechnology'' '''36''', 983-987 (2018).</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_universal_SNP_and_small-indel_variant_caller_using_deep_neural_networks&diff=49105 A universal SNP and small-indel variant caller using deep neural networks 2020-12-04T06:18:15Z <p>Z42qin: /* Preprocessing */</p> <hr /> <div>== Background ==<br /> <br /> Genes determine Biological functions, and mutants or alleles(one of two or more alternative forms of a gene that arise by mutation and are found at the same place on a chromosome) of those genes determine differences within a function. Determining novel alleles is very important in understanding the genetic variation within a species. For example, different alleles of the gene OCA2 determine eyes' color. All animals receive one copy of each gene from each of their parents. Mutations of a gene are classified as either homozygous (both copies are the same) or heterozygous (the two copies are different).<br /> <br /> Next-generation sequencing is a prevalent technique for sequencing or reading DNA. Since all genes are encoded as DNA, sequencing is an essential tool for understanding genes. Next-generation sequencing works by reading short sections of DNA of length k, called k-means, and then piecing them together or aligning them to a reference genome. Next-generation sequencing is relatively fast and inexpensive, although it can randomly misidentify some nucleotides, introducing errors. However, NGS reading is errorful and arises from a complex error process depending on various factors.<br /> <br /> The process of variant calling is determining novel alleles from sequencing data (typically next-generation sequencing data). Some significant alleles only differ from the &quot;standard&quot; version of a gene by only a single base pair, such as the mutation which causes multiple sclerosis. Therefore it is crucial to accurately call single nucleotide swaps/polymorphisms (SNPs), insertions, and deletions (indels). Calling SNPs and small indels are technically challenging since it requires a program to distinguish between genuinely novel mutations and errors in the sequencing data.<br /> <br /> Previous approaches usually involved using various statistical techniques. A widely used one is GATK. GATK uses a combination of logistic regression, hidden Markov models, naive Bayes classification, and Gaussian mixture models to perform the process . However, these methods have their weaknesses as some assumptions do not hold (i.e., independence assumptions), and it's hard to generalize them to other sequencing technologies.<br /> <br /> This paper aims to solve the problem of calling SNPs and small indels using a convolutional neural net by casting the reads as images and classifying whether they contain a mutation. It introduces a variant caller called &quot;DeepVariant&quot;, which requires no specialized knowledge, but performs better than previous state-of-art methods.<br /> <br /> == Overview ==<br /> <br /> In Figure 1, the DeepVariant workflow overview is illustrated.<br /> <br /> [[File:figure 111.JPG|Figure 1. In all panels, blue boxes represent data and red boxes are processes]]<br /> <br /> <br /> Initially, the NGS reads aligned to a reference genome are scanned for candidate variants which are different sites from the reference genome. The read and reference data are encoded as an image for each candidate variant site. Then, trained CNN can compute the genotype likelihoods, (heterozygous or homozygous) for each of the candidate variants (figure1, left box). <br /> To train the CNN for image classification purposes, the DeepVariant machinery makes pileup images for a labeled sample with known genotypes. These labeled images and known genotypes are provided to CNN for training, and a stochastic gradient descent algorithm is used to optimize the CNN parameters to maximize genotype prediction accuracy. After the convergence of the model, the final model is frozen to use for calling mutations for other image classification tests (figure1, middle box).<br /> For example, in figure 1 (right box), the reference and read bases are encoded into a pileup image at a candidate variant site. CNN using this encoded image computes the genotype likelihoods for the three diploid genotype states of homozygous reference (hom-ref), heterozygous (het) or homozygous alternate (hom-alt). In this example, a heterozygous variant call is emitted, as the most probable genotype here is “het”.<br /> <br /> == Preprocessing ==<br /> <br /> Before the sequencing reads can be fed into the classifier, they must be pre-processed. There are many pre-processing steps that are necessary for this algorithm. These steps represent the real novelty in this technique by transforming the data to allow us to use more common neural network architectures for classification. The pre-processing of the data can be broken into three phases: the realignment of reads, finding candidate variants and creating the candidate variants' images.<br /> <br /> The realignment of the pre-processing reads phase is essential to ensure the sequences can be adequately compared to the reference sequences. First, the sequences are aligned to a reference sequence. Reads that align poorly are grouped with other reads around them to build that section, or haplotype, from scratch. If there is strong evidence that the new version of the haplotype fits the reads well, the reads are re-aligned. This process updates the CIGAR (Compact Idiosyncratic Gapped Alignment Report) string to represent a sequence's alignment to a reference for each read.<br /> <br /> Once the reads are correctly aligned, the algorithm then proceeds to find candidate variants, regions in the DNA sequence containing variants. It is these candidate variants that will eventually be passed as input to the neural network. To find these, we need to consider each position in the reference sequence independently. Any unusable reads are filtered at this point. This includes reads that are not appropriately aligned, marked as duplicates, those that fail vendor quality checks, or whose mapping quality is less than ten. For each site in the genome, we collect all the remaining reads that overlap that site. The corresponding allele aligned to that site is then determined by decoding the CIGAR string, which was updated in each read's realignment phase. The alleles are then classified into one of four categories: reference-matching base, reference-mismatching base, insertion with a specific sequence, or deletion with a specific length, and the number of occurrences of each distinct allele across all reads is counted. Read bases are only included as potential alleles if each base in the allele has a quality score of at least 10.<br /> <br /> The last phase of pre-processing is to convert these candidate variants into images representing the data with candidate variants identified. This allows for the use of well established convolutional neural networks for image classification for this technical problem. Each color channel is used to store a different piece of information about a candidate variant. The red channel encodes which base we have (A, G, C, or T) by mapping each base to a particular value. The quality of the read is mapped to the green color channel.<br /> <br /> Moreover, the blue channel encodes whether or not the reference is on the positive strand of the DNA. Each row of the image represents a read, and each column represents a particular base in that read. The reference strand is repeated for the first five rows of the encoded image to maintain its information after a 5x5 convolution is applied. With the data pre-processing complete, the images can then be passed into the neural network for classification.<br /> <br /> == Neural Network ==<br /> <br /> The neural network used is a convolutional neural network. Although the full network architecture is not revealed in the paper, there are several details which we can discuss. The architecture of the network is an input layer attached to an adapted Inception v2 ImageNet model with nine partitions. The inception v2 model in particular uses a series of CNNs. One interesting aspect about the Inception model is that rather than optimizing a series of hyperparameters in order to determine the most optimal parameter configuration, Inception instead concatenates a series of different sizes of filters on the same layer, which acts to learn the best architecture out of these concatenated filters. The input layer takes as input the images representing the candidate variants and rescales them to 299x299 pixels. The output layer is a three-class Softmax layer initialized with Gaussian random weights with a standard deviation of 0.001. This final layer is fully connected to the previous layer. The three classes are the homozygous reference (meaning it is not a variant), heterozygous variant, and homozygous variant. The candidate variant is classified into the class with the highest probability. The model is trained using stochastic gradient descent with a weight decay of 0.00004. The training was done in mini-batches, each with 32 images, using a root mean squared (RMS) decay of 0.9. For the multiple sequencing technologies experiments, a single model was trained with a learning rate of 0.0015 and momentum 0.8 for 250,000 update steps. For all other experiments, multiple models were trained, and the one with the highest accuracy on the training set was chosen as the final model. The multiple models stem from using each combination of the possible parameter values for the learning rate (0.00095, 0.001, 0.0015) and momentum (0.8, 0.85, 0.9). These models were trained for 80 hours, or until the training accuracy converged.<br /> <br /> == Results ==<br /> <br /> DeepVariant was trained using data available from the CEPH (Centre d’Etude du Polymorphism Humain) female sample NA12878 and was evaluated on the unseen Ashkenazi male sample NA24385. The results were compared with other most commonly used bioinformatics methods, such as the GATK, FreeBayes22, SAMtools23, 16GT24 and Strelka25 (Table 1). For better comparison, the overall accuracy (F1), recall, precision, and numbers of true positives (TP), false negatives (FN) and false positives (FP) are illustrated over the whole genome.<br /> <br /> [[File:table 11.JPG]]<br /> <br /> DeepVariant showed the highest accuracy and more than 50% fewer errors per genome compared to the next best algorithm. <br /> <br /> They also evaluated the same set of algorithms using the synthetic diploid sample CHM1-CHM1326 (Table 2).<br /> <br /> [[File:Table 333.JPG]]<br /> <br /> Results illustrated that the DeepVariant method outperformed all other algorithms for variant calling (SNP and indel) and showed the highest accuracy in terms of F1, Recall, precision and TP.<br /> <br /> == Conclusion ==<br /> <br /> This endeavour to further advance a data-centric approach to understanding the gene sequence illustrate the advantages of deep learning over humans. With billions of DNA base pairs, no humans are able to digest that amount of gene expressions. In the past, computational techniques are unfeasible due to the lack of compute power but in the 21st century, it seems that machine learning is the way to go for molecular biology.<br /> <br /> DeepVariant’s strong performance on human data proves that deep learning is a promising technique for variant calling. Perhaps the most exciting feature of DeepVariant is its simplicity. Unlike other states of the art variant callers, DeepVariant has no knowledge of the sequencing technologies that create the reads, or even the biological processes that introduce mutations. This simplifies the problem of variant calling to preprocessing the reads and training a generic deep learning model. It also suggests that DeepVariant could be significantly improved by tailoring the preprocessing to specific sequencing technologies and/or developing a dedicated CNN architecture for the reads, rather than trying to cast them as images.<br /> <br /> == Critique and Discussion==<br /> <br /> The paper presents an interesting method for solving an important problem. Building &quot;images&quot; of reads and running them through a generic image classification CNN seems like a strange approach, and it is interesting that it works well. The biggest issues with the paper are the lack of specific information about how the methods. Some extra information is included in the supplementary material, but there are still some big gaps. In particular:<br /> <br /> 1. What is the structure of the neural net? How many layers, and what sizes? The paper for ConvNet which is cited does not have this information. We suspect that this might be a trade secret that Google is protecting.<br /> <br /> 2. How is the realignment step implemented? The paper mentions that it uses a &quot;De-Bruijn-graph-based read assembly procedure&quot; to realign reads to a new haplotype. This is a non-standard step in most genomics workflows yet the paper does not describe how they do the realignment or how they build the haplotypes.<br /> <br /> 3. How did they settle on the image construction algorithm? The authors provide pseudocode for the construction of pileup images but they do not describe how the decisions for made. For instance, the colour values for different base pairs are not evenly spaced. Also, the image begins with 5 rows of the reference genome.<br /> <br /> One thing we appreciated about the paper was their commentary on future developments. The authors make it very clear that this approach can be improved on and provide specific ideas for next steps.<br /> <br /> Overall, the paper presents an interesting idea with strong results, but lacks detail in some key pieces of the implementation.<br /> <br /> The topic of this project is good but we need to more details of the algorithm. In the neural network part, the details are not enough, Authors should provide a figure to better explain how the model works and the structure of the model. Otherwise we cannot understand how the model works. Also, when we are preprocessing the data, if different data have different lengths, shall we add more information or drop some information so they match?<br /> <br /> 5 Particularly, which package did the researchers use to perform this analysis? Different packages of deep learning can have different accuracy and efficiency while making predictions on this data set. <br /> <br /> Further studies on DeepVariant [https://www.nature.com/articles/s41598-018-36177-7 have shown] that it is a framework with great potential and sets the standard in the medical genetics field.<br /> <br /> 6. It was mentioned that part of the network used is an &quot;adapted Inception v2 ImageNet model&quot; - does this mean that it used an Inception v2 model that was pretrained on ImageNet? This is not clear, but if this is the case then why is this useful? Why would the features that are extracted for an image be useful for genomics? Did they try using a model that wasn't pretrained? Also, they describe the preprocessing method used but were there any alternatives that they considered?<br /> <br /> 7. A more extensive discussion on the &quot;Neural Network&quot; section can be give. For example, the last sentence of the paragraph says that &quot;Models are trained for 80 hours, or until the training accuracy converged.&quot; This sentence implies that if the training accuracy does converge, it usually takes less than 80 hours. More interesting data can thus be presented about the training accuracy converging time. How often do the models converge? How long does it take for the models to converge on average? Moreover, why is the number 80 chosen here? Is it a random upper bound set by people to bound the runtime of the model training, or is the number 80 carefully chosen so that it's twice/three times/ten times of the average training accuracy converging time? I believe that these are interesting facts to look at.<br /> <br /> == References ==<br />  Hartwell, L.H. ''et. al.'' ''Genetics: From Genes to Genomes''. (McGraw-Hill Ryerson, 2014).<br /> <br />  Poplin, R. ''et. al''. A universal SNP and small-indel variant caller using deep neural networks. ''Nature Biotechnology'' '''36''', 983-987 (2018).</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=A_universal_SNP_and_small-indel_variant_caller_using_deep_neural_networks&diff=49104 A universal SNP and small-indel variant caller using deep neural networks 2020-12-04T06:15:01Z <p>Z42qin: /* Background */</p> <hr /> <div>== Background ==<br /> <br /> Genes determine Biological functions, and mutants or alleles(one of two or more alternative forms of a gene that arise by mutation and are found at the same place on a chromosome) of those genes determine differences within a function. Determining novel alleles is very important in understanding the genetic variation within a species. For example, different alleles of the gene OCA2 determine eyes' color. All animals receive one copy of each gene from each of their parents. Mutations of a gene are classified as either homozygous (both copies are the same) or heterozygous (the two copies are different).<br /> <br /> Next-generation sequencing is a prevalent technique for sequencing or reading DNA. Since all genes are encoded as DNA, sequencing is an essential tool for understanding genes. Next-generation sequencing works by reading short sections of DNA of length k, called k-means, and then piecing them together or aligning them to a reference genome. Next-generation sequencing is relatively fast and inexpensive, although it can randomly misidentify some nucleotides, introducing errors. However, NGS reading is errorful and arises from a complex error process depending on various factors.<br /> <br /> The process of variant calling is determining novel alleles from sequencing data (typically next-generation sequencing data). Some significant alleles only differ from the &quot;standard&quot; version of a gene by only a single base pair, such as the mutation which causes multiple sclerosis. Therefore it is crucial to accurately call single nucleotide swaps/polymorphisms (SNPs), insertions, and deletions (indels). Calling SNPs and small indels are technically challenging since it requires a program to distinguish between genuinely novel mutations and errors in the sequencing data.<br /> <br /> Previous approaches usually involved using various statistical techniques. A widely used one is GATK. GATK uses a combination of logistic regression, hidden Markov models, naive Bayes classification, and Gaussian mixture models to perform the process . However, these methods have their weaknesses as some assumptions do not hold (i.e., independence assumptions), and it's hard to generalize them to other sequencing technologies.<br /> <br /> This paper aims to solve the problem of calling SNPs and small indels using a convolutional neural net by casting the reads as images and classifying whether they contain a mutation. It introduces a variant caller called &quot;DeepVariant&quot;, which requires no specialized knowledge, but performs better than previous state-of-art methods.<br /> <br /> == Overview ==<br /> <br /> In Figure 1, the DeepVariant workflow overview is illustrated.<br /> <br /> [[File:figure 111.JPG|Figure 1. In all panels, blue boxes represent data and red boxes are processes]]<br /> <br /> <br /> Initially, the NGS reads aligned to a reference genome are scanned for candidate variants which are different sites from the reference genome. The read and reference data are encoded as an image for each candidate variant site. Then, trained CNN can compute the genotype likelihoods, (heterozygous or homozygous) for each of the candidate variants (figure1, left box). <br /> To train the CNN for image classification purposes, the DeepVariant machinery makes pileup images for a labeled sample with known genotypes. These labeled images and known genotypes are provided to CNN for training, and a stochastic gradient descent algorithm is used to optimize the CNN parameters to maximize genotype prediction accuracy. After the convergence of the model, the final model is frozen to use for calling mutations for other image classification tests (figure1, middle box).<br /> For example, in figure 1 (right box), the reference and read bases are encoded into a pileup image at a candidate variant site. CNN using this encoded image computes the genotype likelihoods for the three diploid genotype states of homozygous reference (hom-ref), heterozygous (het) or homozygous alternate (hom-alt). In this example, a heterozygous variant call is emitted, as the most probable genotype here is “het”.<br /> <br /> == Preprocessing ==<br /> <br /> Before the sequencing reads can be fed into the classifier, they must be preprocessed. There are many pre-processing steps that are necessary for this algorithm. These steps represent the real novelty in this technique, by transforming the data in a way that allows us to use more common neural network architectures for classification. The preprocessing of the data can be broken into three main phases: the realignment of reads, finding candidate variants and creating images of the candidate variants. <br /> <br /> The realignment of the reads phase of the preprocessing is important in order to ensure the sequences can be properly compared to the reference sequences. First, the sequences are aligned to a reference sequence. Reads that align poorly are grouped with other reads around them to build that section, or haplotype, from scratch. If there is strong evidence that the new version of the haplotype fits the reads well, the reads are re-aligned to it. This process updates the CIGAR (Compact Idiosyncratic Gapped Alignment Report) string, a way to represent the alignment of a sequence to a reference, for each read.<br /> <br /> Once the reads are properly aligned, the algorithm then proceeds to find candidate variants, regions in the DNA sequence that may contain variants. It is these candidate variants that will eventually be passed as input to the neural network. To find these, we need to consider each position in the reference sequence independently. Any unusable reads are filtered at this point. This includes reads that are not aligned properly, ones that are marked as duplicates, those that fail vendor quality checks, or whose mapping quality is less than ten. For each site in the genome, we collect all the remaining reads that overlap that site. The corresponding allele aligned to that site is then determined by decoding the CIGAR string, which was updated in the realignment phase, of each read. The alleles are then classified into one of four categories: reference-matching base, reference-mismatching base, insertion with a specific sequence, or deletion with a specific length, and the number of occurrences of each distinct allele across all reads is counted. Read bases are only included as potential alleles if each base in the allele has a quality score of at least 10.<br /> <br /> With candidate variants identified, the last phase of pre-processing is to convert these candidate variants into images representing the data. This allows for the use of well established convolutional neural networks for image classification for this specialized problem. Each colour channel is used to store a different piece of information about a candidate variant. The red channel encodes which base we have (A, G, C, or T), by mapping each base to a particular value. The quality of the read is mapped to the green colour channel. And finally, the blue channel encodes whether or not the reference is on the positive strand of the DNA. Each row of the image represents a read, and each column represents a particular base in that read. The reference strand is repeated for the first five rows of the encoded image, in order to maintain its information after a 5x5 convolution is applied.<br /> With the data preprocessing complete, the images can then be passed into the neural network for classification.<br /> <br /> == Neural Network ==<br /> <br /> The neural network used is a convolutional neural network. Although the full network architecture is not revealed in the paper, there are several details which we can discuss. The architecture of the network is an input layer attached to an adapted Inception v2 ImageNet model with nine partitions. The inception v2 model in particular uses a series of CNNs. One interesting aspect about the Inception model is that rather than optimizing a series of hyperparameters in order to determine the most optimal parameter configuration, Inception instead concatenates a series of different sizes of filters on the same layer, which acts to learn the best architecture out of these concatenated filters. The input layer takes as input the images representing the candidate variants and rescales them to 299x299 pixels. The output layer is a three-class Softmax layer initialized with Gaussian random weights with a standard deviation of 0.001. This final layer is fully connected to the previous layer. The three classes are the homozygous reference (meaning it is not a variant), heterozygous variant, and homozygous variant. The candidate variant is classified into the class with the highest probability. The model is trained using stochastic gradient descent with a weight decay of 0.00004. The training was done in mini-batches, each with 32 images, using a root mean squared (RMS) decay of 0.9. For the multiple sequencing technologies experiments, a single model was trained with a learning rate of 0.0015 and momentum 0.8 for 250,000 update steps. For all other experiments, multiple models were trained, and the one with the highest accuracy on the training set was chosen as the final model. The multiple models stem from using each combination of the possible parameter values for the learning rate (0.00095, 0.001, 0.0015) and momentum (0.8, 0.85, 0.9). These models were trained for 80 hours, or until the training accuracy converged.<br /> <br /> == Results ==<br /> <br /> DeepVariant was trained using data available from the CEPH (Centre d’Etude du Polymorphism Humain) female sample NA12878 and was evaluated on the unseen Ashkenazi male sample NA24385. The results were compared with other most commonly used bioinformatics methods, such as the GATK, FreeBayes22, SAMtools23, 16GT24 and Strelka25 (Table 1). For better comparison, the overall accuracy (F1), recall, precision, and numbers of true positives (TP), false negatives (FN) and false positives (FP) are illustrated over the whole genome.<br /> <br /> [[File:table 11.JPG]]<br /> <br /> DeepVariant showed the highest accuracy and more than 50% fewer errors per genome compared to the next best algorithm. <br /> <br /> They also evaluated the same set of algorithms using the synthetic diploid sample CHM1-CHM1326 (Table 2).<br /> <br /> [[File:Table 333.JPG]]<br /> <br /> Results illustrated that the DeepVariant method outperformed all other algorithms for variant calling (SNP and indel) and showed the highest accuracy in terms of F1, Recall, precision and TP.<br /> <br /> == Conclusion ==<br /> <br /> This endeavour to further advance a data-centric approach to understanding the gene sequence illustrate the advantages of deep learning over humans. With billions of DNA base pairs, no humans are able to digest that amount of gene expressions. In the past, computational techniques are unfeasible due to the lack of compute power but in the 21st century, it seems that machine learning is the way to go for molecular biology.<br /> <br /> DeepVariant’s strong performance on human data proves that deep learning is a promising technique for variant calling. Perhaps the most exciting feature of DeepVariant is its simplicity. Unlike other states of the art variant callers, DeepVariant has no knowledge of the sequencing technologies that create the reads, or even the biological processes that introduce mutations. This simplifies the problem of variant calling to preprocessing the reads and training a generic deep learning model. It also suggests that DeepVariant could be significantly improved by tailoring the preprocessing to specific sequencing technologies and/or developing a dedicated CNN architecture for the reads, rather than trying to cast them as images.<br /> <br /> == Critique and Discussion==<br /> <br /> The paper presents an interesting method for solving an important problem. Building &quot;images&quot; of reads and running them through a generic image classification CNN seems like a strange approach, and it is interesting that it works well. The biggest issues with the paper are the lack of specific information about how the methods. Some extra information is included in the supplementary material, but there are still some big gaps. In particular:<br /> <br /> 1. What is the structure of the neural net? How many layers, and what sizes? The paper for ConvNet which is cited does not have this information. We suspect that this might be a trade secret that Google is protecting.<br /> <br /> 2. How is the realignment step implemented? The paper mentions that it uses a &quot;De-Bruijn-graph-based read assembly procedure&quot; to realign reads to a new haplotype. This is a non-standard step in most genomics workflows yet the paper does not describe how they do the realignment or how they build the haplotypes.<br /> <br /> 3. How did they settle on the image construction algorithm? The authors provide pseudocode for the construction of pileup images but they do not describe how the decisions for made. For instance, the colour values for different base pairs are not evenly spaced. Also, the image begins with 5 rows of the reference genome.<br /> <br /> One thing we appreciated about the paper was their commentary on future developments. The authors make it very clear that this approach can be improved on and provide specific ideas for next steps.<br /> <br /> Overall, the paper presents an interesting idea with strong results, but lacks detail in some key pieces of the implementation.<br /> <br /> The topic of this project is good but we need to more details of the algorithm. In the neural network part, the details are not enough, Authors should provide a figure to better explain how the model works and the structure of the model. Otherwise we cannot understand how the model works. Also, when we are preprocessing the data, if different data have different lengths, shall we add more information or drop some information so they match?<br /> <br /> 5 Particularly, which package did the researchers use to perform this analysis? Different packages of deep learning can have different accuracy and efficiency while making predictions on this data set. <br /> <br /> Further studies on DeepVariant [https://www.nature.com/articles/s41598-018-36177-7 have shown] that it is a framework with great potential and sets the standard in the medical genetics field.<br /> <br /> 6. It was mentioned that part of the network used is an &quot;adapted Inception v2 ImageNet model&quot; - does this mean that it used an Inception v2 model that was pretrained on ImageNet? This is not clear, but if this is the case then why is this useful? Why would the features that are extracted for an image be useful for genomics? Did they try using a model that wasn't pretrained? Also, they describe the preprocessing method used but were there any alternatives that they considered?<br /> <br /> 7. A more extensive discussion on the &quot;Neural Network&quot; section can be give. For example, the last sentence of the paragraph says that &quot;Models are trained for 80 hours, or until the training accuracy converged.&quot; This sentence implies that if the training accuracy does converge, it usually takes less than 80 hours. More interesting data can thus be presented about the training accuracy converging time. How often do the models converge? How long does it take for the models to converge on average? Moreover, why is the number 80 chosen here? Is it a random upper bound set by people to bound the runtime of the model training, or is the number 80 carefully chosen so that it's twice/three times/ten times of the average training accuracy converging time? I believe that these are interesting facts to look at.<br /> <br /> == References ==<br />  Hartwell, L.H. ''et. al.'' ''Genetics: From Genes to Genomes''. (McGraw-Hill Ryerson, 2014).<br /> <br />  Poplin, R. ''et. al''. A universal SNP and small-indel variant caller using deep neural networks. ''Nature Biotechnology'' '''36''', 983-987 (2018).</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Task_Understanding_from_Confusing_Multi-task_Data&diff=49100 Task Understanding from Confusing Multi-task Data 2020-12-04T05:55:27Z <p>Z42qin: /* Introduction */</p> <hr /> <div>'''Presented By'''<br /> <br /> Qianlin Song, William Loh, Junyue Bai, Phoebe Choi<br /> <br /> = Introduction =<br /> <br /> Narrow AI is an artificial intelligence that outperforms humans in a narrowly defined task. The application of Narrow AI is becoming more and more common. For example, Narrow AI can be used for spam filtering, music recommendation services, assist doctors to make data-driven decisions, and even self-driving cars. One of the most famous integrated forms of Narrow AI is Apple's Siri. Siri has no self-awareness or genuine intelligence, and hence often has challenges performing tasks outside its range of abilities. However, the widespread use of Narrow AI in important infrastructure functions raises some concerns. Some people think that the characteristics of Narrow AI make it fragile, and when neural networks can be used to control important systems (such as power grids, financial transactions), alternatives may be more inclined to avoid risks. While these machines help companies improve efficiency and cut costs, the limitations of Narrow AI encouraged researchers to look into General AI. <br /> <br /> General AI is a machine that can apply its learning to different contexts, which closely resembles human intelligence. This paper attempts to generalize the multi-task learning system that learns from data from multiple classification tasks. For an isolated and very difficult task, the artificial intelligence may not learn it very well. For instance, a net with pixel dimension 1000*1000 is less likely to identify complicated objects in real-world situations on the time basis. However, if it could be learned simultaneously, it would be better as the tasks can share what they learned. It is easier for the learner to learn together instead of in isolation, for example, shapes, landmarks, textures, orientation and so on. This is called Multitask Learning. One application is image recognition. In figure 1, an image of an apple corresponds to 3 labels: “red”, “apple” and “sweet”. These labels correspond to 3 different classification tasks: color, fruit, and taste. <br /> <br /> [[File:CSLFigure1.PNG | 500px]]<br /> <br /> Currently, multi-task machines require researchers to construct a task definition. Otherwise, it will end up with different outputs with the same input value. Researchers manually assign tasks to each input in the sample to train the machine. See figure 1(a). This method incurs high annotation costs and restricts the machine’s ability to mirror the human recognition process. This paper is interested in developing an algorithm that understands task concepts and performs multi-task learning without manual task annotations. <br /> <br /> This paper proposed a new learning method called confusing supervised learning (CSL) which includes 2 functions: de-confusing function and mapping function. The de-confusing function allocates samples to respective tasks and the mapping function presents the relation from the input to its label within the allocated tasks. See figure 1(b). To implement the CSL, we use a risk functional to balance the effects of the de-confusing function and mapping function. <br /> <br /> However, simply combining the two functions or networks to a single architecture is impossible, since the one-hot constraint of the outputs for the de-confusing network makes the gradient back-propagation unfeasible. This difficulty is solved by alternatively performing training for the de-confusing net and mapping net optimization in the proposed architecture CLS-Net.<br /> <br /> Experiments for function regression and image recognition problems were constructed and compared with multi-task learning with complete information to test CSL-Net’s performance. Experiment results show that CSL-Net can learn multiple mappings for every task simultaneously and achieve the same cognition result as the current multi-task machine assigned with complete information.<br /> <br /> = Related Work =<br /> <br /> [[File:CSLFigure2.PNG | 700px]]<br /> <br /> ==Latent variable learning==<br /> Latent variable learning aims to estimate the true function with mixed probability models. See '''figure 2a'''. In the multi-task learning problem without task annotations, we know that samples are generated from multiple distinct distributions instead of one distribution combining a mixture of multiple probability models. Thus, the latent variable learning can not fully distinguish labels into different tasks and different distributions, and it is insufficient to classify the multi-task confusing samples. <br /> <br /> ==Multi-task learning==<br /> Multi-task learning aims to learn multiple tasks simultaneously using a shared feature representation. In multi-task learning, the task to which every sample belongs is known. By exploiting similarities and differences between tasks, the learning from one task can improve the learning of another task. (Caruana, 1997) This results in improved the overall learning efficiency, since the labels in different tasks are often correlated: improving the classfication result for one class also help with other classification tasks. In multi-task learning, the input-output mapping of every task can be represented by a unified function. However, these task definitions are manually constructed, and machines need manual task annotations to learn. If such manuual task annotation is abstent, then the algorithm can not be performed. <br /> <br /> ==Multi-label learning==<br /> Multi-label learning aims to assign an input to a set of classes/labels. See '''figure 2b'''. It is a generalization of multi-class classification, which classifies an input into one class. In multi-label learning, an input can be classified into more than one class. Unlike multi-task learning, multi-label does not consider the relationship between different label judgments and it is assumed that each judgment is independent. An example where multi-label learning is applicable is the scenario where a website wants to automatically assign applicable tags/categories to an article. Since an article can be related to multiple categories (eg. an article can be tagged under the politics and business categories) multi-label learning is of primary concern here.<br /> <br /> = Confusing Supervised Learning =<br /> <br /> == Description of the Problem ==<br /> <br /> Confusing supervised learning (CSL) offers a solution to the issue at hand. A major area of improvement can be seen in the choice of risk measure. In traditional supervised learning, let &lt;math&gt; (x,y)&lt;/math&gt; be the training samples from &lt;math&gt;y=f(x)&lt;/math&gt;, which is an identical but unknown mapping relationship. Assuming the risk measure is mean squared error (MSE), the expected risk function is<br /> <br /> $$R(g) = \int_x (f(x) - g(x))^2 p(x) \; \mathrm{d}x$$<br /> <br /> where &lt;math&gt;p(x)&lt;/math&gt; is the data distribution of the input variable &lt;math&gt;x&lt;/math&gt;. In practice, the methods select the optimal function by minimizing the empirical risk:<br /> <br /> $$R_e(g) = \sum_{i=1}^n (y_i - g(x_i))^2$$<br /> <br /> To minimize the risk function, the theoretically optimal solution is &lt;math&gt; f(x) &lt;/math&gt;.<br /> <br /> When the problem involves different tasks, the model should optimize for each data point depending on the given task. Let &lt;math&gt;f_j(x)&lt;/math&gt; be the true ground-truth function for each task &lt;math&gt; j &lt;/math&gt;. Therefore, for some input variable &lt;math&gt; x_i &lt;/math&gt;, an ideal model &lt;math&gt;g&lt;/math&gt; would predict &lt;math&gt; g(x_i) = f_j(x_i) &lt;/math&gt;. With this, the risk function can be modified to fit this new task for traditional supervised learning methods.<br /> <br /> $$R(g) = \int_x \sum_{j=1}^n (f_j(x) - g(x))^2 p(f_j) p(x) \; \mathrm{d}x$$<br /> <br /> We call &lt;math&gt; (f_j(x) - g(x))^2 p(f_j) &lt;/math&gt; the '''confusing multiple mappings'''. Then the optimal solution &lt;math&gt;g^*(x)&lt;/math&gt; is &lt;math&gt;\bar{f}(x) = \sum_{j=1}^n p(f_j) f_j(x)&lt;/math&gt;. However, the optimal solution is not conditional on the specific task at hand but rather on the entire ground-truth functions. The solution represents a mixed probably model instead of knowing the exact tasks and their correpsonding individual probability distribution. Therefore, for every non-trivial set of tasks where &lt;math&gt;f_u(x) \neq f_v(x)&lt;/math&gt; for some input &lt;math&gt;x&lt;/math&gt; and &lt;math&gt;u \neq v&lt;/math&gt;, &lt;math&gt;R(g^*) &gt; 0&lt;/math&gt; which implies that there is an unavoidable confusion risk.<br /> <br /> == Learning Functions of CSL ==<br /> <br /> To overcome this issue, the authors introduce two types of learning functions:<br /> * '''Deconfusing function''' &amp;mdash; allocation of which samples come from the same task<br /> * '''Mapping function''' &amp;mdash; mapping relation from input to the output of every learned task<br /> <br /> Suppose there are &lt;math&gt;n&lt;/math&gt; ground-truth mappings &lt;math&gt;\{f_j : 1 \leq j \leq n\}&lt;/math&gt; that we wish to approximate with a set of mapping functions &lt;math&gt;\{g_k : 1 \leq k \leq l\}&lt;/math&gt;. The authors define the deconfusing function as an indicator function &lt;math&gt;h(x, y, g_k) &lt;/math&gt; which takes some sample &lt;math&gt;(x,y)&lt;/math&gt; and determines whether the sample is assigned to task &lt;math&gt;g_k&lt;/math&gt;. Under the CSL framework, the risk functional (using MSE loss) is <br /> <br /> $$R(g,h) = \int_x \sum_{j,k} (f_j(x) - g_k(x))^2 \; h(x, f_j(x), g_k) \;p(f_j) \; p(x) \;\mathrm{d}x$$<br /> <br /> which can be estimated empirically with<br /> <br /> $$R_e(g,h) = \sum_{i=1}^m \sum_{k=1}^n |y_i - g_k(x_i)|^2 \cdot h(x_i, y_i, g_k)$$<br /> <br /> The risk metric of every sample affects only its assigned task.<br /> <br /> == Theoretical Results ==<br /> <br /> This novel framework yields some theoretical results to show the viability of its construction.<br /> <br /> '''Theorem 1 (Existence of Solution)'''<br /> ''With the confusing supervised learning framework, there is an optimal solution''<br /> $$h^*(x, f_j(x), g_k) = \mathbb{I}[j=k]$$<br /> <br /> $$g_k^*(x) = f_k(x)$$<br /> <br /> ''for each &lt;math&gt;k=1,..., n&lt;/math&gt; that makes the expected risk function of the CSL problem zero.''<br /> <br /> However, necessity constraints are needed to avoid meaningless trivial solutions in all optimal risk solutions.<br /> <br /> '''Theorem 2 (Error Bound of CSL)'''<br /> ''With probability at least &lt;math&gt;1 - \eta&lt;/math&gt; simultaneously with finite VC dimension &lt;math&gt;\tau&lt;/math&gt; of CSL learning framework, the risk measure is bounded by<br /> <br /> $$R(\alpha) \leq R_e(\alpha) + \frac{B\epsilon(m)}{2} \left(1 + \sqrt{1 + \frac{4R_e(\alpha)}{B\epsilon(m)}}\right)$$<br /> <br /> ''where &lt;math&gt;\alpha&lt;/math&gt; is the total parameters of learning functions &lt;math&gt;g, h&lt;/math&gt;, &lt;math&gt;B&lt;/math&gt; is the upper bound of one sample's risk, &lt;math&gt;m&lt;/math&gt; is the size of training data and''<br /> $$\epsilon(m) = 4 \; \frac{\tau (\ln \frac{2m}{\tau} + 1) - \ln \eta / 4}{m}$$<br /> <br /> This theorem shows the method of empirical risk minimization is valid in the CSL framework. Moreover, the assumed number of tasks affects the VC dimension of the learning functions, which is positively related to the generalization error. Therefore, to make the training risk small, we need to choose the ''minimum number'' of tasks when determining the task.<br /> <br /> = CSL-Net =<br /> In this section, the authors describe how to implement and train a network for CSL, including the stucture of CSL-Net and Iterative deconfusing algorithm.<br /> <br /> == The Structure of CSL-Net ==<br /> Two neural networks, deconfusing-net and mapping-net are trained to implement two learning function variables in empirical risk. The optimization target of the training algorithm is:<br /> $$\min_{g, h} R_e = \sum_{i=1}^{m}\sum_{k=1}^{n} (y_i - g_k(x_i))^2 \cdot h(x_i, y_i; g_k)$$<br /> <br /> The mapping-net is corresponding to functions set &lt;math&gt;g_k&lt;/math&gt;, where &lt;math&gt;y_k = g_k(x)&lt;/math&gt; represents the output of one certain task. The deconfusing-net is corresponding to function h, whose input is a sample &lt;math&gt;(x,y)&lt;/math&gt; and output is an n-dimensional one-hot vector. This output vector determines which task the sample &lt;math&gt;(x,y)&lt;/math&gt; should be assigned to. The core difficulty of this algorithm is that the risk function cannot be optimized by gradient back-propagation due to the constraint of one-hot output from deconfusing-net. Approximation of softmax will lead the deconfusing-net output into a non-one-hot form, which results in meaningless trivial solutions.<br /> <br /> == Iterative Deconfusing Algorithm ==<br /> To overcome the training difficulty, the authors divide the empirical risk minimization into two local optimization problems. In each single-network optimization step, the parameters of one network are updated while the parameters of another remain fixed. With one network's parameters unchanged, the problem can be solved by a gradient descent method of neural networks. <br /> <br /> '''Training of Mapping-Net''': With function h from deconfusing-net being determined, the goal is to train every mapping function &lt;math&gt;g_k&lt;/math&gt; with its corresponding sample &lt;math&gt;(x_i^k, y_i^k)&lt;/math&gt;. The optimization problem becomes: &lt;math&gt;\displaystyle \min_{g_k} L_{map}(g_k) = \sum_{i=1}^{m_k} \mid y_i^k - g_k(x_i^k)\mid^2&lt;/math&gt;. Back-propagation algorithm can be applied to solve this optimization problem.<br /> <br /> '''Training of Deconfusing-Net''': The task allocation is re-evaluated during the training phase while the parameters of the mapping-net remain fixed. To minimize the original risk, every sample &lt;math&gt;(x, y)&lt;/math&gt; will be assigned to &lt;math&gt;g_k&lt;/math&gt; that is closest to label y among all different &lt;math&gt;k&lt;/math&gt;s. Mapping-net thus provides a temporary solution for deconfusing-net: &lt;math&gt;\hat{h}(x_i, y_i) = arg \displaystyle\min_{k} \mid y_i - g_k(x_i)\mid^2&lt;/math&gt;. The optimization becomes: &lt;math&gt;\displaystyle \min_{h} L_{dec}(h) = \sum_{i=1}^{m} \mid {h}(x_i, y_i) - \hat{h}(x_i, y_i)\mid^2&lt;/math&gt;. Similarly, the optimization problem can be solved by updating the deconfusing-net with a back-propagation algorithm.<br /> <br /> The two optimization stages are carried out alternately until the solution converges.<br /> <br /> =Experiment=<br /> ==Setup==<br /> <br /> 3 data sets are used to compare CSL to existing methods, 1 function regression task, and 2 image classification tasks. <br /> <br /> '''Function Regression''': The function regression data comes in the form of &lt;math&gt;(x_i,y_i),i=1,...,m&lt;/math&gt; pairs. However, unlike typical regression problems, there are multiple &lt;math&gt;f_j(x),j=1,...,n&lt;/math&gt; mapping functions, so the goal is to reproduce both the mapping functions &lt;math&gt;f_j&lt;/math&gt; as well as determine which mapping function corresponds to each of the &lt;math&gt;m&lt;/math&gt; observations. 3 scalar-valued, scalar-input functions that intersect at several points with each other have been chosen as the different tasks. <br /> <br /> '''Colorful-MNIST''': The first image classification data set consists of digit data in a range of 0 to 9, each of which is in a single color among the eight different colors. Each observation in this modified set consists of a colored image (&lt;math&gt;x_i&lt;/math&gt;) and a label (&lt;math&gt;y_i&lt;/math&gt;) that represents either the corresponding color, or the digit. The goal is to reproduce the classification task (&quot;color&quot; or &quot;digit&quot;) for each observation and construct the 2 classifiers for both tasks. <br /> <br /> '''Kaggle Fashion Product''': The second image classification data set consists of several fashion-related objects labeled from any of the 3 criteria: “gender”, “category”, and “main color”, whose number of observations is larger than that of the &quot;colored-MNIST&quot; data set.<br /> <br /> ==Use of Pre-Trained CNN Feature Layers==<br /> <br /> In the Kaggle Fashion Product experiment, CSL trains fully-connected layers that have been attached to feature-identifying layers from pre-trained Convolutional Neural Networks. The CSL methods autonomously learned three tasks which corresponded exactly to “Gender”,<br /> “Category”, and “Color” as we see it.<br /> <br /> ==Metrics of Confusing Supervised Learning==<br /> <br /> There are two measures of accuracy used to evaluate and compare CSL to other methods, corresponding respectively to the accuracy of the task labeling and the accuracy of the learned mapping function. <br /> <br /> '''Task Prediction Accuracy''': &lt;math&gt;\alpha_T(j)&lt;/math&gt; is the average number of times the learned deconfusing function &lt;math&gt;h&lt;/math&gt; agrees with the task-assignment ability of humans &lt;math&gt;\tilde h&lt;/math&gt; on whether each observation in the data &quot;is&quot; or &quot;is not&quot; in task &lt;math&gt;j&lt;/math&gt;.<br /> <br /> $$\alpha_T(j) = \operatorname{max}_k\frac{1}{m}\sum_{i=1}^m I[h(x_i,y_i;f_k),\tilde h(x_i,y_i;f_j)]$$<br /> <br /> The max over &lt;math&gt;k&lt;/math&gt; is taken because we need to determine which learned task corresponds to which ground-truth task.<br /> <br /> '''Label Prediction Accuracy''': &lt;math&gt;\alpha_L(j)&lt;/math&gt; again chooses &lt;math&gt;f_k&lt;/math&gt;, the learned mapping function that is closest to the ground-truth of task &lt;math&gt;j&lt;/math&gt;, and measures its average absolute accuracy compared to the ground-truth of task &lt;math&gt;j&lt;/math&gt;, &lt;math&gt;f_j&lt;/math&gt;, across all &lt;math&gt;m&lt;/math&gt; observations.<br /> <br /> $$\alpha_L(j) = \operatorname{max}_k\frac{1}{m}\sum_{i=1}^m 1-\dfrac{|g_k(x_i)-f_j(x_i)|}{|f_j(x_i)|}$$<br /> <br /> The purpose of this measure arises from the fact that, in addition to learning mapping allocations like humans, machines should be able to approximate all mapping functions accurately in order to provide corresponding labels. The Label Prediction Accuracy measure captures the exchange equivalence of the following task: each mapping contains its ground-truth output, and machines should be predicting the correct output that is close to the ground-truth. <br /> <br /> ==Results==<br /> <br /> Given confusing data, CSL performs better than traditional supervised learning methods, Pseudo-Label(Lee, 2013), and SMiLE(Tan et al., 2017). This is demonstrated by CSL's &lt;math&gt;\alpha_L&lt;/math&gt; scores of around 95%, compared to &lt;math&gt;\alpha_L&lt;/math&gt; scores of under 50% for the other methods. This supports the assertion that traditional methods only learn the means of all the ground-truth mapping functions when presented with confusing data.<br /> <br /> '''Function Regression''': To &quot;correctly&quot; partition the observations into the correct tasks, a 5-shot warm-up was used. In this situation, the CSL methods work well in learning the ground-truth. That means the initialization of the neural network is set up properly.<br /> <br /> '''Image Classification''': Visualizations created through Spectral embedding confirm the task labelling proficiency of the deconfusing neural network &lt;math&gt;h&lt;/math&gt;.<br /> <br /> The classification and function prediction accuracy of CSL are comparable to supervised learning programs that have been given access to the ground-truth labels.<br /> <br /> ==Application of Multi-label Learning==<br /> <br /> CSL also had better accuracy than traditional supervised learning methods, Pseudo-Label(Lee, 2013), and SMiLE(Tan et al., 2017) when presented with partially labelled multi-label data &lt;math&gt;(x_i,y_i)&lt;/math&gt;, where &lt;math&gt;y_i&lt;/math&gt; is a &lt;math&gt;n&lt;/math&gt;-long indicator vector for whether the image &lt;math&gt;(x_i,y_i)&lt;/math&gt; corresponds to each of the &lt;math&gt;n&lt;/math&gt; labels.<br /> <br /> Applications of multi-label classification include building a recommendation system, social media targeting, as well as detecting adverse drug reactions from the text.<br /> <br /> Multi-label can be used to improve the syndrome diagnosis of a patient by focusing on multiple syndromes instead of a single syndrome.<br /> <br /> ==Limitations==<br /> <br /> '''Number of Tasks''': The number of tasks is determined by increasing the task numbers progressively and testing the performance. Ideally, a better way of deciding the number of tasks is expected rather than increasing it one by one and seeing which is the minimum number of tasks that gives the smallest risk. Adding low-quality constraints to deconfusing-net is a reasonable solution to this problem.<br /> <br /> '''Learning of Basic Features''': The CSL framework is not good at learning features. So far, a pre-trained CNN backbone is needed for complicated image classification problems. Even though the effectiveness of the proposed algorithm in learning confusing data based on pre-trained features hasn't been affected, the full-connect network can only be trained based on learned CNN features. It is still a challenge for the current algorithm to learn basic features directly through a CNN structure and understand tasks simultaneously.<br /> <br /> = Conclusion =<br /> <br /> This paper proposes the CSL method for tackling the multi-task learning problem without manual task annotations from basic input data. The model obtains a basic task concept by learning the minimum risk for confusing samples from differentiating multiple mappings. The paper also demonstrates that the CSL method is an important step to moving from Narrow AI towards General AI for multi-task learning.<br /> <br /> However, some limitations can be improved for future work:<br /> <br /> - The repeated training process of determining the lowest best task number that has the closest to zero causes inefficiency in the learning process; <br /> <br /> - The current algorithm is difficult to learn basic features directly through a CNN structure and understand tasks simultaneously by training a full-connect network. However, this limitation does not affect the effectiveness of our algorithm in learning confusing data based on pre-trained features.<br /> <br /> = Critique =<br /> <br /> The classification accuracy of CSL was made with algorithms not designed to deal with confusing data and which do not first classify the task of each observation.<br /> <br /> Human task annotation is also imperfect, so one additional application of CSL may be to attempt to flag task annotation errors made by humans, such as in sorting comments for items sold by online retailers; concerned customers, in particular, may not correctly label their comments as &quot;refund&quot;, &quot;order didn't arrive&quot;, &quot;order damaged&quot;, &quot;how good the item is&quot; etc.<br /> <br /> Compared to the standard supervised learning, Multi-label learning can associate a training sample with multiple category tags at the same time. It can assign multiple labels to some hidden instances and can be reduced to standard supervised learning by limiting the number of class labels per instance. <br /> <br /> This algorithm will also have a huge issue in scaling, as the proposed method requires repeated training processes, so it might be too expensive for researchers to implement and improve on this algorithm.<br /> <br /> This research paper should have included a plot on loss (of both functions) against epochs in the paper. A common issue with fixing the parameters of one network and updating the other is the variability during training. This is prevalent in other algorithms with similar training methods such as generative adversarial networks (GAN). For instance, ''mode collapse'' is the issue of one network stuck in local minima and other networks that rely on this network may receive incorrect signals during backpropagation. In the case of CSL-Net, since the Deconfusing-Net directly relies on Mapping-Net for training labels, if the Mapping-Net is unable to sufficiently converge, the Deconfusing-Net may incorrectly learn the mapping from inputs to the task. For data with high noise, oscillations may severely prolong the time needed to converge because of the strong correlation in prediction between the two networks.<br /> <br /> - It would be interesting to see this implemented in more examples, to test the robustness of different types of data. The validation tasks chosen by data are all very simple, and CSL is actually not necessary for those tasks. For the colored MNIST data, a simple function can be written to distinguish the color label from the number label. The same problem applied to the Kaggle Fashion product dataset. The candidate label can be easily classified into different tasks by some wording analysis or meaning classification program or even manual classification. Even though the idea discussed by authors are interesting, the examples suggested by authors seem to suggest very limited or even unnecessary application. In most cases, it is more beneficial to treat the Confusing Multi-task Data problems separately into two distinct stages: we classify the tasks first according to the meaning of the label, and then we perform a multi-class/multi-label training process.<br /> <br /> Even though this paper has already included some examples when testing the CSL in experiments, it will be better to include more detailed examples for partial-label in the &quot;Application of Multi-label Learning&quot; section.<br /> <br /> When using this framework for classification, the order of the one-hot classification labels for each task will likely influence the relationships learned between each task, since the same output header is used for all tasks. This may be why this method fails to learn low-level representations and requires pretraining. I would like to see more explanation in the paper about why this isn't a problem if it was investigated.<br /> <br /> It would be a good idea to include comparison details in the summary to make the results and the conclusion more convincing. For instance, though the paper introduced the result generated using confusion data, and provide some applications for multi-label learning, these two sections still fell short and could use some technical details as supporting evidence.<br /> <br /> It is interesting to investigate if the order of adding tasks will influence the model performance.<br /> <br /> It would be interesting to see the effectiveness of applying CSL in face recognition, such that not only does the algorithm map the face to identity, it also categorizes the face based on other features like beard/no beard and glasses/no glasses simultaneously.<br /> <br /> It would be better for the researchers to compare the efficiency of this approach with other models.<br /> <br /> For pattern recognition,pre-trained features were used in the algorithm. It would be interesting to see how the effectiveness of the model changes if we train it with data directly from the CNN structure in the future.<br /> <br /> So basically given a confused dataset CSL finds the important tasks or labels from the dataset as can be seen from the fruit example. In the example, fruits are grouped under their names, their tastes, and their color, when CSL is given a mixed dataset. Hence given an unstructured data, unlabeled, confused dataset CSL helps in finding the labels, which in turn can help in cleaning the dataset and further in preparing high-quality training data set which is very important in different ML algorithms. Since at present preparing these dataset requires manual data annotations, CSL can save time in that process.<br /> <br /> For the Colorful-Mnist data set, the goal is to understand the concept of multiple classification tasks from these examples. All inputs have multiple classification tasks. Each observed sample only represents the classification result of one task, and the task from which the sample comes is unknown.<br /> <br /> It would be nice to know why the given metrics of confusing supervised learning are used. The authors should have used several different metrics and show that CSL's overall performs better than other methods. And what are &quot;the other methods&quot; referring to? algorithm<br /> <br /> In the Iterative Deconfusing algorithm section, the Training of Mapping-Net needs more explanation. The authors should specify what it is doing before showing its equations.<br /> <br /> For the results section, it would be more intuitive and stronger if the author provide more detail on these two methods and add a plot to support the claim. Based on the text, it might not be an obvious comparison.<br /> <br /> = References =<br /> <br />  Su, Xin, et al. &quot;Task Understanding from Confusing Multi-task Data.&quot;<br /> <br />  Caruana, R. (1997) &quot;Multi-task learning&quot;<br /> <br />  Lee, D.-H. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges in representation learning, ICML, vol. 3, 2013, pp. 2–8. <br /> <br />  Tan, Q., Yu, Y., Yu, G., and Wang, J. Semi-supervised multi-label classification using incomplete label information. Neurocomputing, vol. 260, 2017, pp. 192–202.<br /> <br />  Chavdarova, Tatjana, and François Fleuret. &quot;Sgan: An alternative training of generative adversarial networks.&quot; In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9407-9415. 2018.<br /> <br />  Guo-Ping Liu, Jian-Jun Yan, Yi-Qin Wang, Jing-Jing Fu, Zhao-Xia Xu, Rui Guo, Peng Qian, &quot;Application of Multilabel Learning Using the Relevant Feature for Each Label in Chronic Gastritis Syndrome Diagnosis&quot;, Evidence-Based Complementary and Alternative Medicine, vol. 2012, Article ID 135387, 9 pages, 2012. https://doi.org/10.1155/2012/135387</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Neural_Speed_Reading_via_Skim-RNN&diff=49099 Neural Speed Reading via Skim-RNN 2020-12-04T05:45:52Z <p>Z42qin: /* Conclusion */</p> <hr /> <div>== Group ==<br /> <br /> Mingyan Dai, Jerry Huang, Daniel Jiang<br /> <br /> == Introduction ==<br /> <br /> Recurrent Neural Network (RNN) is a class of artificial neural networks where the connection between nodes form a directed graph along with time series and has time dynamic behavior. RNN is derived from a feedforward neural network and can use its memory to process variable-length input sequences. In RNNs, outputs can be fed back to the network as inputs and hence creating a recurrent structure to handle varying lengths of data. This makes it suitable for tasks such as unsegmented, connected handwriting recognition, and speech recognition.<br /> <br /> In Natural Language Processing, Recurrent Neural Network (RNN) is a common architecture used to sequentially ‘read’ input tokens and output a distributed representation for each token. By recurrently updating the hidden state of the neural network, an RNN can inherently require the same computational cost across time. However, when it comes to processing input tokens, some tokens are comparatively less important to the overall representation of a piece of text or query. For example, in the application of RNN to the question answering problem, it is not uncommon to encounter parts of a passage that are irrelevant to answering the query.<br /> <br /> LSTM-Jump (Yu et al., 2017), a variant of LSTMs, was introduced to improve efficiency by skipping multiple tokens at a given step. Skip-RNN skips multiple steps and jumps based on RNNs’ current hidden state. In contrast, Skim-RNN takes advantage of 'skimming' rather than 'skipping tokens'. Skim-RNN predicts each word as important or unimportant. A small RNN is used if the word is not important, and a large RNN is used if the word is important. This paper demonstrates that skimming achieves higher accuracy compared to skipping tokens, implying that paying attention to unimportant tokens is better than completely ignoring them.<br /> <br /> == Model ==<br /> <br /> In this paper, the authors introduce a model called 'skim-RNN', which takes advantage of ‘skimming’ less important tokens or pieces of text rather than ‘skipping’ them entirely. This models the human ability to skim through passages, or to spend less time reading parts that do not affect the reader’s main objective. While this leads to a loss in the comprehension rate of the text , it greatly reduces the amount of time spent reading by not focusing on areas that will not significantly affect efficiency when it comes to the reader's objective.<br /> <br /> 'Skim-RNN' works by rapidly determining the significance of each input and spending less time processing unimportant input tokens by using a smaller RNN to update only a fraction of the hidden state. When the decision is to ‘fully read’, that is to not skim the text, Skim-RNN updates the entire hidden state with the default RNN cell. Since the hard decision function (‘skim’ or ‘read’) is non-differentiable, the authors use a gumbel-softmax  to estimate the gradient of the function, rather than traditional methods such as REINFORCE (policy gradient). Gumbel-softmax provides a differentiable sample compared to the non-differentiable sample of a categorical distribution . The switching mechanism between the two RNN cells enables Skim-RNN to reduce the total number of float operations (Flop reduction, or Flop-R). A high skimming rate often leads to faster inference on CPUs, which makes it very useful for large-scale products and small devices.<br /> <br /> The Skim-RNN has the same input and output interfaces as standard RNNs, so it can be conveniently used to speed up RNNs in existing models. In addition, the speed of Skim-RNN can be dynamically controlled at inference time by adjusting a parameter for the threshold for the ‘skim’ decision.<br /> <br /> === Related Works ===<br /> <br /> As the popularity of neural networks has grown, significant attention has been given to make them faster and lighter. In particular, relevant work focused on reducing the computational cost of recurrent neural networks has been carried out by several other related works. For example, LSTM-Jump (You et al., 2017)  models aim to speed up run times by skipping certain input tokens, as opposed to skimming them. Choi et al. (2017) proposed a model that uses a CNN-based sentence classifier to determine the most relevant sentence(s) to the question and then uses an RNN-based question-answering model. This model focuses on reducing GPU run-times (as opposed to Skim-RNN which focuses on minimizing CPU-time or Flop) and is also focused only on question answering.<br /> <br /> === Implementation ===<br /> <br /> A Skim-RNN consists of two RNN cells, a default (big) RNN cell of hidden state size &lt;math&gt;d&lt;/math&gt; and a small RNN cell of hidden state size &lt;math&gt;d'&lt;/math&gt;, where &lt;math&gt;d&lt;/math&gt; and &lt;math&gt;d'&lt;/math&gt; are parameters defined by the user and &lt;math&gt;d' \ll d&lt;/math&gt;. This follows the fact that there should be a small RNN cell defined for when text is meant to be skimmed and a larger one for when the text should be processed as normal.<br /> <br /> Each RNN cell will have its own set of weights and bias as well as be any variant of an RNN. There is no requirement on how the RNN itself is structured, rather the core concept is to allow the model to dynamically make a decision as to which cell to use when processing input tokens. Note that skipping text can be incorporated by setting &lt;math&gt;d'&lt;/math&gt; to 0, which means that when the input token is deemed irrelevant to a query or classification task, nothing about the information in the token is retained within the model.<br /> <br /> Experimental results suggest that this model is faster than using a single large RNN to process all input tokens, as the smaller RNN requires fewer floating-point operations to process the token. Additionally, higher accuracy and computational efficiency are achieved. <br /> <br /> ==== Inference ====<br /> <br /> At each time step &lt;math&gt;t&lt;/math&gt;, the Skim-RNN unit takes in an input &lt;math&gt;{\bf x}_t \in \mathbb{R}^d&lt;/math&gt; as well as the previous hidden state &lt;math&gt;{\bf h}_{t-1} \in \mathbb{R}^d&lt;/math&gt; and outputs the new state &lt;math&gt;{\bf h}_t &lt;/math&gt; (although the dimensions of the hidden state and input are the same, this process holds for different sizes as well). In the Skim-RNN, there is a hard decision that needs to be made whether to read or skim the input, although there could be potential to include options for multiple levels of skimming.<br /> <br /> The decision to read or skim is done using a multinomial random variable &lt;math&gt;Q_t&lt;/math&gt; over the probability distribution of choices &lt;math&gt;{\bf p}_t&lt;/math&gt;, where<br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;{\bf p}_t = \text{softmax}(\alpha({\bf x}_t, {\bf h}_{t-1})) = \text{softmax}({\bf W}[{\bf x}_t; {\bf h}_{t-1}]+{\bf b}) \in \mathbb{R}^k&lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> where &lt;math&gt;{\bf W} \in \mathbb{R}^{k \times 2d}&lt;/math&gt;, &lt;math&gt;{\bf b} \in \mathbb{R}^{k}&lt;/math&gt; are weights to be learned and &lt;math&gt;[{\bf x}_t; {\bf h}_{t-1}] \in \mathbb{R}^{2d}&lt;/math&gt; indicates the row concatenation of the two vectors. In this case, &lt;math&gt; \alpha &lt;/math&gt; can have any form as long as the complexity of calculating it is less than &lt;math&gt; O(d^2)&lt;/math&gt;. Letting &lt;math&gt;{\bf p}^1_t&lt;/math&gt; indicate the probability for fully reading and &lt;math&gt;{\bf p}^2_t&lt;/math&gt; indicate the probability for skimming the input at time &lt;math&gt; t&lt;/math&gt;, it follows that the decision to read or skim can be modelled using a random variable &lt;math&gt; Q_t&lt;/math&gt; by sampling from the distribution &lt;math&gt;{\bf p}_t&lt;/math&gt; and<br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;Q_t \sim \text{Multinomial}({\bf p}_t)&lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> Without loss of generality, we can define &lt;math&gt; Q_t = 1&lt;/math&gt; to indicate that the input will be read while &lt;math&gt; Q_t = 2&lt;/math&gt; indicates that it will be skimmed. Reading requires applying the full RNN on the input as well as the previous hidden state to modify the entire hidden state while skimming only modifies part of the prior hidden state.<br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;<br /> {\bf h}_t = \begin{cases}<br /> f({\bf x}_t, {\bf h}_{t-1}) &amp; Q_t = 1\\<br /> [f'({\bf x}_t, {\bf h}_{t-1});{\bf h}_{t-1}(d'+1:d)] &amp; Q_t = 2<br /> \end{cases}<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> where &lt;math&gt; f &lt;/math&gt; is a full RNN with output of dimension &lt;math&gt;d&lt;/math&gt; and &lt;math&gt;f'&lt;/math&gt; is a smaller RNN with &lt;math&gt;d'&lt;/math&gt;-dimensional output. This has advantage that when the model decides to skim, then the computational complexity of that step is only &lt;math&gt;O(d'd)&lt;/math&gt;, which is much smaller than &lt;math&gt;O(d^2)&lt;/math&gt; due to previously defining &lt;math&gt; d' \ll d&lt;/math&gt;.<br /> <br /> ==== Training ====<br /> <br /> Since the expected loss/error of the model is a random variable that depends on the sequence of random variables &lt;math&gt; \{Q_t\} &lt;/math&gt;, the loss is minimized with respect to the distribution of the variables. &lt;math&gt; \{Q_t\} &lt;/math&gt; is assumed to have a multinomial distribution with probability of pt. Defining the loss to be minimized while conditioning on a particular sequence of decisions<br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;<br /> L(\theta\vert Q)<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> where &lt;math&gt;Q=Q_1\dots Q_T&lt;/math&gt; is a sequence of decisions of length &lt;math&gt;T&lt;/math&gt;, then the expected loss over the distribution of the sequence of decisions is<br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;<br /> \mathbb{E}[L(\theta)] = \sum_{Q} L(\theta\vert Q)P(Q) = \sum_Q L(\theta\vert Q) \Pi_j {\bf p}_j^{Q_j}<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> Since calculating &lt;math&gt;\delta \mathbb{E}_{Q_t}[L(\theta)]&lt;/math&gt; directly is rather infeasible, it is possible to approximate the gradients with a gumbel-softmax distribution . Reparameterizing &lt;math&gt; {\bf p}_t&lt;/math&gt; as &lt;math&gt; {\bf r}_t&lt;/math&gt;, then the back-propagation can flow to &lt;math&gt; {\bf p}_t&lt;/math&gt; without being blocked by &lt;math&gt; Q_t&lt;/math&gt; and the approximation can arbitrarily approach &lt;math&gt; Q_t&lt;/math&gt; by controlling the parameters. The reparameterized distribution is therefore<br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;<br /> {\bf r}_t^i = \frac{\text{exp}(\log({\bf p}_t^i + {g_t}^i)/\tau)}{\sum_j\text{exp}(\log({\bf p}_t^j + {g_t}^j)/\tau)}<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> where &lt;math&gt;{g_t}^i&lt;/math&gt; is an independent sample from a &lt;math&gt;\text{Gumbel}(0, 1) = -\log(-\log(\text{Uniform}(0, 1))&lt;/math&gt; random variable and &lt;math&gt;\tau&lt;/math&gt; is a parameter that represents a temperature. Then it can be rewritten that<br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;<br /> {\bf h}_t = \sum_i {\bf r}_t^i {\bf \tilde{h}}_t<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> where &lt;math&gt;{\bf \tilde{h}}_t&lt;/math&gt; is the previous equation for &lt;math&gt;{\bf h}_t&lt;/math&gt;. The temperature parameter gradually decreases with time, and &lt;math&gt;{\bf r}_t^i&lt;/math&gt; becomes more discrete as it approaches 0.<br /> <br /> A final addition to the model is to encourage skimming when possible. Therefore an extra term related to the negative log probability of skimming and the sequence length. Therefore the final loss function used for the model is denoted by <br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;<br /> L'(\theta) =L(\theta) + \gamma \cdot\frac{1}{T} \sum_i -\log({\bf \tilde{p}}^i_t)<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> where &lt;math&gt; \gamma &lt;/math&gt; is a parameter used to control the ratio between the main loss function and the negative log probability of skimming.<br /> <br /> == Experiment ==<br /> <br /> The effectiveness of Skim-RNN was measured in terms of accuracy and float operation reduction on four classification tasks and a question-answering task. These tasks were chosen because they do not require one’s full attention to every detail of the text, but rather ask for capturing the high-level information (classification) or focusing on a specific portion (QA) of the text, which a common context for speed reading. The tasks themselves are listed in the table below.<br /> <br /> [[File:Table1SkimRNN.png|center|1000px]]<br /> <br /> === Classification Tasks ===<br /> <br /> In a language classification task, the input is a sequence of words and the output is the vector of categorical probabilities. Each word is embedded into a &lt;math&gt;d&lt;/math&gt;-dimensional vector. We initialize the vector with GloVe  to form representations of the words and use those as the inputs for a long short-term memory (LSTM) architecture. A linear transformation on the last hidden state of the LSTM and then a softmax function is applied to obtain the classification probabilities. Adam  is used for optimization, with an initial learning rate of 0.0001. For Skim-LSTM, &lt;math&gt;\tau = \max(0.5, exp(−rn))&lt;/math&gt; where &lt;math&gt;r = 1e-4&lt;/math&gt; and &lt;math&gt;n&lt;/math&gt; is the global training step, following . We experiment on different sizes of big LSTM (&lt;math&gt;d \in \{100, 200\}&lt;/math&gt;) and small LSTM (&lt;math&gt;d' \in \{5, 10, 20\}&lt;/math&gt;) and the ratio between the model loss and the skim loss (&lt;math&gt;\gamma\in \{0.01, 0.02\}&lt;/math&gt;) for Skim-LSTM. The batch sizes used were 32 for SST and Rotten Tomatoes, and 128 for others. For all models, early stopping is used when the validation accuracy did not increase for 3000 global steps.<br /> <br /> ==== Results ====<br /> <br /> [[File:Table2SkimRNN.png|center|1000px]]<br /> <br /> [[File:Figure2SkimRNN.png|center|1000px]]<br /> <br /> Table 2 shows the accuracy and computational cost of the Skim-RNN model compared with other standard models. It is evident that the Skim-RNN model produces a speed-up on the computational complexity of the task while maintaining a high degree of accuracy. Also, it is interesting to know that the accuracy improvement over LSTM could be due to the increased stability of the hidden state, as the majority of the hidden state is not updated when skimming. Meanwhile, figure 2 demonstrates the effect of varying the size of the small hidden state as well as the parameter &lt;math&gt;\gamma&lt;/math&gt; on the accuracy and computational cost.<br /> <br /> [[File:Table3SkimRNN.png|center|1000px]]<br /> <br /> Table 3 shows an example of a classification task over a IMDb dataset, where Skim-RNN with &lt;math&gt;d = 200&lt;/math&gt;, &lt;math&gt;d' = 10&lt;/math&gt;, and &lt;math&gt;\gamma = 0.01&lt;/math&gt; correctly classifies it with a high skimming rate (92%). The goal is to classify the review as either positive or negative. The black words are skimmed, and the blue words are fully read. The skimmed words are clearly irrelevant and the model learns to only carefully read the important words, such as ‘liked’, ‘dreadful’, and ‘tiresome’.<br /> <br /> === Question Answering Task ===<br /> <br /> In the Stanford Question Answering Dataset, the task was to locate the answer span for a given question in a context paragraph. The effectiveness of Skim-RNN for SQuAD was evaluated using two different models: LSTM+Attention and BiDAF . The first model was inspired by most then-present QA systems consisting of multiple LSTM layers and an attention mechanism. This type of model is complex enough to reach reasonable accuracy on the dataset and simple enough to run well-controlled analyses for the Skim-RNN. The second model was an open-source model designed for SQuAD, used primarily to show that Skim-RNN could replace RNN in existing complex systems.<br /> <br /> ==== Training ==== <br /> <br /> Adam was used with an initial learning rate of 0.0005. For stable training, the model was pre-trained with a standard LSTM for the first 5k steps, and then fine-tuned with Skim-LSTM.<br /> <br /> ==== Results ====<br /> <br /> [[File:Table4SkimRNN.png|center|1000px]]<br /> <br /> Table 4 shows the accuracy (F1 and EM) of LSTM+Attention and Skim-LSTM+Attention models as well as VCRNN . It can be observed from the table that the skimming models achieve higher or similar accuracy scores compared to the non-skimming models while also reducing the computational cost by more than 1.4 times. In addition, decreasing layers (1 layer) or hidden size (&lt;math&gt;d=5&lt;/math&gt;) improved the computational cost but significantly decreases the accuracy compared to skimming. The table also shows that replacing LSTM with Skim-LSTM in an existing complex model (BiDAF) stably gives reduced computational cost without losing much accuracy (only 0.2% drop from 77.3% of BiDAF to 77.1% of Sk-BiDAF with &lt;math&gt;\gamma = 0.001&lt;/math&gt;).<br /> <br /> An explanation for this trend that was given is that the model is more confident about which tokens are important in the second layer. Second, higher &lt;math&gt;\gamma&lt;/math&gt; values lead to a higher skimming rate, which agrees with its intended functionality.<br /> <br /> Figure 4 shows the F1 score of LSTM+Attention model using standard LSTM and Skim LSTM, sorted in ascending order by Flop-R (computational cost). While models tend to perform better with larger computational cost, Skim LSTM (Red) outperforms standard LSTM (Blue) with a comparable computational cost. It can also be seen that the computational cost of Skim-LSTM is more stable across different configurations and computational cost. Moreover, increasing the value of &lt;math&gt;\gamma&lt;/math&gt; for Skim-LSTM gradually increases the skipping rate and Flop-R, while it also led to reduced accuracy.<br /> <br /> === Runtime Benchmark ===<br /> <br /> [[File:Figure6SkimRNN.png|center|1000px]]<br /> <br /> The details of the runtime benchmarks for LSTM and Skim-LSTM, which are used to estimate the speedup of Skim-LSTM-based models in the experiments, are also discussed. A CPU-based benchmark was assumed to be the default benchmark, which has a direct correlation with the number of float operations that can be performed per second. As mentioned previously, the speed-up results in Table 2 (as well as Figure 7) are benchmarked using Python (NumPy), instead of popular frameworks such as TensorFlow or PyTorch.<br /> <br /> Figure 7 shows the relative speed gain of Skim-LSTM compared to standard LSTM with varying hidden state size and skim rate. NumPy was used, with the inferences run on a single thread of CPU. The ratio between the reduction of the number of float operations (Flop-R) of LSTM and Skim-LSTM was plotted, with the ratio acting as a theoretical upper bound of the speed gain on CPUs. From here, it can be noticed that there is a gap between the actual gain and the theoretical gain in speed, with the gap being larger with more overhead of the framework or more parallelization. The gap also decreases as the hidden state size increases because the overhead becomes negligible with very large matrix operations. This indicates that Skim-RNN provides greater benefits for RNNs with larger hidden state size. However, combining Skim-RNN with a CPU-based framework can lead to substantially lower latency than GPUs.<br /> <br /> == Results ==<br /> <br /> The results clearly indicate that the Skim-RNN model provides features that are suitable for general reading tasks, which include classification and question answering. While the tables indicate that minor losses in accuracy occasionally did result when parameters were set at specific values, they were minor and were acceptable given the improvement in runtime.<br /> <br /> '''Controlling skim rate:''' An important advantage of Skim-RNN is that the skim rate (and thus computational cost) can be dynamically controlled at inference time by adjusting the threshold for<br /> ‘skim’ decision probability &lt;math&gt;{\bf p}^1_t&lt;/math&gt;. Figure 5 shows the trade-off between the accuracy and computational cost for two settings, confirming the importance of skimming (&lt;math&gt;d' &gt; 0&lt;/math&gt;) compared to skipping (&lt;math&gt;d' = 0&lt;/math&gt;).<br /> <br /> '''Visualization:''' Figure 6 shows that the model does not skim when the input seems to be relevant to answering the question, which was as expected by the design of the model. In addition, the LSTM in the second layer skims more than that in the first layer mainly because the second layer is more confident about the importance of each token.<br /> <br /> == Conclusion ==<br /> <br /> A Skim-RNN can offer better latency results on a CPU compared to a standard RNN on a GPU, with lower computational cost and better latency, as demonstrated through the results of this study. Compared to RNN, Skim-RNN takes the advantage of &quot;skimming&quot; rather than &quot;reading&quot;, spends less time on parts of the input that is unimportant. Future work (as stated by the authors) involves using Skim-RNN for applications that require much higher hidden state size, such as video understanding, and using multiple small RNN cells for varying degrees of skimming. Further, since it has the same input and output interface as a regular RNN, it can replace RNNs in existing applications.<br /> <br /> == Critiques ==<br /> <br /> 1. It seems like Skim-RNN is using the not full RNN of processing words that are not important, thus it can increase speed in some very particular circumstances (ie, only small networks). The extra model complexity did slow down the speed while trying to &quot;optimizing&quot; the efficiency and sacrifice part of accuracy while doing so. It is only trying to target a very specific situation (classification/question-answering) and made comparisons only with the baseline LSTM model. It would be definitely more persuasive if the model can compare with some of the state of art neural network models.<br /> <br /> 2. This model of Skim-RNN is pretty good to extract binary classification type of text, thus it would be interesting for this to be applied to stock market news analysis. For example, a press release from a company can be analyzed quickly using this model and immediately give the trader a positive or negative summary of the news. Would be beneficial in trading since time and speed is an important factor when executing a trade.<br /> <br /> 3. An appropriate application for Skim-RNN could be customer service chatbots as they can analyze a customer's message and skim associated company policies to craft a response. In this circumstance, quickly analyzing text is ideal to not waste customers' time.<br /> <br /> 4. This could be applied to news apps to improve readability by highlighting important sections.<br /> <br /> 5. This summary describes an interesting and useful model that can save readers time for reading an article. I think it will be interesting that discuss more on training a model by Skim-RNN to highlight the important sections in very long textbooks. As a student, having highlights in the textbook is really helpful to study. But highlight the important parts in a time-consuming work for the author, maybe using Skim-RNN can provide a nice model to do this job. <br /> <br /> 6. Besides the good training performance of Skim-RNN, it's good to see the algorithm even performs well simply by training with CPU. It would make it possible to perform the result on lite-platforms.<br /> <br /> 7. Another good application of Skim-RNN could be in reading the terms and conditions of websites. A lot of the terms and conditions documents tend to be painfully long and majority of customers because of the length of the document tend to sign them without reading them. Since these websites can compromise the personal information of the customers by giving it to third parties, it is worthwhile for the customers to know at least what they are signing. Infact, this can also be applied to read important legal official documents like lease agreements etc.<br /> <br /> 8. Another [https://arxiv.org/abs/1904.00761 paper], written by Christian Hansen et al., also discussed Neural Speed Reading. However, conducting Neural Speed reading via a Skim-RNN, this paper suggested using a Structural-Jump-LSTM. The Structural-Jump-LSTM can both skip and jump test during inference. This model consisted of a standard LSTM with 2 agents: 1 capable of bypassing single words and another capable of bypassing punctuation.<br /> <br /> 9. Another method of increasing the speed of human reading is to quickly flash each of the words in an article in quick succession in the middle of the screen. However, it is important that relative duration each word spends on the screen is adjusted, so that a human reader will feel natural reading it. Could the skimming results from this article be incorporated into the display-one-word-at-a-time method to even further increase the speed one can read an article? Would adding color to the unskimmed words also help increase reading speed?<br /> <br /> 10. It is interesting how the paper mentions the correlation between a system’s performance in a given reading-based task and the speed of reading. This conclusion was reached by the human reading pattern with neural attention in an unsupervised learning approach.<br /> <br /> 11. With the comparison in figure 7, skim-LSTM has a definite advantage over the original LSTM model on the speed which give us an inspiration that we can boost the speed of original LSTM model used in daily life applications such as stock forecasting.<br /> <br /> 12. The paper never specifies which type of CPU they tested on (Intel vs AMD). Would be interesting to see which manufacturer's CPU Skim-RNN works better on as each CPU has different FLOP values.<br /> <br /> == Applications ==<br /> <br /> Recurrent architectures are used in many other applications:<br /> <br /> 1. '''Real-time video processing''' is an exceedingly demanding and resource-constrained task, particularly in edge settings.<br /> <br /> 2. '''Music Recommend system''' is which many music platforms such as Spotify and Pandora used for recommending music based on the user's information.<br /> <br /> 3. '''Speech recognition''' enables the recognition and translation of spoken language into text by computers. <br /> <br /> It would be interesting to see if this method could be applied to those cases for more efficient inference, such as on drones or self-driving cars. Another possible application is real-time edge processing of game video for sports arenas.<br /> <br /> == References ==<br /> <br />  Patricia Anderson Carpenter Marcel Adam Just. The Psychology of Reading and Language Comprehension. 1987.<br /> <br />  Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.<br /> <br />  Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992.<br /> <br />  Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.<br /> <br />  Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.<br /> <br />  Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. In ICLR, 2017a.<br /> <br />  Yacine Jernite, Edouard Grave, Armand Joulin, and Tomas Mikolov. Variable computation in recurrent neural networks. In ICLR, 2017.<br /> <br />  Adams Wei Yu, Hongrae Lee, and Quoc V Le. Learning to skim text. In ACL, 2017.<br /> <br />  Eunsol Choi, Daniel Hewlett, Alexandre Lacoste, Illia Polosukhin, Jakob Uszkoreit, and Jonathan Berant. Coarse-to-fine question answering for long documents. In ACL, 2017.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Neural_Speed_Reading_via_Skim-RNN&diff=49098 Neural Speed Reading via Skim-RNN 2020-12-04T05:38:11Z <p>Z42qin: /* Training */</p> <hr /> <div>== Group ==<br /> <br /> Mingyan Dai, Jerry Huang, Daniel Jiang<br /> <br /> == Introduction ==<br /> <br /> Recurrent Neural Network (RNN) is a class of artificial neural networks where the connection between nodes form a directed graph along with time series and has time dynamic behavior. RNN is derived from a feedforward neural network and can use its memory to process variable-length input sequences. In RNNs, outputs can be fed back to the network as inputs and hence creating a recurrent structure to handle varying lengths of data. This makes it suitable for tasks such as unsegmented, connected handwriting recognition, and speech recognition.<br /> <br /> In Natural Language Processing, Recurrent Neural Network (RNN) is a common architecture used to sequentially ‘read’ input tokens and output a distributed representation for each token. By recurrently updating the hidden state of the neural network, an RNN can inherently require the same computational cost across time. However, when it comes to processing input tokens, some tokens are comparatively less important to the overall representation of a piece of text or query. For example, in the application of RNN to the question answering problem, it is not uncommon to encounter parts of a passage that are irrelevant to answering the query.<br /> <br /> LSTM-Jump (Yu et al., 2017), a variant of LSTMs, was introduced to improve efficiency by skipping multiple tokens at a given step. Skip-RNN skips multiple steps and jumps based on RNNs’ current hidden state. In contrast, Skim-RNN takes advantage of 'skimming' rather than 'skipping tokens'. Skim-RNN predicts each word as important or unimportant. A small RNN is used if the word is not important, and a large RNN is used if the word is important. This paper demonstrates that skimming achieves higher accuracy compared to skipping tokens, implying that paying attention to unimportant tokens is better than completely ignoring them.<br /> <br /> == Model ==<br /> <br /> In this paper, the authors introduce a model called 'skim-RNN', which takes advantage of ‘skimming’ less important tokens or pieces of text rather than ‘skipping’ them entirely. This models the human ability to skim through passages, or to spend less time reading parts that do not affect the reader’s main objective. While this leads to a loss in the comprehension rate of the text , it greatly reduces the amount of time spent reading by not focusing on areas that will not significantly affect efficiency when it comes to the reader's objective.<br /> <br /> 'Skim-RNN' works by rapidly determining the significance of each input and spending less time processing unimportant input tokens by using a smaller RNN to update only a fraction of the hidden state. When the decision is to ‘fully read’, that is to not skim the text, Skim-RNN updates the entire hidden state with the default RNN cell. Since the hard decision function (‘skim’ or ‘read’) is non-differentiable, the authors use a gumbel-softmax  to estimate the gradient of the function, rather than traditional methods such as REINFORCE (policy gradient). Gumbel-softmax provides a differentiable sample compared to the non-differentiable sample of a categorical distribution . The switching mechanism between the two RNN cells enables Skim-RNN to reduce the total number of float operations (Flop reduction, or Flop-R). A high skimming rate often leads to faster inference on CPUs, which makes it very useful for large-scale products and small devices.<br /> <br /> The Skim-RNN has the same input and output interfaces as standard RNNs, so it can be conveniently used to speed up RNNs in existing models. In addition, the speed of Skim-RNN can be dynamically controlled at inference time by adjusting a parameter for the threshold for the ‘skim’ decision.<br /> <br /> === Related Works ===<br /> <br /> As the popularity of neural networks has grown, significant attention has been given to make them faster and lighter. In particular, relevant work focused on reducing the computational cost of recurrent neural networks has been carried out by several other related works. For example, LSTM-Jump (You et al., 2017)  models aim to speed up run times by skipping certain input tokens, as opposed to skimming them. Choi et al. (2017) proposed a model that uses a CNN-based sentence classifier to determine the most relevant sentence(s) to the question and then uses an RNN-based question-answering model. This model focuses on reducing GPU run-times (as opposed to Skim-RNN which focuses on minimizing CPU-time or Flop) and is also focused only on question answering.<br /> <br /> === Implementation ===<br /> <br /> A Skim-RNN consists of two RNN cells, a default (big) RNN cell of hidden state size &lt;math&gt;d&lt;/math&gt; and a small RNN cell of hidden state size &lt;math&gt;d'&lt;/math&gt;, where &lt;math&gt;d&lt;/math&gt; and &lt;math&gt;d'&lt;/math&gt; are parameters defined by the user and &lt;math&gt;d' \ll d&lt;/math&gt;. This follows the fact that there should be a small RNN cell defined for when text is meant to be skimmed and a larger one for when the text should be processed as normal.<br /> <br /> Each RNN cell will have its own set of weights and bias as well as be any variant of an RNN. There is no requirement on how the RNN itself is structured, rather the core concept is to allow the model to dynamically make a decision as to which cell to use when processing input tokens. Note that skipping text can be incorporated by setting &lt;math&gt;d'&lt;/math&gt; to 0, which means that when the input token is deemed irrelevant to a query or classification task, nothing about the information in the token is retained within the model.<br /> <br /> Experimental results suggest that this model is faster than using a single large RNN to process all input tokens, as the smaller RNN requires fewer floating-point operations to process the token. Additionally, higher accuracy and computational efficiency are achieved. <br /> <br /> ==== Inference ====<br /> <br /> At each time step &lt;math&gt;t&lt;/math&gt;, the Skim-RNN unit takes in an input &lt;math&gt;{\bf x}_t \in \mathbb{R}^d&lt;/math&gt; as well as the previous hidden state &lt;math&gt;{\bf h}_{t-1} \in \mathbb{R}^d&lt;/math&gt; and outputs the new state &lt;math&gt;{\bf h}_t &lt;/math&gt; (although the dimensions of the hidden state and input are the same, this process holds for different sizes as well). In the Skim-RNN, there is a hard decision that needs to be made whether to read or skim the input, although there could be potential to include options for multiple levels of skimming.<br /> <br /> The decision to read or skim is done using a multinomial random variable &lt;math&gt;Q_t&lt;/math&gt; over the probability distribution of choices &lt;math&gt;{\bf p}_t&lt;/math&gt;, where<br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;{\bf p}_t = \text{softmax}(\alpha({\bf x}_t, {\bf h}_{t-1})) = \text{softmax}({\bf W}[{\bf x}_t; {\bf h}_{t-1}]+{\bf b}) \in \mathbb{R}^k&lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> where &lt;math&gt;{\bf W} \in \mathbb{R}^{k \times 2d}&lt;/math&gt;, &lt;math&gt;{\bf b} \in \mathbb{R}^{k}&lt;/math&gt; are weights to be learned and &lt;math&gt;[{\bf x}_t; {\bf h}_{t-1}] \in \mathbb{R}^{2d}&lt;/math&gt; indicates the row concatenation of the two vectors. In this case, &lt;math&gt; \alpha &lt;/math&gt; can have any form as long as the complexity of calculating it is less than &lt;math&gt; O(d^2)&lt;/math&gt;. Letting &lt;math&gt;{\bf p}^1_t&lt;/math&gt; indicate the probability for fully reading and &lt;math&gt;{\bf p}^2_t&lt;/math&gt; indicate the probability for skimming the input at time &lt;math&gt; t&lt;/math&gt;, it follows that the decision to read or skim can be modelled using a random variable &lt;math&gt; Q_t&lt;/math&gt; by sampling from the distribution &lt;math&gt;{\bf p}_t&lt;/math&gt; and<br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;Q_t \sim \text{Multinomial}({\bf p}_t)&lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> Without loss of generality, we can define &lt;math&gt; Q_t = 1&lt;/math&gt; to indicate that the input will be read while &lt;math&gt; Q_t = 2&lt;/math&gt; indicates that it will be skimmed. Reading requires applying the full RNN on the input as well as the previous hidden state to modify the entire hidden state while skimming only modifies part of the prior hidden state.<br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;<br /> {\bf h}_t = \begin{cases}<br /> f({\bf x}_t, {\bf h}_{t-1}) &amp; Q_t = 1\\<br /> [f'({\bf x}_t, {\bf h}_{t-1});{\bf h}_{t-1}(d'+1:d)] &amp; Q_t = 2<br /> \end{cases}<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> where &lt;math&gt; f &lt;/math&gt; is a full RNN with output of dimension &lt;math&gt;d&lt;/math&gt; and &lt;math&gt;f'&lt;/math&gt; is a smaller RNN with &lt;math&gt;d'&lt;/math&gt;-dimensional output. This has advantage that when the model decides to skim, then the computational complexity of that step is only &lt;math&gt;O(d'd)&lt;/math&gt;, which is much smaller than &lt;math&gt;O(d^2)&lt;/math&gt; due to previously defining &lt;math&gt; d' \ll d&lt;/math&gt;.<br /> <br /> ==== Training ====<br /> <br /> Since the expected loss/error of the model is a random variable that depends on the sequence of random variables &lt;math&gt; \{Q_t\} &lt;/math&gt;, the loss is minimized with respect to the distribution of the variables. &lt;math&gt; \{Q_t\} &lt;/math&gt; is assumed to have a multinomial distribution with probability of pt. Defining the loss to be minimized while conditioning on a particular sequence of decisions<br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;<br /> L(\theta\vert Q)<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> where &lt;math&gt;Q=Q_1\dots Q_T&lt;/math&gt; is a sequence of decisions of length &lt;math&gt;T&lt;/math&gt;, then the expected loss over the distribution of the sequence of decisions is<br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;<br /> \mathbb{E}[L(\theta)] = \sum_{Q} L(\theta\vert Q)P(Q) = \sum_Q L(\theta\vert Q) \Pi_j {\bf p}_j^{Q_j}<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> Since calculating &lt;math&gt;\delta \mathbb{E}_{Q_t}[L(\theta)]&lt;/math&gt; directly is rather infeasible, it is possible to approximate the gradients with a gumbel-softmax distribution . Reparameterizing &lt;math&gt; {\bf p}_t&lt;/math&gt; as &lt;math&gt; {\bf r}_t&lt;/math&gt;, then the back-propagation can flow to &lt;math&gt; {\bf p}_t&lt;/math&gt; without being blocked by &lt;math&gt; Q_t&lt;/math&gt; and the approximation can arbitrarily approach &lt;math&gt; Q_t&lt;/math&gt; by controlling the parameters. The reparameterized distribution is therefore<br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;<br /> {\bf r}_t^i = \frac{\text{exp}(\log({\bf p}_t^i + {g_t}^i)/\tau)}{\sum_j\text{exp}(\log({\bf p}_t^j + {g_t}^j)/\tau)}<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> where &lt;math&gt;{g_t}^i&lt;/math&gt; is an independent sample from a &lt;math&gt;\text{Gumbel}(0, 1) = -\log(-\log(\text{Uniform}(0, 1))&lt;/math&gt; random variable and &lt;math&gt;\tau&lt;/math&gt; is a parameter that represents a temperature. Then it can be rewritten that<br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;<br /> {\bf h}_t = \sum_i {\bf r}_t^i {\bf \tilde{h}}_t<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> <br /> where &lt;math&gt;{\bf \tilde{h}}_t&lt;/math&gt; is the previous equation for &lt;math&gt;{\bf h}_t&lt;/math&gt;. The temperature parameter gradually decreases with time, and &lt;math&gt;{\bf r}_t^i&lt;/math&gt; becomes more discrete as it approaches 0.<br /> <br /> A final addition to the model is to encourage skimming when possible. Therefore an extra term related to the negative log probability of skimming and the sequence length. Therefore the final loss function used for the model is denoted by <br /> <br /> &lt;div class=&quot;center&quot; style=&quot;width: auto; margin-left: auto; margin-right: auto;&quot;&gt;<br /> &lt;math&gt;<br /> L'(\theta) =L(\theta) + \gamma \cdot\frac{1}{T} \sum_i -\log({\bf \tilde{p}}^i_t)<br /> &lt;/math&gt;<br /> &lt;/div&gt;<br /> where &lt;math&gt; \gamma &lt;/math&gt; is a parameter used to control the ratio between the main loss function and the negative log probability of skimming.<br /> <br /> == Experiment ==<br /> <br /> The effectiveness of Skim-RNN was measured in terms of accuracy and float operation reduction on four classification tasks and a question-answering task. These tasks were chosen because they do not require one’s full attention to every detail of the text, but rather ask for capturing the high-level information (classification) or focusing on a specific portion (QA) of the text, which a common context for speed reading. The tasks themselves are listed in the table below.<br /> <br /> [[File:Table1SkimRNN.png|center|1000px]]<br /> <br /> === Classification Tasks ===<br /> <br /> In a language classification task, the input is a sequence of words and the output is the vector of categorical probabilities. Each word is embedded into a &lt;math&gt;d&lt;/math&gt;-dimensional vector. We initialize the vector with GloVe  to form representations of the words and use those as the inputs for a long short-term memory (LSTM) architecture. A linear transformation on the last hidden state of the LSTM and then a softmax function is applied to obtain the classification probabilities. Adam  is used for optimization, with an initial learning rate of 0.0001. For Skim-LSTM, &lt;math&gt;\tau = \max(0.5, exp(−rn))&lt;/math&gt; where &lt;math&gt;r = 1e-4&lt;/math&gt; and &lt;math&gt;n&lt;/math&gt; is the global training step, following . We experiment on different sizes of big LSTM (&lt;math&gt;d \in \{100, 200\}&lt;/math&gt;) and small LSTM (&lt;math&gt;d' \in \{5, 10, 20\}&lt;/math&gt;) and the ratio between the model loss and the skim loss (&lt;math&gt;\gamma\in \{0.01, 0.02\}&lt;/math&gt;) for Skim-LSTM. The batch sizes used were 32 for SST and Rotten Tomatoes, and 128 for others. For all models, early stopping is used when the validation accuracy did not increase for 3000 global steps.<br /> <br /> ==== Results ====<br /> <br /> [[File:Table2SkimRNN.png|center|1000px]]<br /> <br /> [[File:Figure2SkimRNN.png|center|1000px]]<br /> <br /> Table 2 shows the accuracy and computational cost of the Skim-RNN model compared with other standard models. It is evident that the Skim-RNN model produces a speed-up on the computational complexity of the task while maintaining a high degree of accuracy. Also, it is interesting to know that the accuracy improvement over LSTM could be due to the increased stability of the hidden state, as the majority of the hidden state is not updated when skimming. Meanwhile, figure 2 demonstrates the effect of varying the size of the small hidden state as well as the parameter &lt;math&gt;\gamma&lt;/math&gt; on the accuracy and computational cost.<br /> <br /> [[File:Table3SkimRNN.png|center|1000px]]<br /> <br /> Table 3 shows an example of a classification task over a IMDb dataset, where Skim-RNN with &lt;math&gt;d = 200&lt;/math&gt;, &lt;math&gt;d' = 10&lt;/math&gt;, and &lt;math&gt;\gamma = 0.01&lt;/math&gt; correctly classifies it with a high skimming rate (92%). The goal is to classify the review as either positive or negative. The black words are skimmed, and the blue words are fully read. The skimmed words are clearly irrelevant and the model learns to only carefully read the important words, such as ‘liked’, ‘dreadful’, and ‘tiresome’.<br /> <br /> === Question Answering Task ===<br /> <br /> In the Stanford Question Answering Dataset, the task was to locate the answer span for a given question in a context paragraph. The effectiveness of Skim-RNN for SQuAD was evaluated using two different models: LSTM+Attention and BiDAF . The first model was inspired by most then-present QA systems consisting of multiple LSTM layers and an attention mechanism. This type of model is complex enough to reach reasonable accuracy on the dataset and simple enough to run well-controlled analyses for the Skim-RNN. The second model was an open-source model designed for SQuAD, used primarily to show that Skim-RNN could replace RNN in existing complex systems.<br /> <br /> ==== Training ==== <br /> <br /> Adam was used with an initial learning rate of 0.0005. For stable training, the model was pre-trained with a standard LSTM for the first 5k steps, and then fine-tuned with Skim-LSTM.<br /> <br /> ==== Results ====<br /> <br /> [[File:Table4SkimRNN.png|center|1000px]]<br /> <br /> Table 4 shows the accuracy (F1 and EM) of LSTM+Attention and Skim-LSTM+Attention models as well as VCRNN . It can be observed from the table that the skimming models achieve higher or similar accuracy scores compared to the non-skimming models while also reducing the computational cost by more than 1.4 times. In addition, decreasing layers (1 layer) or hidden size (&lt;math&gt;d=5&lt;/math&gt;) improved the computational cost but significantly decreases the accuracy compared to skimming. The table also shows that replacing LSTM with Skim-LSTM in an existing complex model (BiDAF) stably gives reduced computational cost without losing much accuracy (only 0.2% drop from 77.3% of BiDAF to 77.1% of Sk-BiDAF with &lt;math&gt;\gamma = 0.001&lt;/math&gt;).<br /> <br /> An explanation for this trend that was given is that the model is more confident about which tokens are important in the second layer. Second, higher &lt;math&gt;\gamma&lt;/math&gt; values lead to a higher skimming rate, which agrees with its intended functionality.<br /> <br /> Figure 4 shows the F1 score of LSTM+Attention model using standard LSTM and Skim LSTM, sorted in ascending order by Flop-R (computational cost). While models tend to perform better with larger computational cost, Skim LSTM (Red) outperforms standard LSTM (Blue) with a comparable computational cost. It can also be seen that the computational cost of Skim-LSTM is more stable across different configurations and computational cost. Moreover, increasing the value of &lt;math&gt;\gamma&lt;/math&gt; for Skim-LSTM gradually increases the skipping rate and Flop-R, while it also led to reduced accuracy.<br /> <br /> === Runtime Benchmark ===<br /> <br /> [[File:Figure6SkimRNN.png|center|1000px]]<br /> <br /> The details of the runtime benchmarks for LSTM and Skim-LSTM, which are used to estimate the speedup of Skim-LSTM-based models in the experiments, are also discussed. A CPU-based benchmark was assumed to be the default benchmark, which has a direct correlation with the number of float operations that can be performed per second. As mentioned previously, the speed-up results in Table 2 (as well as Figure 7) are benchmarked using Python (NumPy), instead of popular frameworks such as TensorFlow or PyTorch.<br /> <br /> Figure 7 shows the relative speed gain of Skim-LSTM compared to standard LSTM with varying hidden state size and skim rate. NumPy was used, with the inferences run on a single thread of CPU. The ratio between the reduction of the number of float operations (Flop-R) of LSTM and Skim-LSTM was plotted, with the ratio acting as a theoretical upper bound of the speed gain on CPUs. From here, it can be noticed that there is a gap between the actual gain and the theoretical gain in speed, with the gap being larger with more overhead of the framework or more parallelization. The gap also decreases as the hidden state size increases because the overhead becomes negligible with very large matrix operations. This indicates that Skim-RNN provides greater benefits for RNNs with larger hidden state size. However, combining Skim-RNN with a CPU-based framework can lead to substantially lower latency than GPUs.<br /> <br /> == Results ==<br /> <br /> The results clearly indicate that the Skim-RNN model provides features that are suitable for general reading tasks, which include classification and question answering. While the tables indicate that minor losses in accuracy occasionally did result when parameters were set at specific values, they were minor and were acceptable given the improvement in runtime.<br /> <br /> '''Controlling skim rate:''' An important advantage of Skim-RNN is that the skim rate (and thus computational cost) can be dynamically controlled at inference time by adjusting the threshold for<br /> ‘skim’ decision probability &lt;math&gt;{\bf p}^1_t&lt;/math&gt;. Figure 5 shows the trade-off between the accuracy and computational cost for two settings, confirming the importance of skimming (&lt;math&gt;d' &gt; 0&lt;/math&gt;) compared to skipping (&lt;math&gt;d' = 0&lt;/math&gt;).<br /> <br /> '''Visualization:''' Figure 6 shows that the model does not skim when the input seems to be relevant to answering the question, which was as expected by the design of the model. In addition, the LSTM in the second layer skims more than that in the first layer mainly because the second layer is more confident about the importance of each token.<br /> <br /> == Conclusion ==<br /> <br /> A Skim-RNN can offer better latency results on a CPU compared to a standard RNN on a GPU, with lower computational cost, as demonstrated through the results of this study. Compared to RNN, Skim-RNN takes the advantage of &quot;skimming&quot; rather than &quot;reading&quot;, spends less time on parts of the input that is unimportant. Future work (as stated by the authors) involves using Skim-RNN for applications that require much higher hidden state size, such as video understanding, and using multiple small RNN cells for varying degrees of skimming. Further, since it has the same input and output interface as a regular RNN, it can replace RNNs in existing applications.<br /> <br /> == Critiques ==<br /> <br /> 1. It seems like Skim-RNN is using the not full RNN of processing words that are not important, thus it can increase speed in some very particular circumstances (ie, only small networks). The extra model complexity did slow down the speed while trying to &quot;optimizing&quot; the efficiency and sacrifice part of accuracy while doing so. It is only trying to target a very specific situation (classification/question-answering) and made comparisons only with the baseline LSTM model. It would be definitely more persuasive if the model can compare with some of the state of art neural network models.<br /> <br /> 2. This model of Skim-RNN is pretty good to extract binary classification type of text, thus it would be interesting for this to be applied to stock market news analysis. For example, a press release from a company can be analyzed quickly using this model and immediately give the trader a positive or negative summary of the news. Would be beneficial in trading since time and speed is an important factor when executing a trade.<br /> <br /> 3. An appropriate application for Skim-RNN could be customer service chatbots as they can analyze a customer's message and skim associated company policies to craft a response. In this circumstance, quickly analyzing text is ideal to not waste customers' time.<br /> <br /> 4. This could be applied to news apps to improve readability by highlighting important sections.<br /> <br /> 5. This summary describes an interesting and useful model that can save readers time for reading an article. I think it will be interesting that discuss more on training a model by Skim-RNN to highlight the important sections in very long textbooks. As a student, having highlights in the textbook is really helpful to study. But highlight the important parts in a time-consuming work for the author, maybe using Skim-RNN can provide a nice model to do this job. <br /> <br /> 6. Besides the good training performance of Skim-RNN, it's good to see the algorithm even performs well simply by training with CPU. It would make it possible to perform the result on lite-platforms.<br /> <br /> 7. Another good application of Skim-RNN could be in reading the terms and conditions of websites. A lot of the terms and conditions documents tend to be painfully long and majority of customers because of the length of the document tend to sign them without reading them. Since these websites can compromise the personal information of the customers by giving it to third parties, it is worthwhile for the customers to know at least what they are signing. Infact, this can also be applied to read important legal official documents like lease agreements etc.<br /> <br /> 8. Another [https://arxiv.org/abs/1904.00761 paper], written by Christian Hansen et al., also discussed Neural Speed Reading. However, conducting Neural Speed reading via a Skim-RNN, this paper suggested using a Structural-Jump-LSTM. The Structural-Jump-LSTM can both skip and jump test during inference. This model consisted of a standard LSTM with 2 agents: 1 capable of bypassing single words and another capable of bypassing punctuation.<br /> <br /> 9. Another method of increasing the speed of human reading is to quickly flash each of the words in an article in quick succession in the middle of the screen. However, it is important that relative duration each word spends on the screen is adjusted, so that a human reader will feel natural reading it. Could the skimming results from this article be incorporated into the display-one-word-at-a-time method to even further increase the speed one can read an article? Would adding color to the unskimmed words also help increase reading speed?<br /> <br /> 10. It is interesting how the paper mentions the correlation between a system’s performance in a given reading-based task and the speed of reading. This conclusion was reached by the human reading pattern with neural attention in an unsupervised learning approach.<br /> <br /> 11. With the comparison in figure 7, skim-LSTM has a definite advantage over the original LSTM model on the speed which give us an inspiration that we can boost the speed of original LSTM model used in daily life applications such as stock forecasting.<br /> <br /> 12. The paper never specifies which type of CPU they tested on (Intel vs AMD). Would be interesting to see which manufacturer's CPU Skim-RNN works better on as each CPU has different FLOP values.<br /> <br /> == Applications ==<br /> <br /> Recurrent architectures are used in many other applications:<br /> <br /> 1. '''Real-time video processing''' is an exceedingly demanding and resource-constrained task, particularly in edge settings.<br /> <br /> 2. '''Music Recommend system''' is which many music platforms such as Spotify and Pandora used for recommending music based on the user's information.<br /> <br /> 3. '''Speech recognition''' enables the recognition and translation of spoken language into text by computers. <br /> <br /> It would be interesting to see if this method could be applied to those cases for more efficient inference, such as on drones or self-driving cars. Another possible application is real-time edge processing of game video for sports arenas.<br /> <br /> == References ==<br /> <br />  Patricia Anderson Carpenter Marcel Adam Just. The Psychology of Reading and Language Comprehension. 1987.<br /> <br />  Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.<br /> <br />  Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992.<br /> <br />  Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.<br /> <br />  Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.<br /> <br />  Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. In ICLR, 2017a.<br /> <br />  Yacine Jernite, Edouard Grave, Armand Joulin, and Tomas Mikolov. Variable computation in recurrent neural networks. In ICLR, 2017.<br /> <br />  Adams Wei Yu, Hongrae Lee, and Quoc V Le. Learning to skim text. In ACL, 2017.<br /> <br />  Eunsol Choi, Daniel Hewlett, Alexandre Lacoste, Illia Polosukhin, Jakob Uszkoreit, and Jonathan Berant. Coarse-to-fine question answering for long documents. In ACL, 2017.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=User:Yktan&diff=49096 User:Yktan 2020-12-04T05:29:44Z <p>Z42qin: /* Model Architecture and Algorithm */</p> <hr /> <div><br /> == Introduction ==<br /> <br /> Much of the success in training deep neural networks (DNNs) is due to the collection of large datasets with human-annotated labels. However, human annotation is both a time-consuming and expensive task, especially for data that requires expertise such as medical data. Furthermore, certain datasets will be noisy due to the biases introduced by different annotators. Data obtained in large quantities through searching for images in search engines and data downloaded from social media sites (in a manner abiding by privacy and copyright laws) are especially noisy, since the labels are generally inferred from tags to save on human-annotation cost. <br /> <br /> There are a few existing approaches to use datasets with noisy labels. In learning with noisy labels (LNL), most methods take a loss correction approach. Other LNL methods estimate a noise transition matrix and employ it to correct the loss function. An example of a popular loss correction approach is the bootstrapping loss approach. Another approach to reduce annotation cost is semi-supervised learning (SSL), where the training data consists of labeled and unlabeled samples. The main limitation of these methods is that they do not perform well under high noise ratio and cause overfitting.<br /> <br /> This paper introduces DivideMix, which combines approaches from LNL and SSL. One unique thing about DivideMix is that it discards sample labels that are highly likely to be noisy and leverages these noisy samples as unlabeled data instead. This prevents the model from overfitting and improves generalization performance. Key contributions of this work are:<br /> 1) Co-divide, which trains two networks simultaneously, aims to improve generalization and avoid confirmation bias.<br /> 2) During the SSL phase, an improvement is made on an existing method (MixMatch) by combining it with another method (MixUp).<br /> 3) Significant improvements to state-of-the-art results on multiple conditions are experimentally shown while using DivideMix. Extensive ablation study and qualitative results are also shown to examine the effect of different components.<br /> <br /> == Motivation ==<br /> <br /> While much has been achieved in training DNNs with noisy labels and SSL methods individually, not much progress has been made in exploring their underlying connections and building on top of the two approaches simultaneously. <br /> <br /> Existing LNL methods aim to correct the loss function by:<br /> &lt;ol&gt;<br /> &lt;li&gt; Treating all samples equally and correcting loss explicitly or implicitly through relabelling of the noisy samples<br /> &lt;li&gt; Reweighting training samples or separating clean and noisy samples, which results in correction of the loss function<br /> &lt;/ol&gt;<br /> <br /> A few examples of LNL methods include:<br /> &lt;ol&gt;<br /> &lt;li&gt; Estimating the noise transition matrix, which denotes the probability of clean labels flipping to noisy labels, to correct the loss function<br /> &lt;li&gt; Leveraging the predictions from DNNs to correct labels and using them to modify the loss<br /> &lt;li&gt; Reweighting samples so that noisy labels contribute less to the loss<br /> &lt;/ol&gt;<br /> <br /> However, these methods all have downsides: it is very challenging to correctly estimate the noise transition matrix in the first method; for the second method, DNNs tend to overfit to datasets with high noise ratio; and for the third method, we need to be able to identify clean samples, which has also proven to be challenging.<br /> <br /> On the other hand, SSL methods mostly leverage unlabeled data using regularization to improve model performance. A recently proposed method, MixMatch, incorporates the two classes of regularization. These classes are consistency regularization which enforces the model to produce consistent predictions on augmented input data, and entropy minimization which encourages the model to give high-confidence predictions on unlabeled data, as well as MixUp regularization. <br /> <br /> DivideMix partially adopts LNL in that it removes the labels that are highly likely to be noisy by using co-divide to avoid the confirmation bias problem. It then utilizes the noisy samples as unlabeled data and adopts an improved version of MixMatch (an SSL technique) which accounts for the label noise during the label co-refinement and co-guessing phase. By incorporating SSL techniques into LNL and taking the best of both worlds, DivideMix aims to produce highly promising results in training DNNs by better addressing the confirmation bias problem, more accurately distinguishing and utilizing noisy samples, and performing well under high levels of noise.<br /> <br /> == Model Architecture and Algorithm ==<br /> <br /> DivideMix leverages semi-supervised learning to achieve effective modeling. The sample is first split into a labeled set and an unlabeled set. This is achieved by fitting a Gaussian Mixture Model as a per-sample loss distribution. The unlabeled set is made up of data points with discarded labels deemed noisy. <br /> <br /> Then, to avoid confirmation bias, which is typical when a model is self-training, two models are being trained simultaneously to filter error for each other. This is done by dividing the data into clean labeled set (X)and a noisy unlabeled set (U) using one model and then training the other model on the data set U. This algorithm, known as Co-divide, keeps the two networks from converging when training, which avoids the bias from occurring. Gaussian Mixture Model is better at distinguishing X and U, whereas Beta Mixture Model produces flat distribution and fails to label correctly. Being diverged also offers the two networks distinct abilities to filter different types of error, making the model more robust to noise. However, the model could still have confirmation error where both model would prone to make and confirm the same mistake. Figure 1 describes the algorithm in graphical form.<br /> <br /> [[File:ModelArchitecture.PNG | center]]<br /> <br /> &lt;div align=&quot;center&quot;&gt;Figure 1: Model Architecture of DivideMix&lt;/div&gt;<br /> <br /> For each epoch, the network divides the dataset into a labeled set consisting of clean data, and an unlabeled set consisting of noisy data, which is then used as training data for the other network, where training is done in mini-batches. For each batch of the labelled samples, co-refinement is performed by using the ground truth label &lt;math&gt; y_b &lt;/math&gt;, the predicted label &lt;math&gt; p_b &lt;/math&gt;, and the posterior is used as the weight, &lt;math&gt; w_b &lt;/math&gt;. <br /> <br /> &lt;center&gt;&lt;math&gt; \bar{y}_b = w_b y_b + (1-w_b) p_b &lt;/math&gt;&lt;/center&gt; <br /> <br /> Then, a sharpening function is implemented on this weighted sum to produce the estimate with reduced temperature, &lt;math&gt; \hat{y}_b &lt;/math&gt;. <br /> <br /> &lt;center&gt;&lt;math&gt; \hat{y}_b=Sharpen(\bar{y}_b,T)={\bar{y}^{c{\frac{1}{T}}}_b}/{\sum_{c=1}^C\bar{y}^{c{\frac{1}{T}}}_b} &lt;/math&gt;, for &lt;math&gt;c = 1, 2,..,C&lt;/math&gt;&lt;/center&gt;<br /> <br /> Using all these predicted labels, the unlabeled samples will then be assigned a &quot;co-guessed&quot; label, which should produce a more accurate prediction. Having calculated all these labels, MixMatch is applied to the combined mini-batch of labeled, &lt;math&gt; \hat{X} &lt;/math&gt; and unlabeled data, &lt;math&gt; \hat{U} &lt;/math&gt;, where, for a pair of samples and their labels, one new sample and new label is produced. More specifically, for a pair of samples &lt;math&gt; (x_1,x_2) &lt;/math&gt; and their labels &lt;math&gt; (p_1,p_2) &lt;/math&gt;, the mixed sample &lt;math&gt; (x',p') &lt;/math&gt; is:<br /> <br /> &lt;center&gt;<br /> &lt;math&gt;<br /> \begin{alignat}{2}<br /> <br /> \lambda &amp;\sim Beta(\alpha, \alpha) \\<br /> \lambda ' &amp;= max(\lambda, 1 - \lambda) \\<br /> x' &amp;= \lambda ' x_1 + (1 - \lambda ' ) x_2 \\<br /> p' &amp;= \lambda ' p_1 + (1 - \lambda ' ) p_2 \\<br /> <br /> \end{alignat}<br /> &lt;/math&gt;<br /> &lt;/center&gt; <br /> <br /> MixMatch transforms &lt;math&gt; \hat{X} &lt;/math&gt; and &lt;math&gt; \hat{U} &lt;/math&gt; into &lt;math&gt; X' &lt;/math&gt; and &lt;math&gt; U' &lt;/math&gt;. Then, the loss on &lt;math&gt; X' &lt;/math&gt;, &lt;math&gt; L_X &lt;/math&gt; (Cross-entropy loss) and the loss on &lt;math&gt; U' &lt;/math&gt;, &lt;math&gt; L_U &lt;/math&gt; (Mean Squared Error) are calculated. A regularization term, &lt;math&gt; L_{reg} &lt;/math&gt;, is introduced to regularize the model's average output across all samples in the mini-batch. Then, the total loss is calculated as:<br /> <br /> &lt;center&gt;&lt;math&gt; L = L_X + \lambda_u L_U + \lambda_r L_{reg} &lt;/math&gt;&lt;/center&gt; <br /> <br /> where &lt;math&gt; \lambda_r &lt;/math&gt; is set to 1, and &lt;math&gt; \lambda_u &lt;/math&gt; is used to control the unsupervised loss.<br /> <br /> Lastly, the stochastic gradient descent formula is updated with the calculated loss, &lt;math&gt; L &lt;/math&gt;, and the estimated parameters, &lt;math&gt; \boldsymbol{ \theta } &lt;/math&gt;.<br /> <br /> The full algorithm is shown below. [[File:dividemix.jpg|600px| | center]]<br /> &lt;div align=&quot;center&quot;&gt;Algorithm1: DivideMix. Line 4-8: co-divide; Line 17-18: label co-refinement; Line 20: co-guessing.&lt;/div&gt;<br /> <br /> The when the model is warmed up, it is trained on all data using standard cross-entropy to initially converge the model, but with a regulatory negative entropy term &lt;math&gt;\mathcal{H} = -\sum_{c}\text{p}^\text{c}_\text{model}(x;\theta)\log(\text{p}^\text{c}_\text{model}(x;\theta))&lt;/math&gt;, where &lt;math&gt;\text{p}^\text{c}_\text{model}&lt;/math&gt; is the softmax output probability for class c. This term penalizes confident predictions during the warm up to prevent overfitting to noise during the warm up, which can happen when there is asymmetric noise.<br /> <br /> == Results ==<br /> '''Applications'''<br /> <br /> The method was validated using four benchmark datasets: CIFAR-10, CIFAR100 (Krizhevsky &amp; Hinton, 2009) which contain 50K training images and 10K test images of size 32 × 32), Clothing1M (Xiao et al., 2015), and WebVision (Li et al., 2017a).<br /> Two types of label noise are used in the experiments: symmetric and asymmetric.<br /> An 18-layer PreAct Resnet (He et al., 2016) is trained using SGD with a momentum of 0.9, a weight decay of 0.0005, and a batch size of 128. The network is trained for 300 epochs. The initial learning rate was set to 0.02 and reduced by a factor of 10 after 150 epochs. Before applying the Co-divide and MixMatch strategies, the models were first independently trained over the entire dataset using cross-entropy loss during a &quot;warm-up&quot; period. Initially, training the models in this way prepares a more regular distribution of losses to improve upon in subsequent epochs. The warm-up period is 10 epochs for CIFAR-10 and 30 epochs for CIFAR-100. For all CIFAR experiments, we use the same hyperparameters M = 2, T = 0.5, and α = 4. τ is set as 0.5 except for 90% noise ratio when it is set as 0.6.<br /> <br /> <br /> '''Comparison of State-of-the-Art Methods'''<br /> <br /> The effectiveness of DivideMix was shown by comparing the test accuracy with the most recent state-of-the-art methods: <br /> Meta-Learning (Li et al., 2019) proposes a gradient-based method to find model parameters that are more noise-tolerant; <br /> Joint-Optim (Tanaka et al., 2018) and P-correction (Yi &amp; Wu, 2019) jointly optimize the sample labels and the network parameters;<br /> M-correction (Arazo et al., 2019) models sample loss with BMM and apply MixUp.<br /> The following are the results on CIFAR-10 and CIFAR-100 with different levels of symmetric label noise ranging from 20% to 90%. Both the best test accuracy across all epochs and the averaged test accuracy over the last 10 epochs were recorded in the following table:<br /> <br /> <br /> [[File:divideMixtable1.PNG | center]]<br /> <br /> From table 1, the author noticed that none of these methods can consistently outperform others across different datasets. M-correction excels at symmetric noise, whereas Meta-Learning performs better for asymmetric noise. DivideMix outperforms state-of-the-art methods by a large margin across all noise ratios. The improvement is substantial (∼10% of accuracy) for the more challenging CIFAR-100 with high noise ratios.<br /> <br /> DivideMix was compared with the state-of-the-art methods with the other two datasets: Clothing1M and WebVision. It also shows that DivideMix consistently outperforms state-of-the-art methods across all datasets with different types of label noise. For WebVision, DivideMix achieves more than 12% improvement in top-1 accuracy. <br /> <br /> <br /> '''Ablation Study'''<br /> <br /> The effect of removing different components to provide insights into what makes DivideMix successful. We analyze the results in Table 5 as follows.<br /> <br /> <br /> [[File:DivideMixtable5.PNG | center]]<br /> <br /> The authors combined self-divide with the original MixMatch as a naive baseline for using SLL in LNL.<br /> They also find that both label refinement and input augmentation are beneficial for DivideMix. ''Label refinement'' is important for high noise ratio due because samples that are noisier would be incorrectly divided into the labeled set. ''Augmentation'' upgrades model performance by creating more reliable predictions and by achieving consistent regularization. In addition, the performance drop was seen in the ''DivideMix w/o co-training'' highlights the disadvantage of self-training; the model still has dataset division, label refinement and label guessing, but they are all performed by the same model.<br /> <br /> == Conclusion ==<br /> <br /> This paper provides a new and effective algorithm for learning with noisy labels by using highly noisy data unlabelled data in a Semi-Supervised Learning framework. The DivideMix method trains two networks simultaneously and utilizes co-guessing and co-labeling effectively, therefore it is a robust approach to deal with noise in datasets. Also, the DivideMix method has been tested using various datasets with the results consistently being one of the best when compared to the state-of-the-art methods through extensive experiments.<br /> <br /> Future work of DivideMix is to create an adaptation for other applications such as Natural Language Processing, and incorporating the ideas of SSL and LNL into DivideMix architecture.<br /> <br /> == Critiques/ Insights ==<br /> <br /> 1. While combining both models makes the result better, the author did not show the relative time increase using this new combined methodology, which is very crucial considering training a large amount of data, especially for images. In addition, it seems that the author did not perform much on hyperparameters tuning for the combined model.<br /> <br /> 2. There is an interesting insight, which is when the noise ratio increases from 80% to 90%, the accuracy of DivideMix drops dramatically in both datasets.<br /> <br /> 3. There should be a further explanation of why the learning rate drops by a factor of 10 after 150 epochs.<br /> <br /> 4. It would be interesting to see the effectiveness of this method in other domains such as NLP. I am not aware of noisy training datasets available in NLP, but surely this is an important area to focus on, as much of the available data is collected from noisy sources from the web.<br /> <br /> 5. The paper implicitly assumes that a Gaussian mixture model (GMM) is sufficiently capable of identifying noise. Given the nature of a GMM, it would work well for noise that is distributed by a Gaussian distribution but for all other noise, it would probably be only asymptotic. The paper should present theoretical results on the noise that are Exponential, Rayleigh, etc. This is particularly important because the experiments were done on massive datasets, but they do not directly address the case when there are not many data points. <br /> <br /> 6. Comparing the training result on these benchmark datasets makes the algorithm quite comprehensive. This is a very insightful idea to maintain two networks to avoid bias from occurring.<br /> <br /> 7. The current benchmark accuracy for CIFAR-10 is 99.7, CIFAR-100 is 96.08 using EffNet-L2 in 2020. In 2019, CIFAR-10 is 99.37, CIFAR-100 is 93.51 using BiT-L.(based on paperswithcode.com) As there exists better methods, it would be nice to know why the authors chose these state-of-the-art methods to compare the test accuracy.<br /> <br /> 8. Another interesting observation is that DivideMix seems to maintain a similar accuracy while some methods give unstable results. That shows the reliability of the proposed algorithm.<br /> <br /> 9. It would be interesting to see if the drop in accuracy from increasing the noise ratio to 90% is a result of a low porportion or low number of clean labels. That is, would increasing the size of the training set but keeping the noise ratio at 90% result in increased accuracy?<br /> <br /> 10. For Ablation Study part, the paper also introduced a study on the Robustness of Testing Marking Methods Noise, including AUC for classification of clean/noisy samples of CIFAR-10 training data. And it shows that the method can effectively separate clean and noisy samples as training proceeds.<br /> <br /> 11. It is interesting how unlike common methods, the method in this paper discards the labels that are highly likely to be<br /> noisy. It also utilizes the noisy samples as unlabeled data to regularize training in a SSL manner. This model can better distinguish and utilize noisy samples.<br /> <br /> 12. In the result section, the author gives us a comprehensive understanding of this algorithm by introducing the applications and the comparison of it with respect to similar methods. It would be attractive if in the application part, the author could indicate how the application relative to our daily life.<br /> <br /> 13. High quality data is very important for training Machine learning systems. Preparing the data to train ML systems requires data annotations which are prone to errors and are time-consuming. It is interesting to note how paper 14 and this paper aims to approach this problem from different perspectives. Paper 14 introduces CSL algorithm that learns from confused or Noisy data to find the tasks associated with them. And this paper proposes an algorithm that shows good performance when learning from noisy data. Hence both the papers seem to tackle similar problem and implementing the approaches described in both the papers when handling noisy data can be twice helpful.<br /> <br /> 14. Noise exists in all big data, and big data is what we are dealing with in real life nowadays. Having an effective noise eliminating method such as Dividemix is important to us.<br /> <br /> 15. The DivideMix consistently outperforms state-of-the-art methods across the given datasets, but how about some other potential datasets? If it can be given that it has advantages for a certain type of potential dataset, it will be a better discussion.<br /> <br /> == References ==<br />  Eric Arazo, Diego Ortego, Paul Albert, Noel E. O’Connor, and Kevin McGuinness. Unsupervised<br /> label noise modeling and loss correction. In ICML, pp. 312–321, 2019.<br /> <br />  David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin<br /> Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurIPS, 2019.<br /> <br />  Yifan Ding, Liqiang Wang, Deliang Fan, and Boqing Gong. A semi-supervised two-stage approach<br /> to learning from noisy labels. In WACV, pp. 1215–1224, 2018.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Music_Recommender_System_Based_using_CRNN&diff=49072 Music Recommender System Based using CRNN 2020-12-04T04:00:31Z <p>Z42qin: /* Modeling Neural Networks */</p> <hr /> <div>==Introduction and Objective:==<br /> <br /> In the digital era of music streaming, companies, such as Spotify and Pandora, are faced with the following challenge: can they provide users with relevant and personalized music recommendations amidst the ever-growing abundance of music and user data.<br /> <br /> The objective of this paper is to implement a personalized music recommender system that takes user listening history as input and continually finds new music that captures individual user preferences.<br /> <br /> This paper argues that a music recommendation system should vary from the general recommendation system used in practice since it should combine music feature recognition and audio processing technologies to extract music features, and combine them with data on user preferences.<br /> <br /> The authors of this paper took a content-based music approach to build the recommendation system - specifically, comparing the similarity of features based on the audio signal.<br /> <br /> The following two-method approach for building the recommendation system was followed:<br /> #Make recommendations including genre information extracted from classification algorithms.<br /> #Make recommendations without genre information.<br /> <br /> The authors used convolutional recurrent neural networks (CRNN), which is a combination of convolutional neural networks (CNN) and recurrent neural network(RNN), as their main classification model.<br /> <br /> ==Methods and Techniques:==<br /> Collaborative filtering method is used by many music recommendation systems, which was explicitly based on ratings of a song from users to find similar users or items. However, people tend not to rate a song when listening to music. Nowadays, deep learning techniques are employed to music recommendation systems. Content based recommendation systems use item content to find similarities between items and the recommended items for users. <br /> <br /> Generally, a music recommender can be divided into three main parts: (I) users, (ii) items, and (iii) user-item matching algorithms. First, we generated users' music tastes based on their profiles. Second, item profiling includes editorial, cultural, and acoustic metadata and they were collected for listeners' satisfaction. Finally, we come to the matching algorithm that suggests recommended personalized music to listeners. <br /> <br /> To classify music, the original music’s audio signal is converted into a spectrogram image. Using the image and the Short Time Fourier Transform (STFT), we convert the data into the Mel scale which is used in the CNN and CRNN models. <br /> === Mel Scale: === <br /> The scale of pitches that are heard by listeners, which translates to equal pitch increments.<br /> <br /> [[File:Mel.png|frame|none|Mel Scale on Spectrogram]]<br /> <br /> === Short Time Fourier Transform (STFT): ===<br /> The transformation that determines the sinusoidal frequency of the audio, with a Hanning smoothing function. In the continuous case this is written as: &lt;math&gt;\mathbf{STFT}\{x(t)\}(\tau,\omega) \equiv X(\tau, \omega) = \int_{-\infty}^{\infty} x(t) w(t-\tau) e^{-i \omega t} \, d t &lt;/math&gt;<br /> <br /> where: &lt;math&gt;w(\tau)&lt;/math&gt; is the Hanning smoothing function. The STFT is applied over a specified window length at a certain time allowing the frequency to represented for that given window rather than the entire signal as a typical Fourier Transform would.<br /> <br /> === Convolutional Neural Network (CNN): ===<br /> A Convolutional Neural Network is a Neural Network that uses convolution in place of matrix multiplication for some layer calculations. By training the data, weights for inputs are updated to find the most significant data relevant to classification. These convolutional layers gather small groups of data with kernels and try to find patterns that can help find features in the overall data. The features are then used for classification. Padding is another technique used to extend the pixels on the edge of the original image to allow the kernel to more accurately capture the borderline pixels. Padding is also used if one wishes the convolved output image to have a certain size. The image on the left represents the mathematical expression of a convolution operation, while the right image demonstrates an application of a kernel on the data.<br /> <br /> [[File:Convolution.png|thumb|400px|left|Convolution Operation]]<br /> [[File:PaddingKernels.png|thumb|400px|center|Example of Padding (white 0s) and Kernels (blue square)]]<br /> <br /> === Convolutional Recurrent Neural Network (CRNN): === <br /> The CRNN is similar to the architecture of a CNN, but with the addition of a GRU, which is a Recurrent Neural Network (RNN). An RNN is used to treat sequential data, by reusing the activation function of previous nodes to update the output. A Gated Recurrent Unit (GRU) is used to store more long-term memory and will help train the early hidden layers. GRUs can be thought of as LSTMs but with a forget gate, and has fewer parameters than an LSTM. These gates are used to determine how much information from the past should be passed along onto the future. They are originally aimed to prevent the vanishing gradient problem, since deeper networks will result in smaller and smaller gradients at each layer. The GRU can choose to copy over all the information in the past, thus eliminating the risk of vanishing gradients.<br /> <br /> [[File:GRU441.png|thumb|400px|left|Gated Recurrent Unit (GRU)]]<br /> [[File:Recurrent441.png|thumb|400px|center|Diagram of General Recurrent Neural Network]]<br /> <br /> ==Data Screening:==<br /> <br /> The authors of this paper used a publicly available music dataset made up of 25,000 30-second songs from the Free Music Archives which contains 16 different genres. The data is cleaned up by removing low audio quality songs, wrongly labelled genres and those that have multiple genres. To ensure a balanced dataset, only 1000 songs each from the genres of classical, electronic, folk, hip-hop, instrumental, jazz and rock were used in the final model. <br /> <br /> [[File:Data441.png|thumb|200px|none|Data sorted by music genre]]<br /> <br /> ==Implementation:==<br /> <br /> === Modeling Neural Networks ===<br /> <br /> As noted previously, both CNNs and CRNNs were used to model the data. The advantage of CRNNs is that they are able to model time sequence patterns in addition to frequency features from the spectrogram, allowing for greater identification of important features. Furthermore, feature vectors produced before the classification stage could be used to improve accuracy. <br /> <br /> In implementing the neural networks, the Mel-spectrogram data was split up into training, validation, and test sets at a ratio of 8:1:1 respectively and labelled via one-hot encoding. This made it possible for the categorical data to be labelled correctly for binary classification. As opposed to classical stochastic gradient descent, the authors opted to use binary classier and ADAM optimization to update weights in the training phase, and parameters of &lt;math&gt;\alpha = 0.001, \beta_1 = 0.9, \beta_2 = 0.999&lt;\math&gt;. Binary cross-entropy was used as the loss function. <br /> Input spectrogram image are 96x1366. In both the CNN and CRNN models, the data was trained over 100 epochs and batch size of 50 (limited computing power) with a binary cross-entropy loss function. Notable model specific details are below:<br /> <br /> '''CNN'''<br /> * Five convolutional layers with 3x3 kernel, stride 1, padding, batch normalization, and ReLU activation<br /> * Max pooling layers <br /> * The sigmoid function was used as the output layer<br /> <br /> '''CRNN'''<br /> * Four convolutional layers with 3x3 kernel (which construct a 2D temporal pattern - two layers of RNNs with Gated Recurrent Units), stride 1, padding, batch normalization, ReLU activation, and dropout rate 0.1<br /> * Feature maps are N x1x15 (N = number of features maps, 68 feature maps in this case) is used for RNNs.<br /> * 4 Max pooling layers for four convolutional layers with kernel ((2x2)-(3x3)-(4x4)-(4x4)) and same stride<br /> * The sigmoid function was used as the output layer<br /> <br /> The CNN and CRNN architecture is also given in the charts below.<br /> <br /> [[File:CNN441.png|thumb|800px|none|Implementation of CNN Model]]<br /> [[File:CRNN441.png|thumb|800px|none|Implementation of CRNN Model]]<br /> <br /> === Music Recommendation System ===<br /> <br /> The recommendation system is computed by the cosine similarity of the extraction features from the neural network. Each genre will have a song act as a centre point for each class. The final inputs of the trained neural networks will be the feature variables. The featured variables will be used in the cosine similarity to find the best recommendations. <br /> <br /> The values are between [-1,1], where larger values are songs that have similar features. When the user inputs five songs, those songs become the new inputs in the neural networks and the features are used by the cosine similarity with other music. The largest five cosine similarities are used as recommendations.<br /> [[File:Cosine441.png|frame|100px|none|Cosine Similarity]]<br /> <br /> == Evaluation Metrics ==<br /> === Precision: ===<br /> * The proportion of True Positives with respect to the '''predicted''' positive cases (true positives and false positives)<br /> * For example, out of all the songs that the classifier '''predicted''' as Classical, how many are actually Classical?<br /> * Describes the rate at which the classifier predicts the true genre of songs among those predicted to be of that certain genre<br /> <br /> === Recall: ===<br /> * The proportion of True Positives with respect to the '''actual''' positive cases (true positives and false negatives)<br /> * For example, out of all the songs that are '''actually''' Classical, how many are correctly predicted to be Classical?<br /> * Describes the rate at which the classifier predicts the true genre of songs among the correct instances of that genre<br /> <br /> === F1-Score: ===<br /> An accuracy metric that combines the classifier’s precision and recall scores by taking the harmonic mean between the two metrics:<br /> <br /> [[File:F1441.png|frame|100px|none|F1-Score]]<br /> <br /> === Receiver operating characteristics (ROC): ===<br /> * A graphical metric that is used to assess a classification model at different classification thresholds <br /> * In the case of a classification threshold of 0.5, this means that if &lt;math&gt;P(Y = k | X = x) &gt; 0.5&lt;/math&gt; then we classify this instance as class k<br /> * Plots the true positive rate versus false positive rate as the classification threshold is varied<br /> <br /> [[File:ROCGraph.jpg|thumb|400px|none|ROC Graph. Comparison of True Positive Rate and False Positive Rate]]<br /> <br /> === Area Under the Curve (AUC) ===<br /> AUC is the area under the ROC in doing so, the ROC provides an aggregate measure across all possible classification thresholds.<br /> <br /> In the context of the paper: When scoring all songs as &lt;math&gt;Prob(Classical | X=x)&lt;/math&gt;, it is the probability that the model ranks a random Classical song at a higher probability than a random non-Classical song.<br /> <br /> [[File:AUCGraph.jpg|thumb|400px|none|Area under the ROC curve.]]<br /> <br /> == Results ==<br /> === Accuracy Metrics ===<br /> The table below is the accuracy metrics with the classification threshold of 0.5.<br /> <br /> [[File:TruePositiveChart.jpg|thumb|none|True Positive / False Positive Chart]]<br /> On average, CRNN outperforms CNN in true positive and false positive cases. In addition, it is very apparent that false positives are much more frequent for songs in the Instrumental genre, perhaps indicating that more pre-processing needs to be done for songs in this genre or that it should be excluded from analysis completely given how most music has instrumental components.<br /> <br /> <br /> [[File:F1Chart441.jpg|thumb|400px|none|F1 Chart]]<br /> On average, CRNN outperforms CNN in F1-score. <br /> <br /> <br /> [[File:AUCChart.jpg|thumb|400px|none|AUC Chart]]<br /> On average, CRNN also outperforms CNN in AUC metric.<br /> <br /> <br /> CRNN models that consider the frequency features and time sequence patterns of songs have a better classification performance through metrics such as F1 score and AUC when comparing to CNN classifier.<br /> <br /> === Evaluation of Music Recommendation System: ===<br /> <br /> * A listening experiment was performed with 30 participants to access user responses to given music recommendations.<br /> * Participants choose 5 pieces of music they enjoyed and the recommender system generated 5 new recommendations. The participants then evaluated the music recommendation by recording whether the song was liked or disliked.<br /> * The recommendation system takes two approaches to the recommendation:<br /> ** Method one uses only the value of cosine similarity.<br /> ** Method two uses the value of cosine similarity and information on music genre.<br /> *Perform test of significance of differences in average user likes between the two methods using a t-statistic:<br /> [[File:H0441.png|frame|100px|none|Hypothesis test between method 1 and method 2]]<br /> <br /> Comparing the two methods, &lt;math&gt; H_0: u_1 - u_2 = 0&lt;/math&gt;, we have &lt;math&gt; t_{stat} = -4.743 &lt; -2.037 &lt;/math&gt;, which demonstrates that the increase in average user likes with the addition of music genre information is statistically significant.<br /> <br /> == Conclusion: ==<br /> <br /> Here are two main conclusions obtained from this paper:<br /> <br /> - To increase the predictive capabilities of the music recommendation system, the music genre should be a key feature to analyze.<br /> <br /> - To extract the song genre from a song’s audio signals and get overall better performance, CRNN’s are superior to CNN’s as they consider frequency in features and time sequence patterns of audio signals. <br /> <br /> According to analyses in the paper, the authors also suggested adding other music features like tempo gram for capturing local tempo to improve the accuracy of the recommender system.<br /> <br /> == Critiques/ Insights: ==<br /> # The authors fail to give reference to the performance of current recommendation algorithms used in the industry; my critique would be for the authors to bench-mark their novel approach with other recommendation algorithms such as collaborative filtering to see if there is a lift in predictive capabilities.<br /> # The listening experiment used to evaluate the recommendation system only includes songs that are outputted by the model. Users may be biased if they believe all songs have come from a recommendation system. To remove bias, we suggest having 15 songs where 5 songs are recommended and 10 songs are set. With this in the user’s mind, it may remove some bias in response and give more accurate predictive capabilities. <br /> # They could go into more details about how CRNN makes it perform better than CNN, in terms of attributes of each network.<br /> # The methodology introduced in this paper is probably also suitable for movie recommendations. As music is presented as spectrograms (images) in a time sequence, and it is very similar to a movie. <br /> # The way of evaluation is a very interesting approach. Since it's usually not easy to evaluate the testing result when it's subjective. By listing all these evaluations' performance, the result would be more comprehensive. A practice that might reduce bias is by coming back to the participants after a couple of days and asking whether they liked the music that was recommended. Often times music &quot;grows&quot; on people and their opinion of a new song may change after some time has passed. <br /> # The paper lacks the comparison between the proposed algorithm and the music recommendation algorithms being used now. It will be clearer to show the superiority of this algorithm.<br /> # The GAN neural network has been proposed to enhance the performance of the neural network, so an improved result may appear after considering using GAN.<br /> # The limitation of CNN and CRNN could be that they are only able to process the spectrograms with single labels rather than multiple labels. This is far from enough for the music recommender systems in today's music industry since the edges between various genres are blurred.<br /> # Is it possible for CNN and CRNN to identify different songs? The model would be harder to train, based on my experience, the efficiency of CNN in R is not very high, which can be improved for future work.<br /> # according to the author, the recommender system is done by calculating the cosine similarity of extraction features from one music to another music. Is possible to represent it by Euclidean distance or p-norm distances?<br /> # In real-life application, most of the music software will have the ability to recommend music to the listener and ask do they like the music that was recommended. It would be a nice application by involving some new information from the listener.<br /> # This paper is very similar to another [https://link.springer.com/chapter/10.1007/978-3-319-46131-1_29 paper], written by Bruce Fewerda and Markus Schedl. Both papers are suggesting methods of building music recommendation systems. However, this paper recommends music based on genre, but the paper written by Fewerda and Schedl suggests a personality-based user modeling for music recommender systems.<br /> # Actual music listeners do not listen to one genre of music, and in fact listening to the same track or the same genre would be somewhat unusual. Could this method be used to make recommendations not on genre, but based on other categories? (Such as the theme of the lyrics, the pitch of the singer, or the date published). Would this model be able to differentiate between tracks of varying &quot;lyric vocabulation difficulty&quot;? Or would NLP algorithms be needed to consider lyrics?<br /> # This model can be applied to many other fields such as recommending the news in the news app, recommending things to buy in the amazon, recommending videos to watch in YOUTUBE and so on based on the user information.<br /> # Looks like for the most genres, CRNN outperforms CNN, but CNN did do better on a few genres (like Jazz), so it might be better to mix them together or might use CNN for some genres and CRNN for the rest.<br /> # Cosine similarity is used to find songs with similar patterns as the input ones from users. That is, feature variables are extracted from the trained neural network model before the classification layer, and used as the basis to find similar songs. One potential problem of this approach is that if the neural network classifies an input song incorrectly, the extracted feature vector will not be a good representation of the input song. Thus, a song that is in fact really similar to the input song may have a small cosine similarity value, i.e. not be recommended. In conclusion, if the first classification is wrong, future inferences based on that is going to make it deviate further from the true answer. A possible future improvement will be how to offset this inference error.<br /> # In the tables when comparing performance and accuracies of the CNN and CRNN models on different genres of music, the researchers claimed that CRNN had superior performance to CNN models. This seemed intuitive, especially in the cases when the differences in accuracies were large. However, maybe the researchers should consider including some hypothesis testing statistics in such tables, which would support such claims in a more rigorous manner.<br /> # A music recommender system that doesn't use the song's meta data such as artist and genre and rather tries to classify genre itself seems unproductive. I also believe that the specific artist matters much more than the genre since within a genre you have many different styles. It just seems like the authors hamstring their recommender system by excluding other relevant data.<br /> # The genres that are posed in the paper are very broad and may not be specific enough to distinguish a listeners actual tastes (ie, I like rock and roll, but not punk rock, which could both be in the &quot;rock&quot; category). It would be interesting to run similar experiments with more concrete and specific genres to study the possibility of improving accuracy in the model.<br /> # This summary is well organized with detailed explanation to the music recommendation algorithm. However, since the data used in this paper is cleaned to buffer the efficiency of the recommendation, there should be a section evaluating the impact of noise on the performance this algorithm and how to minimize the impact.<br /> # This method will be better if the user choose some certain music genres that they like while doing the sign-up process. This is similar to recommending articles on twitter.<br /> # In the</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Efficient_kNN_Classification_with_Different_Numbers_of_Nearest_Neighbors&diff=49068 Efficient kNN Classification with Different Numbers of Nearest Neighbors 2020-12-04T03:43:04Z <p>Z42qin: /* Reconstruction */</p> <hr /> <div>== Presented by == <br /> Cooper Brooke, Daniel Fagan, Maya Perelman<br /> <br /> == Introduction == <br /> Traditional model-based classification approaches first use training observations to fit a model before predicting test samples. In contrast, the model-free k-nearest neighbors (KNNs) method classifies observations with a majority rule approach, labeling each piece of test data based on its k closest training observations (neighbors). This method has become very popular due to its strong performance and simple implementation. <br /> <br /> There are two main approaches to conduct kNN classification. The first is to use a fixed k value to classify all test samples, and the second is to use a different k value for each test sample. The former, while easy to implement, has proven to be impractical in machine learning applications. Therefore, interest lies in developing an efficient way to apply a different optimal k value for each test sample. The authors of this paper presented the kTree and k*Tree methods to solve this research question.<br /> <br /> == Previous Work == <br /> <br /> Previous work on finding an optimal fixed k value for all test samples is well-studied. Zhang et al.  incorporated a certainty factor measure to solve for an optimal fixed k. This resulted in the conclusion that k should be &lt;math&gt;\sqrt{n}&lt;/math&gt; (where n is the number of training samples) when n &gt; 100. The method Song et al. explored involved selecting a subset of the most informative samples from neighbourhoods. Vincent and Bengio  took the unique approach of designing a k-local hyperplane distance to solve for k. Premachandran and Kakarala  had the solution of selecting a robust k using the consensus of multiple rounds of kNNs. These fixed k methods are valuable however are impractical for data mining and machine learning applications. <br /> <br /> Finding an efficient approach to assigning varied k values has also been previously studied. Tuning approaches such as the ones taken by Zhu et al. as well as Sahugara et al. have been popular. Zhu et al.  determined that optimal k values should be chosen using cross validation while Sahugara et al.  proposed using Monte Carlo validation to select varied k parameters. Other learning approaches such as those taken by Zheng et al. and Góra and Wojna also show promise. Zheng et al.  applied a reconstruction framework to learn suitable k values. Góra and Wojna  proposed using rule induction and instance-based learning to learn optimal k-values for each test sample. While all these methods are valid, their processes of either learning varied k values or scanning all training samples are time-consuming.<br /> <br /> == Motivation == <br /> <br /> Due to the previously mentioned drawbacks of fixed-k and current varied-k kNN classification, the paper’s authors sought to design a new approach to solve for different k values. The kTree and k*Tree approach seeks to calculate optimal values of k while avoiding computationally costly steps such as cross-validation.<br /> <br /> A secondary motivation of this research was to ensure that the kTree method would perform better than kNN using fixed values of k given that running costs would be similar in this instance.<br /> <br /> == Approach == <br /> <br /> <br /> === kTree Classification ===<br /> <br /> The proposed kTree method is illustrated by the following flow chart:<br /> <br /> [[File:Approach_Figure_1.png | center | 800x800px]]<br /> <br /> ==== Reconstruction ====<br /> <br /> The first step is to use the training samples to reconstruct themselves. The goal of this is to find the matrix of correlations between the training samples themselves, &lt;math&gt;\textbf{W}&lt;/math&gt;, such that the distance between an individual training sample and the corresponding correlation vector multiplied by the entire training set is minimized. This least square loss function where &lt;math&gt;\mathbf{X}\in \mathbb{R}^{d\times n} = [x_1,...,x_n]&lt;/math&gt; represents the training set can be written as:<br /> <br /> \begin{aligned}<br /> \mathop{min}_{\textbf{W}} \sum_{i=1}^n ||Xw_i - x_i||^2<br /> \end{aligned}<br /> <br /> In addition, an &lt;math&gt;l_1&lt;/math&gt; regularization term multiplied by a tuning parameter, &lt;math&gt;\rho_1&lt;/math&gt;, is added to ensure that sparse results are generated as the objective is to minimize the number of training samples that will eventually be depended on by the test samples. <br /> <br /> \begin{aligned}<br /> \mathop{min}_{\textbf{W}} \sum_{i=1}^n ||Xw_i - x_i||^2 + \rho||\textbf{W}||^2_2<br /> \end{aligned}<br /> <br /> This is called ridge regression and it has a close solution where $$W = (X^TX+\rho I)^{-1}X^TX$$<br /> <br /> However, this objective function does not provide a sparse result, there we further employe a sparse objective function: <br /> <br /> $$W = (X^TX+\rho I)^{-1}X^TX, W &gt;= 0$$<br /> <br /> <br /> The least square loss function is then further modified to account for samples that have similar values for certain features yielding similar results. It is penalized with the function: <br /> <br /> $$\frac{1}{2} \sum^{d}_{i,j} ||x^iW-x^jW||^2_2$$<br /> <br /> with sij denotes the relation between feature vectors. It uses a radial basis function kernel to calculate Sij. After some transformations, this second regularization term that has tuning parameter &lt;math&gt;\rho_2&lt;/math&gt; is:<br /> <br /> \begin{aligned}<br /> R(W) = Tr(\textbf{W}^T \textbf{X}^T \textbf{LXW})<br /> \end{aligned}<br /> <br /> where &lt;math&gt;\mathbf{L}&lt;/math&gt; is a Laplacian matrix that indicates the relationship between features. The Laplacian matrix, also called the graph Laplacian, is a matrix representation of a graph. <br /> <br /> This gives a final objective function of:<br /> <br /> \begin{aligned}<br /> \mathop{min}_{\textbf{W}} \sum_{i=1}^n ||Xw_i - x_i||^2 + \rho_1||\textbf{W}|| + \rho_2R(\textbf{W})<br /> \end{aligned}<br /> <br /> Since this is a convex function, an iterative method can be used to optimize it to find the optimal solution &lt;math&gt;\mathbf{W^*}&lt;/math&gt;.<br /> <br /> ==== Calculate ''k'' for training set ====<br /> <br /> Each element &lt;math&gt;w_{ij}&lt;/math&gt; in &lt;math&gt;\textbf{W*}&lt;/math&gt; represents the correlation between the ith and jth training sample so if a value is 0, it can be concluded that the jth training sample has no effect on the ith training sample which means that it should not be used in the prediction of the ith training sample. Consequently, all non-zero values in the &lt;math&gt;w_{.j}&lt;/math&gt; vector would be useful in predicting the ith training sample which gives the result that the number of these non-zero elements for each sample is equal to the optimal ''k'' value for each sample.<br /> <br /> For example, if there was a 4x4 training set where &lt;math&gt;\textbf{W*}&lt;/math&gt; had the form:<br /> <br /> [[File:Approach_Figure_2.png | center | 300x300px]]<br /> <br /> The optimal ''k'' value for training sample 1 would be 2 since the correlation between training sample 1 and both training samples 2 and 4 are non-zero.<br /> <br /> ==== Train a Decision Tree using ''k'' as the label ====<br /> <br /> In a normal decision tree, the target data is the labels themselves. In contrast, in the kTree method, the target data is the optimal ''k'' value for each sample that was solved for in the previous step. So this decision tree has the following form:<br /> <br /> [[File:Approach_Figure_3.png | center | 300x300px]]<br /> <br /> ==== Making Predictions for Test Data ====<br /> <br /> The optimal ''k'' values for each testing sample are easily obtainable using the kTree solved for in the previous step. The only remaining step is to predict the labels of the testing samples by finding the majority class of the optimal ''k'' nearest neighbors across '''all''' of the training data.<br /> <br /> === k*Tree Classification ===<br /> <br /> The proposed k*Tree method is illustrated by the following flow chart:<br /> <br /> [[File:Approach_Figure_4.png | center | 1000x1000px]]<br /> <br /> Clearly, this is a very similar approach to the kTree as the k*Tree method attempts to sacrifice very little in predictive power in return for a substantial decrease in complexity when actually implementing the traditional kNN on the testing data once the optimal ''k'' values have been found.<br /> <br /> While all steps previous are the exact same, the difference comes from additional data stored in the leaf nodes. k*Tree method not only stores the optimal ''k'' value but also the following information:<br /> <br /> * The training samples that have the same optimal ''k''<br /> * The ''k'' nearest neighbours of the previously identified training samples<br /> * The nearest neighbor of each of the previously identified ''k'' nearest neighbours<br /> <br /> The data stored in each node is summarized in the following figure:<br /> <br /> [[File:Approach_Figure_5.png | center | 800x800px]]<br /> <br /> When testing, the constructed k*Tree is searched for its optimal k values well as its nearest neighbours in the leaf node. It then selects a number of its nearest neighbours from the subset of training samples and assigns the test sample with the majority label of these nearest neighbours.<br /> <br /> In the kTree method, predictions were made based on all of the training data, whereas in the k*Tree method, predicting the test labels will only be done using the samples stored in the applicable node of the tree.<br /> <br /> == Experiments == <br /> <br /> In order to assess the performance of the proposed method against existing methods, a number of experiments were performed to measure classification accuracy and run time. The experiments were run on twenty public datasets provided by the UCI Repository of Machine Learning Data, and contained a mix of data types varying in size, in dimensionality, in the number of classes, and in imbalanced nature of the data. Ten-fold cross-validation was used to measure classification accuracy, and the following methods were compared against:<br /> <br /> # k-Nearest Neighbor: The classical kNN approach with k set to k=1,5,10,20 and square root of the sample size ; the best result was reported.<br /> # kNN-Based Applicability Domain Approach (AD-kNN) <br /> # kNN Method Based on Sparse Learning (S-kNN) <br /> # kNN Based on Graph Sparse Reconstruction (GS-kNN) <br /> # Filtered Attribute Subspace-based Bagging with Injected Randomness (FASBIR) , <br /> # Landmark-based Spectral Clustering kNN (LC-kNN) <br /> <br /> The experimental results were then assessed based on classification tasks that focused on different sample sizes, and tasks that focused on different numbers of features. <br /> <br /> <br /> '''A. Experimental Results on Different Sample Sizes'''<br /> <br /> The running cost and (cross-validation) classification accuracy based on experiments on ten UCI datasets can be seen in Table I below.<br /> <br /> [[File:Table_I_kNN.png | center | 1000x1000px]]<br /> <br /> The following key results are noted:<br /> * Regarding classification accuracy, the proposed methods (kTree and k*Tree) outperformed kNN, AD-KNN, FASBIR, and LC-kNN on all datasets by 1.5%-4.5%, but had no notable improvements compared to GS-kNN and S-kNN.<br /> * Classification methods which involved learning optimal k-values (for example the proposed kTree and k*Tree methods, or S-kNN, GS-kNN, AD-kNN) outperformed the methods with predefined k-values, such as traditional kNN.<br /> * The proposed k*Tree method had the lowest running cost of all methods. However, the k*Tree method was still outperformed in terms of classification accuracy by GS-kNN and S-kNN, but ran on average 15 000 times faster than either method. In addition, the kTree had the highest accuracy and it's running cost was lower than any other methods except the k*Tree method.<br /> <br /> <br /> '''B. Experimental Results on Different Feature Numbers'''<br /> <br /> The goal of this section was to evaluate the robustness of all methods under differing numbers of features; results can be seen in Table II below. The Fisher score, an algorithm that solves maximum likelihood equations numerically , was used to rank and select the most information features in the datasets. <br /> <br /> [[File:Table_II_kNN.png | center | 1000x1000px]]<br /> <br /> From Table II, the proposed kTree and k*Tree approaches outperformed kNN, AD-kNN, FASBIR and LC-KNN when tested for varying feature numbers. The S-kNN and GS-kNN approaches remained the best in terms of classification accuracy, but were greatly outperformed in terms of running cost by k*Tree. The cause for this is that k*Tree only scans a subsample of the training samples for kNN classification, while S-kNN and GS-kNN scan all training samples.<br /> <br /> == Conclusion == <br /> <br /> This paper introduced two novel approaches for kNN classification algorithms that can determine optimal k-values for each test sample. The proposed kTree and k*Tree methods can classify the test samples efficiently and effectively, by designing a training step that reduces the run time of the test stage and thus enhances the performance. Based on the experimental results for varying sample sizes and differing feature numbers, it was observed that the proposed methods outperformed existing ones in terms of running cost while still achieving similar or better classification accuracies. Future areas of investigation could focus on the improvement of kTree and k*Tree for data with large numbers of features.<br /> <br /> == Critiques == <br /> <br /> *The paper only assessed classification accuracy through cross-validation accuracy. However, it would be interesting to investigate how the proposed methods perform using different metrics, such as AUC, precision-recall curves, or in terms of holdout test data set accuracy. <br /> * The authors addressed that some of the UCI datasets contained imbalanced data (such as the Climate and German data sets) while others did not. However, the nature of the class imbalance was not extreme, and the effect of imbalanced data on algorithm performance was not discussed or assessed. Moreover, it would have been interesting to see how the proposed algorithms performed on highly imbalanced datasets in conjunction with common techniques to address imbalance (e.g. oversampling, undersampling, etc.). <br /> *While the authors contrast their kTree and k*Tree approach with different kNN methods, the paper could contrast their results with more of the approaches discussed in the Related Work section of their paper. For example, it would be interesting to see how the kTree and k*Tree results compared to Góra and Wojna varied optimal k method.<br /> <br /> * The paper conducted an experiment on kNN, AD-kNN, S-kNN, GS-kNN,FASBIR and LC-kNN with different sample sizes and feature numbers. It would be interesting to discuss why the running cost of FASBIR is between that of kTree and k*Tree in figure 21.<br /> <br /> * A different [https://iopscience.iop.org/article/10.1088/1757-899X/725/1/012133/pdf paper] also discusses optimizing the K value for the kNN algorithm in clustering. However, this paper suggests using the expectation-maximization algorithm as a means of finding the optimal k value.<br /> <br /> * It would be really helpful if kTrees method can be explained at the very beginning. The transition from KNN to kTrees is not very smooth.<br /> <br /> * It would be nice to have a comparison of the running costs of different methods to see how much faster kTree and k*Tree performed<br /> <br /> * It would be better to show the key result only on a summary rather than stacking up all results without screening.<br /> <br /> * In the results section, it was mentioned that in the experiment on data sets with different numbers of features, the kTree and k*Tree model did not achieve GS-kNN or S-kNN's accuracies, but was faster in terms of running cost. It might be helpful here if the authors add some more supporting arguments about the benefit of this tradeoff, which appears to be a minor decrease in accuracy for a large improvement in speed. This could further showcase the advantages of the kTree and k*Tree models. More quantitative analysis or real-life scenario examples could be some choices here.<br /> <br /> * An interesting thing to notice while solving for the optimal matrix &lt;math&gt;W^*&lt;/math&gt; that minimizes the loss function is that &lt;math&gt;W^*&lt;/math&gt; is not necessarily a symmetric matrix. That is, the correlation between the &lt;math&gt;i^{th}&lt;/math&gt; entry and the &lt;math&gt;j^{th}&lt;/math&gt; entry is different from that between the &lt;math&gt;j^{th}&lt;/math&gt; entry and the &lt;math&gt;i^{th}&lt;/math&gt; entry, which makes the resulting W* not really semantically meaningful. Therefore, it would be interesting if we may set a threshold on the allowing difference between the &lt;math&gt;ij^{th}&lt;/math&gt; entry and the &lt;math&gt;ji^{th}&lt;/math&gt; entry in &lt;math&gt;W^*&lt;/math&gt; and see if this new configuration will give better or worse results compared to current ones, which will provide better insights of the algorithm.<br /> <br /> * It would be interesting to see how the proposed model works with highly non-linear datasets. In the event it does not work well, it would pose the question: would replacing the k*Tree with a SVM or a neural network improve the accuracy? There could be experiments to show if this variant would prove superior over the original models.<br /> <br /> * The key results are a little misleading - for example they claim &quot;the kTree had the highest accuracy and it's running cost was lower than any other methods except the k*Tree method&quot; is false. The kTree method had slightly lower accuracy than both GS-kNN and S-kNN and kTree was also slower than LC-kNN<br /> <br /> == References == <br /> <br />  C. Zhang, Y. Qin, X. Zhu, and J. Zhang, “Clustering-based missing value imputation for data preprocessing,” in Proc. IEEE Int. Conf., Aug. 2006, pp. 1081–1086.<br /> <br />  Y. Song, J. Huang, D. Zhou, H. Zha, and C. L. Giles, “IKNN: Informative K-nearest neighbor pattern classification,” in Knowledge Discovery in Databases. Berlin, Germany: Springer, 2007, pp. 248–264.<br /> <br />  P. Vincent and Y. Bengio, “K-local hyperplane and convex distance nearest neighbor algorithms,” in Proc. NIPS, 2001, pp. 985–992.<br /> <br />  V. Premachandran and R. Kakarala, “Consensus of k-NNs for robust neighborhood selection on graph-based manifolds,” in Proc. CVPR, Jun. 2013, pp. 1594–1601.<br /> <br />  X. Zhu, S. Zhang, Z. Jin, Z. Zhang, and Z. Xu, “Missing value estimation for mixed-attribute data sets,” IEEE Trans. Knowl. Data Eng., vol. 23, no. 1, pp. 110–121, Jan. 2011.<br /> <br />  F. Sahigara, D. Ballabio, R. Todeschini, and V. Consonni, “Assessing the validity of QSARS for ready biodegradability of chemicals: An applicability domain perspective,” Current Comput.-Aided Drug Design, vol. 10, no. 2, pp. 137–147, 2013.<br /> <br />  S. Zhang, M. Zong, K. Sun, Y. Liu, and D. Cheng, “Efficient kNN algorithm based on graph sparse reconstruction,” in Proc. ADMA, 2014, pp. 356–369.<br /> <br />  X. Zhu, L. Zhang, and Z. Huang, “A sparse embedding and least variance encoding approach to hashing,” IEEE Trans. Image Process., vol. 23, no. 9, pp. 3737–3750, Sep. 2014.<br /> <br />  U. Lall and A. Sharma, “A nearest neighbor bootstrap for resampling hydrologic time series,” Water Resour. Res., vol. 32, no. 3, pp. 679–693, 1996.<br /> <br />  D. Cheng, S. Zhang, Z. Deng, Y. Zhu, and M. Zong, “KNN algorithm with data-driven k value,” in Proc. ADMA, 2014, pp. 499–512.<br /> <br />  F. Sahigara, D. Ballabio, R. Todeschini, and V. Consonni, “Assessing the validity of QSARS for ready biodegradability of chemicals: An applicability domain perspective,” Current Comput.-Aided Drug Design, vol. 10, no. 2, pp. 137–147, 2013. <br /> <br />  Z. H. Zhou and Y. Yu, “Ensembling local learners throughmultimodal perturbation,” IEEE Trans. Syst. Man, B, vol. 35, no. 4, pp. 725–735, Apr. 2005.<br /> <br />  Z. H. Zhou, Ensemble Methods: Foundations and Algorithms. London, U.K.: Chapman &amp; Hall, 2012.<br /> <br />  Z. Deng, X. Zhu, D. Cheng, M. Zong, and S. Zhang, “Efficient kNN classification algorithm for big data,” Neurocomputing, vol. 195, pp. 143–148, Jun. 2016.<br /> <br />  K. Tsuda, M. Kawanabe, and K.-R. Müller, “Clustering with the fisher score,” in Proc. NIPS, 2002, pp. 729–736.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Efficient_kNN_Classification_with_Different_Numbers_of_Nearest_Neighbors&diff=49067 Efficient kNN Classification with Different Numbers of Nearest Neighbors 2020-12-04T03:42:39Z <p>Z42qin: /* kTree Classification */</p> <hr /> <div>== Presented by == <br /> Cooper Brooke, Daniel Fagan, Maya Perelman<br /> <br /> == Introduction == <br /> Traditional model-based classification approaches first use training observations to fit a model before predicting test samples. In contrast, the model-free k-nearest neighbors (KNNs) method classifies observations with a majority rule approach, labeling each piece of test data based on its k closest training observations (neighbors). This method has become very popular due to its strong performance and simple implementation. <br /> <br /> There are two main approaches to conduct kNN classification. The first is to use a fixed k value to classify all test samples, and the second is to use a different k value for each test sample. The former, while easy to implement, has proven to be impractical in machine learning applications. Therefore, interest lies in developing an efficient way to apply a different optimal k value for each test sample. The authors of this paper presented the kTree and k*Tree methods to solve this research question.<br /> <br /> == Previous Work == <br /> <br /> Previous work on finding an optimal fixed k value for all test samples is well-studied. Zhang et al.  incorporated a certainty factor measure to solve for an optimal fixed k. This resulted in the conclusion that k should be &lt;math&gt;\sqrt{n}&lt;/math&gt; (where n is the number of training samples) when n &gt; 100. The method Song et al. explored involved selecting a subset of the most informative samples from neighbourhoods. Vincent and Bengio  took the unique approach of designing a k-local hyperplane distance to solve for k. Premachandran and Kakarala  had the solution of selecting a robust k using the consensus of multiple rounds of kNNs. These fixed k methods are valuable however are impractical for data mining and machine learning applications. <br /> <br /> Finding an efficient approach to assigning varied k values has also been previously studied. Tuning approaches such as the ones taken by Zhu et al. as well as Sahugara et al. have been popular. Zhu et al.  determined that optimal k values should be chosen using cross validation while Sahugara et al.  proposed using Monte Carlo validation to select varied k parameters. Other learning approaches such as those taken by Zheng et al. and Góra and Wojna also show promise. Zheng et al.  applied a reconstruction framework to learn suitable k values. Góra and Wojna  proposed using rule induction and instance-based learning to learn optimal k-values for each test sample. While all these methods are valid, their processes of either learning varied k values or scanning all training samples are time-consuming.<br /> <br /> == Motivation == <br /> <br /> Due to the previously mentioned drawbacks of fixed-k and current varied-k kNN classification, the paper’s authors sought to design a new approach to solve for different k values. The kTree and k*Tree approach seeks to calculate optimal values of k while avoiding computationally costly steps such as cross-validation.<br /> <br /> A secondary motivation of this research was to ensure that the kTree method would perform better than kNN using fixed values of k given that running costs would be similar in this instance.<br /> <br /> == Approach == <br /> <br /> <br /> === kTree Classification ===<br /> <br /> The proposed kTree method is illustrated by the following flow chart:<br /> <br /> [[File:Approach_Figure_1.png | center | 800x800px]]<br /> <br /> ==== Reconstruction ====<br /> <br /> The first step is to use the training samples to reconstruct themselves. The goal of this is to find the matrix of correlations between the training samples themselves, &lt;math&gt;\textbf{W}&lt;/math&gt;, such that the distance between an individual training sample and the corresponding correlation vector multiplied by the entire training set is minimized. This least square loss function where &lt;math&gt;\mathbf{X}\in \mathbb{R}^{d\times n} = [x_1,...,x_n]&lt;/math&gt; represents the training set can be written as:<br /> <br /> \begin{aligned}<br /> \mathop{min}_{\textbf{W}} \sum_{i=1}^n ||Xw_i - x_i||^2<br /> \end{aligned}<br /> <br /> In addition, an &lt;math&gt;l_1&lt;/math&gt; regularization term multiplied by a tuning parameter, &lt;math&gt;\rho_1&lt;/math&gt;, is added to ensure that sparse results are generated as the objective is to minimize the number of training samples that will eventually be depended on by the test samples. <br /> <br /> \begin{aligned}<br /> \mathop{min}_{\textbf{W}} \sum_{i=1}^n ||Xw_i - x_i||^2 + \rho||\textbf{W}||^2_2<br /> \end{aligned}<br /> <br /> This is called ridge regression and it has a close solution where $$W = (X^TX+\rho I)^{-1}X^TX$$<br /> <br /> However, this objective function does not provide a sparse result, there we further employe a sparse objective function: <br /> <br /> $$W = (X^TX+\rho I)^{-1}X^TX, W &gt;= 0$$<br /> <br /> <br /> The least square loss function is then further modified to account for samples that have similar values for certain features yielding similar results. It is penalized with the function: <br /> <br /> $$\frac{1}{2} \sum{d}{i,j} ||x^iW-x^jW||^2_2$$<br /> <br /> with sij denotes the relation between feature vectors. It uses a radial basis function kernel to calculate Sij. After some transformations, this second regularization term that has tuning parameter &lt;math&gt;\rho_2&lt;/math&gt; is:<br /> <br /> \begin{aligned}<br /> R(W) = Tr(\textbf{W}^T \textbf{X}^T \textbf{LXW})<br /> \end{aligned}<br /> <br /> where &lt;math&gt;\mathbf{L}&lt;/math&gt; is a Laplacian matrix that indicates the relationship between features. The Laplacian matrix, also called the graph Laplacian, is a matrix representation of a graph. <br /> <br /> This gives a final objective function of:<br /> <br /> \begin{aligned}<br /> \mathop{min}_{\textbf{W}} \sum_{i=1}^n ||Xw_i - x_i||^2 + \rho_1||\textbf{W}|| + \rho_2R(\textbf{W})<br /> \end{aligned}<br /> <br /> Since this is a convex function, an iterative method can be used to optimize it to find the optimal solution &lt;math&gt;\mathbf{W^*}&lt;/math&gt;.<br /> <br /> ==== Calculate ''k'' for training set ====<br /> <br /> Each element &lt;math&gt;w_{ij}&lt;/math&gt; in &lt;math&gt;\textbf{W*}&lt;/math&gt; represents the correlation between the ith and jth training sample so if a value is 0, it can be concluded that the jth training sample has no effect on the ith training sample which means that it should not be used in the prediction of the ith training sample. Consequently, all non-zero values in the &lt;math&gt;w_{.j}&lt;/math&gt; vector would be useful in predicting the ith training sample which gives the result that the number of these non-zero elements for each sample is equal to the optimal ''k'' value for each sample.<br /> <br /> For example, if there was a 4x4 training set where &lt;math&gt;\textbf{W*}&lt;/math&gt; had the form:<br /> <br /> [[File:Approach_Figure_2.png | center | 300x300px]]<br /> <br /> The optimal ''k'' value for training sample 1 would be 2 since the correlation between training sample 1 and both training samples 2 and 4 are non-zero.<br /> <br /> ==== Train a Decision Tree using ''k'' as the label ====<br /> <br /> In a normal decision tree, the target data is the labels themselves. In contrast, in the kTree method, the target data is the optimal ''k'' value for each sample that was solved for in the previous step. So this decision tree has the following form:<br /> <br /> [[File:Approach_Figure_3.png | center | 300x300px]]<br /> <br /> ==== Making Predictions for Test Data ====<br /> <br /> The optimal ''k'' values for each testing sample are easily obtainable using the kTree solved for in the previous step. The only remaining step is to predict the labels of the testing samples by finding the majority class of the optimal ''k'' nearest neighbors across '''all''' of the training data.<br /> <br /> === k*Tree Classification ===<br /> <br /> The proposed k*Tree method is illustrated by the following flow chart:<br /> <br /> [[File:Approach_Figure_4.png | center | 1000x1000px]]<br /> <br /> Clearly, this is a very similar approach to the kTree as the k*Tree method attempts to sacrifice very little in predictive power in return for a substantial decrease in complexity when actually implementing the traditional kNN on the testing data once the optimal ''k'' values have been found.<br /> <br /> While all steps previous are the exact same, the difference comes from additional data stored in the leaf nodes. k*Tree method not only stores the optimal ''k'' value but also the following information:<br /> <br /> * The training samples that have the same optimal ''k''<br /> * The ''k'' nearest neighbours of the previously identified training samples<br /> * The nearest neighbor of each of the previously identified ''k'' nearest neighbours<br /> <br /> The data stored in each node is summarized in the following figure:<br /> <br /> [[File:Approach_Figure_5.png | center | 800x800px]]<br /> <br /> When testing, the constructed k*Tree is searched for its optimal k values well as its nearest neighbours in the leaf node. It then selects a number of its nearest neighbours from the subset of training samples and assigns the test sample with the majority label of these nearest neighbours.<br /> <br /> In the kTree method, predictions were made based on all of the training data, whereas in the k*Tree method, predicting the test labels will only be done using the samples stored in the applicable node of the tree.<br /> <br /> == Experiments == <br /> <br /> In order to assess the performance of the proposed method against existing methods, a number of experiments were performed to measure classification accuracy and run time. The experiments were run on twenty public datasets provided by the UCI Repository of Machine Learning Data, and contained a mix of data types varying in size, in dimensionality, in the number of classes, and in imbalanced nature of the data. Ten-fold cross-validation was used to measure classification accuracy, and the following methods were compared against:<br /> <br /> # k-Nearest Neighbor: The classical kNN approach with k set to k=1,5,10,20 and square root of the sample size ; the best result was reported.<br /> # kNN-Based Applicability Domain Approach (AD-kNN) <br /> # kNN Method Based on Sparse Learning (S-kNN) <br /> # kNN Based on Graph Sparse Reconstruction (GS-kNN) <br /> # Filtered Attribute Subspace-based Bagging with Injected Randomness (FASBIR) , <br /> # Landmark-based Spectral Clustering kNN (LC-kNN) <br /> <br /> The experimental results were then assessed based on classification tasks that focused on different sample sizes, and tasks that focused on different numbers of features. <br /> <br /> <br /> '''A. Experimental Results on Different Sample Sizes'''<br /> <br /> The running cost and (cross-validation) classification accuracy based on experiments on ten UCI datasets can be seen in Table I below.<br /> <br /> [[File:Table_I_kNN.png | center | 1000x1000px]]<br /> <br /> The following key results are noted:<br /> * Regarding classification accuracy, the proposed methods (kTree and k*Tree) outperformed kNN, AD-KNN, FASBIR, and LC-kNN on all datasets by 1.5%-4.5%, but had no notable improvements compared to GS-kNN and S-kNN.<br /> * Classification methods which involved learning optimal k-values (for example the proposed kTree and k*Tree methods, or S-kNN, GS-kNN, AD-kNN) outperformed the methods with predefined k-values, such as traditional kNN.<br /> * The proposed k*Tree method had the lowest running cost of all methods. However, the k*Tree method was still outperformed in terms of classification accuracy by GS-kNN and S-kNN, but ran on average 15 000 times faster than either method. In addition, the kTree had the highest accuracy and it's running cost was lower than any other methods except the k*Tree method.<br /> <br /> <br /> '''B. Experimental Results on Different Feature Numbers'''<br /> <br /> The goal of this section was to evaluate the robustness of all methods under differing numbers of features; results can be seen in Table II below. The Fisher score, an algorithm that solves maximum likelihood equations numerically , was used to rank and select the most information features in the datasets. <br /> <br /> [[File:Table_II_kNN.png | center | 1000x1000px]]<br /> <br /> From Table II, the proposed kTree and k*Tree approaches outperformed kNN, AD-kNN, FASBIR and LC-KNN when tested for varying feature numbers. The S-kNN and GS-kNN approaches remained the best in terms of classification accuracy, but were greatly outperformed in terms of running cost by k*Tree. The cause for this is that k*Tree only scans a subsample of the training samples for kNN classification, while S-kNN and GS-kNN scan all training samples.<br /> <br /> == Conclusion == <br /> <br /> This paper introduced two novel approaches for kNN classification algorithms that can determine optimal k-values for each test sample. The proposed kTree and k*Tree methods can classify the test samples efficiently and effectively, by designing a training step that reduces the run time of the test stage and thus enhances the performance. Based on the experimental results for varying sample sizes and differing feature numbers, it was observed that the proposed methods outperformed existing ones in terms of running cost while still achieving similar or better classification accuracies. Future areas of investigation could focus on the improvement of kTree and k*Tree for data with large numbers of features.<br /> <br /> == Critiques == <br /> <br /> *The paper only assessed classification accuracy through cross-validation accuracy. However, it would be interesting to investigate how the proposed methods perform using different metrics, such as AUC, precision-recall curves, or in terms of holdout test data set accuracy. <br /> * The authors addressed that some of the UCI datasets contained imbalanced data (such as the Climate and German data sets) while others did not. However, the nature of the class imbalance was not extreme, and the effect of imbalanced data on algorithm performance was not discussed or assessed. Moreover, it would have been interesting to see how the proposed algorithms performed on highly imbalanced datasets in conjunction with common techniques to address imbalance (e.g. oversampling, undersampling, etc.). <br /> *While the authors contrast their kTree and k*Tree approach with different kNN methods, the paper could contrast their results with more of the approaches discussed in the Related Work section of their paper. For example, it would be interesting to see how the kTree and k*Tree results compared to Góra and Wojna varied optimal k method.<br /> <br /> * The paper conducted an experiment on kNN, AD-kNN, S-kNN, GS-kNN,FASBIR and LC-kNN with different sample sizes and feature numbers. It would be interesting to discuss why the running cost of FASBIR is between that of kTree and k*Tree in figure 21.<br /> <br /> * A different [https://iopscience.iop.org/article/10.1088/1757-899X/725/1/012133/pdf paper] also discusses optimizing the K value for the kNN algorithm in clustering. However, this paper suggests using the expectation-maximization algorithm as a means of finding the optimal k value.<br /> <br /> * It would be really helpful if kTrees method can be explained at the very beginning. The transition from KNN to kTrees is not very smooth.<br /> <br /> * It would be nice to have a comparison of the running costs of different methods to see how much faster kTree and k*Tree performed<br /> <br /> * It would be better to show the key result only on a summary rather than stacking up all results without screening.<br /> <br /> * In the results section, it was mentioned that in the experiment on data sets with different numbers of features, the kTree and k*Tree model did not achieve GS-kNN or S-kNN's accuracies, but was faster in terms of running cost. It might be helpful here if the authors add some more supporting arguments about the benefit of this tradeoff, which appears to be a minor decrease in accuracy for a large improvement in speed. This could further showcase the advantages of the kTree and k*Tree models. More quantitative analysis or real-life scenario examples could be some choices here.<br /> <br /> * An interesting thing to notice while solving for the optimal matrix &lt;math&gt;W^*&lt;/math&gt; that minimizes the loss function is that &lt;math&gt;W^*&lt;/math&gt; is not necessarily a symmetric matrix. That is, the correlation between the &lt;math&gt;i^{th}&lt;/math&gt; entry and the &lt;math&gt;j^{th}&lt;/math&gt; entry is different from that between the &lt;math&gt;j^{th}&lt;/math&gt; entry and the &lt;math&gt;i^{th}&lt;/math&gt; entry, which makes the resulting W* not really semantically meaningful. Therefore, it would be interesting if we may set a threshold on the allowing difference between the &lt;math&gt;ij^{th}&lt;/math&gt; entry and the &lt;math&gt;ji^{th}&lt;/math&gt; entry in &lt;math&gt;W^*&lt;/math&gt; and see if this new configuration will give better or worse results compared to current ones, which will provide better insights of the algorithm.<br /> <br /> * It would be interesting to see how the proposed model works with highly non-linear datasets. In the event it does not work well, it would pose the question: would replacing the k*Tree with a SVM or a neural network improve the accuracy? There could be experiments to show if this variant would prove superior over the original models.<br /> <br /> * The key results are a little misleading - for example they claim &quot;the kTree had the highest accuracy and it's running cost was lower than any other methods except the k*Tree method&quot; is false. The kTree method had slightly lower accuracy than both GS-kNN and S-kNN and kTree was also slower than LC-kNN<br /> <br /> == References == <br /> <br />  C. Zhang, Y. Qin, X. Zhu, and J. Zhang, “Clustering-based missing value imputation for data preprocessing,” in Proc. IEEE Int. Conf., Aug. 2006, pp. 1081–1086.<br /> <br />  Y. Song, J. Huang, D. Zhou, H. Zha, and C. L. Giles, “IKNN: Informative K-nearest neighbor pattern classification,” in Knowledge Discovery in Databases. Berlin, Germany: Springer, 2007, pp. 248–264.<br /> <br />  P. Vincent and Y. Bengio, “K-local hyperplane and convex distance nearest neighbor algorithms,” in Proc. NIPS, 2001, pp. 985–992.<br /> <br />  V. Premachandran and R. Kakarala, “Consensus of k-NNs for robust neighborhood selection on graph-based manifolds,” in Proc. CVPR, Jun. 2013, pp. 1594–1601.<br /> <br />  X. Zhu, S. Zhang, Z. Jin, Z. Zhang, and Z. Xu, “Missing value estimation for mixed-attribute data sets,” IEEE Trans. Knowl. Data Eng., vol. 23, no. 1, pp. 110–121, Jan. 2011.<br /> <br />  F. Sahigara, D. Ballabio, R. Todeschini, and V. Consonni, “Assessing the validity of QSARS for ready biodegradability of chemicals: An applicability domain perspective,” Current Comput.-Aided Drug Design, vol. 10, no. 2, pp. 137–147, 2013.<br /> <br />  S. Zhang, M. Zong, K. Sun, Y. Liu, and D. Cheng, “Efficient kNN algorithm based on graph sparse reconstruction,” in Proc. ADMA, 2014, pp. 356–369.<br /> <br />  X. Zhu, L. Zhang, and Z. Huang, “A sparse embedding and least variance encoding approach to hashing,” IEEE Trans. Image Process., vol. 23, no. 9, pp. 3737–3750, Sep. 2014.<br /> <br />  U. Lall and A. Sharma, “A nearest neighbor bootstrap for resampling hydrologic time series,” Water Resour. Res., vol. 32, no. 3, pp. 679–693, 1996.<br /> <br />  D. Cheng, S. Zhang, Z. Deng, Y. Zhu, and M. Zong, “KNN algorithm with data-driven k value,” in Proc. ADMA, 2014, pp. 499–512.<br /> <br />  F. Sahigara, D. Ballabio, R. Todeschini, and V. Consonni, “Assessing the validity of QSARS for ready biodegradability of chemicals: An applicability domain perspective,” Current Comput.-Aided Drug Design, vol. 10, no. 2, pp. 137–147, 2013. <br /> <br />  Z. H. Zhou and Y. Yu, “Ensembling local learners throughmultimodal perturbation,” IEEE Trans. Syst. Man, B, vol. 35, no. 4, pp. 725–735, Apr. 2005.<br /> <br />  Z. H. Zhou, Ensemble Methods: Foundations and Algorithms. London, U.K.: Chapman &amp; Hall, 2012.<br /> <br />  Z. Deng, X. Zhu, D. Cheng, M. Zong, and S. Zhang, “Efficient kNN classification algorithm for big data,” Neurocomputing, vol. 195, pp. 143–148, Jun. 2016.<br /> <br />  K. Tsuda, M. Kawanabe, and K.-R. Müller, “Clustering with the fisher score,” in Proc. NIPS, 2002, pp. 729–736.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Efficient_kNN_Classification_with_Different_Numbers_of_Nearest_Neighbors&diff=49066 Efficient kNN Classification with Different Numbers of Nearest Neighbors 2020-12-04T03:42:14Z <p>Z42qin: /* Reconstruction */</p> <hr /> <div>== Presented by == <br /> Cooper Brooke, Daniel Fagan, Maya Perelman<br /> <br /> == Introduction == <br /> Traditional model-based classification approaches first use training observations to fit a model before predicting test samples. In contrast, the model-free k-nearest neighbors (KNNs) method classifies observations with a majority rule approach, labeling each piece of test data based on its k closest training observations (neighbors). This method has become very popular due to its strong performance and simple implementation. <br /> <br /> There are two main approaches to conduct kNN classification. The first is to use a fixed k value to classify all test samples, and the second is to use a different k value for each test sample. The former, while easy to implement, has proven to be impractical in machine learning applications. Therefore, interest lies in developing an efficient way to apply a different optimal k value for each test sample. The authors of this paper presented the kTree and k*Tree methods to solve this research question.<br /> <br /> == Previous Work == <br /> <br /> Previous work on finding an optimal fixed k value for all test samples is well-studied. Zhang et al.  incorporated a certainty factor measure to solve for an optimal fixed k. This resulted in the conclusion that k should be &lt;math&gt;\sqrt{n}&lt;/math&gt; (where n is the number of training samples) when n &gt; 100. The method Song et al. explored involved selecting a subset of the most informative samples from neighbourhoods. Vincent and Bengio  took the unique approach of designing a k-local hyperplane distance to solve for k. Premachandran and Kakarala  had the solution of selecting a robust k using the consensus of multiple rounds of kNNs. These fixed k methods are valuable however are impractical for data mining and machine learning applications. <br /> <br /> Finding an efficient approach to assigning varied k values has also been previously studied. Tuning approaches such as the ones taken by Zhu et al. as well as Sahugara et al. have been popular. Zhu et al.  determined that optimal k values should be chosen using cross validation while Sahugara et al.  proposed using Monte Carlo validation to select varied k parameters. Other learning approaches such as those taken by Zheng et al. and Góra and Wojna also show promise. Zheng et al.  applied a reconstruction framework to learn suitable k values. Góra and Wojna  proposed using rule induction and instance-based learning to learn optimal k-values for each test sample. While all these methods are valid, their processes of either learning varied k values or scanning all training samples are time-consuming.<br /> <br /> == Motivation == <br /> <br /> Due to the previously mentioned drawbacks of fixed-k and current varied-k kNN classification, the paper’s authors sought to design a new approach to solve for different k values. The kTree and k*Tree approach seeks to calculate optimal values of k while avoiding computationally costly steps such as cross-validation.<br /> <br /> A secondary motivation of this research was to ensure that the kTree method would perform better than kNN using fixed values of k given that running costs would be similar in this instance.<br /> <br /> == Approach == <br /> <br /> <br /> === kTree Classification ===<br /> <br /> The proposed kTree method is illustrated by the following flow chart:<br /> <br /> [[File:Approach_Figure_1.png | center | 800x800px]]<br /> <br /> ==== Reconstruction ====<br /> <br /> The first step is to use the training samples to reconstruct themselves. The goal of this is to find the matrix of correlations between the training samples themselves, &lt;math&gt;\textbf{W}&lt;/math&gt;, such that the distance between an individual training sample and the corresponding correlation vector multiplied by the entire training set is minimized. This least square loss function where &lt;math&gt;\mathbf{X}\in \mathbb{R}^{d\times n} = [x_1,...,x_n]&lt;/math&gt; represents the training set can be written as:<br /> <br /> \begin{aligned}<br /> \mathop{min}_{\textbf{W}} \sum_{i=1}^n ||Xw_i - x_i||^2<br /> \end{aligned}<br /> <br /> In addition, an &lt;math&gt;l_1&lt;/math&gt; regularization term multiplied by a tuning parameter, &lt;math&gt;\rho_1&lt;/math&gt;, is added to ensure that sparse results are generated as the objective is to minimize the number of training samples that will eventually be depended on by the test samples. <br /> <br /> \begin{aligned}<br /> \mathop{min}_{\textbf{W}} \sum_{i=1}^n ||Xw_i - x_i||^2 + \rho||\textbf{W}||^2_2<br /> \end{aligned}<br /> <br /> This is called ridge regression and it has a close solution where $$W = (X^TX+\rhoI)^{-1}X^TX$$<br /> <br /> However, this objective function does not provide a sparse result, there we further employe a sparse objective function: <br /> <br /> $$W = (X^TX+\rhoI)^{-1}X^TX, W &gt;= 0$$<br /> <br /> <br /> The least square loss function is then further modified to account for samples that have similar values for certain features yielding similar results. It is penalized with the function: <br /> <br /> $$\frac{1}{2} \sum{d}{i,j} ||x^iW-x^jW||^2_2$$<br /> <br /> with sij denotes the relation between feature vectors. It uses a radial basis function kernel to calculate Sij. After some transformations, this second regularization term that has tuning parameter &lt;math&gt;\rho_2&lt;/math&gt; is:<br /> <br /> \begin{aligned}<br /> R(W) = Tr(\textbf{W}^T \textbf{X}^T \textbf{LXW})<br /> \end{aligned}<br /> <br /> where &lt;math&gt;\mathbf{L}&lt;/math&gt; is a Laplacian matrix that indicates the relationship between features. The Laplacian matrix, also called the graph Laplacian, is a matrix representation of a graph. <br /> <br /> This gives a final objective function of:<br /> <br /> \begin{aligned}<br /> \mathop{min}_{\textbf{W}} \sum_{i=1}^n ||Xw_i - x_i||^2 + \rho_1||\textbf{W}|| + \rho_2R(\textbf{W})<br /> \end{aligned}<br /> <br /> Since this is a convex function, an iterative method can be used to optimize it to find the optimal solution &lt;math&gt;\mathbf{W^*}&lt;/math&gt;.<br /> <br /> ==== Calculate ''k'' for training set ====<br /> <br /> Each element &lt;math&gt;w_{ij}&lt;/math&gt; in &lt;math&gt;\textbf{W*}&lt;/math&gt; represents the correlation between the ith and jth training sample so if a value is 0, it can be concluded that the jth training sample has no effect on the ith training sample which means that it should not be used in the prediction of the ith training sample. Consequently, all non-zero values in the &lt;math&gt;w_{.j}&lt;/math&gt; vector would be useful in predicting the ith training sample which gives the result that the number of these non-zero elements for each sample is equal to the optimal ''k'' value for each sample.<br /> <br /> For example, if there was a 4x4 training set where &lt;math&gt;\textbf{W*}&lt;/math&gt; had the form:<br /> <br /> [[File:Approach_Figure_2.png | center | 300x300px]]<br /> <br /> The optimal ''k'' value for training sample 1 would be 2 since the correlation between training sample 1 and both training samples 2 and 4 are non-zero.<br /> <br /> ==== Train a Decision Tree using ''k'' as the label ====<br /> <br /> In a normal decision tree, the target data is the labels themselves. In contrast, in the kTree method, the target data is the optimal ''k'' value for each sample that was solved for in the previous step. So this decision tree has the following form:<br /> <br /> [[File:Approach_Figure_3.png | center | 300x300px]]<br /> <br /> ==== Making Predictions for Test Data ====<br /> <br /> The optimal ''k'' values for each testing sample are easily obtainable using the kTree solved for in the previous step. The only remaining step is to predict the labels of the testing samples by finding the majority class of the optimal ''k'' nearest neighbors across '''all''' of the training data.<br /> <br /> === k*Tree Classification ===<br /> <br /> The proposed k*Tree method is illustrated by the following flow chart:<br /> <br /> [[File:Approach_Figure_4.png | center | 1000x1000px]]<br /> <br /> Clearly, this is a very similar approach to the kTree as the k*Tree method attempts to sacrifice very little in predictive power in return for a substantial decrease in complexity when actually implementing the traditional kNN on the testing data once the optimal ''k'' values have been found.<br /> <br /> While all steps previous are the exact same, the difference comes from additional data stored in the leaf nodes. k*Tree method not only stores the optimal ''k'' value but also the following information:<br /> <br /> * The training samples that have the same optimal ''k''<br /> * The ''k'' nearest neighbours of the previously identified training samples<br /> * The nearest neighbor of each of the previously identified ''k'' nearest neighbours<br /> <br /> The data stored in each node is summarized in the following figure:<br /> <br /> [[File:Approach_Figure_5.png | center | 800x800px]]<br /> <br /> When testing, the constructed k*Tree is searched for its optimal k values well as its nearest neighbours in the leaf node. It then selects a number of its nearest neighbours from the subset of training samples and assigns the test sample with the majority label of these nearest neighbours.<br /> <br /> In the kTree method, predictions were made based on all of the training data, whereas in the k*Tree method, predicting the test labels will only be done using the samples stored in the applicable node of the tree.<br /> <br /> == Experiments == <br /> <br /> In order to assess the performance of the proposed method against existing methods, a number of experiments were performed to measure classification accuracy and run time. The experiments were run on twenty public datasets provided by the UCI Repository of Machine Learning Data, and contained a mix of data types varying in size, in dimensionality, in the number of classes, and in imbalanced nature of the data. Ten-fold cross-validation was used to measure classification accuracy, and the following methods were compared against:<br /> <br /> # k-Nearest Neighbor: The classical kNN approach with k set to k=1,5,10,20 and square root of the sample size ; the best result was reported.<br /> # kNN-Based Applicability Domain Approach (AD-kNN) <br /> # kNN Method Based on Sparse Learning (S-kNN) <br /> # kNN Based on Graph Sparse Reconstruction (GS-kNN) <br /> # Filtered Attribute Subspace-based Bagging with Injected Randomness (FASBIR) , <br /> # Landmark-based Spectral Clustering kNN (LC-kNN) <br /> <br /> The experimental results were then assessed based on classification tasks that focused on different sample sizes, and tasks that focused on different numbers of features. <br /> <br /> <br /> '''A. Experimental Results on Different Sample Sizes'''<br /> <br /> The running cost and (cross-validation) classification accuracy based on experiments on ten UCI datasets can be seen in Table I below.<br /> <br /> [[File:Table_I_kNN.png | center | 1000x1000px]]<br /> <br /> The following key results are noted:<br /> * Regarding classification accuracy, the proposed methods (kTree and k*Tree) outperformed kNN, AD-KNN, FASBIR, and LC-kNN on all datasets by 1.5%-4.5%, but had no notable improvements compared to GS-kNN and S-kNN.<br /> * Classification methods which involved learning optimal k-values (for example the proposed kTree and k*Tree methods, or S-kNN, GS-kNN, AD-kNN) outperformed the methods with predefined k-values, such as traditional kNN.<br /> * The proposed k*Tree method had the lowest running cost of all methods. However, the k*Tree method was still outperformed in terms of classification accuracy by GS-kNN and S-kNN, but ran on average 15 000 times faster than either method. In addition, the kTree had the highest accuracy and it's running cost was lower than any other methods except the k*Tree method.<br /> <br /> <br /> '''B. Experimental Results on Different Feature Numbers'''<br /> <br /> The goal of this section was to evaluate the robustness of all methods under differing numbers of features; results can be seen in Table II below. The Fisher score, an algorithm that solves maximum likelihood equations numerically , was used to rank and select the most information features in the datasets. <br /> <br /> [[File:Table_II_kNN.png | center | 1000x1000px]]<br /> <br /> From Table II, the proposed kTree and k*Tree approaches outperformed kNN, AD-kNN, FASBIR and LC-KNN when tested for varying feature numbers. The S-kNN and GS-kNN approaches remained the best in terms of classification accuracy, but were greatly outperformed in terms of running cost by k*Tree. The cause for this is that k*Tree only scans a subsample of the training samples for kNN classification, while S-kNN and GS-kNN scan all training samples.<br /> <br /> == Conclusion == <br /> <br /> This paper introduced two novel approaches for kNN classification algorithms that can determine optimal k-values for each test sample. The proposed kTree and k*Tree methods can classify the test samples efficiently and effectively, by designing a training step that reduces the run time of the test stage and thus enhances the performance. Based on the experimental results for varying sample sizes and differing feature numbers, it was observed that the proposed methods outperformed existing ones in terms of running cost while still achieving similar or better classification accuracies. Future areas of investigation could focus on the improvement of kTree and k*Tree for data with large numbers of features.<br /> <br /> == Critiques == <br /> <br /> *The paper only assessed classification accuracy through cross-validation accuracy. However, it would be interesting to investigate how the proposed methods perform using different metrics, such as AUC, precision-recall curves, or in terms of holdout test data set accuracy. <br /> * The authors addressed that some of the UCI datasets contained imbalanced data (such as the Climate and German data sets) while others did not. However, the nature of the class imbalance was not extreme, and the effect of imbalanced data on algorithm performance was not discussed or assessed. Moreover, it would have been interesting to see how the proposed algorithms performed on highly imbalanced datasets in conjunction with common techniques to address imbalance (e.g. oversampling, undersampling, etc.). <br /> *While the authors contrast their kTree and k*Tree approach with different kNN methods, the paper could contrast their results with more of the approaches discussed in the Related Work section of their paper. For example, it would be interesting to see how the kTree and k*Tree results compared to Góra and Wojna varied optimal k method.<br /> <br /> * The paper conducted an experiment on kNN, AD-kNN, S-kNN, GS-kNN,FASBIR and LC-kNN with different sample sizes and feature numbers. It would be interesting to discuss why the running cost of FASBIR is between that of kTree and k*Tree in figure 21.<br /> <br /> * A different [https://iopscience.iop.org/article/10.1088/1757-899X/725/1/012133/pdf paper] also discusses optimizing the K value for the kNN algorithm in clustering. However, this paper suggests using the expectation-maximization algorithm as a means of finding the optimal k value.<br /> <br /> * It would be really helpful if kTrees method can be explained at the very beginning. The transition from KNN to kTrees is not very smooth.<br /> <br /> * It would be nice to have a comparison of the running costs of different methods to see how much faster kTree and k*Tree performed<br /> <br /> * It would be better to show the key result only on a summary rather than stacking up all results without screening.<br /> <br /> * In the results section, it was mentioned that in the experiment on data sets with different numbers of features, the kTree and k*Tree model did not achieve GS-kNN or S-kNN's accuracies, but was faster in terms of running cost. It might be helpful here if the authors add some more supporting arguments about the benefit of this tradeoff, which appears to be a minor decrease in accuracy for a large improvement in speed. This could further showcase the advantages of the kTree and k*Tree models. More quantitative analysis or real-life scenario examples could be some choices here.<br /> <br /> * An interesting thing to notice while solving for the optimal matrix &lt;math&gt;W^*&lt;/math&gt; that minimizes the loss function is that &lt;math&gt;W^*&lt;/math&gt; is not necessarily a symmetric matrix. That is, the correlation between the &lt;math&gt;i^{th}&lt;/math&gt; entry and the &lt;math&gt;j^{th}&lt;/math&gt; entry is different from that between the &lt;math&gt;j^{th}&lt;/math&gt; entry and the &lt;math&gt;i^{th}&lt;/math&gt; entry, which makes the resulting W* not really semantically meaningful. Therefore, it would be interesting if we may set a threshold on the allowing difference between the &lt;math&gt;ij^{th}&lt;/math&gt; entry and the &lt;math&gt;ji^{th}&lt;/math&gt; entry in &lt;math&gt;W^*&lt;/math&gt; and see if this new configuration will give better or worse results compared to current ones, which will provide better insights of the algorithm.<br /> <br /> * It would be interesting to see how the proposed model works with highly non-linear datasets. In the event it does not work well, it would pose the question: would replacing the k*Tree with a SVM or a neural network improve the accuracy? There could be experiments to show if this variant would prove superior over the original models.<br /> <br /> * The key results are a little misleading - for example they claim &quot;the kTree had the highest accuracy and it's running cost was lower than any other methods except the k*Tree method&quot; is false. The kTree method had slightly lower accuracy than both GS-kNN and S-kNN and kTree was also slower than LC-kNN<br /> <br /> == References == <br /> <br />  C. Zhang, Y. Qin, X. Zhu, and J. Zhang, “Clustering-based missing value imputation for data preprocessing,” in Proc. IEEE Int. Conf., Aug. 2006, pp. 1081–1086.<br /> <br />  Y. Song, J. Huang, D. Zhou, H. Zha, and C. L. Giles, “IKNN: Informative K-nearest neighbor pattern classification,” in Knowledge Discovery in Databases. Berlin, Germany: Springer, 2007, pp. 248–264.<br /> <br />  P. Vincent and Y. Bengio, “K-local hyperplane and convex distance nearest neighbor algorithms,” in Proc. NIPS, 2001, pp. 985–992.<br /> <br />  V. Premachandran and R. Kakarala, “Consensus of k-NNs for robust neighborhood selection on graph-based manifolds,” in Proc. CVPR, Jun. 2013, pp. 1594–1601.<br /> <br />  X. Zhu, S. Zhang, Z. Jin, Z. Zhang, and Z. Xu, “Missing value estimation for mixed-attribute data sets,” IEEE Trans. Knowl. Data Eng., vol. 23, no. 1, pp. 110–121, Jan. 2011.<br /> <br />  F. Sahigara, D. Ballabio, R. Todeschini, and V. Consonni, “Assessing the validity of QSARS for ready biodegradability of chemicals: An applicability domain perspective,” Current Comput.-Aided Drug Design, vol. 10, no. 2, pp. 137–147, 2013.<br /> <br />  S. Zhang, M. Zong, K. Sun, Y. Liu, and D. Cheng, “Efficient kNN algorithm based on graph sparse reconstruction,” in Proc. ADMA, 2014, pp. 356–369.<br /> <br />  X. Zhu, L. Zhang, and Z. Huang, “A sparse embedding and least variance encoding approach to hashing,” IEEE Trans. Image Process., vol. 23, no. 9, pp. 3737–3750, Sep. 2014.<br /> <br />  U. Lall and A. Sharma, “A nearest neighbor bootstrap for resampling hydrologic time series,” Water Resour. Res., vol. 32, no. 3, pp. 679–693, 1996.<br /> <br />  D. Cheng, S. Zhang, Z. Deng, Y. Zhu, and M. Zong, “KNN algorithm with data-driven k value,” in Proc. ADMA, 2014, pp. 499–512.<br /> <br />  F. Sahigara, D. Ballabio, R. Todeschini, and V. Consonni, “Assessing the validity of QSARS for ready biodegradability of chemicals: An applicability domain perspective,” Current Comput.-Aided Drug Design, vol. 10, no. 2, pp. 137–147, 2013. <br /> <br />  Z. H. Zhou and Y. Yu, “Ensembling local learners throughmultimodal perturbation,” IEEE Trans. Syst. Man, B, vol. 35, no. 4, pp. 725–735, Apr. 2005.<br /> <br />  Z. H. Zhou, Ensemble Methods: Foundations and Algorithms. London, U.K.: Chapman &amp; Hall, 2012.<br /> <br />  Z. Deng, X. Zhu, D. Cheng, M. Zong, and S. Zhang, “Efficient kNN classification algorithm for big data,” Neurocomputing, vol. 195, pp. 143–148, Jun. 2016.<br /> <br />  K. Tsuda, M. Kawanabe, and K.-R. Müller, “Clustering with the fisher score,” in Proc. NIPS, 2002, pp. 729–736.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Efficient_kNN_Classification_with_Different_Numbers_of_Nearest_Neighbors&diff=49065 Efficient kNN Classification with Different Numbers of Nearest Neighbors 2020-12-04T03:41:43Z <p>Z42qin: /* Reconstruction */</p> <hr /> <div>== Presented by == <br /> Cooper Brooke, Daniel Fagan, Maya Perelman<br /> <br /> == Introduction == <br /> Traditional model-based classification approaches first use training observations to fit a model before predicting test samples. In contrast, the model-free k-nearest neighbors (KNNs) method classifies observations with a majority rule approach, labeling each piece of test data based on its k closest training observations (neighbors). This method has become very popular due to its strong performance and simple implementation. <br /> <br /> There are two main approaches to conduct kNN classification. The first is to use a fixed k value to classify all test samples, and the second is to use a different k value for each test sample. The former, while easy to implement, has proven to be impractical in machine learning applications. Therefore, interest lies in developing an efficient way to apply a different optimal k value for each test sample. The authors of this paper presented the kTree and k*Tree methods to solve this research question.<br /> <br /> == Previous Work == <br /> <br /> Previous work on finding an optimal fixed k value for all test samples is well-studied. Zhang et al.  incorporated a certainty factor measure to solve for an optimal fixed k. This resulted in the conclusion that k should be &lt;math&gt;\sqrt{n}&lt;/math&gt; (where n is the number of training samples) when n &gt; 100. The method Song et al. explored involved selecting a subset of the most informative samples from neighbourhoods. Vincent and Bengio  took the unique approach of designing a k-local hyperplane distance to solve for k. Premachandran and Kakarala  had the solution of selecting a robust k using the consensus of multiple rounds of kNNs. These fixed k methods are valuable however are impractical for data mining and machine learning applications. <br /> <br /> Finding an efficient approach to assigning varied k values has also been previously studied. Tuning approaches such as the ones taken by Zhu et al. as well as Sahugara et al. have been popular. Zhu et al.  determined that optimal k values should be chosen using cross validation while Sahugara et al.  proposed using Monte Carlo validation to select varied k parameters. Other learning approaches such as those taken by Zheng et al. and Góra and Wojna also show promise. Zheng et al.  applied a reconstruction framework to learn suitable k values. Góra and Wojna  proposed using rule induction and instance-based learning to learn optimal k-values for each test sample. While all these methods are valid, their processes of either learning varied k values or scanning all training samples are time-consuming.<br /> <br /> == Motivation == <br /> <br /> Due to the previously mentioned drawbacks of fixed-k and current varied-k kNN classification, the paper’s authors sought to design a new approach to solve for different k values. The kTree and k*Tree approach seeks to calculate optimal values of k while avoiding computationally costly steps such as cross-validation.<br /> <br /> A secondary motivation of this research was to ensure that the kTree method would perform better than kNN using fixed values of k given that running costs would be similar in this instance.<br /> <br /> == Approach == <br /> <br /> <br /> === kTree Classification ===<br /> <br /> The proposed kTree method is illustrated by the following flow chart:<br /> <br /> [[File:Approach_Figure_1.png | center | 800x800px]]<br /> <br /> ==== Reconstruction ====<br /> <br /> The first step is to use the training samples to reconstruct themselves. The goal of this is to find the matrix of correlations between the training samples themselves, &lt;math&gt;\textbf{W}&lt;/math&gt;, such that the distance between an individual training sample and the corresponding correlation vector multiplied by the entire training set is minimized. This least square loss function where &lt;math&gt;\mathbf{X}\in \mathbb{R}^{d\times n} = [x_1,...,x_n]&lt;/math&gt; represents the training set can be written as:<br /> <br /> \begin{aligned}<br /> \mathop{min}_{\textbf{W}} \sum_{i=1}^n ||Xw_i - x_i||^2<br /> \end{aligned}<br /> <br /> In addition, an &lt;math&gt;l_1&lt;/math&gt; regularization term multiplied by a tuning parameter, &lt;math&gt;\rho_1&lt;/math&gt;, is added to ensure that sparse results are generated as the objective is to minimize the number of training samples that will eventually be depended on by the test samples. <br /> <br /> \begin{aligned}<br /> \mathop{min}_{\textbf{W}} \sum_{i=1}^n ||Xw_i - x_i||^2 + \rho||\textbf{W}||^2_2<br /> \end{aligned}<br /> <br /> This is called ridge regression and it has a close solution where $W = (X^TX+\rhoI)^{-1}X^TX$<br /> <br /> However, this objective function does not provide a sparse result, there we further employe a sparse objective function: <br /> <br /> $W = (X^TX+\rhoI)^{-1}X^TX, W &gt;= 0$<br /> <br /> <br /> The least square loss function is then further modified to account for samples that have similar values for certain features yielding similar results. It is penalized with the function: <br /> <br /> $\frac{1}{2} \sum{d}{i,j} ||x^iW-x^jW||^2_2$<br /> <br /> with sij denotes the relation between feature vectors. It uses a radial basis function kernel to calculate Sij. After some transformations, this second regularization term that has tuning parameter &lt;math&gt;\rho_2&lt;/math&gt; is:<br /> <br /> \begin{aligned}<br /> R(W) = Tr(\textbf{W}^T \textbf{X}^T \textbf{LXW})<br /> \end{aligned}<br /> <br /> where &lt;math&gt;\mathbf{L}&lt;/math&gt; is a Laplacian matrix that indicates the relationship between features. The Laplacian matrix, also called the graph Laplacian, is a matrix representation of a graph. <br /> <br /> This gives a final objective function of:<br /> <br /> \begin{aligned}<br /> \mathop{min}_{\textbf{W}} \sum_{i=1}^n ||Xw_i - x_i||^2 + \rho_1||\textbf{W}|| + \rho_2R(\textbf{W})<br /> \end{aligned}<br /> <br /> Since this is a convex function, an iterative method can be used to optimize it to find the optimal solution &lt;math&gt;\mathbf{W^*}&lt;/math&gt;.<br /> <br /> ==== Calculate ''k'' for training set ====<br /> <br /> Each element &lt;math&gt;w_{ij}&lt;/math&gt; in &lt;math&gt;\textbf{W*}&lt;/math&gt; represents the correlation between the ith and jth training sample so if a value is 0, it can be concluded that the jth training sample has no effect on the ith training sample which means that it should not be used in the prediction of the ith training sample. Consequently, all non-zero values in the &lt;math&gt;w_{.j}&lt;/math&gt; vector would be useful in predicting the ith training sample which gives the result that the number of these non-zero elements for each sample is equal to the optimal ''k'' value for each sample.<br /> <br /> For example, if there was a 4x4 training set where &lt;math&gt;\textbf{W*}&lt;/math&gt; had the form:<br /> <br /> [[File:Approach_Figure_2.png | center | 300x300px]]<br /> <br /> The optimal ''k'' value for training sample 1 would be 2 since the correlation between training sample 1 and both training samples 2 and 4 are non-zero.<br /> <br /> ==== Train a Decision Tree using ''k'' as the label ====<br /> <br /> In a normal decision tree, the target data is the labels themselves. In contrast, in the kTree method, the target data is the optimal ''k'' value for each sample that was solved for in the previous step. So this decision tree has the following form:<br /> <br /> [[File:Approach_Figure_3.png | center | 300x300px]]<br /> <br /> ==== Making Predictions for Test Data ====<br /> <br /> The optimal ''k'' values for each testing sample are easily obtainable using the kTree solved for in the previous step. The only remaining step is to predict the labels of the testing samples by finding the majority class of the optimal ''k'' nearest neighbors across '''all''' of the training data.<br /> <br /> === k*Tree Classification ===<br /> <br /> The proposed k*Tree method is illustrated by the following flow chart:<br /> <br /> [[File:Approach_Figure_4.png | center | 1000x1000px]]<br /> <br /> Clearly, this is a very similar approach to the kTree as the k*Tree method attempts to sacrifice very little in predictive power in return for a substantial decrease in complexity when actually implementing the traditional kNN on the testing data once the optimal ''k'' values have been found.<br /> <br /> While all steps previous are the exact same, the difference comes from additional data stored in the leaf nodes. k*Tree method not only stores the optimal ''k'' value but also the following information:<br /> <br /> * The training samples that have the same optimal ''k''<br /> * The ''k'' nearest neighbours of the previously identified training samples<br /> * The nearest neighbor of each of the previously identified ''k'' nearest neighbours<br /> <br /> The data stored in each node is summarized in the following figure:<br /> <br /> [[File:Approach_Figure_5.png | center | 800x800px]]<br /> <br /> When testing, the constructed k*Tree is searched for its optimal k values well as its nearest neighbours in the leaf node. It then selects a number of its nearest neighbours from the subset of training samples and assigns the test sample with the majority label of these nearest neighbours.<br /> <br /> In the kTree method, predictions were made based on all of the training data, whereas in the k*Tree method, predicting the test labels will only be done using the samples stored in the applicable node of the tree.<br /> <br /> == Experiments == <br /> <br /> In order to assess the performance of the proposed method against existing methods, a number of experiments were performed to measure classification accuracy and run time. The experiments were run on twenty public datasets provided by the UCI Repository of Machine Learning Data, and contained a mix of data types varying in size, in dimensionality, in the number of classes, and in imbalanced nature of the data. Ten-fold cross-validation was used to measure classification accuracy, and the following methods were compared against:<br /> <br /> # k-Nearest Neighbor: The classical kNN approach with k set to k=1,5,10,20 and square root of the sample size ; the best result was reported.<br /> # kNN-Based Applicability Domain Approach (AD-kNN) <br /> # kNN Method Based on Sparse Learning (S-kNN) <br /> # kNN Based on Graph Sparse Reconstruction (GS-kNN) <br /> # Filtered Attribute Subspace-based Bagging with Injected Randomness (FASBIR) , <br /> # Landmark-based Spectral Clustering kNN (LC-kNN) <br /> <br /> The experimental results were then assessed based on classification tasks that focused on different sample sizes, and tasks that focused on different numbers of features. <br /> <br /> <br /> '''A. Experimental Results on Different Sample Sizes'''<br /> <br /> The running cost and (cross-validation) classification accuracy based on experiments on ten UCI datasets can be seen in Table I below.<br /> <br /> [[File:Table_I_kNN.png | center | 1000x1000px]]<br /> <br /> The following key results are noted:<br /> * Regarding classification accuracy, the proposed methods (kTree and k*Tree) outperformed kNN, AD-KNN, FASBIR, and LC-kNN on all datasets by 1.5%-4.5%, but had no notable improvements compared to GS-kNN and S-kNN.<br /> * Classification methods which involved learning optimal k-values (for example the proposed kTree and k*Tree methods, or S-kNN, GS-kNN, AD-kNN) outperformed the methods with predefined k-values, such as traditional kNN.<br /> * The proposed k*Tree method had the lowest running cost of all methods. However, the k*Tree method was still outperformed in terms of classification accuracy by GS-kNN and S-kNN, but ran on average 15 000 times faster than either method. In addition, the kTree had the highest accuracy and it's running cost was lower than any other methods except the k*Tree method.<br /> <br /> <br /> '''B. Experimental Results on Different Feature Numbers'''<br /> <br /> The goal of this section was to evaluate the robustness of all methods under differing numbers of features; results can be seen in Table II below. The Fisher score, an algorithm that solves maximum likelihood equations numerically , was used to rank and select the most information features in the datasets. <br /> <br /> [[File:Table_II_kNN.png | center | 1000x1000px]]<br /> <br /> From Table II, the proposed kTree and k*Tree approaches outperformed kNN, AD-kNN, FASBIR and LC-KNN when tested for varying feature numbers. The S-kNN and GS-kNN approaches remained the best in terms of classification accuracy, but were greatly outperformed in terms of running cost by k*Tree. The cause for this is that k*Tree only scans a subsample of the training samples for kNN classification, while S-kNN and GS-kNN scan all training samples.<br /> <br /> == Conclusion == <br /> <br /> This paper introduced two novel approaches for kNN classification algorithms that can determine optimal k-values for each test sample. The proposed kTree and k*Tree methods can classify the test samples efficiently and effectively, by designing a training step that reduces the run time of the test stage and thus enhances the performance. Based on the experimental results for varying sample sizes and differing feature numbers, it was observed that the proposed methods outperformed existing ones in terms of running cost while still achieving similar or better classification accuracies. Future areas of investigation could focus on the improvement of kTree and k*Tree for data with large numbers of features.<br /> <br /> == Critiques == <br /> <br /> *The paper only assessed classification accuracy through cross-validation accuracy. However, it would be interesting to investigate how the proposed methods perform using different metrics, such as AUC, precision-recall curves, or in terms of holdout test data set accuracy. <br /> * The authors addressed that some of the UCI datasets contained imbalanced data (such as the Climate and German data sets) while others did not. However, the nature of the class imbalance was not extreme, and the effect of imbalanced data on algorithm performance was not discussed or assessed. Moreover, it would have been interesting to see how the proposed algorithms performed on highly imbalanced datasets in conjunction with common techniques to address imbalance (e.g. oversampling, undersampling, etc.). <br /> *While the authors contrast their kTree and k*Tree approach with different kNN methods, the paper could contrast their results with more of the approaches discussed in the Related Work section of their paper. For example, it would be interesting to see how the kTree and k*Tree results compared to Góra and Wojna varied optimal k method.<br /> <br /> * The paper conducted an experiment on kNN, AD-kNN, S-kNN, GS-kNN,FASBIR and LC-kNN with different sample sizes and feature numbers. It would be interesting to discuss why the running cost of FASBIR is between that of kTree and k*Tree in figure 21.<br /> <br /> * A different [https://iopscience.iop.org/article/10.1088/1757-899X/725/1/012133/pdf paper] also discusses optimizing the K value for the kNN algorithm in clustering. However, this paper suggests using the expectation-maximization algorithm as a means of finding the optimal k value.<br /> <br /> * It would be really helpful if kTrees method can be explained at the very beginning. The transition from KNN to kTrees is not very smooth.<br /> <br /> * It would be nice to have a comparison of the running costs of different methods to see how much faster kTree and k*Tree performed<br /> <br /> * It would be better to show the key result only on a summary rather than stacking up all results without screening.<br /> <br /> * In the results section, it was mentioned that in the experiment on data sets with different numbers of features, the kTree and k*Tree model did not achieve GS-kNN or S-kNN's accuracies, but was faster in terms of running cost. It might be helpful here if the authors add some more supporting arguments about the benefit of this tradeoff, which appears to be a minor decrease in accuracy for a large improvement in speed. This could further showcase the advantages of the kTree and k*Tree models. More quantitative analysis or real-life scenario examples could be some choices here.<br /> <br /> * An interesting thing to notice while solving for the optimal matrix &lt;math&gt;W^*&lt;/math&gt; that minimizes the loss function is that &lt;math&gt;W^*&lt;/math&gt; is not necessarily a symmetric matrix. That is, the correlation between the &lt;math&gt;i^{th}&lt;/math&gt; entry and the &lt;math&gt;j^{th}&lt;/math&gt; entry is different from that between the &lt;math&gt;j^{th}&lt;/math&gt; entry and the &lt;math&gt;i^{th}&lt;/math&gt; entry, which makes the resulting W* not really semantically meaningful. Therefore, it would be interesting if we may set a threshold on the allowing difference between the &lt;math&gt;ij^{th}&lt;/math&gt; entry and the &lt;math&gt;ji^{th}&lt;/math&gt; entry in &lt;math&gt;W^*&lt;/math&gt; and see if this new configuration will give better or worse results compared to current ones, which will provide better insights of the algorithm.<br /> <br /> * It would be interesting to see how the proposed model works with highly non-linear datasets. In the event it does not work well, it would pose the question: would replacing the k*Tree with a SVM or a neural network improve the accuracy? There could be experiments to show if this variant would prove superior over the original models.<br /> <br /> * The key results are a little misleading - for example they claim &quot;the kTree had the highest accuracy and it's running cost was lower than any other methods except the k*Tree method&quot; is false. The kTree method had slightly lower accuracy than both GS-kNN and S-kNN and kTree was also slower than LC-kNN<br /> <br /> == References == <br /> <br />  C. Zhang, Y. Qin, X. Zhu, and J. Zhang, “Clustering-based missing value imputation for data preprocessing,” in Proc. IEEE Int. Conf., Aug. 2006, pp. 1081–1086.<br /> <br />  Y. Song, J. Huang, D. Zhou, H. Zha, and C. L. Giles, “IKNN: Informative K-nearest neighbor pattern classification,” in Knowledge Discovery in Databases. Berlin, Germany: Springer, 2007, pp. 248–264.<br /> <br />  P. Vincent and Y. Bengio, “K-local hyperplane and convex distance nearest neighbor algorithms,” in Proc. NIPS, 2001, pp. 985–992.<br /> <br />  V. Premachandran and R. Kakarala, “Consensus of k-NNs for robust neighborhood selection on graph-based manifolds,” in Proc. CVPR, Jun. 2013, pp. 1594–1601.<br /> <br />  X. Zhu, S. Zhang, Z. Jin, Z. Zhang, and Z. Xu, “Missing value estimation for mixed-attribute data sets,” IEEE Trans. Knowl. Data Eng., vol. 23, no. 1, pp. 110–121, Jan. 2011.<br /> <br />  F. Sahigara, D. Ballabio, R. Todeschini, and V. Consonni, “Assessing the validity of QSARS for ready biodegradability of chemicals: An applicability domain perspective,” Current Comput.-Aided Drug Design, vol. 10, no. 2, pp. 137–147, 2013.<br /> <br />  S. Zhang, M. Zong, K. Sun, Y. Liu, and D. Cheng, “Efficient kNN algorithm based on graph sparse reconstruction,” in Proc. ADMA, 2014, pp. 356–369.<br /> <br />  X. Zhu, L. Zhang, and Z. Huang, “A sparse embedding and least variance encoding approach to hashing,” IEEE Trans. Image Process., vol. 23, no. 9, pp. 3737–3750, Sep. 2014.<br /> <br />  U. Lall and A. Sharma, “A nearest neighbor bootstrap for resampling hydrologic time series,” Water Resour. Res., vol. 32, no. 3, pp. 679–693, 1996.<br /> <br />  D. Cheng, S. Zhang, Z. Deng, Y. Zhu, and M. Zong, “KNN algorithm with data-driven k value,” in Proc. ADMA, 2014, pp. 499–512.<br /> <br />  F. Sahigara, D. Ballabio, R. Todeschini, and V. Consonni, “Assessing the validity of QSARS for ready biodegradability of chemicals: An applicability domain perspective,” Current Comput.-Aided Drug Design, vol. 10, no. 2, pp. 137–147, 2013. <br /> <br />  Z. H. Zhou and Y. Yu, “Ensembling local learners throughmultimodal perturbation,” IEEE Trans. Syst. Man, B, vol. 35, no. 4, pp. 725–735, Apr. 2005.<br /> <br />  Z. H. Zhou, Ensemble Methods: Foundations and Algorithms. London, U.K.: Chapman &amp; Hall, 2012.<br /> <br />  Z. Deng, X. Zhu, D. Cheng, M. Zong, and S. Zhang, “Efficient kNN classification algorithm for big data,” Neurocomputing, vol. 195, pp. 143–148, Jun. 2016.<br /> <br />  K. Tsuda, M. Kawanabe, and K.-R. Müller, “Clustering with the fisher score,” in Proc. NIPS, 2002, pp. 729–736.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Being_Bayesian_about_Categorical_Probability&diff=49058 Being Bayesian about Categorical Probability 2020-12-04T01:04:10Z <p>Z42qin: /* Classification With a Neural Network */</p> <hr /> <div>== Presented By ==<br /> Evan Li, Jason Pu, Karam Abuaisha, Nicholas Vadivelu<br /> <br /> == Introduction ==<br /> <br /> Since the outputs of neural networks are not probabilities, Softmax (Bridle, 1990) is a staple for neural network’s performing classification--it exponentiates each logit then normalizes by the sum, giving a distribution over the target classes. However, networks with softmax outputs give no information about uncertainty (Blundell et al., 2015; Gal &amp; Ghahramani, 2016), and the resulting distribution over classes is poorly calibrated (Guo et al., 2017), often giving overconfident predictions even when the classification is wrong. In addition, softmax also raises concerns about overfitting NNs due to its confident predictive behaviors(Xie et al., 2016; Pereyra et al., 2017). To achieve better generalization performance, this may require some effective regularization techniques. <br /> <br /> Bayesian Neural Networks (BNNs; MacKay, 1992) can alleviate these issues, but the resulting posteriors over the parameters are often intractable. Approximations such as variational inference (Graves, 2011; Blundell et al., 2015) and Monte Carlo Dropout (Gal &amp; Ghahramani, 2016) can still be expensive or give poor estimates for the posteriors. This work proposes a Bayesian treatment of the output logits of the neural network, treating the targets as a categorical random variable instead of a fixed label. This gives a computationally cheap way to get well-calibrated uncertainty estimates on neural network classifications.<br /> <br /> == Related Work ==<br /> <br /> Using Bayesian Neural Networks is the dominant way of applying Bayesian techniques to neural networks. Many techniques have been developed to make posterior approximation more accurate and scalable, despite these, BNNs do not scale to the state of the art techniques or large data sets. There are techniques to explicitly avoid modeling the full weight posterior that are more scalable, such as with Monte Carlo Dropout (Gal &amp; Ghahramani, 2016) or tracking mean/covariance of the posterior during training (Mandt et al., 2017; Zhang et al., 2018; Maddox et al., 2019; Osawa et al., 2019). Non-Bayesian uncertainty estimation techniques such as deep ensembles (Lakshminarayanan et al., 2017) and temperature scaling (Guo et al., 2017; Neumann et al., 2018).<br /> <br /> == Preliminaries ==<br /> === Definitions ===<br /> Let's formalize our classification problem and define some notations for the rest of this summary:<br /> <br /> ::Dataset:<br /> $$\mathcal D = \{(x_i,y_i)\} \in (\mathcal X \times \mathcal Y)^N$$<br /> ::General classification model<br /> $$f^W: \mathcal X \to \mathbb R^K$$<br /> ::Softmax function: <br /> $$\phi(x): \mathbb R^K \to [0,1]^K \;\;|\;\; \phi_k(X) = \frac{\exp(f_k^W(x))}{\sum_{k \in K} \exp(f_k^W(x))}$$<br /> ::Softmax activated NN:<br /> $$\phi \;\circ\; f^W: \chi \to \Delta^{K-1}$$<br /> ::NN as a true classifier:<br /> $$arg\max_i \;\circ\; \phi_i \;\circ\; f^W \;:\; \mathcal X \to \mathcal Y$$<br /> <br /> We'll also define the '''count function''' - a &lt;math&gt;K&lt;/math&gt;-vector valued function that outputs the occurences of each class coincident with &lt;math&gt;x&lt;/math&gt;:<br /> $$c^{\mathcal D}(x) = \sum_{(x',y') \in \mathcal D} \mathbb y' I(x' = x)$$<br /> <br /> === Classification With a Neural Network ===<br /> A typical loss function used in classification is cross-entropy. It's well known that optimizing &lt;math&gt;f^W&lt;/math&gt; for &lt;math&gt;l_{CE}&lt;/math&gt; is equivalent to optimizing for &lt;math&gt;l_{KL}&lt;/math&gt;, the &lt;math&gt;KL&lt;/math&gt; divergence between the true distribution and the distribution modeled by NN, that is:<br /> $$l_{KL}(W) = KL(\text{true distribution} \;|\; \text{distribution encoded by }NN(W))$$<br /> Let's introduce notations for the underlying (true) distributions of our problem. Let &lt;math&gt;(x_0,y_0) \sim (\mathcal X \times \mathcal Y)&lt;/math&gt;:<br /> $$\text{Full Distribution} = F(x,y) = P(x_0 = x,y_0 = y)$$<br /> $$\text{Marginal Distribution} = P(x) = F(x_0 = x)$$<br /> $$\text{Point Class Distribution} = P(y_0 = y \;|\; x_0 = x) = F_x(y)$$<br /> Then we have the following factorization:<br /> $$F(x,y) = P(x,y) = P(y|x)P(x) = F_x(y)F(x)$$<br /> Substitute this into the definition of KL divergence:<br /> $$= \sum_{(x,y) \in \mathcal X \times \mathcal Y} F(x,y) \log\left(\frac{F(x,y)}{\phi_y(f^W(x))}\right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) \sum_{y \in \mathcal Y} F(y|x) \log\left( \frac{F(y|x)}{\phi_y(f^W(x))} \right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) \sum_{y \in \mathcal Y} F_x(y) \log\left( \frac{F_x(y)}{\phi_y(f^W(x))} \right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) KL(F_x \;||\; \phi\left( f^W(x) \right))$$<br /> As usual, we don't have an analytic form for &lt;math&gt;l&lt;/math&gt; (if we did, this would imply we know &lt;math&gt;F_X&lt;/math&gt; meaning we knew the distribution in the first place). Instead, estimate from &lt;math&gt;\mathcal D&lt;/math&gt;:<br /> $$F(x) \approx \hat F(x) = \frac{||c^{\mathcal D}(x)||_1}{N}$$<br /> $$F_x(y) \approx \hat F_x(y) = \frac{c^{\mathcal D}(x)}{|| c^{\mathcal D}(x) ||_1}$$<br /> $$\to l_{KL}(W) = \sum_{x \in \mathcal D} \frac{||c^{\mathcal D}(x)||_1}{N} KL \left( \frac{c^{\mathcal D}(x)}{||c^{\mathcal D}(x)||_1} \;||\; \phi(f^W(x)) \right)$$<br /> The approximations &lt;math&gt;\hat F, \hat F_X&lt;/math&gt; are often not very good though: consider a typical classification such as MNIST, we would never expect two handwritten digits to produce the exact same image. Hence &lt;math&gt;c^{\mathcal D}(x)&lt;/math&gt; is (almost) always going to have a single index 1 and the rest 0. This has implications for our approximations:<br /> $$\hat F(x) \text{ is uniform for all } x \in \mathcal D$$<br /> $$\hat F_x(y) \text{ is degenerate for all } x \in \mathcal D$$<br /> This clearly has implications for overfitting: to minimize the KL term in &lt;math&gt;l_{KL}(W)&lt;/math&gt; we want &lt;math&gt;\phi(f^W(x))&lt;/math&gt; to be very close to &lt;math&gt;\hat F_x(y)&lt;/math&gt; at each point - this means that the loss function is in fact encouraging the neural network to output near degenerate distributions! <br /> <br /> '''Labal Smoothing'''<br /> One form of regularization to help this problem is called label smoothing. Instead of using the degenerate $$F_x(y)$$ as a target function, let's &quot;smooth&quot; it (by adding a scaled uniform distribution to it) so it's no longer degenerate:<br /> $$F'_x(y) = (1-\lambda)\hat F_x(y) + \frac \lambda K \vec 1$$<br /> <br /> '''BNNs'''<br /> BBNs balances the complexity of the model and the distance to target distribution without choosing a single beset configuration (one-hot encoding). Specifically, BNNs with the Gaussian Weight prior $$F_x(y) = N (0,T^{-1} I)$$ has score of configuration W measured by the posterior density $$p_W(W|D) = p(D|W)p_W(W), \log(p_W(W)) = T||W||^2_2$$<br /> Here ||W||^2_2 could be a poor proxy to penalized for the model complexity due to its linear nature.<br /> <br /> == Method ==<br /> The main technical proposal of the paper is a Bayesian framework to estimate the (former) target distribution &lt;math&gt;F_x(y)&lt;/math&gt;. That is, we construct a posterior distribution of &lt;math&gt; F_x(y) &lt;/math&gt; and use that as our new target distribution. We call it the ''belief matching'' (BM) framework.<br /> <br /> === Constructing Target Distribution ===<br /> Recall that &lt;math&gt;F_x(y)&lt;/math&gt; is a k-categorical probability distribution - it's PMF can be fully characterized by k numbers that sum to 1. Hence we can encode any such &lt;math&gt;F_x&lt;/math&gt; as a point in &lt;math&gt;\Delta^{k-1}&lt;/math&gt;. We'll do exactly that - let's call this vecor &lt;math&gt;z&lt;/math&gt;:<br /> $$z \in \Delta^{k-1}$$<br /> $$\text{prior} = p_{z|x}(z)$$<br /> $$\text{conditional} = p_{y|z,x}(y)$$<br /> $$\text{posterior} = p_{z|x,y}(z)$$<br /> Then if we perform inference:<br /> $$p_{z|x,y}(z) \propto p_{z|x}(z)p_{y|z,x}(y)$$<br /> The distribution chosen to model prior was &lt;math&gt;dir_K(\beta)&lt;/math&gt;:<br /> $$p_{z|x}(z) = \frac{\Gamma(||\beta||_1)}{\prod_{k=1}^K \Gamma(\beta_k)} \prod_{k=1}^K z_k^{\beta_k - 1}$$<br /> Note that by definition of &lt;math&gt;z&lt;/math&gt;: &lt;math&gt; p_{y|x,z} = z_y &lt;/math&gt;. Since the Dirichlet is a conjugate prior to categorical distributions we have a convenient form for the mean of the posterior:<br /> $$\bar{p_{z|x,y}}(z) = \frac{\beta + c^{\mathcal D}(x)}{||\beta + c^{\mathcal D}(x)||_1} \propto \beta + c^{\mathcal D}(x)$$<br /> This is in fact a generalization of (uniform) label smoothing (label smoothing is a special case where &lt;math&gt;\beta = \frac 1 K \vec{1} &lt;/math&gt;).<br /> <br /> === Representing Approximate Distribution ===<br /> Our new target distribution is &lt;math&gt;p_{z|x,y}(z)&lt;/math&gt; (as opposed to &lt;math&gt;F_x(y)&lt;/math&gt;). That is, we want to construct an interpretation of our neural network weights to construct a distribution with support in &lt;math&gt; \Delta^{K-1} &lt;/math&gt; - the NN can then be trained so this encoded distribution closely approximates &lt;math&gt;p_{z|x,y}&lt;/math&gt;. Let's denote the PMF of this encoded distribution &lt;math&gt;q_{z|x}^W&lt;/math&gt;. This is how the BM framework defines it:<br /> $$\alpha^W(x) := \exp(f^W(x))$$<br /> $$q_{z|x}^W(z) = \frac{\Gamma(||\alpha^W(x)||_1)}{\sum_{k=1}^K \Gamma(\alpha_k^W(x))} \prod_{k=1}^K z_{k}^{\alpha_k^W(x) - 1}$$<br /> $$\to Z^W_x \sim dir(\alpha^W(x))$$<br /> Apply &lt;math&gt;\log&lt;/math&gt; then &lt;math&gt;\exp&lt;/math&gt; to &lt;math&gt;q_{z|x}^W&lt;/math&gt;:<br /> $$q^W_{z|x}(z) \propto \exp \left( \sum_k (\alpha_k^W(x) \log(z_k)) - \sum_k \log(z_k) \right)$$<br /> $$\propto -l_{CE}(\phi(f^W(x)),z) + \frac{K}{||\alpha^W(x)||}KL(\mathcal U_k \;||\; z)$$<br /> It can actually be shown that the mean of &lt;math&gt;Z_x^W&lt;/math&gt; is identical to &lt;math&gt;\phi(f^W(x))&lt;/math&gt; - in other words, if we output the mean of the encoded distribution of our neural network under the BM framework, it is theoretically identical to a traditional neural network.<br /> <br /> === Distribution Matching ===<br /> <br /> We now need a way to fit our approximate distribution from our neural network &lt;math&gt;q_{\mathbf{z | x}}^{\mathbf{W}}&lt;/math&gt; to our target distribution &lt;math&gt;p_{\mathbf{z|x},y}&lt;/math&gt;. The authors achieve this by maximizing the evidence lower bound (ELBO):<br /> <br /> $$l_{EB}(\mathbf y, \alpha^{\mathbf W}(\mathbf x)) = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] - KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; p_{\mathbf{z|x}})$$<br /> <br /> Each term can be computed analytically:<br /> <br /> $$\mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf W }} \left[\log z_y \right] = \psi(\alpha_y^{\mathbf W} ( \mathbf x )) - \psi(\alpha_0^{\mathbf W} ( \mathbf x ))$$<br /> <br /> Where &lt;math&gt;\psi(\cdot)&lt;/math&gt; represents the digamma function (logarithmic derivative of gamma function). Intuitively, we maximize the probability of the correct label. For the KL term:<br /> <br /> $$KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; p_{\mathbf{z|x}}) = \log \frac{\Gamma(a_0^{\mathbf W}(\mathbf x)) \prod_k \Gamma(\beta_k)}{\prod_k \Gamma(\alpha_k^{\mathbf W}(x)) \Gamma (\beta_0)} + \sum_k (\alpha_k^{\mathbf W}(x)-\beta_k)(\psi(\alpha_k^{\mathbf W}(\mathbf x)) - \psi(\alpha_0^{\mathbf W}(\mathbf x))$$<br /> <br /> In the first term, for intuition, we can ignore &lt;math&gt;\alpha_0&lt;/math&gt; and &lt;math&gt;\beta_0&lt;/math&gt; since those just calibrate the distributions. Otherwise, we want the ratio of the products to be as close to 1 as possible to minimize the KL. In the second term, we want to minimize the difference between each individual &lt;math&gt;\alpha_k&lt;/math&gt; and &lt;math&gt;\beta_k&lt;/math&gt;, scaled by the normalized output of the neural network. <br /> <br /> This loss function can be used as a drop-in replacement for the standard softmax cross-entropy, as it has an analytic form and the same time complexity as typical softmax-cross entropy with respect to the number of classes (&lt;math&gt;O(K)&lt;/math&gt;).<br /> <br /> === On Prior Distributions ===<br /> <br /> We must choose our concentration parameter, &lt;math&gt;\beta&lt;/math&gt;, for our dirichlet prior. We see our prior essentially disappears as &lt;math&gt;\beta_0 \to 0&lt;/math&gt; and becomes stronger as &lt;math&gt;\beta_0 \to \infty&lt;/math&gt;. Thus, we want a small &lt;math&gt;\beta_0&lt;/math&gt; so the posterior isn't dominated by the prior. But, the authors claim that a small &lt;math&gt;\beta_0&lt;/math&gt; makes &lt;math&gt;\alpha_0^{\mathbf W}(\mathbf x)&lt;/math&gt; small, which causes &lt;math&gt;\psi (\alpha_0^{\mathbf W}(\mathbf x))&lt;/math&gt; to be large, which is problematic for gradient based optimization. In practice, many neural network techniques aim to make &lt;math&gt;\mathbb E [f^{\mathbf W} (\mathbf x)] \approx \mathbf 0&lt;/math&gt; and thus &lt;math&gt;\mathbb E [\alpha^{\mathbf W} (\mathbf x)] \approx \mathbf 1&lt;/math&gt;, which means making &lt;math&gt;\alpha_0^{\mathbf W}(\mathbf x)&lt;/math&gt; small can be counterproductive.<br /> <br /> So, the authors set &lt;math&gt;\beta = \mathbf 1&lt;/math&gt; and introduce a new hyperparameter &lt;math&gt;\lambda&lt;/math&gt; which is multiplied with the KL term in the ELBO:<br /> <br /> $$l^\lambda_{EB}(\mathbf y, \alpha^{\mathbf W}(\mathbf x)) = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] - \lambda KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; \mathcal P^D (\mathbf 1))$$<br /> <br /> This stabilizes the optimization, as we can tell from the gradients:<br /> <br /> $$\frac{\partial l_{E B}\left(\mathbf{y}, \alpha^{\mathbf W}(\mathbf{x})\right)}{\partial \alpha_{k}^{\mathbf W}(\mathbf {x})}=\left(\tilde{\mathbf{y}}_{k}-\left(\alpha_{k}^{\mathbf W}(\mathbf{x})-\beta_{k}\right)\right) \psi^{\prime}\left(\alpha_{k}^{\mathbf{W}}(\boldsymbol{x})\right)<br /> -\left(1-\left(\alpha_{0}^{\boldsymbol{W}}(\boldsymbol{x})-\beta_{0}\right)\right) \psi^{\prime}\left(\alpha_{0}^{\boldsymbol{W}}(\boldsymbol{x})\right)$$<br /> <br /> $$\frac{\partial l_{E B}^{\lambda}\left(\mathbf{y}, \alpha^{\mathbf{W}}(\mathbf{x})\right)}{\partial \alpha_{k}^{W}(\mathbf{x})}=\left(\tilde{\mathbf{y}}_{k}-\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})-\lambda\right)\right) \frac{\psi^{\prime}\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})\right)}{\psi^{\prime}\left(\tilde{\alpha}_{0}^{\mathbf W}(\mathbf{x})\right)}<br /> -\left(1-\left(\tilde{\alpha}_{0}^{W}(\mathbf{x})-\lambda K\right)\right)$$<br /> <br /> As we can see, the first expression is affected by the magnitude of &lt;math&gt;\alpha^{\boldsymbol{W}}(\boldsymbol{x})&lt;/math&gt;, whereas the second expression is not due to the &lt;math&gt;\frac{\psi^{\prime}\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})\right)}{\psi^{\prime}\left(\tilde{\alpha}_{0}^{\mathbf W}(\mathbf{x})\right)}&lt;/math&gt; ratio.<br /> <br /> == Experiments ==<br /> <br /> Throughout the experiments in this paper, the authors employ various models based on residual connections (He et al., 2016 ) which are the models used for benchmarking in practice. We will first demonstrate improvements provided by BM, then we will show versatility in other applications. For fairness of comparisons, all configurations in the reference implementation will be fixed. The only additions in the experiments are initial learning rate warm-up and gradient clipping which are extremely helpful for stable training of BM. <br /> <br /> === Generalization performance === <br /> The paper compares the generalization performance of BM with softmax and MC dropout on CIFAR-10 and CIFAR-100 benchmarks.<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_T1.png]]<br /> <br /> The next comparison was performed between BM and softmax on the ImageNet benchmark. <br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_T2.png]]<br /> <br /> For both datasets and In all configurations, BM achieves the best generalization and outperforms softmax and MC dropout.<br /> <br /> ===== Regularization effect of prior =====<br /> <br /> In theory, BM has 2 regularization effects:<br /> The prior distribution, which smooths the target posterior<br /> Averaging all of the possible categorical probabilities to compute the distribution matching loss<br /> The authors perform an ablation study to examine the 2 effects separately - removing the KL term in the ELBO removes the effect of the prior distribution.<br /> For ResNet-50 on CIFAR-100 and CIFAR-10 the resulting test error rates were 24.69% and 5.68% respectively. <br /> <br /> This demonstrates that both regularization effects are significant since just having one of them improves the generalization performance compared to the softmax baseline, and having both improves the performance even more.<br /> <br /> ===== Impact of &lt;math&gt;\beta&lt;/math&gt; =====<br /> <br /> The effect of β on generalization performance is studied by training ResNet-18 on CIFAR-10 by tuning the value of β on its own, as well as jointly with λ. It was found that robust generalization performance is obtained for β ∈ [&lt;math&gt;e^{−1}, e^4&lt;/math&gt;] when tuning β on its own; and β ∈ [&lt;math&gt;e^{−4}, e^{8}&lt;/math&gt;] when tuning β jointly with λ. The figure below shows a plot of the error rate with varying β.<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F3.png]]<br /> <br /> === Uncertainty Representation ===<br /> <br /> One of the big advantages of BM is the ability to represent uncertainty about the prediction. The authors evaluate the uncertainty representation on in-distribution (ID) and out-of-distribution (OOD) samples. <br /> <br /> ===== ID uncertainty =====<br /> <br /> For ID (in-distribution) samples, calibration performance is measured, which is a measure of how well the model’s confidence matches its actual accuracy. This measure can be visualized using reliability plots and quantified using a metric called expected calibration error (ECE). ECE is calculated by grouping predictions into M groups based on their confidence score and then finding the absolute difference between the average accuracy and average confidence for each group.<br /> The figure below is a reliability plot of ResNet-50 on CIFAR-10 and CIFAR-100 with 15 groups. It shows that BM has a significantly better calibration performance than softmax since the confidence matches the accuracy more closely (this is also reflected in the lower ECE).<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F4.png]]<br /> <br /> ===== OOD uncertainty =====<br /> <br /> Here, the authors quantify uncertainty using predictive entropy - the larger the predictive entropy, the larger the uncertainty about a prediction. <br /> <br /> The figure below is a density plot of the predictive entropy of ResNet-50 on CIFAR-10. It shows that BM provides significantly better uncertainty estimation compared to other methods since BM is the only method that has a clear peak of high predictive entropy for OOD samples which should have high uncertainty. <br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F5.png]]<br /> <br /> === Transfer learning ===<br /> <br /> Belief matching applies the Bayesian principle outside the neural network, which means it can easily be applied to already trained models. Thus, belief matching can be employed in transfer learning scenarios. The authors downloaded the ImageNet pre-trained ResNet-50 weights and fine-tuned the weights of the last linear layer for 100 epochs using an Adam optimizer.<br /> <br /> This table shows the test error rates from transfer learning on CIFAR-10, Food-101, and Cars datasets. Belief matching consistently performs better than softmax. <br /> <br /> [[File:being_bayesian_about_categorical_probability_transfer_learning.png]]<br /> <br /> Belief matching was also tested for the predictive uncertainty for out of dataset samples based on CIFAR-10 as the in distribution sample. Looking at the figure below, it is observed that belief matching significantly improves the uncertainty representation of pre-trained models by only fine-tuning the last layer’s weights. Note that belief matching confidently predicts examples in Cars since CIFAR-10 contains the object category automobiles. In comparison, softmax produces confident predictions on all datasets. Thus, belief matching could also be used to enhance the uncertainty representation ability of pre-trained models without sacrificing their generalization performance.<br /> <br /> [[File: being_bayesian_about_categorical_probability_transfer_learning_uncertainty.png]]<br /> <br /> === Semi-Supervised Learning ===<br /> <br /> Belief matching’s ability to allow neural networks to represent rich information in their predictions can be exploited to aid consistency based loss function for semi-supervised learning. Consistency-based loss functions use unlabelled samples to determine where to promote the robustness of predictions based on stochastic perturbations. This can be done by perturbing the inputs (which is the VAT model) or the networks (which is the pi-model). Both methods minimize the divergence between two categorical probabilities under some perturbations, thus belief matching can be used by the following replacements in the loss functions. The hope is that belief matching can provide better prediction consistencies using its Dirichlet distributions.<br /> <br /> [[File: being_bayesian_about_categorical_probability_semi_supervised_equation.png]]<br /> <br /> The results of training on ResNet28-2 with consistency based loss functions on CIFAR-10 are shown in this table. Belief matching does have lower classification error rates compared to using a softmax.<br /> <br /> [[File:being_bayesian_about_categorical_probability_semi_supervised_table.png]]<br /> <br /> == Conclusion ==<br /> <br /> Bayesian principles can be used to construct the target distribution by using the categorical probability as a random variable rather than a training label. This can be applied to neural network models by replacing only the softmax and cross-entropy loss, while improving the generalization performance and uncertainty estimation. <br /> <br /> In the future, the authors would like to allow for more expressive distributions in the belief matching framework, such as logistic normal distributions to capture strong semantic similarities among class labels. Furthermore, using input dependent priors would allow for interesting properties that would aid imbalanced datasets and multi-domain learning.<br /> <br /> == Citations ==<br /> <br />  Bridle, J. S. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pp. 227–236. Springer, 1990.<br /> <br />  Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural networks. In International Conference on Machine Learning, 2015.<br /> <br />  Gal, Y. and Ghahramani, Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.<br /> <br />  Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning, 2017. <br /> <br />  MacKay, D. J. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448– 472, 1992.<br /> <br />  Graves, A. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, 2011. <br /> <br />  Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate Bayesian inference. Journal of Machine Learning Research, 18(1):4873–4907, 2017.<br /> <br />  Zhang, G., Sun, S., Duvenaud, D., and Grosse, R. Noisy natural gradient as variational inference. In International Conference of Machine Learning, 2018.<br /> <br />  Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, 2019.<br /> <br />  Osawa, K., Swaroop, S., Jain, A., Eschenhagen, R., Turner, R. E., Yokota, R., and Khan, M. E. Practical deep learning with Bayesian principles. In Advances in Neural Information Processing Systems, 2019.<br /> <br />  Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.<br /> <br />  Neumann, L., Zisserman, A., and Vedaldi, A. Relaxed softmax: Efficient confidence auto-calibration for safe pedestrian detection. In NIPS Workshop on Machine Learning for Intelligent Transportation Systems, 2018.<br /> <br />  Xie, L., Wang, J., Wei, Z., Wang, M., and Tian, Q. Disturblabel: Regularizing cnn on the loss layer. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.<br /> <br />  Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., and Hinton, G. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Being_Bayesian_about_Categorical_Probability&diff=49056 Being Bayesian about Categorical Probability 2020-12-04T01:01:49Z <p>Z42qin: /* Classification With a Neural Network */</p> <hr /> <div>== Presented By ==<br /> Evan Li, Jason Pu, Karam Abuaisha, Nicholas Vadivelu<br /> <br /> == Introduction ==<br /> <br /> Since the outputs of neural networks are not probabilities, Softmax (Bridle, 1990) is a staple for neural network’s performing classification--it exponentiates each logit then normalizes by the sum, giving a distribution over the target classes. However, networks with softmax outputs give no information about uncertainty (Blundell et al., 2015; Gal &amp; Ghahramani, 2016), and the resulting distribution over classes is poorly calibrated (Guo et al., 2017), often giving overconfident predictions even when the classification is wrong. In addition, softmax also raises concerns about overfitting NNs due to its confident predictive behaviors(Xie et al., 2016; Pereyra et al., 2017). To achieve better generalization performance, this may require some effective regularization techniques. <br /> <br /> Bayesian Neural Networks (BNNs; MacKay, 1992) can alleviate these issues, but the resulting posteriors over the parameters are often intractable. Approximations such as variational inference (Graves, 2011; Blundell et al., 2015) and Monte Carlo Dropout (Gal &amp; Ghahramani, 2016) can still be expensive or give poor estimates for the posteriors. This work proposes a Bayesian treatment of the output logits of the neural network, treating the targets as a categorical random variable instead of a fixed label. This gives a computationally cheap way to get well-calibrated uncertainty estimates on neural network classifications.<br /> <br /> == Related Work ==<br /> <br /> Using Bayesian Neural Networks is the dominant way of applying Bayesian techniques to neural networks. Many techniques have been developed to make posterior approximation more accurate and scalable, despite these, BNNs do not scale to the state of the art techniques or large data sets. There are techniques to explicitly avoid modeling the full weight posterior that are more scalable, such as with Monte Carlo Dropout (Gal &amp; Ghahramani, 2016) or tracking mean/covariance of the posterior during training (Mandt et al., 2017; Zhang et al., 2018; Maddox et al., 2019; Osawa et al., 2019). Non-Bayesian uncertainty estimation techniques such as deep ensembles (Lakshminarayanan et al., 2017) and temperature scaling (Guo et al., 2017; Neumann et al., 2018).<br /> <br /> == Preliminaries ==<br /> === Definitions ===<br /> Let's formalize our classification problem and define some notations for the rest of this summary:<br /> <br /> ::Dataset:<br /> $$\mathcal D = \{(x_i,y_i)\} \in (\mathcal X \times \mathcal Y)^N$$<br /> ::General classification model<br /> $$f^W: \mathcal X \to \mathbb R^K$$<br /> ::Softmax function: <br /> $$\phi(x): \mathbb R^K \to [0,1]^K \;\;|\;\; \phi_k(X) = \frac{\exp(f_k^W(x))}{\sum_{k \in K} \exp(f_k^W(x))}$$<br /> ::Softmax activated NN:<br /> $$\phi \;\circ\; f^W: \chi \to \Delta^{K-1}$$<br /> ::NN as a true classifier:<br /> $$arg\max_i \;\circ\; \phi_i \;\circ\; f^W \;:\; \mathcal X \to \mathcal Y$$<br /> <br /> We'll also define the '''count function''' - a &lt;math&gt;K&lt;/math&gt;-vector valued function that outputs the occurences of each class coincident with &lt;math&gt;x&lt;/math&gt;:<br /> $$c^{\mathcal D}(x) = \sum_{(x',y') \in \mathcal D} \mathbb y' I(x' = x)$$<br /> <br /> === Classification With a Neural Network ===<br /> A typical loss function used in classification is cross-entropy. It's well known that optimizing &lt;math&gt;f^W&lt;/math&gt; for &lt;math&gt;l_{CE}&lt;/math&gt; is equivalent to optimizing for &lt;math&gt;l_{KL}&lt;/math&gt;, the &lt;math&gt;KL&lt;/math&gt; divergence between the true distribution and the distribution modeled by NN, that is:<br /> $$l_{KL}(W) = KL(\text{true distribution} \;|\; \text{distribution encoded by }NN(W))$$<br /> Let's introduce notations for the underlying (true) distributions of our problem. Let &lt;math&gt;(x_0,y_0) \sim (\mathcal X \times \mathcal Y)&lt;/math&gt;:<br /> $$\text{Full Distribution} = F(x,y) = P(x_0 = x,y_0 = y)$$<br /> $$\text{Marginal Distribution} = P(x) = F(x_0 = x)$$<br /> $$\text{Point Class Distribution} = P(y_0 = y \;|\; x_0 = x) = F_x(y)$$<br /> Then we have the following factorization:<br /> $$F(x,y) = P(x,y) = P(y|x)P(x) = F_x(y)F(x)$$<br /> Substitute this into the definition of KL divergence:<br /> $$= \sum_{(x,y) \in \mathcal X \times \mathcal Y} F(x,y) \log\left(\frac{F(x,y)}{\phi_y(f^W(x))}\right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) \sum_{y \in \mathcal Y} F(y|x) \log\left( \frac{F(y|x)}{\phi_y(f^W(x))} \right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) \sum_{y \in \mathcal Y} F_x(y) \log\left( \frac{F_x(y)}{\phi_y(f^W(x))} \right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) KL(F_x \;||\; \phi\left( f^W(x) \right))$$<br /> As usual, we don't have an analytic form for &lt;math&gt;l&lt;/math&gt; (if we did, this would imply we know &lt;math&gt;F_X&lt;/math&gt; meaning we knew the distribution in the first place). Instead, estimate from &lt;math&gt;\mathcal D&lt;/math&gt;:<br /> $$F(x) \approx \hat F(x) = \frac{||c^{\mathcal D}(x)||_1}{N}$$<br /> $$F_x(y) \approx \hat F_x(y) = \frac{c^{\mathcal D}(x)}{|| c^{\mathcal D}(x) ||_1}$$<br /> $$\to l_{KL}(W) = \sum_{x \in \mathcal D} \frac{||c^{\mathcal D}(x)||_1}{N} KL \left( \frac{c^{\mathcal D}(x)}{||c^{\mathcal D}(x)||_1} \;||\; \phi(f^W(x)) \right)$$<br /> The approximations &lt;math&gt;\hat F, \hat F_X&lt;/math&gt; are often not very good though: consider a typical classification such as MNIST, we would never expect two handwritten digits to produce the exact same image. Hence &lt;math&gt;c^{\mathcal D}(x)&lt;/math&gt; is (almost) always going to have a single index 1 and the rest 0. This has implications for our approximations:<br /> $$\hat F(x) \text{ is uniform for all } x \in \mathcal D$$<br /> $$\hat F_x(y) \text{ is degenerate for all } x \in \mathcal D$$<br /> This clearly has implications for overfitting: to minimize the KL term in &lt;math&gt;l_{KL}(W)&lt;/math&gt; we want &lt;math&gt;\phi(f^W(x))&lt;/math&gt; to be very close to &lt;math&gt;\hat F_x(y)&lt;/math&gt; at each point - this means that the loss function is in fact encouraging the neural network to output near degenerate distributions! <br /> <br /> '''Labal Smoothing'''<br /> One form of regularization to help this problem is called label smoothing. Instead of using the degenerate $$F_x(y)$$ as a target function, let's &quot;smooth&quot; it (by adding a scaled uniform distribution to it) so it's no longer degenerate:<br /> $$F'_x(y) = (1-\lambda)\hat F_x(y) + \frac \lambda K \vec 1$$<br /> <br /> '''BNNs'''<br /> BBNs balances the complexity of the model and the distance to target distribution without choosing a single beset configuration (one-hot encoding). Specifically, BNNs with the Gaussian Weight prior $$F_x(y) ~ N (0,T^{-1} I)$$ has score of configuration W measured by the posterior density $$p_W(W|D) ~ p(D|W)p_W(W), \log(p_W(W))~ T||W||^2_2$$<br /> <br /> == Method ==<br /> The main technical proposal of the paper is a Bayesian framework to estimate the (former) target distribution &lt;math&gt;F_x(y)&lt;/math&gt;. That is, we construct a posterior distribution of &lt;math&gt; F_x(y) &lt;/math&gt; and use that as our new target distribution. We call it the ''belief matching'' (BM) framework.<br /> <br /> === Constructing Target Distribution ===<br /> Recall that &lt;math&gt;F_x(y)&lt;/math&gt; is a k-categorical probability distribution - it's PMF can be fully characterized by k numbers that sum to 1. Hence we can encode any such &lt;math&gt;F_x&lt;/math&gt; as a point in &lt;math&gt;\Delta^{k-1}&lt;/math&gt;. We'll do exactly that - let's call this vecor &lt;math&gt;z&lt;/math&gt;:<br /> $$z \in \Delta^{k-1}$$<br /> $$\text{prior} = p_{z|x}(z)$$<br /> $$\text{conditional} = p_{y|z,x}(y)$$<br /> $$\text{posterior} = p_{z|x,y}(z)$$<br /> Then if we perform inference:<br /> $$p_{z|x,y}(z) \propto p_{z|x}(z)p_{y|z,x}(y)$$<br /> The distribution chosen to model prior was &lt;math&gt;dir_K(\beta)&lt;/math&gt;:<br /> $$p_{z|x}(z) = \frac{\Gamma(||\beta||_1)}{\prod_{k=1}^K \Gamma(\beta_k)} \prod_{k=1}^K z_k^{\beta_k - 1}$$<br /> Note that by definition of &lt;math&gt;z&lt;/math&gt;: &lt;math&gt; p_{y|x,z} = z_y &lt;/math&gt;. Since the Dirichlet is a conjugate prior to categorical distributions we have a convenient form for the mean of the posterior:<br /> $$\bar{p_{z|x,y}}(z) = \frac{\beta + c^{\mathcal D}(x)}{||\beta + c^{\mathcal D}(x)||_1} \propto \beta + c^{\mathcal D}(x)$$<br /> This is in fact a generalization of (uniform) label smoothing (label smoothing is a special case where &lt;math&gt;\beta = \frac 1 K \vec{1} &lt;/math&gt;).<br /> <br /> === Representing Approximate Distribution ===<br /> Our new target distribution is &lt;math&gt;p_{z|x,y}(z)&lt;/math&gt; (as opposed to &lt;math&gt;F_x(y)&lt;/math&gt;). That is, we want to construct an interpretation of our neural network weights to construct a distribution with support in &lt;math&gt; \Delta^{K-1} &lt;/math&gt; - the NN can then be trained so this encoded distribution closely approximates &lt;math&gt;p_{z|x,y}&lt;/math&gt;. Let's denote the PMF of this encoded distribution &lt;math&gt;q_{z|x}^W&lt;/math&gt;. This is how the BM framework defines it:<br /> $$\alpha^W(x) := \exp(f^W(x))$$<br /> $$q_{z|x}^W(z) = \frac{\Gamma(||\alpha^W(x)||_1)}{\sum_{k=1}^K \Gamma(\alpha_k^W(x))} \prod_{k=1}^K z_{k}^{\alpha_k^W(x) - 1}$$<br /> $$\to Z^W_x \sim dir(\alpha^W(x))$$<br /> Apply &lt;math&gt;\log&lt;/math&gt; then &lt;math&gt;\exp&lt;/math&gt; to &lt;math&gt;q_{z|x}^W&lt;/math&gt;:<br /> $$q^W_{z|x}(z) \propto \exp \left( \sum_k (\alpha_k^W(x) \log(z_k)) - \sum_k \log(z_k) \right)$$<br /> $$\propto -l_{CE}(\phi(f^W(x)),z) + \frac{K}{||\alpha^W(x)||}KL(\mathcal U_k \;||\; z)$$<br /> It can actually be shown that the mean of &lt;math&gt;Z_x^W&lt;/math&gt; is identical to &lt;math&gt;\phi(f^W(x))&lt;/math&gt; - in other words, if we output the mean of the encoded distribution of our neural network under the BM framework, it is theoretically identical to a traditional neural network.<br /> <br /> === Distribution Matching ===<br /> <br /> We now need a way to fit our approximate distribution from our neural network &lt;math&gt;q_{\mathbf{z | x}}^{\mathbf{W}}&lt;/math&gt; to our target distribution &lt;math&gt;p_{\mathbf{z|x},y}&lt;/math&gt;. The authors achieve this by maximizing the evidence lower bound (ELBO):<br /> <br /> $$l_{EB}(\mathbf y, \alpha^{\mathbf W}(\mathbf x)) = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] - KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; p_{\mathbf{z|x}})$$<br /> <br /> Each term can be computed analytically:<br /> <br /> $$\mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf W }} \left[\log z_y \right] = \psi(\alpha_y^{\mathbf W} ( \mathbf x )) - \psi(\alpha_0^{\mathbf W} ( \mathbf x ))$$<br /> <br /> Where &lt;math&gt;\psi(\cdot)&lt;/math&gt; represents the digamma function (logarithmic derivative of gamma function). Intuitively, we maximize the probability of the correct label. For the KL term:<br /> <br /> $$KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; p_{\mathbf{z|x}}) = \log \frac{\Gamma(a_0^{\mathbf W}(\mathbf x)) \prod_k \Gamma(\beta_k)}{\prod_k \Gamma(\alpha_k^{\mathbf W}(x)) \Gamma (\beta_0)} + \sum_k (\alpha_k^{\mathbf W}(x)-\beta_k)(\psi(\alpha_k^{\mathbf W}(\mathbf x)) - \psi(\alpha_0^{\mathbf W}(\mathbf x))$$<br /> <br /> In the first term, for intuition, we can ignore &lt;math&gt;\alpha_0&lt;/math&gt; and &lt;math&gt;\beta_0&lt;/math&gt; since those just calibrate the distributions. Otherwise, we want the ratio of the products to be as close to 1 as possible to minimize the KL. In the second term, we want to minimize the difference between each individual &lt;math&gt;\alpha_k&lt;/math&gt; and &lt;math&gt;\beta_k&lt;/math&gt;, scaled by the normalized output of the neural network. <br /> <br /> This loss function can be used as a drop-in replacement for the standard softmax cross-entropy, as it has an analytic form and the same time complexity as typical softmax-cross entropy with respect to the number of classes (&lt;math&gt;O(K)&lt;/math&gt;).<br /> <br /> === On Prior Distributions ===<br /> <br /> We must choose our concentration parameter, &lt;math&gt;\beta&lt;/math&gt;, for our dirichlet prior. We see our prior essentially disappears as &lt;math&gt;\beta_0 \to 0&lt;/math&gt; and becomes stronger as &lt;math&gt;\beta_0 \to \infty&lt;/math&gt;. Thus, we want a small &lt;math&gt;\beta_0&lt;/math&gt; so the posterior isn't dominated by the prior. But, the authors claim that a small &lt;math&gt;\beta_0&lt;/math&gt; makes &lt;math&gt;\alpha_0^{\mathbf W}(\mathbf x)&lt;/math&gt; small, which causes &lt;math&gt;\psi (\alpha_0^{\mathbf W}(\mathbf x))&lt;/math&gt; to be large, which is problematic for gradient based optimization. In practice, many neural network techniques aim to make &lt;math&gt;\mathbb E [f^{\mathbf W} (\mathbf x)] \approx \mathbf 0&lt;/math&gt; and thus &lt;math&gt;\mathbb E [\alpha^{\mathbf W} (\mathbf x)] \approx \mathbf 1&lt;/math&gt;, which means making &lt;math&gt;\alpha_0^{\mathbf W}(\mathbf x)&lt;/math&gt; small can be counterproductive.<br /> <br /> So, the authors set &lt;math&gt;\beta = \mathbf 1&lt;/math&gt; and introduce a new hyperparameter &lt;math&gt;\lambda&lt;/math&gt; which is multiplied with the KL term in the ELBO:<br /> <br /> $$l^\lambda_{EB}(\mathbf y, \alpha^{\mathbf W}(\mathbf x)) = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] - \lambda KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; \mathcal P^D (\mathbf 1))$$<br /> <br /> This stabilizes the optimization, as we can tell from the gradients:<br /> <br /> $$\frac{\partial l_{E B}\left(\mathbf{y}, \alpha^{\mathbf W}(\mathbf{x})\right)}{\partial \alpha_{k}^{\mathbf W}(\mathbf {x})}=\left(\tilde{\mathbf{y}}_{k}-\left(\alpha_{k}^{\mathbf W}(\mathbf{x})-\beta_{k}\right)\right) \psi^{\prime}\left(\alpha_{k}^{\mathbf{W}}(\boldsymbol{x})\right)<br /> -\left(1-\left(\alpha_{0}^{\boldsymbol{W}}(\boldsymbol{x})-\beta_{0}\right)\right) \psi^{\prime}\left(\alpha_{0}^{\boldsymbol{W}}(\boldsymbol{x})\right)$$<br /> <br /> $$\frac{\partial l_{E B}^{\lambda}\left(\mathbf{y}, \alpha^{\mathbf{W}}(\mathbf{x})\right)}{\partial \alpha_{k}^{W}(\mathbf{x})}=\left(\tilde{\mathbf{y}}_{k}-\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})-\lambda\right)\right) \frac{\psi^{\prime}\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})\right)}{\psi^{\prime}\left(\tilde{\alpha}_{0}^{\mathbf W}(\mathbf{x})\right)}<br /> -\left(1-\left(\tilde{\alpha}_{0}^{W}(\mathbf{x})-\lambda K\right)\right)$$<br /> <br /> As we can see, the first expression is affected by the magnitude of &lt;math&gt;\alpha^{\boldsymbol{W}}(\boldsymbol{x})&lt;/math&gt;, whereas the second expression is not due to the &lt;math&gt;\frac{\psi^{\prime}\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})\right)}{\psi^{\prime}\left(\tilde{\alpha}_{0}^{\mathbf W}(\mathbf{x})\right)}&lt;/math&gt; ratio.<br /> <br /> == Experiments ==<br /> <br /> Throughout the experiments in this paper, the authors employ various models based on residual connections (He et al., 2016 ) which are the models used for benchmarking in practice. We will first demonstrate improvements provided by BM, then we will show versatility in other applications. For fairness of comparisons, all configurations in the reference implementation will be fixed. The only additions in the experiments are initial learning rate warm-up and gradient clipping which are extremely helpful for stable training of BM. <br /> <br /> === Generalization performance === <br /> The paper compares the generalization performance of BM with softmax and MC dropout on CIFAR-10 and CIFAR-100 benchmarks.<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_T1.png]]<br /> <br /> The next comparison was performed between BM and softmax on the ImageNet benchmark. <br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_T2.png]]<br /> <br /> For both datasets and In all configurations, BM achieves the best generalization and outperforms softmax and MC dropout.<br /> <br /> ===== Regularization effect of prior =====<br /> <br /> In theory, BM has 2 regularization effects:<br /> The prior distribution, which smooths the target posterior<br /> Averaging all of the possible categorical probabilities to compute the distribution matching loss<br /> The authors perform an ablation study to examine the 2 effects separately - removing the KL term in the ELBO removes the effect of the prior distribution.<br /> For ResNet-50 on CIFAR-100 and CIFAR-10 the resulting test error rates were 24.69% and 5.68% respectively. <br /> <br /> This demonstrates that both regularization effects are significant since just having one of them improves the generalization performance compared to the softmax baseline, and having both improves the performance even more.<br /> <br /> ===== Impact of &lt;math&gt;\beta&lt;/math&gt; =====<br /> <br /> The effect of β on generalization performance is studied by training ResNet-18 on CIFAR-10 by tuning the value of β on its own, as well as jointly with λ. It was found that robust generalization performance is obtained for β ∈ [&lt;math&gt;e^{−1}, e^4&lt;/math&gt;] when tuning β on its own; and β ∈ [&lt;math&gt;e^{−4}, e^{8}&lt;/math&gt;] when tuning β jointly with λ. The figure below shows a plot of the error rate with varying β.<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F3.png]]<br /> <br /> === Uncertainty Representation ===<br /> <br /> One of the big advantages of BM is the ability to represent uncertainty about the prediction. The authors evaluate the uncertainty representation on in-distribution (ID) and out-of-distribution (OOD) samples. <br /> <br /> ===== ID uncertainty =====<br /> <br /> For ID (in-distribution) samples, calibration performance is measured, which is a measure of how well the model’s confidence matches its actual accuracy. This measure can be visualized using reliability plots and quantified using a metric called expected calibration error (ECE). ECE is calculated by grouping predictions into M groups based on their confidence score and then finding the absolute difference between the average accuracy and average confidence for each group.<br /> The figure below is a reliability plot of ResNet-50 on CIFAR-10 and CIFAR-100 with 15 groups. It shows that BM has a significantly better calibration performance than softmax since the confidence matches the accuracy more closely (this is also reflected in the lower ECE).<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F4.png]]<br /> <br /> ===== OOD uncertainty =====<br /> <br /> Here, the authors quantify uncertainty using predictive entropy - the larger the predictive entropy, the larger the uncertainty about a prediction. <br /> <br /> The figure below is a density plot of the predictive entropy of ResNet-50 on CIFAR-10. It shows that BM provides significantly better uncertainty estimation compared to other methods since BM is the only method that has a clear peak of high predictive entropy for OOD samples which should have high uncertainty. <br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F5.png]]<br /> <br /> === Transfer learning ===<br /> <br /> Belief matching applies the Bayesian principle outside the neural network, which means it can easily be applied to already trained models. Thus, belief matching can be employed in transfer learning scenarios. The authors downloaded the ImageNet pre-trained ResNet-50 weights and fine-tuned the weights of the last linear layer for 100 epochs using an Adam optimizer.<br /> <br /> This table shows the test error rates from transfer learning on CIFAR-10, Food-101, and Cars datasets. Belief matching consistently performs better than softmax. <br /> <br /> [[File:being_bayesian_about_categorical_probability_transfer_learning.png]]<br /> <br /> Belief matching was also tested for the predictive uncertainty for out of dataset samples based on CIFAR-10 as the in distribution sample. Looking at the figure below, it is observed that belief matching significantly improves the uncertainty representation of pre-trained models by only fine-tuning the last layer’s weights. Note that belief matching confidently predicts examples in Cars since CIFAR-10 contains the object category automobiles. In comparison, softmax produces confident predictions on all datasets. Thus, belief matching could also be used to enhance the uncertainty representation ability of pre-trained models without sacrificing their generalization performance.<br /> <br /> [[File: being_bayesian_about_categorical_probability_transfer_learning_uncertainty.png]]<br /> <br /> === Semi-Supervised Learning ===<br /> <br /> Belief matching’s ability to allow neural networks to represent rich information in their predictions can be exploited to aid consistency based loss function for semi-supervised learning. Consistency-based loss functions use unlabelled samples to determine where to promote the robustness of predictions based on stochastic perturbations. This can be done by perturbing the inputs (which is the VAT model) or the networks (which is the pi-model). Both methods minimize the divergence between two categorical probabilities under some perturbations, thus belief matching can be used by the following replacements in the loss functions. The hope is that belief matching can provide better prediction consistencies using its Dirichlet distributions.<br /> <br /> [[File: being_bayesian_about_categorical_probability_semi_supervised_equation.png]]<br /> <br /> The results of training on ResNet28-2 with consistency based loss functions on CIFAR-10 are shown in this table. Belief matching does have lower classification error rates compared to using a softmax.<br /> <br /> [[File:being_bayesian_about_categorical_probability_semi_supervised_table.png]]<br /> <br /> == Conclusion ==<br /> <br /> Bayesian principles can be used to construct the target distribution by using the categorical probability as a random variable rather than a training label. This can be applied to neural network models by replacing only the softmax and cross-entropy loss, while improving the generalization performance and uncertainty estimation. <br /> <br /> In the future, the authors would like to allow for more expressive distributions in the belief matching framework, such as logistic normal distributions to capture strong semantic similarities among class labels. Furthermore, using input dependent priors would allow for interesting properties that would aid imbalanced datasets and multi-domain learning.<br /> <br /> == Citations ==<br /> <br />  Bridle, J. S. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pp. 227–236. Springer, 1990.<br /> <br />  Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural networks. In International Conference on Machine Learning, 2015.<br /> <br />  Gal, Y. and Ghahramani, Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.<br /> <br />  Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning, 2017. <br /> <br />  MacKay, D. J. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448– 472, 1992.<br /> <br />  Graves, A. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, 2011. <br /> <br />  Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate Bayesian inference. Journal of Machine Learning Research, 18(1):4873–4907, 2017.<br /> <br />  Zhang, G., Sun, S., Duvenaud, D., and Grosse, R. Noisy natural gradient as variational inference. In International Conference of Machine Learning, 2018.<br /> <br />  Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, 2019.<br /> <br />  Osawa, K., Swaroop, S., Jain, A., Eschenhagen, R., Turner, R. E., Yokota, R., and Khan, M. E. Practical deep learning with Bayesian principles. In Advances in Neural Information Processing Systems, 2019.<br /> <br />  Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.<br /> <br />  Neumann, L., Zisserman, A., and Vedaldi, A. Relaxed softmax: Efficient confidence auto-calibration for safe pedestrian detection. In NIPS Workshop on Machine Learning for Intelligent Transportation Systems, 2018.<br /> <br />  Xie, L., Wang, J., Wei, Z., Wang, M., and Tian, Q. Disturblabel: Regularizing cnn on the loss layer. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.<br /> <br />  Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., and Hinton, G. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Being_Bayesian_about_Categorical_Probability&diff=49055 Being Bayesian about Categorical Probability 2020-12-04T01:01:33Z <p>Z42qin: /* Classification With a Neural Network */</p> <hr /> <div>== Presented By ==<br /> Evan Li, Jason Pu, Karam Abuaisha, Nicholas Vadivelu<br /> <br /> == Introduction ==<br /> <br /> Since the outputs of neural networks are not probabilities, Softmax (Bridle, 1990) is a staple for neural network’s performing classification--it exponentiates each logit then normalizes by the sum, giving a distribution over the target classes. However, networks with softmax outputs give no information about uncertainty (Blundell et al., 2015; Gal &amp; Ghahramani, 2016), and the resulting distribution over classes is poorly calibrated (Guo et al., 2017), often giving overconfident predictions even when the classification is wrong. In addition, softmax also raises concerns about overfitting NNs due to its confident predictive behaviors(Xie et al., 2016; Pereyra et al., 2017). To achieve better generalization performance, this may require some effective regularization techniques. <br /> <br /> Bayesian Neural Networks (BNNs; MacKay, 1992) can alleviate these issues, but the resulting posteriors over the parameters are often intractable. Approximations such as variational inference (Graves, 2011; Blundell et al., 2015) and Monte Carlo Dropout (Gal &amp; Ghahramani, 2016) can still be expensive or give poor estimates for the posteriors. This work proposes a Bayesian treatment of the output logits of the neural network, treating the targets as a categorical random variable instead of a fixed label. This gives a computationally cheap way to get well-calibrated uncertainty estimates on neural network classifications.<br /> <br /> == Related Work ==<br /> <br /> Using Bayesian Neural Networks is the dominant way of applying Bayesian techniques to neural networks. Many techniques have been developed to make posterior approximation more accurate and scalable, despite these, BNNs do not scale to the state of the art techniques or large data sets. There are techniques to explicitly avoid modeling the full weight posterior that are more scalable, such as with Monte Carlo Dropout (Gal &amp; Ghahramani, 2016) or tracking mean/covariance of the posterior during training (Mandt et al., 2017; Zhang et al., 2018; Maddox et al., 2019; Osawa et al., 2019). Non-Bayesian uncertainty estimation techniques such as deep ensembles (Lakshminarayanan et al., 2017) and temperature scaling (Guo et al., 2017; Neumann et al., 2018).<br /> <br /> == Preliminaries ==<br /> === Definitions ===<br /> Let's formalize our classification problem and define some notations for the rest of this summary:<br /> <br /> ::Dataset:<br /> $$\mathcal D = \{(x_i,y_i)\} \in (\mathcal X \times \mathcal Y)^N$$<br /> ::General classification model<br /> $$f^W: \mathcal X \to \mathbb R^K$$<br /> ::Softmax function: <br /> $$\phi(x): \mathbb R^K \to [0,1]^K \;\;|\;\; \phi_k(X) = \frac{\exp(f_k^W(x))}{\sum_{k \in K} \exp(f_k^W(x))}$$<br /> ::Softmax activated NN:<br /> $$\phi \;\circ\; f^W: \chi \to \Delta^{K-1}$$<br /> ::NN as a true classifier:<br /> $$arg\max_i \;\circ\; \phi_i \;\circ\; f^W \;:\; \mathcal X \to \mathcal Y$$<br /> <br /> We'll also define the '''count function''' - a &lt;math&gt;K&lt;/math&gt;-vector valued function that outputs the occurences of each class coincident with &lt;math&gt;x&lt;/math&gt;:<br /> $$c^{\mathcal D}(x) = \sum_{(x',y') \in \mathcal D} \mathbb y' I(x' = x)$$<br /> <br /> === Classification With a Neural Network ===<br /> A typical loss function used in classification is cross-entropy. It's well known that optimizing &lt;math&gt;f^W&lt;/math&gt; for &lt;math&gt;l_{CE}&lt;/math&gt; is equivalent to optimizing for &lt;math&gt;l_{KL}&lt;/math&gt;, the &lt;math&gt;KL&lt;/math&gt; divergence between the true distribution and the distribution modeled by NN, that is:<br /> $$l_{KL}(W) = KL(\text{true distribution} \;|\; \text{distribution encoded by }NN(W))$$<br /> Let's introduce notations for the underlying (true) distributions of our problem. Let &lt;math&gt;(x_0,y_0) \sim (\mathcal X \times \mathcal Y)&lt;/math&gt;:<br /> $$\text{Full Distribution} = F(x,y) = P(x_0 = x,y_0 = y)$$<br /> $$\text{Marginal Distribution} = P(x) = F(x_0 = x)$$<br /> $$\text{Point Class Distribution} = P(y_0 = y \;|\; x_0 = x) = F_x(y)$$<br /> Then we have the following factorization:<br /> $$F(x,y) = P(x,y) = P(y|x)P(x) = F_x(y)F(x)$$<br /> Substitute this into the definition of KL divergence:<br /> $$= \sum_{(x,y) \in \mathcal X \times \mathcal Y} F(x,y) \log\left(\frac{F(x,y)}{\phi_y(f^W(x))}\right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) \sum_{y \in \mathcal Y} F(y|x) \log\left( \frac{F(y|x)}{\phi_y(f^W(x))} \right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) \sum_{y \in \mathcal Y} F_x(y) \log\left( \frac{F_x(y)}{\phi_y(f^W(x))} \right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) KL(F_x \;||\; \phi\left( f^W(x) \right))$$<br /> As usual, we don't have an analytic form for &lt;math&gt;l&lt;/math&gt; (if we did, this would imply we know &lt;math&gt;F_X&lt;/math&gt; meaning we knew the distribution in the first place). Instead, estimate from &lt;math&gt;\mathcal D&lt;/math&gt;:<br /> $$F(x) \approx \hat F(x) = \frac{||c^{\mathcal D}(x)||_1}{N}$$<br /> $$F_x(y) \approx \hat F_x(y) = \frac{c^{\mathcal D}(x)}{|| c^{\mathcal D}(x) ||_1}$$<br /> $$\to l_{KL}(W) = \sum_{x \in \mathcal D} \frac{||c^{\mathcal D}(x)||_1}{N} KL \left( \frac{c^{\mathcal D}(x)}{||c^{\mathcal D}(x)||_1} \;||\; \phi(f^W(x)) \right)$$<br /> The approximations &lt;math&gt;\hat F, \hat F_X&lt;/math&gt; are often not very good though: consider a typical classification such as MNIST, we would never expect two handwritten digits to produce the exact same image. Hence &lt;math&gt;c^{\mathcal D}(x)&lt;/math&gt; is (almost) always going to have a single index 1 and the rest 0. This has implications for our approximations:<br /> $$\hat F(x) \text{ is uniform for all } x \in \mathcal D$$<br /> $$\hat F_x(y) \text{ is degenerate for all } x \in \mathcal D$$<br /> This clearly has implications for overfitting: to minimize the KL term in &lt;math&gt;l_{KL}(W)&lt;/math&gt; we want &lt;math&gt;\phi(f^W(x))&lt;/math&gt; to be very close to &lt;math&gt;\hat F_x(y)&lt;/math&gt; at each point - this means that the loss function is in fact encouraging the neural network to output near degenerate distributions! <br /> <br /> '''Labal Smoothing'''<br /> One form of regularization to help this problem is called label smoothing. Instead of using the degenerate $$F_x(y)$$ as a target function, let's &quot;smooth&quot; it (by adding a scaled uniform distribution to it) so it's no longer degenerate:<br /> $$F'_x(y) = (1-\lambda)\hat F_x(y) + \frac \lambda K \vec 1$$<br /> <br /> '''BNNs'''<br /> BBNs balances the complexity of the model and the distance to target distribution without choosing a single beset configuration (one-hot encoding). Specifically, BNNs with the Gaussian Weight prior $$F_x(y) ~ N (0,T^{-1} I)$$ has score of configuration W measured by the posterior density $$p_W(W|D) ~ p(D|W)p_W(W), \log(p_W(W))~ T||W||^(2)_2$$<br /> <br /> == Method ==<br /> The main technical proposal of the paper is a Bayesian framework to estimate the (former) target distribution &lt;math&gt;F_x(y)&lt;/math&gt;. That is, we construct a posterior distribution of &lt;math&gt; F_x(y) &lt;/math&gt; and use that as our new target distribution. We call it the ''belief matching'' (BM) framework.<br /> <br /> === Constructing Target Distribution ===<br /> Recall that &lt;math&gt;F_x(y)&lt;/math&gt; is a k-categorical probability distribution - it's PMF can be fully characterized by k numbers that sum to 1. Hence we can encode any such &lt;math&gt;F_x&lt;/math&gt; as a point in &lt;math&gt;\Delta^{k-1}&lt;/math&gt;. We'll do exactly that - let's call this vecor &lt;math&gt;z&lt;/math&gt;:<br /> $$z \in \Delta^{k-1}$$<br /> $$\text{prior} = p_{z|x}(z)$$<br /> $$\text{conditional} = p_{y|z,x}(y)$$<br /> $$\text{posterior} = p_{z|x,y}(z)$$<br /> Then if we perform inference:<br /> $$p_{z|x,y}(z) \propto p_{z|x}(z)p_{y|z,x}(y)$$<br /> The distribution chosen to model prior was &lt;math&gt;dir_K(\beta)&lt;/math&gt;:<br /> $$p_{z|x}(z) = \frac{\Gamma(||\beta||_1)}{\prod_{k=1}^K \Gamma(\beta_k)} \prod_{k=1}^K z_k^{\beta_k - 1}$$<br /> Note that by definition of &lt;math&gt;z&lt;/math&gt;: &lt;math&gt; p_{y|x,z} = z_y &lt;/math&gt;. Since the Dirichlet is a conjugate prior to categorical distributions we have a convenient form for the mean of the posterior:<br /> $$\bar{p_{z|x,y}}(z) = \frac{\beta + c^{\mathcal D}(x)}{||\beta + c^{\mathcal D}(x)||_1} \propto \beta + c^{\mathcal D}(x)$$<br /> This is in fact a generalization of (uniform) label smoothing (label smoothing is a special case where &lt;math&gt;\beta = \frac 1 K \vec{1} &lt;/math&gt;).<br /> <br /> === Representing Approximate Distribution ===<br /> Our new target distribution is &lt;math&gt;p_{z|x,y}(z)&lt;/math&gt; (as opposed to &lt;math&gt;F_x(y)&lt;/math&gt;). That is, we want to construct an interpretation of our neural network weights to construct a distribution with support in &lt;math&gt; \Delta^{K-1} &lt;/math&gt; - the NN can then be trained so this encoded distribution closely approximates &lt;math&gt;p_{z|x,y}&lt;/math&gt;. Let's denote the PMF of this encoded distribution &lt;math&gt;q_{z|x}^W&lt;/math&gt;. This is how the BM framework defines it:<br /> $$\alpha^W(x) := \exp(f^W(x))$$<br /> $$q_{z|x}^W(z) = \frac{\Gamma(||\alpha^W(x)||_1)}{\sum_{k=1}^K \Gamma(\alpha_k^W(x))} \prod_{k=1}^K z_{k}^{\alpha_k^W(x) - 1}$$<br /> $$\to Z^W_x \sim dir(\alpha^W(x))$$<br /> Apply &lt;math&gt;\log&lt;/math&gt; then &lt;math&gt;\exp&lt;/math&gt; to &lt;math&gt;q_{z|x}^W&lt;/math&gt;:<br /> $$q^W_{z|x}(z) \propto \exp \left( \sum_k (\alpha_k^W(x) \log(z_k)) - \sum_k \log(z_k) \right)$$<br /> $$\propto -l_{CE}(\phi(f^W(x)),z) + \frac{K}{||\alpha^W(x)||}KL(\mathcal U_k \;||\; z)$$<br /> It can actually be shown that the mean of &lt;math&gt;Z_x^W&lt;/math&gt; is identical to &lt;math&gt;\phi(f^W(x))&lt;/math&gt; - in other words, if we output the mean of the encoded distribution of our neural network under the BM framework, it is theoretically identical to a traditional neural network.<br /> <br /> === Distribution Matching ===<br /> <br /> We now need a way to fit our approximate distribution from our neural network &lt;math&gt;q_{\mathbf{z | x}}^{\mathbf{W}}&lt;/math&gt; to our target distribution &lt;math&gt;p_{\mathbf{z|x},y}&lt;/math&gt;. The authors achieve this by maximizing the evidence lower bound (ELBO):<br /> <br /> $$l_{EB}(\mathbf y, \alpha^{\mathbf W}(\mathbf x)) = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] - KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; p_{\mathbf{z|x}})$$<br /> <br /> Each term can be computed analytically:<br /> <br /> $$\mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf W }} \left[\log z_y \right] = \psi(\alpha_y^{\mathbf W} ( \mathbf x )) - \psi(\alpha_0^{\mathbf W} ( \mathbf x ))$$<br /> <br /> Where &lt;math&gt;\psi(\cdot)&lt;/math&gt; represents the digamma function (logarithmic derivative of gamma function). Intuitively, we maximize the probability of the correct label. For the KL term:<br /> <br /> $$KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; p_{\mathbf{z|x}}) = \log \frac{\Gamma(a_0^{\mathbf W}(\mathbf x)) \prod_k \Gamma(\beta_k)}{\prod_k \Gamma(\alpha_k^{\mathbf W}(x)) \Gamma (\beta_0)} + \sum_k (\alpha_k^{\mathbf W}(x)-\beta_k)(\psi(\alpha_k^{\mathbf W}(\mathbf x)) - \psi(\alpha_0^{\mathbf W}(\mathbf x))$$<br /> <br /> In the first term, for intuition, we can ignore &lt;math&gt;\alpha_0&lt;/math&gt; and &lt;math&gt;\beta_0&lt;/math&gt; since those just calibrate the distributions. Otherwise, we want the ratio of the products to be as close to 1 as possible to minimize the KL. In the second term, we want to minimize the difference between each individual &lt;math&gt;\alpha_k&lt;/math&gt; and &lt;math&gt;\beta_k&lt;/math&gt;, scaled by the normalized output of the neural network. <br /> <br /> This loss function can be used as a drop-in replacement for the standard softmax cross-entropy, as it has an analytic form and the same time complexity as typical softmax-cross entropy with respect to the number of classes (&lt;math&gt;O(K)&lt;/math&gt;).<br /> <br /> === On Prior Distributions ===<br /> <br /> We must choose our concentration parameter, &lt;math&gt;\beta&lt;/math&gt;, for our dirichlet prior. We see our prior essentially disappears as &lt;math&gt;\beta_0 \to 0&lt;/math&gt; and becomes stronger as &lt;math&gt;\beta_0 \to \infty&lt;/math&gt;. Thus, we want a small &lt;math&gt;\beta_0&lt;/math&gt; so the posterior isn't dominated by the prior. But, the authors claim that a small &lt;math&gt;\beta_0&lt;/math&gt; makes &lt;math&gt;\alpha_0^{\mathbf W}(\mathbf x)&lt;/math&gt; small, which causes &lt;math&gt;\psi (\alpha_0^{\mathbf W}(\mathbf x))&lt;/math&gt; to be large, which is problematic for gradient based optimization. In practice, many neural network techniques aim to make &lt;math&gt;\mathbb E [f^{\mathbf W} (\mathbf x)] \approx \mathbf 0&lt;/math&gt; and thus &lt;math&gt;\mathbb E [\alpha^{\mathbf W} (\mathbf x)] \approx \mathbf 1&lt;/math&gt;, which means making &lt;math&gt;\alpha_0^{\mathbf W}(\mathbf x)&lt;/math&gt; small can be counterproductive.<br /> <br /> So, the authors set &lt;math&gt;\beta = \mathbf 1&lt;/math&gt; and introduce a new hyperparameter &lt;math&gt;\lambda&lt;/math&gt; which is multiplied with the KL term in the ELBO:<br /> <br /> $$l^\lambda_{EB}(\mathbf y, \alpha^{\mathbf W}(\mathbf x)) = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] - \lambda KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; \mathcal P^D (\mathbf 1))$$<br /> <br /> This stabilizes the optimization, as we can tell from the gradients:<br /> <br /> $$\frac{\partial l_{E B}\left(\mathbf{y}, \alpha^{\mathbf W}(\mathbf{x})\right)}{\partial \alpha_{k}^{\mathbf W}(\mathbf {x})}=\left(\tilde{\mathbf{y}}_{k}-\left(\alpha_{k}^{\mathbf W}(\mathbf{x})-\beta_{k}\right)\right) \psi^{\prime}\left(\alpha_{k}^{\mathbf{W}}(\boldsymbol{x})\right)<br /> -\left(1-\left(\alpha_{0}^{\boldsymbol{W}}(\boldsymbol{x})-\beta_{0}\right)\right) \psi^{\prime}\left(\alpha_{0}^{\boldsymbol{W}}(\boldsymbol{x})\right)$$<br /> <br /> $$\frac{\partial l_{E B}^{\lambda}\left(\mathbf{y}, \alpha^{\mathbf{W}}(\mathbf{x})\right)}{\partial \alpha_{k}^{W}(\mathbf{x})}=\left(\tilde{\mathbf{y}}_{k}-\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})-\lambda\right)\right) \frac{\psi^{\prime}\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})\right)}{\psi^{\prime}\left(\tilde{\alpha}_{0}^{\mathbf W}(\mathbf{x})\right)}<br /> -\left(1-\left(\tilde{\alpha}_{0}^{W}(\mathbf{x})-\lambda K\right)\right)$$<br /> <br /> As we can see, the first expression is affected by the magnitude of &lt;math&gt;\alpha^{\boldsymbol{W}}(\boldsymbol{x})&lt;/math&gt;, whereas the second expression is not due to the &lt;math&gt;\frac{\psi^{\prime}\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})\right)}{\psi^{\prime}\left(\tilde{\alpha}_{0}^{\mathbf W}(\mathbf{x})\right)}&lt;/math&gt; ratio.<br /> <br /> == Experiments ==<br /> <br /> Throughout the experiments in this paper, the authors employ various models based on residual connections (He et al., 2016 ) which are the models used for benchmarking in practice. We will first demonstrate improvements provided by BM, then we will show versatility in other applications. For fairness of comparisons, all configurations in the reference implementation will be fixed. The only additions in the experiments are initial learning rate warm-up and gradient clipping which are extremely helpful for stable training of BM. <br /> <br /> === Generalization performance === <br /> The paper compares the generalization performance of BM with softmax and MC dropout on CIFAR-10 and CIFAR-100 benchmarks.<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_T1.png]]<br /> <br /> The next comparison was performed between BM and softmax on the ImageNet benchmark. <br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_T2.png]]<br /> <br /> For both datasets and In all configurations, BM achieves the best generalization and outperforms softmax and MC dropout.<br /> <br /> ===== Regularization effect of prior =====<br /> <br /> In theory, BM has 2 regularization effects:<br /> The prior distribution, which smooths the target posterior<br /> Averaging all of the possible categorical probabilities to compute the distribution matching loss<br /> The authors perform an ablation study to examine the 2 effects separately - removing the KL term in the ELBO removes the effect of the prior distribution.<br /> For ResNet-50 on CIFAR-100 and CIFAR-10 the resulting test error rates were 24.69% and 5.68% respectively. <br /> <br /> This demonstrates that both regularization effects are significant since just having one of them improves the generalization performance compared to the softmax baseline, and having both improves the performance even more.<br /> <br /> ===== Impact of &lt;math&gt;\beta&lt;/math&gt; =====<br /> <br /> The effect of β on generalization performance is studied by training ResNet-18 on CIFAR-10 by tuning the value of β on its own, as well as jointly with λ. It was found that robust generalization performance is obtained for β ∈ [&lt;math&gt;e^{−1}, e^4&lt;/math&gt;] when tuning β on its own; and β ∈ [&lt;math&gt;e^{−4}, e^{8}&lt;/math&gt;] when tuning β jointly with λ. The figure below shows a plot of the error rate with varying β.<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F3.png]]<br /> <br /> === Uncertainty Representation ===<br /> <br /> One of the big advantages of BM is the ability to represent uncertainty about the prediction. The authors evaluate the uncertainty representation on in-distribution (ID) and out-of-distribution (OOD) samples. <br /> <br /> ===== ID uncertainty =====<br /> <br /> For ID (in-distribution) samples, calibration performance is measured, which is a measure of how well the model’s confidence matches its actual accuracy. This measure can be visualized using reliability plots and quantified using a metric called expected calibration error (ECE). ECE is calculated by grouping predictions into M groups based on their confidence score and then finding the absolute difference between the average accuracy and average confidence for each group.<br /> The figure below is a reliability plot of ResNet-50 on CIFAR-10 and CIFAR-100 with 15 groups. It shows that BM has a significantly better calibration performance than softmax since the confidence matches the accuracy more closely (this is also reflected in the lower ECE).<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F4.png]]<br /> <br /> ===== OOD uncertainty =====<br /> <br /> Here, the authors quantify uncertainty using predictive entropy - the larger the predictive entropy, the larger the uncertainty about a prediction. <br /> <br /> The figure below is a density plot of the predictive entropy of ResNet-50 on CIFAR-10. It shows that BM provides significantly better uncertainty estimation compared to other methods since BM is the only method that has a clear peak of high predictive entropy for OOD samples which should have high uncertainty. <br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F5.png]]<br /> <br /> === Transfer learning ===<br /> <br /> Belief matching applies the Bayesian principle outside the neural network, which means it can easily be applied to already trained models. Thus, belief matching can be employed in transfer learning scenarios. The authors downloaded the ImageNet pre-trained ResNet-50 weights and fine-tuned the weights of the last linear layer for 100 epochs using an Adam optimizer.<br /> <br /> This table shows the test error rates from transfer learning on CIFAR-10, Food-101, and Cars datasets. Belief matching consistently performs better than softmax. <br /> <br /> [[File:being_bayesian_about_categorical_probability_transfer_learning.png]]<br /> <br /> Belief matching was also tested for the predictive uncertainty for out of dataset samples based on CIFAR-10 as the in distribution sample. Looking at the figure below, it is observed that belief matching significantly improves the uncertainty representation of pre-trained models by only fine-tuning the last layer’s weights. Note that belief matching confidently predicts examples in Cars since CIFAR-10 contains the object category automobiles. In comparison, softmax produces confident predictions on all datasets. Thus, belief matching could also be used to enhance the uncertainty representation ability of pre-trained models without sacrificing their generalization performance.<br /> <br /> [[File: being_bayesian_about_categorical_probability_transfer_learning_uncertainty.png]]<br /> <br /> === Semi-Supervised Learning ===<br /> <br /> Belief matching’s ability to allow neural networks to represent rich information in their predictions can be exploited to aid consistency based loss function for semi-supervised learning. Consistency-based loss functions use unlabelled samples to determine where to promote the robustness of predictions based on stochastic perturbations. This can be done by perturbing the inputs (which is the VAT model) or the networks (which is the pi-model). Both methods minimize the divergence between two categorical probabilities under some perturbations, thus belief matching can be used by the following replacements in the loss functions. The hope is that belief matching can provide better prediction consistencies using its Dirichlet distributions.<br /> <br /> [[File: being_bayesian_about_categorical_probability_semi_supervised_equation.png]]<br /> <br /> The results of training on ResNet28-2 with consistency based loss functions on CIFAR-10 are shown in this table. Belief matching does have lower classification error rates compared to using a softmax.<br /> <br /> [[File:being_bayesian_about_categorical_probability_semi_supervised_table.png]]<br /> <br /> == Conclusion ==<br /> <br /> Bayesian principles can be used to construct the target distribution by using the categorical probability as a random variable rather than a training label. This can be applied to neural network models by replacing only the softmax and cross-entropy loss, while improving the generalization performance and uncertainty estimation. <br /> <br /> In the future, the authors would like to allow for more expressive distributions in the belief matching framework, such as logistic normal distributions to capture strong semantic similarities among class labels. Furthermore, using input dependent priors would allow for interesting properties that would aid imbalanced datasets and multi-domain learning.<br /> <br /> == Citations ==<br /> <br />  Bridle, J. S. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pp. 227–236. Springer, 1990.<br /> <br />  Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural networks. In International Conference on Machine Learning, 2015.<br /> <br />  Gal, Y. and Ghahramani, Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.<br /> <br />  Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning, 2017. <br /> <br />  MacKay, D. J. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448– 472, 1992.<br /> <br />  Graves, A. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, 2011. <br /> <br />  Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate Bayesian inference. Journal of Machine Learning Research, 18(1):4873–4907, 2017.<br /> <br />  Zhang, G., Sun, S., Duvenaud, D., and Grosse, R. Noisy natural gradient as variational inference. In International Conference of Machine Learning, 2018.<br /> <br />  Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, 2019.<br /> <br />  Osawa, K., Swaroop, S., Jain, A., Eschenhagen, R., Turner, R. E., Yokota, R., and Khan, M. E. Practical deep learning with Bayesian principles. In Advances in Neural Information Processing Systems, 2019.<br /> <br />  Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.<br /> <br />  Neumann, L., Zisserman, A., and Vedaldi, A. Relaxed softmax: Efficient confidence auto-calibration for safe pedestrian detection. In NIPS Workshop on Machine Learning for Intelligent Transportation Systems, 2018.<br /> <br />  Xie, L., Wang, J., Wei, Z., Wang, M., and Tian, Q. Disturblabel: Regularizing cnn on the loss layer. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.<br /> <br />  Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., and Hinton, G. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Being_Bayesian_about_Categorical_Probability&diff=49054 Being Bayesian about Categorical Probability 2020-12-04T00:59:20Z <p>Z42qin: /* Classification With a Neural Network */</p> <hr /> <div>== Presented By ==<br /> Evan Li, Jason Pu, Karam Abuaisha, Nicholas Vadivelu<br /> <br /> == Introduction ==<br /> <br /> Since the outputs of neural networks are not probabilities, Softmax (Bridle, 1990) is a staple for neural network’s performing classification--it exponentiates each logit then normalizes by the sum, giving a distribution over the target classes. However, networks with softmax outputs give no information about uncertainty (Blundell et al., 2015; Gal &amp; Ghahramani, 2016), and the resulting distribution over classes is poorly calibrated (Guo et al., 2017), often giving overconfident predictions even when the classification is wrong. In addition, softmax also raises concerns about overfitting NNs due to its confident predictive behaviors(Xie et al., 2016; Pereyra et al., 2017). To achieve better generalization performance, this may require some effective regularization techniques. <br /> <br /> Bayesian Neural Networks (BNNs; MacKay, 1992) can alleviate these issues, but the resulting posteriors over the parameters are often intractable. Approximations such as variational inference (Graves, 2011; Blundell et al., 2015) and Monte Carlo Dropout (Gal &amp; Ghahramani, 2016) can still be expensive or give poor estimates for the posteriors. This work proposes a Bayesian treatment of the output logits of the neural network, treating the targets as a categorical random variable instead of a fixed label. This gives a computationally cheap way to get well-calibrated uncertainty estimates on neural network classifications.<br /> <br /> == Related Work ==<br /> <br /> Using Bayesian Neural Networks is the dominant way of applying Bayesian techniques to neural networks. Many techniques have been developed to make posterior approximation more accurate and scalable, despite these, BNNs do not scale to the state of the art techniques or large data sets. There are techniques to explicitly avoid modeling the full weight posterior that are more scalable, such as with Monte Carlo Dropout (Gal &amp; Ghahramani, 2016) or tracking mean/covariance of the posterior during training (Mandt et al., 2017; Zhang et al., 2018; Maddox et al., 2019; Osawa et al., 2019). Non-Bayesian uncertainty estimation techniques such as deep ensembles (Lakshminarayanan et al., 2017) and temperature scaling (Guo et al., 2017; Neumann et al., 2018).<br /> <br /> == Preliminaries ==<br /> === Definitions ===<br /> Let's formalize our classification problem and define some notations for the rest of this summary:<br /> <br /> ::Dataset:<br /> $$\mathcal D = \{(x_i,y_i)\} \in (\mathcal X \times \mathcal Y)^N$$<br /> ::General classification model<br /> $$f^W: \mathcal X \to \mathbb R^K$$<br /> ::Softmax function: <br /> $$\phi(x): \mathbb R^K \to [0,1]^K \;\;|\;\; \phi_k(X) = \frac{\exp(f_k^W(x))}{\sum_{k \in K} \exp(f_k^W(x))}$$<br /> ::Softmax activated NN:<br /> $$\phi \;\circ\; f^W: \chi \to \Delta^{K-1}$$<br /> ::NN as a true classifier:<br /> $$arg\max_i \;\circ\; \phi_i \;\circ\; f^W \;:\; \mathcal X \to \mathcal Y$$<br /> <br /> We'll also define the '''count function''' - a &lt;math&gt;K&lt;/math&gt;-vector valued function that outputs the occurences of each class coincident with &lt;math&gt;x&lt;/math&gt;:<br /> $$c^{\mathcal D}(x) = \sum_{(x',y') \in \mathcal D} \mathbb y' I(x' = x)$$<br /> <br /> === Classification With a Neural Network ===<br /> A typical loss function used in classification is cross-entropy. It's well known that optimizing &lt;math&gt;f^W&lt;/math&gt; for &lt;math&gt;l_{CE}&lt;/math&gt; is equivalent to optimizing for &lt;math&gt;l_{KL}&lt;/math&gt;, the &lt;math&gt;KL&lt;/math&gt; divergence between the true distribution and the distribution modeled by NN, that is:<br /> $$l_{KL}(W) = KL(\text{true distribution} \;|\; \text{distribution encoded by }NN(W))$$<br /> Let's introduce notations for the underlying (true) distributions of our problem. Let &lt;math&gt;(x_0,y_0) \sim (\mathcal X \times \mathcal Y)&lt;/math&gt;:<br /> $$\text{Full Distribution} = F(x,y) = P(x_0 = x,y_0 = y)$$<br /> $$\text{Marginal Distribution} = P(x) = F(x_0 = x)$$<br /> $$\text{Point Class Distribution} = P(y_0 = y \;|\; x_0 = x) = F_x(y)$$<br /> Then we have the following factorization:<br /> $$F(x,y) = P(x,y) = P(y|x)P(x) = F_x(y)F(x)$$<br /> Substitute this into the definition of KL divergence:<br /> $$= \sum_{(x,y) \in \mathcal X \times \mathcal Y} F(x,y) \log\left(\frac{F(x,y)}{\phi_y(f^W(x))}\right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) \sum_{y \in \mathcal Y} F(y|x) \log\left( \frac{F(y|x)}{\phi_y(f^W(x))} \right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) \sum_{y \in \mathcal Y} F_x(y) \log\left( \frac{F_x(y)}{\phi_y(f^W(x))} \right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) KL(F_x \;||\; \phi\left( f^W(x) \right))$$<br /> As usual, we don't have an analytic form for &lt;math&gt;l&lt;/math&gt; (if we did, this would imply we know &lt;math&gt;F_X&lt;/math&gt; meaning we knew the distribution in the first place). Instead, estimate from &lt;math&gt;\mathcal D&lt;/math&gt;:<br /> $$F(x) \approx \hat F(x) = \frac{||c^{\mathcal D}(x)||_1}{N}$$<br /> $$F_x(y) \approx \hat F_x(y) = \frac{c^{\mathcal D}(x)}{|| c^{\mathcal D}(x) ||_1}$$<br /> $$\to l_{KL}(W) = \sum_{x \in \mathcal D} \frac{||c^{\mathcal D}(x)||_1}{N} KL \left( \frac{c^{\mathcal D}(x)}{||c^{\mathcal D}(x)||_1} \;||\; \phi(f^W(x)) \right)$$<br /> The approximations &lt;math&gt;\hat F, \hat F_X&lt;/math&gt; are often not very good though: consider a typical classification such as MNIST, we would never expect two handwritten digits to produce the exact same image. Hence &lt;math&gt;c^{\mathcal D}(x)&lt;/math&gt; is (almost) always going to have a single index 1 and the rest 0. This has implications for our approximations:<br /> $$\hat F(x) \text{ is uniform for all } x \in \mathcal D$$<br /> $$\hat F_x(y) \text{ is degenerate for all } x \in \mathcal D$$<br /> This clearly has implications for overfitting: to minimize the KL term in &lt;math&gt;l_{KL}(W)&lt;/math&gt; we want &lt;math&gt;\phi(f^W(x))&lt;/math&gt; to be very close to &lt;math&gt;\hat F_x(y)&lt;/math&gt; at each point - this means that the loss function is in fact encouraging the neural network to output near degenerate distributions! <br /> <br /> '''Labal Smoothing'''<br /> One form of regularization to help this problem is called label smoothing. Instead of using the degenerate $$F_x(y)$$ as a target function, let's &quot;smooth&quot; it (by adding a scaled uniform distribution to it) so it's no longer degenerate:<br /> $$F'_x(y) = (1-\lambda)\hat F_x(y) + \frac \lambda K \vec 1$$<br /> <br /> '''BNNs'''<br /> BBNs balances the complexity of the model and the distance to target distribution without choosing a single beset configuration (one-hot encoding). Specifically, BNNs with the Gaussian Weight prior $$F_x(y) ~ N (0,T^(-1) I)$$ has score of configuration W measured by the posterior density $$p_W(W|D)$$<br /> <br /> == Method ==<br /> The main technical proposal of the paper is a Bayesian framework to estimate the (former) target distribution &lt;math&gt;F_x(y)&lt;/math&gt;. That is, we construct a posterior distribution of &lt;math&gt; F_x(y) &lt;/math&gt; and use that as our new target distribution. We call it the ''belief matching'' (BM) framework.<br /> <br /> === Constructing Target Distribution ===<br /> Recall that &lt;math&gt;F_x(y)&lt;/math&gt; is a k-categorical probability distribution - it's PMF can be fully characterized by k numbers that sum to 1. Hence we can encode any such &lt;math&gt;F_x&lt;/math&gt; as a point in &lt;math&gt;\Delta^{k-1}&lt;/math&gt;. We'll do exactly that - let's call this vecor &lt;math&gt;z&lt;/math&gt;:<br /> $$z \in \Delta^{k-1}$$<br /> $$\text{prior} = p_{z|x}(z)$$<br /> $$\text{conditional} = p_{y|z,x}(y)$$<br /> $$\text{posterior} = p_{z|x,y}(z)$$<br /> Then if we perform inference:<br /> $$p_{z|x,y}(z) \propto p_{z|x}(z)p_{y|z,x}(y)$$<br /> The distribution chosen to model prior was &lt;math&gt;dir_K(\beta)&lt;/math&gt;:<br /> $$p_{z|x}(z) = \frac{\Gamma(||\beta||_1)}{\prod_{k=1}^K \Gamma(\beta_k)} \prod_{k=1}^K z_k^{\beta_k - 1}$$<br /> Note that by definition of &lt;math&gt;z&lt;/math&gt;: &lt;math&gt; p_{y|x,z} = z_y &lt;/math&gt;. Since the Dirichlet is a conjugate prior to categorical distributions we have a convenient form for the mean of the posterior:<br /> $$\bar{p_{z|x,y}}(z) = \frac{\beta + c^{\mathcal D}(x)}{||\beta + c^{\mathcal D}(x)||_1} \propto \beta + c^{\mathcal D}(x)$$<br /> This is in fact a generalization of (uniform) label smoothing (label smoothing is a special case where &lt;math&gt;\beta = \frac 1 K \vec{1} &lt;/math&gt;).<br /> <br /> === Representing Approximate Distribution ===<br /> Our new target distribution is &lt;math&gt;p_{z|x,y}(z)&lt;/math&gt; (as opposed to &lt;math&gt;F_x(y)&lt;/math&gt;). That is, we want to construct an interpretation of our neural network weights to construct a distribution with support in &lt;math&gt; \Delta^{K-1} &lt;/math&gt; - the NN can then be trained so this encoded distribution closely approximates &lt;math&gt;p_{z|x,y}&lt;/math&gt;. Let's denote the PMF of this encoded distribution &lt;math&gt;q_{z|x}^W&lt;/math&gt;. This is how the BM framework defines it:<br /> $$\alpha^W(x) := \exp(f^W(x))$$<br /> $$q_{z|x}^W(z) = \frac{\Gamma(||\alpha^W(x)||_1)}{\sum_{k=1}^K \Gamma(\alpha_k^W(x))} \prod_{k=1}^K z_{k}^{\alpha_k^W(x) - 1}$$<br /> $$\to Z^W_x \sim dir(\alpha^W(x))$$<br /> Apply &lt;math&gt;\log&lt;/math&gt; then &lt;math&gt;\exp&lt;/math&gt; to &lt;math&gt;q_{z|x}^W&lt;/math&gt;:<br /> $$q^W_{z|x}(z) \propto \exp \left( \sum_k (\alpha_k^W(x) \log(z_k)) - \sum_k \log(z_k) \right)$$<br /> $$\propto -l_{CE}(\phi(f^W(x)),z) + \frac{K}{||\alpha^W(x)||}KL(\mathcal U_k \;||\; z)$$<br /> It can actually be shown that the mean of &lt;math&gt;Z_x^W&lt;/math&gt; is identical to &lt;math&gt;\phi(f^W(x))&lt;/math&gt; - in other words, if we output the mean of the encoded distribution of our neural network under the BM framework, it is theoretically identical to a traditional neural network.<br /> <br /> === Distribution Matching ===<br /> <br /> We now need a way to fit our approximate distribution from our neural network &lt;math&gt;q_{\mathbf{z | x}}^{\mathbf{W}}&lt;/math&gt; to our target distribution &lt;math&gt;p_{\mathbf{z|x},y}&lt;/math&gt;. The authors achieve this by maximizing the evidence lower bound (ELBO):<br /> <br /> $$l_{EB}(\mathbf y, \alpha^{\mathbf W}(\mathbf x)) = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] - KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; p_{\mathbf{z|x}})$$<br /> <br /> Each term can be computed analytically:<br /> <br /> $$\mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf W }} \left[\log z_y \right] = \psi(\alpha_y^{\mathbf W} ( \mathbf x )) - \psi(\alpha_0^{\mathbf W} ( \mathbf x ))$$<br /> <br /> Where &lt;math&gt;\psi(\cdot)&lt;/math&gt; represents the digamma function (logarithmic derivative of gamma function). Intuitively, we maximize the probability of the correct label. For the KL term:<br /> <br /> $$KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; p_{\mathbf{z|x}}) = \log \frac{\Gamma(a_0^{\mathbf W}(\mathbf x)) \prod_k \Gamma(\beta_k)}{\prod_k \Gamma(\alpha_k^{\mathbf W}(x)) \Gamma (\beta_0)} + \sum_k (\alpha_k^{\mathbf W}(x)-\beta_k)(\psi(\alpha_k^{\mathbf W}(\mathbf x)) - \psi(\alpha_0^{\mathbf W}(\mathbf x))$$<br /> <br /> In the first term, for intuition, we can ignore &lt;math&gt;\alpha_0&lt;/math&gt; and &lt;math&gt;\beta_0&lt;/math&gt; since those just calibrate the distributions. Otherwise, we want the ratio of the products to be as close to 1 as possible to minimize the KL. In the second term, we want to minimize the difference between each individual &lt;math&gt;\alpha_k&lt;/math&gt; and &lt;math&gt;\beta_k&lt;/math&gt;, scaled by the normalized output of the neural network. <br /> <br /> This loss function can be used as a drop-in replacement for the standard softmax cross-entropy, as it has an analytic form and the same time complexity as typical softmax-cross entropy with respect to the number of classes (&lt;math&gt;O(K)&lt;/math&gt;).<br /> <br /> === On Prior Distributions ===<br /> <br /> We must choose our concentration parameter, &lt;math&gt;\beta&lt;/math&gt;, for our dirichlet prior. We see our prior essentially disappears as &lt;math&gt;\beta_0 \to 0&lt;/math&gt; and becomes stronger as &lt;math&gt;\beta_0 \to \infty&lt;/math&gt;. Thus, we want a small &lt;math&gt;\beta_0&lt;/math&gt; so the posterior isn't dominated by the prior. But, the authors claim that a small &lt;math&gt;\beta_0&lt;/math&gt; makes &lt;math&gt;\alpha_0^{\mathbf W}(\mathbf x)&lt;/math&gt; small, which causes &lt;math&gt;\psi (\alpha_0^{\mathbf W}(\mathbf x))&lt;/math&gt; to be large, which is problematic for gradient based optimization. In practice, many neural network techniques aim to make &lt;math&gt;\mathbb E [f^{\mathbf W} (\mathbf x)] \approx \mathbf 0&lt;/math&gt; and thus &lt;math&gt;\mathbb E [\alpha^{\mathbf W} (\mathbf x)] \approx \mathbf 1&lt;/math&gt;, which means making &lt;math&gt;\alpha_0^{\mathbf W}(\mathbf x)&lt;/math&gt; small can be counterproductive.<br /> <br /> So, the authors set &lt;math&gt;\beta = \mathbf 1&lt;/math&gt; and introduce a new hyperparameter &lt;math&gt;\lambda&lt;/math&gt; which is multiplied with the KL term in the ELBO:<br /> <br /> $$l^\lambda_{EB}(\mathbf y, \alpha^{\mathbf W}(\mathbf x)) = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] - \lambda KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; \mathcal P^D (\mathbf 1))$$<br /> <br /> This stabilizes the optimization, as we can tell from the gradients:<br /> <br /> $$\frac{\partial l_{E B}\left(\mathbf{y}, \alpha^{\mathbf W}(\mathbf{x})\right)}{\partial \alpha_{k}^{\mathbf W}(\mathbf {x})}=\left(\tilde{\mathbf{y}}_{k}-\left(\alpha_{k}^{\mathbf W}(\mathbf{x})-\beta_{k}\right)\right) \psi^{\prime}\left(\alpha_{k}^{\mathbf{W}}(\boldsymbol{x})\right)<br /> -\left(1-\left(\alpha_{0}^{\boldsymbol{W}}(\boldsymbol{x})-\beta_{0}\right)\right) \psi^{\prime}\left(\alpha_{0}^{\boldsymbol{W}}(\boldsymbol{x})\right)$$<br /> <br /> $$\frac{\partial l_{E B}^{\lambda}\left(\mathbf{y}, \alpha^{\mathbf{W}}(\mathbf{x})\right)}{\partial \alpha_{k}^{W}(\mathbf{x})}=\left(\tilde{\mathbf{y}}_{k}-\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})-\lambda\right)\right) \frac{\psi^{\prime}\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})\right)}{\psi^{\prime}\left(\tilde{\alpha}_{0}^{\mathbf W}(\mathbf{x})\right)}<br /> -\left(1-\left(\tilde{\alpha}_{0}^{W}(\mathbf{x})-\lambda K\right)\right)$$<br /> <br /> As we can see, the first expression is affected by the magnitude of &lt;math&gt;\alpha^{\boldsymbol{W}}(\boldsymbol{x})&lt;/math&gt;, whereas the second expression is not due to the &lt;math&gt;\frac{\psi^{\prime}\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})\right)}{\psi^{\prime}\left(\tilde{\alpha}_{0}^{\mathbf W}(\mathbf{x})\right)}&lt;/math&gt; ratio.<br /> <br /> == Experiments ==<br /> <br /> Throughout the experiments in this paper, the authors employ various models based on residual connections (He et al., 2016 ) which are the models used for benchmarking in practice. We will first demonstrate improvements provided by BM, then we will show versatility in other applications. For fairness of comparisons, all configurations in the reference implementation will be fixed. The only additions in the experiments are initial learning rate warm-up and gradient clipping which are extremely helpful for stable training of BM. <br /> <br /> === Generalization performance === <br /> The paper compares the generalization performance of BM with softmax and MC dropout on CIFAR-10 and CIFAR-100 benchmarks.<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_T1.png]]<br /> <br /> The next comparison was performed between BM and softmax on the ImageNet benchmark. <br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_T2.png]]<br /> <br /> For both datasets and In all configurations, BM achieves the best generalization and outperforms softmax and MC dropout.<br /> <br /> ===== Regularization effect of prior =====<br /> <br /> In theory, BM has 2 regularization effects:<br /> The prior distribution, which smooths the target posterior<br /> Averaging all of the possible categorical probabilities to compute the distribution matching loss<br /> The authors perform an ablation study to examine the 2 effects separately - removing the KL term in the ELBO removes the effect of the prior distribution.<br /> For ResNet-50 on CIFAR-100 and CIFAR-10 the resulting test error rates were 24.69% and 5.68% respectively. <br /> <br /> This demonstrates that both regularization effects are significant since just having one of them improves the generalization performance compared to the softmax baseline, and having both improves the performance even more.<br /> <br /> ===== Impact of &lt;math&gt;\beta&lt;/math&gt; =====<br /> <br /> The effect of β on generalization performance is studied by training ResNet-18 on CIFAR-10 by tuning the value of β on its own, as well as jointly with λ. It was found that robust generalization performance is obtained for β ∈ [&lt;math&gt;e^{−1}, e^4&lt;/math&gt;] when tuning β on its own; and β ∈ [&lt;math&gt;e^{−4}, e^{8}&lt;/math&gt;] when tuning β jointly with λ. The figure below shows a plot of the error rate with varying β.<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F3.png]]<br /> <br /> === Uncertainty Representation ===<br /> <br /> One of the big advantages of BM is the ability to represent uncertainty about the prediction. The authors evaluate the uncertainty representation on in-distribution (ID) and out-of-distribution (OOD) samples. <br /> <br /> ===== ID uncertainty =====<br /> <br /> For ID (in-distribution) samples, calibration performance is measured, which is a measure of how well the model’s confidence matches its actual accuracy. This measure can be visualized using reliability plots and quantified using a metric called expected calibration error (ECE). ECE is calculated by grouping predictions into M groups based on their confidence score and then finding the absolute difference between the average accuracy and average confidence for each group.<br /> The figure below is a reliability plot of ResNet-50 on CIFAR-10 and CIFAR-100 with 15 groups. It shows that BM has a significantly better calibration performance than softmax since the confidence matches the accuracy more closely (this is also reflected in the lower ECE).<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F4.png]]<br /> <br /> ===== OOD uncertainty =====<br /> <br /> Here, the authors quantify uncertainty using predictive entropy - the larger the predictive entropy, the larger the uncertainty about a prediction. <br /> <br /> The figure below is a density plot of the predictive entropy of ResNet-50 on CIFAR-10. It shows that BM provides significantly better uncertainty estimation compared to other methods since BM is the only method that has a clear peak of high predictive entropy for OOD samples which should have high uncertainty. <br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F5.png]]<br /> <br /> === Transfer learning ===<br /> <br /> Belief matching applies the Bayesian principle outside the neural network, which means it can easily be applied to already trained models. Thus, belief matching can be employed in transfer learning scenarios. The authors downloaded the ImageNet pre-trained ResNet-50 weights and fine-tuned the weights of the last linear layer for 100 epochs using an Adam optimizer.<br /> <br /> This table shows the test error rates from transfer learning on CIFAR-10, Food-101, and Cars datasets. Belief matching consistently performs better than softmax. <br /> <br /> [[File:being_bayesian_about_categorical_probability_transfer_learning.png]]<br /> <br /> Belief matching was also tested for the predictive uncertainty for out of dataset samples based on CIFAR-10 as the in distribution sample. Looking at the figure below, it is observed that belief matching significantly improves the uncertainty representation of pre-trained models by only fine-tuning the last layer’s weights. Note that belief matching confidently predicts examples in Cars since CIFAR-10 contains the object category automobiles. In comparison, softmax produces confident predictions on all datasets. Thus, belief matching could also be used to enhance the uncertainty representation ability of pre-trained models without sacrificing their generalization performance.<br /> <br /> [[File: being_bayesian_about_categorical_probability_transfer_learning_uncertainty.png]]<br /> <br /> === Semi-Supervised Learning ===<br /> <br /> Belief matching’s ability to allow neural networks to represent rich information in their predictions can be exploited to aid consistency based loss function for semi-supervised learning. Consistency-based loss functions use unlabelled samples to determine where to promote the robustness of predictions based on stochastic perturbations. This can be done by perturbing the inputs (which is the VAT model) or the networks (which is the pi-model). Both methods minimize the divergence between two categorical probabilities under some perturbations, thus belief matching can be used by the following replacements in the loss functions. The hope is that belief matching can provide better prediction consistencies using its Dirichlet distributions.<br /> <br /> [[File: being_bayesian_about_categorical_probability_semi_supervised_equation.png]]<br /> <br /> The results of training on ResNet28-2 with consistency based loss functions on CIFAR-10 are shown in this table. Belief matching does have lower classification error rates compared to using a softmax.<br /> <br /> [[File:being_bayesian_about_categorical_probability_semi_supervised_table.png]]<br /> <br /> == Conclusion ==<br /> <br /> Bayesian principles can be used to construct the target distribution by using the categorical probability as a random variable rather than a training label. This can be applied to neural network models by replacing only the softmax and cross-entropy loss, while improving the generalization performance and uncertainty estimation. <br /> <br /> In the future, the authors would like to allow for more expressive distributions in the belief matching framework, such as logistic normal distributions to capture strong semantic similarities among class labels. Furthermore, using input dependent priors would allow for interesting properties that would aid imbalanced datasets and multi-domain learning.<br /> <br /> == Citations ==<br /> <br />  Bridle, J. S. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pp. 227–236. Springer, 1990.<br /> <br />  Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural networks. In International Conference on Machine Learning, 2015.<br /> <br />  Gal, Y. and Ghahramani, Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.<br /> <br />  Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning, 2017. <br /> <br />  MacKay, D. J. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448– 472, 1992.<br /> <br />  Graves, A. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, 2011. <br /> <br />  Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate Bayesian inference. Journal of Machine Learning Research, 18(1):4873–4907, 2017.<br /> <br />  Zhang, G., Sun, S., Duvenaud, D., and Grosse, R. Noisy natural gradient as variational inference. In International Conference of Machine Learning, 2018.<br /> <br />  Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, 2019.<br /> <br />  Osawa, K., Swaroop, S., Jain, A., Eschenhagen, R., Turner, R. E., Yokota, R., and Khan, M. E. Practical deep learning with Bayesian principles. In Advances in Neural Information Processing Systems, 2019.<br /> <br />  Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.<br /> <br />  Neumann, L., Zisserman, A., and Vedaldi, A. Relaxed softmax: Efficient confidence auto-calibration for safe pedestrian detection. In NIPS Workshop on Machine Learning for Intelligent Transportation Systems, 2018.<br /> <br />  Xie, L., Wang, J., Wei, Z., Wang, M., and Tian, Q. Disturblabel: Regularizing cnn on the loss layer. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.<br /> <br />  Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., and Hinton, G. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Being_Bayesian_about_Categorical_Probability&diff=49053 Being Bayesian about Categorical Probability 2020-12-04T00:58:56Z <p>Z42qin: /* Classification With a Neural Network */</p> <hr /> <div>== Presented By ==<br /> Evan Li, Jason Pu, Karam Abuaisha, Nicholas Vadivelu<br /> <br /> == Introduction ==<br /> <br /> Since the outputs of neural networks are not probabilities, Softmax (Bridle, 1990) is a staple for neural network’s performing classification--it exponentiates each logit then normalizes by the sum, giving a distribution over the target classes. However, networks with softmax outputs give no information about uncertainty (Blundell et al., 2015; Gal &amp; Ghahramani, 2016), and the resulting distribution over classes is poorly calibrated (Guo et al., 2017), often giving overconfident predictions even when the classification is wrong. In addition, softmax also raises concerns about overfitting NNs due to its confident predictive behaviors(Xie et al., 2016; Pereyra et al., 2017). To achieve better generalization performance, this may require some effective regularization techniques. <br /> <br /> Bayesian Neural Networks (BNNs; MacKay, 1992) can alleviate these issues, but the resulting posteriors over the parameters are often intractable. Approximations such as variational inference (Graves, 2011; Blundell et al., 2015) and Monte Carlo Dropout (Gal &amp; Ghahramani, 2016) can still be expensive or give poor estimates for the posteriors. This work proposes a Bayesian treatment of the output logits of the neural network, treating the targets as a categorical random variable instead of a fixed label. This gives a computationally cheap way to get well-calibrated uncertainty estimates on neural network classifications.<br /> <br /> == Related Work ==<br /> <br /> Using Bayesian Neural Networks is the dominant way of applying Bayesian techniques to neural networks. Many techniques have been developed to make posterior approximation more accurate and scalable, despite these, BNNs do not scale to the state of the art techniques or large data sets. There are techniques to explicitly avoid modeling the full weight posterior that are more scalable, such as with Monte Carlo Dropout (Gal &amp; Ghahramani, 2016) or tracking mean/covariance of the posterior during training (Mandt et al., 2017; Zhang et al., 2018; Maddox et al., 2019; Osawa et al., 2019). Non-Bayesian uncertainty estimation techniques such as deep ensembles (Lakshminarayanan et al., 2017) and temperature scaling (Guo et al., 2017; Neumann et al., 2018).<br /> <br /> == Preliminaries ==<br /> === Definitions ===<br /> Let's formalize our classification problem and define some notations for the rest of this summary:<br /> <br /> ::Dataset:<br /> $$\mathcal D = \{(x_i,y_i)\} \in (\mathcal X \times \mathcal Y)^N$$<br /> ::General classification model<br /> $$f^W: \mathcal X \to \mathbb R^K$$<br /> ::Softmax function: <br /> $$\phi(x): \mathbb R^K \to [0,1]^K \;\;|\;\; \phi_k(X) = \frac{\exp(f_k^W(x))}{\sum_{k \in K} \exp(f_k^W(x))}$$<br /> ::Softmax activated NN:<br /> $$\phi \;\circ\; f^W: \chi \to \Delta^{K-1}$$<br /> ::NN as a true classifier:<br /> $$arg\max_i \;\circ\; \phi_i \;\circ\; f^W \;:\; \mathcal X \to \mathcal Y$$<br /> <br /> We'll also define the '''count function''' - a &lt;math&gt;K&lt;/math&gt;-vector valued function that outputs the occurences of each class coincident with &lt;math&gt;x&lt;/math&gt;:<br /> $$c^{\mathcal D}(x) = \sum_{(x',y') \in \mathcal D} \mathbb y' I(x' = x)$$<br /> <br /> === Classification With a Neural Network ===<br /> A typical loss function used in classification is cross-entropy. It's well known that optimizing &lt;math&gt;f^W&lt;/math&gt; for &lt;math&gt;l_{CE}&lt;/math&gt; is equivalent to optimizing for &lt;math&gt;l_{KL}&lt;/math&gt;, the &lt;math&gt;KL&lt;/math&gt; divergence between the true distribution and the distribution modeled by NN, that is:<br /> $$l_{KL}(W) = KL(\text{true distribution} \;|\; \text{distribution encoded by }NN(W))$$<br /> Let's introduce notations for the underlying (true) distributions of our problem. Let &lt;math&gt;(x_0,y_0) \sim (\mathcal X \times \mathcal Y)&lt;/math&gt;:<br /> $$\text{Full Distribution} = F(x,y) = P(x_0 = x,y_0 = y)$$<br /> $$\text{Marginal Distribution} = P(x) = F(x_0 = x)$$<br /> $$\text{Point Class Distribution} = P(y_0 = y \;|\; x_0 = x) = F_x(y)$$<br /> Then we have the following factorization:<br /> $$F(x,y) = P(x,y) = P(y|x)P(x) = F_x(y)F(x)$$<br /> Substitute this into the definition of KL divergence:<br /> $$= \sum_{(x,y) \in \mathcal X \times \mathcal Y} F(x,y) \log\left(\frac{F(x,y)}{\phi_y(f^W(x))}\right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) \sum_{y \in \mathcal Y} F(y|x) \log\left( \frac{F(y|x)}{\phi_y(f^W(x))} \right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) \sum_{y \in \mathcal Y} F_x(y) \log\left( \frac{F_x(y)}{\phi_y(f^W(x))} \right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) KL(F_x \;||\; \phi\left( f^W(x) \right))$$<br /> As usual, we don't have an analytic form for &lt;math&gt;l&lt;/math&gt; (if we did, this would imply we know &lt;math&gt;F_X&lt;/math&gt; meaning we knew the distribution in the first place). Instead, estimate from &lt;math&gt;\mathcal D&lt;/math&gt;:<br /> $$F(x) \approx \hat F(x) = \frac{||c^{\mathcal D}(x)||_1}{N}$$<br /> $$F_x(y) \approx \hat F_x(y) = \frac{c^{\mathcal D}(x)}{|| c^{\mathcal D}(x) ||_1}$$<br /> $$\to l_{KL}(W) = \sum_{x \in \mathcal D} \frac{||c^{\mathcal D}(x)||_1}{N} KL \left( \frac{c^{\mathcal D}(x)}{||c^{\mathcal D}(x)||_1} \;||\; \phi(f^W(x)) \right)$$<br /> The approximations &lt;math&gt;\hat F, \hat F_X&lt;/math&gt; are often not very good though: consider a typical classification such as MNIST, we would never expect two handwritten digits to produce the exact same image. Hence &lt;math&gt;c^{\mathcal D}(x)&lt;/math&gt; is (almost) always going to have a single index 1 and the rest 0. This has implications for our approximations:<br /> $$\hat F(x) \text{ is uniform for all } x \in \mathcal D$$<br /> $$\hat F_x(y) \text{ is degenerate for all } x \in \mathcal D$$<br /> This clearly has implications for overfitting: to minimize the KL term in &lt;math&gt;l_{KL}(W)&lt;/math&gt; we want &lt;math&gt;\phi(f^W(x))&lt;/math&gt; to be very close to &lt;math&gt;\hat F_x(y)&lt;/math&gt; at each point - this means that the loss function is in fact encouraging the neural network to output near degenerate distributions! <br /> <br /> '''Labal Smoothing'''<br /> One form of regularization to help this problem is called label smoothing. Instead of using the degenerate $$F_x(y)$$ as a target function, let's &quot;smooth&quot; it (by adding a scaled uniform distribution to it) so it's no longer degenerate:<br /> $$F'_x(y) = (1-\lambda)\hat F_x(y) + \frac \lambda K \vec 1$$<br /> <br /> '''BNNs'''<br /> BBNs balances the complexity of the model and the distance to target distribution without choosing a single beset configuration (one-hot encoding). Specifically, BNNs with the Gaussian Weight prior $$F_x(y) ~ N (0,T^-1 I)$$ has score of configuration W measured by the posterior density $$p_W(W|D)$$<br /> <br /> == Method ==<br /> The main technical proposal of the paper is a Bayesian framework to estimate the (former) target distribution &lt;math&gt;F_x(y)&lt;/math&gt;. That is, we construct a posterior distribution of &lt;math&gt; F_x(y) &lt;/math&gt; and use that as our new target distribution. We call it the ''belief matching'' (BM) framework.<br /> <br /> === Constructing Target Distribution ===<br /> Recall that &lt;math&gt;F_x(y)&lt;/math&gt; is a k-categorical probability distribution - it's PMF can be fully characterized by k numbers that sum to 1. Hence we can encode any such &lt;math&gt;F_x&lt;/math&gt; as a point in &lt;math&gt;\Delta^{k-1}&lt;/math&gt;. We'll do exactly that - let's call this vecor &lt;math&gt;z&lt;/math&gt;:<br /> $$z \in \Delta^{k-1}$$<br /> $$\text{prior} = p_{z|x}(z)$$<br /> $$\text{conditional} = p_{y|z,x}(y)$$<br /> $$\text{posterior} = p_{z|x,y}(z)$$<br /> Then if we perform inference:<br /> $$p_{z|x,y}(z) \propto p_{z|x}(z)p_{y|z,x}(y)$$<br /> The distribution chosen to model prior was &lt;math&gt;dir_K(\beta)&lt;/math&gt;:<br /> $$p_{z|x}(z) = \frac{\Gamma(||\beta||_1)}{\prod_{k=1}^K \Gamma(\beta_k)} \prod_{k=1}^K z_k^{\beta_k - 1}$$<br /> Note that by definition of &lt;math&gt;z&lt;/math&gt;: &lt;math&gt; p_{y|x,z} = z_y &lt;/math&gt;. Since the Dirichlet is a conjugate prior to categorical distributions we have a convenient form for the mean of the posterior:<br /> $$\bar{p_{z|x,y}}(z) = \frac{\beta + c^{\mathcal D}(x)}{||\beta + c^{\mathcal D}(x)||_1} \propto \beta + c^{\mathcal D}(x)$$<br /> This is in fact a generalization of (uniform) label smoothing (label smoothing is a special case where &lt;math&gt;\beta = \frac 1 K \vec{1} &lt;/math&gt;).<br /> <br /> === Representing Approximate Distribution ===<br /> Our new target distribution is &lt;math&gt;p_{z|x,y}(z)&lt;/math&gt; (as opposed to &lt;math&gt;F_x(y)&lt;/math&gt;). That is, we want to construct an interpretation of our neural network weights to construct a distribution with support in &lt;math&gt; \Delta^{K-1} &lt;/math&gt; - the NN can then be trained so this encoded distribution closely approximates &lt;math&gt;p_{z|x,y}&lt;/math&gt;. Let's denote the PMF of this encoded distribution &lt;math&gt;q_{z|x}^W&lt;/math&gt;. This is how the BM framework defines it:<br /> $$\alpha^W(x) := \exp(f^W(x))$$<br /> $$q_{z|x}^W(z) = \frac{\Gamma(||\alpha^W(x)||_1)}{\sum_{k=1}^K \Gamma(\alpha_k^W(x))} \prod_{k=1}^K z_{k}^{\alpha_k^W(x) - 1}$$<br /> $$\to Z^W_x \sim dir(\alpha^W(x))$$<br /> Apply &lt;math&gt;\log&lt;/math&gt; then &lt;math&gt;\exp&lt;/math&gt; to &lt;math&gt;q_{z|x}^W&lt;/math&gt;:<br /> $$q^W_{z|x}(z) \propto \exp \left( \sum_k (\alpha_k^W(x) \log(z_k)) - \sum_k \log(z_k) \right)$$<br /> $$\propto -l_{CE}(\phi(f^W(x)),z) + \frac{K}{||\alpha^W(x)||}KL(\mathcal U_k \;||\; z)$$<br /> It can actually be shown that the mean of &lt;math&gt;Z_x^W&lt;/math&gt; is identical to &lt;math&gt;\phi(f^W(x))&lt;/math&gt; - in other words, if we output the mean of the encoded distribution of our neural network under the BM framework, it is theoretically identical to a traditional neural network.<br /> <br /> === Distribution Matching ===<br /> <br /> We now need a way to fit our approximate distribution from our neural network &lt;math&gt;q_{\mathbf{z | x}}^{\mathbf{W}}&lt;/math&gt; to our target distribution &lt;math&gt;p_{\mathbf{z|x},y}&lt;/math&gt;. The authors achieve this by maximizing the evidence lower bound (ELBO):<br /> <br /> $$l_{EB}(\mathbf y, \alpha^{\mathbf W}(\mathbf x)) = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] - KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; p_{\mathbf{z|x}})$$<br /> <br /> Each term can be computed analytically:<br /> <br /> $$\mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf W }} \left[\log z_y \right] = \psi(\alpha_y^{\mathbf W} ( \mathbf x )) - \psi(\alpha_0^{\mathbf W} ( \mathbf x ))$$<br /> <br /> Where &lt;math&gt;\psi(\cdot)&lt;/math&gt; represents the digamma function (logarithmic derivative of gamma function). Intuitively, we maximize the probability of the correct label. For the KL term:<br /> <br /> $$KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; p_{\mathbf{z|x}}) = \log \frac{\Gamma(a_0^{\mathbf W}(\mathbf x)) \prod_k \Gamma(\beta_k)}{\prod_k \Gamma(\alpha_k^{\mathbf W}(x)) \Gamma (\beta_0)} + \sum_k (\alpha_k^{\mathbf W}(x)-\beta_k)(\psi(\alpha_k^{\mathbf W}(\mathbf x)) - \psi(\alpha_0^{\mathbf W}(\mathbf x))$$<br /> <br /> In the first term, for intuition, we can ignore &lt;math&gt;\alpha_0&lt;/math&gt; and &lt;math&gt;\beta_0&lt;/math&gt; since those just calibrate the distributions. Otherwise, we want the ratio of the products to be as close to 1 as possible to minimize the KL. In the second term, we want to minimize the difference between each individual &lt;math&gt;\alpha_k&lt;/math&gt; and &lt;math&gt;\beta_k&lt;/math&gt;, scaled by the normalized output of the neural network. <br /> <br /> This loss function can be used as a drop-in replacement for the standard softmax cross-entropy, as it has an analytic form and the same time complexity as typical softmax-cross entropy with respect to the number of classes (&lt;math&gt;O(K)&lt;/math&gt;).<br /> <br /> === On Prior Distributions ===<br /> <br /> We must choose our concentration parameter, &lt;math&gt;\beta&lt;/math&gt;, for our dirichlet prior. We see our prior essentially disappears as &lt;math&gt;\beta_0 \to 0&lt;/math&gt; and becomes stronger as &lt;math&gt;\beta_0 \to \infty&lt;/math&gt;. Thus, we want a small &lt;math&gt;\beta_0&lt;/math&gt; so the posterior isn't dominated by the prior. But, the authors claim that a small &lt;math&gt;\beta_0&lt;/math&gt; makes &lt;math&gt;\alpha_0^{\mathbf W}(\mathbf x)&lt;/math&gt; small, which causes &lt;math&gt;\psi (\alpha_0^{\mathbf W}(\mathbf x))&lt;/math&gt; to be large, which is problematic for gradient based optimization. In practice, many neural network techniques aim to make &lt;math&gt;\mathbb E [f^{\mathbf W} (\mathbf x)] \approx \mathbf 0&lt;/math&gt; and thus &lt;math&gt;\mathbb E [\alpha^{\mathbf W} (\mathbf x)] \approx \mathbf 1&lt;/math&gt;, which means making &lt;math&gt;\alpha_0^{\mathbf W}(\mathbf x)&lt;/math&gt; small can be counterproductive.<br /> <br /> So, the authors set &lt;math&gt;\beta = \mathbf 1&lt;/math&gt; and introduce a new hyperparameter &lt;math&gt;\lambda&lt;/math&gt; which is multiplied with the KL term in the ELBO:<br /> <br /> $$l^\lambda_{EB}(\mathbf y, \alpha^{\mathbf W}(\mathbf x)) = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] - \lambda KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; \mathcal P^D (\mathbf 1))$$<br /> <br /> This stabilizes the optimization, as we can tell from the gradients:<br /> <br /> $$\frac{\partial l_{E B}\left(\mathbf{y}, \alpha^{\mathbf W}(\mathbf{x})\right)}{\partial \alpha_{k}^{\mathbf W}(\mathbf {x})}=\left(\tilde{\mathbf{y}}_{k}-\left(\alpha_{k}^{\mathbf W}(\mathbf{x})-\beta_{k}\right)\right) \psi^{\prime}\left(\alpha_{k}^{\mathbf{W}}(\boldsymbol{x})\right)<br /> -\left(1-\left(\alpha_{0}^{\boldsymbol{W}}(\boldsymbol{x})-\beta_{0}\right)\right) \psi^{\prime}\left(\alpha_{0}^{\boldsymbol{W}}(\boldsymbol{x})\right)$$<br /> <br /> $$\frac{\partial l_{E B}^{\lambda}\left(\mathbf{y}, \alpha^{\mathbf{W}}(\mathbf{x})\right)}{\partial \alpha_{k}^{W}(\mathbf{x})}=\left(\tilde{\mathbf{y}}_{k}-\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})-\lambda\right)\right) \frac{\psi^{\prime}\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})\right)}{\psi^{\prime}\left(\tilde{\alpha}_{0}^{\mathbf W}(\mathbf{x})\right)}<br /> -\left(1-\left(\tilde{\alpha}_{0}^{W}(\mathbf{x})-\lambda K\right)\right)$$<br /> <br /> As we can see, the first expression is affected by the magnitude of &lt;math&gt;\alpha^{\boldsymbol{W}}(\boldsymbol{x})&lt;/math&gt;, whereas the second expression is not due to the &lt;math&gt;\frac{\psi^{\prime}\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})\right)}{\psi^{\prime}\left(\tilde{\alpha}_{0}^{\mathbf W}(\mathbf{x})\right)}&lt;/math&gt; ratio.<br /> <br /> == Experiments ==<br /> <br /> Throughout the experiments in this paper, the authors employ various models based on residual connections (He et al., 2016 ) which are the models used for benchmarking in practice. We will first demonstrate improvements provided by BM, then we will show versatility in other applications. For fairness of comparisons, all configurations in the reference implementation will be fixed. The only additions in the experiments are initial learning rate warm-up and gradient clipping which are extremely helpful for stable training of BM. <br /> <br /> === Generalization performance === <br /> The paper compares the generalization performance of BM with softmax and MC dropout on CIFAR-10 and CIFAR-100 benchmarks.<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_T1.png]]<br /> <br /> The next comparison was performed between BM and softmax on the ImageNet benchmark. <br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_T2.png]]<br /> <br /> For both datasets and In all configurations, BM achieves the best generalization and outperforms softmax and MC dropout.<br /> <br /> ===== Regularization effect of prior =====<br /> <br /> In theory, BM has 2 regularization effects:<br /> The prior distribution, which smooths the target posterior<br /> Averaging all of the possible categorical probabilities to compute the distribution matching loss<br /> The authors perform an ablation study to examine the 2 effects separately - removing the KL term in the ELBO removes the effect of the prior distribution.<br /> For ResNet-50 on CIFAR-100 and CIFAR-10 the resulting test error rates were 24.69% and 5.68% respectively. <br /> <br /> This demonstrates that both regularization effects are significant since just having one of them improves the generalization performance compared to the softmax baseline, and having both improves the performance even more.<br /> <br /> ===== Impact of &lt;math&gt;\beta&lt;/math&gt; =====<br /> <br /> The effect of β on generalization performance is studied by training ResNet-18 on CIFAR-10 by tuning the value of β on its own, as well as jointly with λ. It was found that robust generalization performance is obtained for β ∈ [&lt;math&gt;e^{−1}, e^4&lt;/math&gt;] when tuning β on its own; and β ∈ [&lt;math&gt;e^{−4}, e^{8}&lt;/math&gt;] when tuning β jointly with λ. The figure below shows a plot of the error rate with varying β.<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F3.png]]<br /> <br /> === Uncertainty Representation ===<br /> <br /> One of the big advantages of BM is the ability to represent uncertainty about the prediction. The authors evaluate the uncertainty representation on in-distribution (ID) and out-of-distribution (OOD) samples. <br /> <br /> ===== ID uncertainty =====<br /> <br /> For ID (in-distribution) samples, calibration performance is measured, which is a measure of how well the model’s confidence matches its actual accuracy. This measure can be visualized using reliability plots and quantified using a metric called expected calibration error (ECE). ECE is calculated by grouping predictions into M groups based on their confidence score and then finding the absolute difference between the average accuracy and average confidence for each group.<br /> The figure below is a reliability plot of ResNet-50 on CIFAR-10 and CIFAR-100 with 15 groups. It shows that BM has a significantly better calibration performance than softmax since the confidence matches the accuracy more closely (this is also reflected in the lower ECE).<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F4.png]]<br /> <br /> ===== OOD uncertainty =====<br /> <br /> Here, the authors quantify uncertainty using predictive entropy - the larger the predictive entropy, the larger the uncertainty about a prediction. <br /> <br /> The figure below is a density plot of the predictive entropy of ResNet-50 on CIFAR-10. It shows that BM provides significantly better uncertainty estimation compared to other methods since BM is the only method that has a clear peak of high predictive entropy for OOD samples which should have high uncertainty. <br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F5.png]]<br /> <br /> === Transfer learning ===<br /> <br /> Belief matching applies the Bayesian principle outside the neural network, which means it can easily be applied to already trained models. Thus, belief matching can be employed in transfer learning scenarios. The authors downloaded the ImageNet pre-trained ResNet-50 weights and fine-tuned the weights of the last linear layer for 100 epochs using an Adam optimizer.<br /> <br /> This table shows the test error rates from transfer learning on CIFAR-10, Food-101, and Cars datasets. Belief matching consistently performs better than softmax. <br /> <br /> [[File:being_bayesian_about_categorical_probability_transfer_learning.png]]<br /> <br /> Belief matching was also tested for the predictive uncertainty for out of dataset samples based on CIFAR-10 as the in distribution sample. Looking at the figure below, it is observed that belief matching significantly improves the uncertainty representation of pre-trained models by only fine-tuning the last layer’s weights. Note that belief matching confidently predicts examples in Cars since CIFAR-10 contains the object category automobiles. In comparison, softmax produces confident predictions on all datasets. Thus, belief matching could also be used to enhance the uncertainty representation ability of pre-trained models without sacrificing their generalization performance.<br /> <br /> [[File: being_bayesian_about_categorical_probability_transfer_learning_uncertainty.png]]<br /> <br /> === Semi-Supervised Learning ===<br /> <br /> Belief matching’s ability to allow neural networks to represent rich information in their predictions can be exploited to aid consistency based loss function for semi-supervised learning. Consistency-based loss functions use unlabelled samples to determine where to promote the robustness of predictions based on stochastic perturbations. This can be done by perturbing the inputs (which is the VAT model) or the networks (which is the pi-model). Both methods minimize the divergence between two categorical probabilities under some perturbations, thus belief matching can be used by the following replacements in the loss functions. The hope is that belief matching can provide better prediction consistencies using its Dirichlet distributions.<br /> <br /> [[File: being_bayesian_about_categorical_probability_semi_supervised_equation.png]]<br /> <br /> The results of training on ResNet28-2 with consistency based loss functions on CIFAR-10 are shown in this table. Belief matching does have lower classification error rates compared to using a softmax.<br /> <br /> [[File:being_bayesian_about_categorical_probability_semi_supervised_table.png]]<br /> <br /> == Conclusion ==<br /> <br /> Bayesian principles can be used to construct the target distribution by using the categorical probability as a random variable rather than a training label. This can be applied to neural network models by replacing only the softmax and cross-entropy loss, while improving the generalization performance and uncertainty estimation. <br /> <br /> In the future, the authors would like to allow for more expressive distributions in the belief matching framework, such as logistic normal distributions to capture strong semantic similarities among class labels. Furthermore, using input dependent priors would allow for interesting properties that would aid imbalanced datasets and multi-domain learning.<br /> <br /> == Citations ==<br /> <br />  Bridle, J. S. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pp. 227–236. Springer, 1990.<br /> <br />  Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural networks. In International Conference on Machine Learning, 2015.<br /> <br />  Gal, Y. and Ghahramani, Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.<br /> <br />  Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning, 2017. <br /> <br />  MacKay, D. J. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448– 472, 1992.<br /> <br />  Graves, A. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, 2011. <br /> <br />  Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate Bayesian inference. Journal of Machine Learning Research, 18(1):4873–4907, 2017.<br /> <br />  Zhang, G., Sun, S., Duvenaud, D., and Grosse, R. Noisy natural gradient as variational inference. In International Conference of Machine Learning, 2018.<br /> <br />  Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, 2019.<br /> <br />  Osawa, K., Swaroop, S., Jain, A., Eschenhagen, R., Turner, R. E., Yokota, R., and Khan, M. E. Practical deep learning with Bayesian principles. In Advances in Neural Information Processing Systems, 2019.<br /> <br />  Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.<br /> <br />  Neumann, L., Zisserman, A., and Vedaldi, A. Relaxed softmax: Efficient confidence auto-calibration for safe pedestrian detection. In NIPS Workshop on Machine Learning for Intelligent Transportation Systems, 2018.<br /> <br />  Xie, L., Wang, J., Wei, Z., Wang, M., and Tian, Q. Disturblabel: Regularizing cnn on the loss layer. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.<br /> <br />  Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., and Hinton, G. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Being_Bayesian_about_Categorical_Probability&diff=49052 Being Bayesian about Categorical Probability 2020-12-04T00:58:05Z <p>Z42qin: /* Classification With a Neural Network */</p> <hr /> <div>== Presented By ==<br /> Evan Li, Jason Pu, Karam Abuaisha, Nicholas Vadivelu<br /> <br /> == Introduction ==<br /> <br /> Since the outputs of neural networks are not probabilities, Softmax (Bridle, 1990) is a staple for neural network’s performing classification--it exponentiates each logit then normalizes by the sum, giving a distribution over the target classes. However, networks with softmax outputs give no information about uncertainty (Blundell et al., 2015; Gal &amp; Ghahramani, 2016), and the resulting distribution over classes is poorly calibrated (Guo et al., 2017), often giving overconfident predictions even when the classification is wrong. In addition, softmax also raises concerns about overfitting NNs due to its confident predictive behaviors(Xie et al., 2016; Pereyra et al., 2017). To achieve better generalization performance, this may require some effective regularization techniques. <br /> <br /> Bayesian Neural Networks (BNNs; MacKay, 1992) can alleviate these issues, but the resulting posteriors over the parameters are often intractable. Approximations such as variational inference (Graves, 2011; Blundell et al., 2015) and Monte Carlo Dropout (Gal &amp; Ghahramani, 2016) can still be expensive or give poor estimates for the posteriors. This work proposes a Bayesian treatment of the output logits of the neural network, treating the targets as a categorical random variable instead of a fixed label. This gives a computationally cheap way to get well-calibrated uncertainty estimates on neural network classifications.<br /> <br /> == Related Work ==<br /> <br /> Using Bayesian Neural Networks is the dominant way of applying Bayesian techniques to neural networks. Many techniques have been developed to make posterior approximation more accurate and scalable, despite these, BNNs do not scale to the state of the art techniques or large data sets. There are techniques to explicitly avoid modeling the full weight posterior that are more scalable, such as with Monte Carlo Dropout (Gal &amp; Ghahramani, 2016) or tracking mean/covariance of the posterior during training (Mandt et al., 2017; Zhang et al., 2018; Maddox et al., 2019; Osawa et al., 2019). Non-Bayesian uncertainty estimation techniques such as deep ensembles (Lakshminarayanan et al., 2017) and temperature scaling (Guo et al., 2017; Neumann et al., 2018).<br /> <br /> == Preliminaries ==<br /> === Definitions ===<br /> Let's formalize our classification problem and define some notations for the rest of this summary:<br /> <br /> ::Dataset:<br /> $$\mathcal D = \{(x_i,y_i)\} \in (\mathcal X \times \mathcal Y)^N$$<br /> ::General classification model<br /> $$f^W: \mathcal X \to \mathbb R^K$$<br /> ::Softmax function: <br /> $$\phi(x): \mathbb R^K \to [0,1]^K \;\;|\;\; \phi_k(X) = \frac{\exp(f_k^W(x))}{\sum_{k \in K} \exp(f_k^W(x))}$$<br /> ::Softmax activated NN:<br /> $$\phi \;\circ\; f^W: \chi \to \Delta^{K-1}$$<br /> ::NN as a true classifier:<br /> $$arg\max_i \;\circ\; \phi_i \;\circ\; f^W \;:\; \mathcal X \to \mathcal Y$$<br /> <br /> We'll also define the '''count function''' - a &lt;math&gt;K&lt;/math&gt;-vector valued function that outputs the occurences of each class coincident with &lt;math&gt;x&lt;/math&gt;:<br /> $$c^{\mathcal D}(x) = \sum_{(x',y') \in \mathcal D} \mathbb y' I(x' = x)$$<br /> <br /> === Classification With a Neural Network ===<br /> A typical loss function used in classification is cross-entropy. It's well known that optimizing &lt;math&gt;f^W&lt;/math&gt; for &lt;math&gt;l_{CE}&lt;/math&gt; is equivalent to optimizing for &lt;math&gt;l_{KL}&lt;/math&gt;, the &lt;math&gt;KL&lt;/math&gt; divergence between the true distribution and the distribution modeled by NN, that is:<br /> $$l_{KL}(W) = KL(\text{true distribution} \;|\; \text{distribution encoded by }NN(W))$$<br /> Let's introduce notations for the underlying (true) distributions of our problem. Let &lt;math&gt;(x_0,y_0) \sim (\mathcal X \times \mathcal Y)&lt;/math&gt;:<br /> $$\text{Full Distribution} = F(x,y) = P(x_0 = x,y_0 = y)$$<br /> $$\text{Marginal Distribution} = P(x) = F(x_0 = x)$$<br /> $$\text{Point Class Distribution} = P(y_0 = y \;|\; x_0 = x) = F_x(y)$$<br /> Then we have the following factorization:<br /> $$F(x,y) = P(x,y) = P(y|x)P(x) = F_x(y)F(x)$$<br /> Substitute this into the definition of KL divergence:<br /> $$= \sum_{(x,y) \in \mathcal X \times \mathcal Y} F(x,y) \log\left(\frac{F(x,y)}{\phi_y(f^W(x))}\right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) \sum_{y \in \mathcal Y} F(y|x) \log\left( \frac{F(y|x)}{\phi_y(f^W(x))} \right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) \sum_{y \in \mathcal Y} F_x(y) \log\left( \frac{F_x(y)}{\phi_y(f^W(x))} \right)$$<br /> $$= \sum_{x \in \mathcal X} F(x) KL(F_x \;||\; \phi\left( f^W(x) \right))$$<br /> As usual, we don't have an analytic form for &lt;math&gt;l&lt;/math&gt; (if we did, this would imply we know &lt;math&gt;F_X&lt;/math&gt; meaning we knew the distribution in the first place). Instead, estimate from &lt;math&gt;\mathcal D&lt;/math&gt;:<br /> $$F(x) \approx \hat F(x) = \frac{||c^{\mathcal D}(x)||_1}{N}$$<br /> $$F_x(y) \approx \hat F_x(y) = \frac{c^{\mathcal D}(x)}{|| c^{\mathcal D}(x) ||_1}$$<br /> $$\to l_{KL}(W) = \sum_{x \in \mathcal D} \frac{||c^{\mathcal D}(x)||_1}{N} KL \left( \frac{c^{\mathcal D}(x)}{||c^{\mathcal D}(x)||_1} \;||\; \phi(f^W(x)) \right)$$<br /> The approximations &lt;math&gt;\hat F, \hat F_X&lt;/math&gt; are often not very good though: consider a typical classification such as MNIST, we would never expect two handwritten digits to produce the exact same image. Hence &lt;math&gt;c^{\mathcal D}(x)&lt;/math&gt; is (almost) always going to have a single index 1 and the rest 0. This has implications for our approximations:<br /> $$\hat F(x) \text{ is uniform for all } x \in \mathcal D$$<br /> $$\hat F_x(y) \text{ is degenerate for all } x \in \mathcal D$$<br /> This clearly has implications for overfitting: to minimize the KL term in &lt;math&gt;l_{KL}(W)&lt;/math&gt; we want &lt;math&gt;\phi(f^W(x))&lt;/math&gt; to be very close to &lt;math&gt;\hat F_x(y)&lt;/math&gt; at each point - this means that the loss function is in fact encouraging the neural network to output near degenerate distributions! <br /> <br /> '''Labal Smoothing'''<br /> One form of regularization to help this problem is called label smoothing. Instead of using the degenerate $F_x(y)$ as a target function, let's &quot;smooth&quot; it (by adding a scaled uniform distribution to it) so it's no longer degenerate:<br /> $$F'_x(y) = (1-\lambda)\hat F_x(y) + \frac \lambda K \vec 1$$<br /> <br /> '''BNNs'''<br /> BBNs balances the complexity of the model and the distance to target distribution without choosing a single beset configuration (one-hot encoding). Specifically, BNNs with the Gaussian Weight prior $F_x(y) ~ N (0,T^-1 I)$ has score of configuration W measured by the posterior density $p_W(W|D)$<br /> <br /> == Method ==<br /> The main technical proposal of the paper is a Bayesian framework to estimate the (former) target distribution &lt;math&gt;F_x(y)&lt;/math&gt;. That is, we construct a posterior distribution of &lt;math&gt; F_x(y) &lt;/math&gt; and use that as our new target distribution. We call it the ''belief matching'' (BM) framework.<br /> <br /> === Constructing Target Distribution ===<br /> Recall that &lt;math&gt;F_x(y)&lt;/math&gt; is a k-categorical probability distribution - it's PMF can be fully characterized by k numbers that sum to 1. Hence we can encode any such &lt;math&gt;F_x&lt;/math&gt; as a point in &lt;math&gt;\Delta^{k-1}&lt;/math&gt;. We'll do exactly that - let's call this vecor &lt;math&gt;z&lt;/math&gt;:<br /> $$z \in \Delta^{k-1}$$<br /> $$\text{prior} = p_{z|x}(z)$$<br /> $$\text{conditional} = p_{y|z,x}(y)$$<br /> $$\text{posterior} = p_{z|x,y}(z)$$<br /> Then if we perform inference:<br /> $$p_{z|x,y}(z) \propto p_{z|x}(z)p_{y|z,x}(y)$$<br /> The distribution chosen to model prior was &lt;math&gt;dir_K(\beta)&lt;/math&gt;:<br /> $$p_{z|x}(z) = \frac{\Gamma(||\beta||_1)}{\prod_{k=1}^K \Gamma(\beta_k)} \prod_{k=1}^K z_k^{\beta_k - 1}$$<br /> Note that by definition of &lt;math&gt;z&lt;/math&gt;: &lt;math&gt; p_{y|x,z} = z_y &lt;/math&gt;. Since the Dirichlet is a conjugate prior to categorical distributions we have a convenient form for the mean of the posterior:<br /> $$\bar{p_{z|x,y}}(z) = \frac{\beta + c^{\mathcal D}(x)}{||\beta + c^{\mathcal D}(x)||_1} \propto \beta + c^{\mathcal D}(x)$$<br /> This is in fact a generalization of (uniform) label smoothing (label smoothing is a special case where &lt;math&gt;\beta = \frac 1 K \vec{1} &lt;/math&gt;).<br /> <br /> === Representing Approximate Distribution ===<br /> Our new target distribution is &lt;math&gt;p_{z|x,y}(z)&lt;/math&gt; (as opposed to &lt;math&gt;F_x(y)&lt;/math&gt;). That is, we want to construct an interpretation of our neural network weights to construct a distribution with support in &lt;math&gt; \Delta^{K-1} &lt;/math&gt; - the NN can then be trained so this encoded distribution closely approximates &lt;math&gt;p_{z|x,y}&lt;/math&gt;. Let's denote the PMF of this encoded distribution &lt;math&gt;q_{z|x}^W&lt;/math&gt;. This is how the BM framework defines it:<br /> $$\alpha^W(x) := \exp(f^W(x))$$<br /> $$q_{z|x}^W(z) = \frac{\Gamma(||\alpha^W(x)||_1)}{\sum_{k=1}^K \Gamma(\alpha_k^W(x))} \prod_{k=1}^K z_{k}^{\alpha_k^W(x) - 1}$$<br /> $$\to Z^W_x \sim dir(\alpha^W(x))$$<br /> Apply &lt;math&gt;\log&lt;/math&gt; then &lt;math&gt;\exp&lt;/math&gt; to &lt;math&gt;q_{z|x}^W&lt;/math&gt;:<br /> $$q^W_{z|x}(z) \propto \exp \left( \sum_k (\alpha_k^W(x) \log(z_k)) - \sum_k \log(z_k) \right)$$<br /> $$\propto -l_{CE}(\phi(f^W(x)),z) + \frac{K}{||\alpha^W(x)||}KL(\mathcal U_k \;||\; z)$$<br /> It can actually be shown that the mean of &lt;math&gt;Z_x^W&lt;/math&gt; is identical to &lt;math&gt;\phi(f^W(x))&lt;/math&gt; - in other words, if we output the mean of the encoded distribution of our neural network under the BM framework, it is theoretically identical to a traditional neural network.<br /> <br /> === Distribution Matching ===<br /> <br /> We now need a way to fit our approximate distribution from our neural network &lt;math&gt;q_{\mathbf{z | x}}^{\mathbf{W}}&lt;/math&gt; to our target distribution &lt;math&gt;p_{\mathbf{z|x},y}&lt;/math&gt;. The authors achieve this by maximizing the evidence lower bound (ELBO):<br /> <br /> $$l_{EB}(\mathbf y, \alpha^{\mathbf W}(\mathbf x)) = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] - KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; p_{\mathbf{z|x}})$$<br /> <br /> Each term can be computed analytically:<br /> <br /> $$\mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf W }} \left[\log z_y \right] = \psi(\alpha_y^{\mathbf W} ( \mathbf x )) - \psi(\alpha_0^{\mathbf W} ( \mathbf x ))$$<br /> <br /> Where &lt;math&gt;\psi(\cdot)&lt;/math&gt; represents the digamma function (logarithmic derivative of gamma function). Intuitively, we maximize the probability of the correct label. For the KL term:<br /> <br /> $$KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; p_{\mathbf{z|x}}) = \log \frac{\Gamma(a_0^{\mathbf W}(\mathbf x)) \prod_k \Gamma(\beta_k)}{\prod_k \Gamma(\alpha_k^{\mathbf W}(x)) \Gamma (\beta_0)} + \sum_k (\alpha_k^{\mathbf W}(x)-\beta_k)(\psi(\alpha_k^{\mathbf W}(\mathbf x)) - \psi(\alpha_0^{\mathbf W}(\mathbf x))$$<br /> <br /> In the first term, for intuition, we can ignore &lt;math&gt;\alpha_0&lt;/math&gt; and &lt;math&gt;\beta_0&lt;/math&gt; since those just calibrate the distributions. Otherwise, we want the ratio of the products to be as close to 1 as possible to minimize the KL. In the second term, we want to minimize the difference between each individual &lt;math&gt;\alpha_k&lt;/math&gt; and &lt;math&gt;\beta_k&lt;/math&gt;, scaled by the normalized output of the neural network. <br /> <br /> This loss function can be used as a drop-in replacement for the standard softmax cross-entropy, as it has an analytic form and the same time complexity as typical softmax-cross entropy with respect to the number of classes (&lt;math&gt;O(K)&lt;/math&gt;).<br /> <br /> === On Prior Distributions ===<br /> <br /> We must choose our concentration parameter, &lt;math&gt;\beta&lt;/math&gt;, for our dirichlet prior. We see our prior essentially disappears as &lt;math&gt;\beta_0 \to 0&lt;/math&gt; and becomes stronger as &lt;math&gt;\beta_0 \to \infty&lt;/math&gt;. Thus, we want a small &lt;math&gt;\beta_0&lt;/math&gt; so the posterior isn't dominated by the prior. But, the authors claim that a small &lt;math&gt;\beta_0&lt;/math&gt; makes &lt;math&gt;\alpha_0^{\mathbf W}(\mathbf x)&lt;/math&gt; small, which causes &lt;math&gt;\psi (\alpha_0^{\mathbf W}(\mathbf x))&lt;/math&gt; to be large, which is problematic for gradient based optimization. In practice, many neural network techniques aim to make &lt;math&gt;\mathbb E [f^{\mathbf W} (\mathbf x)] \approx \mathbf 0&lt;/math&gt; and thus &lt;math&gt;\mathbb E [\alpha^{\mathbf W} (\mathbf x)] \approx \mathbf 1&lt;/math&gt;, which means making &lt;math&gt;\alpha_0^{\mathbf W}(\mathbf x)&lt;/math&gt; small can be counterproductive.<br /> <br /> So, the authors set &lt;math&gt;\beta = \mathbf 1&lt;/math&gt; and introduce a new hyperparameter &lt;math&gt;\lambda&lt;/math&gt; which is multiplied with the KL term in the ELBO:<br /> <br /> $$l^\lambda_{EB}(\mathbf y, \alpha^{\mathbf W}(\mathbf x)) = \mathbb E_{q_{\mathbf{z | x}}^{\mathbf{W}}} \left[\log p(\mathbf {y | x, z})\right] - \lambda KL (q_{\mathbf{z | x}}^{\mathbf W} \; || \; \mathcal P^D (\mathbf 1))$$<br /> <br /> This stabilizes the optimization, as we can tell from the gradients:<br /> <br /> $$\frac{\partial l_{E B}\left(\mathbf{y}, \alpha^{\mathbf W}(\mathbf{x})\right)}{\partial \alpha_{k}^{\mathbf W}(\mathbf {x})}=\left(\tilde{\mathbf{y}}_{k}-\left(\alpha_{k}^{\mathbf W}(\mathbf{x})-\beta_{k}\right)\right) \psi^{\prime}\left(\alpha_{k}^{\mathbf{W}}(\boldsymbol{x})\right)<br /> -\left(1-\left(\alpha_{0}^{\boldsymbol{W}}(\boldsymbol{x})-\beta_{0}\right)\right) \psi^{\prime}\left(\alpha_{0}^{\boldsymbol{W}}(\boldsymbol{x})\right)$$<br /> <br /> $$\frac{\partial l_{E B}^{\lambda}\left(\mathbf{y}, \alpha^{\mathbf{W}}(\mathbf{x})\right)}{\partial \alpha_{k}^{W}(\mathbf{x})}=\left(\tilde{\mathbf{y}}_{k}-\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})-\lambda\right)\right) \frac{\psi^{\prime}\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})\right)}{\psi^{\prime}\left(\tilde{\alpha}_{0}^{\mathbf W}(\mathbf{x})\right)}<br /> -\left(1-\left(\tilde{\alpha}_{0}^{W}(\mathbf{x})-\lambda K\right)\right)$$<br /> <br /> As we can see, the first expression is affected by the magnitude of &lt;math&gt;\alpha^{\boldsymbol{W}}(\boldsymbol{x})&lt;/math&gt;, whereas the second expression is not due to the &lt;math&gt;\frac{\psi^{\prime}\left(\tilde{\alpha}_{k}^{\mathbf W}(\mathbf{x})\right)}{\psi^{\prime}\left(\tilde{\alpha}_{0}^{\mathbf W}(\mathbf{x})\right)}&lt;/math&gt; ratio.<br /> <br /> == Experiments ==<br /> <br /> Throughout the experiments in this paper, the authors employ various models based on residual connections (He et al., 2016 ) which are the models used for benchmarking in practice. We will first demonstrate improvements provided by BM, then we will show versatility in other applications. For fairness of comparisons, all configurations in the reference implementation will be fixed. The only additions in the experiments are initial learning rate warm-up and gradient clipping which are extremely helpful for stable training of BM. <br /> <br /> === Generalization performance === <br /> The paper compares the generalization performance of BM with softmax and MC dropout on CIFAR-10 and CIFAR-100 benchmarks.<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_T1.png]]<br /> <br /> The next comparison was performed between BM and softmax on the ImageNet benchmark. <br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_T2.png]]<br /> <br /> For both datasets and In all configurations, BM achieves the best generalization and outperforms softmax and MC dropout.<br /> <br /> ===== Regularization effect of prior =====<br /> <br /> In theory, BM has 2 regularization effects:<br /> The prior distribution, which smooths the target posterior<br /> Averaging all of the possible categorical probabilities to compute the distribution matching loss<br /> The authors perform an ablation study to examine the 2 effects separately - removing the KL term in the ELBO removes the effect of the prior distribution.<br /> For ResNet-50 on CIFAR-100 and CIFAR-10 the resulting test error rates were 24.69% and 5.68% respectively. <br /> <br /> This demonstrates that both regularization effects are significant since just having one of them improves the generalization performance compared to the softmax baseline, and having both improves the performance even more.<br /> <br /> ===== Impact of &lt;math&gt;\beta&lt;/math&gt; =====<br /> <br /> The effect of β on generalization performance is studied by training ResNet-18 on CIFAR-10 by tuning the value of β on its own, as well as jointly with λ. It was found that robust generalization performance is obtained for β ∈ [&lt;math&gt;e^{−1}, e^4&lt;/math&gt;] when tuning β on its own; and β ∈ [&lt;math&gt;e^{−4}, e^{8}&lt;/math&gt;] when tuning β jointly with λ. The figure below shows a plot of the error rate with varying β.<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F3.png]]<br /> <br /> === Uncertainty Representation ===<br /> <br /> One of the big advantages of BM is the ability to represent uncertainty about the prediction. The authors evaluate the uncertainty representation on in-distribution (ID) and out-of-distribution (OOD) samples. <br /> <br /> ===== ID uncertainty =====<br /> <br /> For ID (in-distribution) samples, calibration performance is measured, which is a measure of how well the model’s confidence matches its actual accuracy. This measure can be visualized using reliability plots and quantified using a metric called expected calibration error (ECE). ECE is calculated by grouping predictions into M groups based on their confidence score and then finding the absolute difference between the average accuracy and average confidence for each group.<br /> The figure below is a reliability plot of ResNet-50 on CIFAR-10 and CIFAR-100 with 15 groups. It shows that BM has a significantly better calibration performance than softmax since the confidence matches the accuracy more closely (this is also reflected in the lower ECE).<br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F4.png]]<br /> <br /> ===== OOD uncertainty =====<br /> <br /> Here, the authors quantify uncertainty using predictive entropy - the larger the predictive entropy, the larger the uncertainty about a prediction. <br /> <br /> The figure below is a density plot of the predictive entropy of ResNet-50 on CIFAR-10. It shows that BM provides significantly better uncertainty estimation compared to other methods since BM is the only method that has a clear peak of high predictive entropy for OOD samples which should have high uncertainty. <br /> <br /> [[File:Being_Bayesian_about_Categorical_Probability_F5.png]]<br /> <br /> === Transfer learning ===<br /> <br /> Belief matching applies the Bayesian principle outside the neural network, which means it can easily be applied to already trained models. Thus, belief matching can be employed in transfer learning scenarios. The authors downloaded the ImageNet pre-trained ResNet-50 weights and fine-tuned the weights of the last linear layer for 100 epochs using an Adam optimizer.<br /> <br /> This table shows the test error rates from transfer learning on CIFAR-10, Food-101, and Cars datasets. Belief matching consistently performs better than softmax. <br /> <br /> [[File:being_bayesian_about_categorical_probability_transfer_learning.png]]<br /> <br /> Belief matching was also tested for the predictive uncertainty for out of dataset samples based on CIFAR-10 as the in distribution sample. Looking at the figure below, it is observed that belief matching significantly improves the uncertainty representation of pre-trained models by only fine-tuning the last layer’s weights. Note that belief matching confidently predicts examples in Cars since CIFAR-10 contains the object category automobiles. In comparison, softmax produces confident predictions on all datasets. Thus, belief matching could also be used to enhance the uncertainty representation ability of pre-trained models without sacrificing their generalization performance.<br /> <br /> [[File: being_bayesian_about_categorical_probability_transfer_learning_uncertainty.png]]<br /> <br /> === Semi-Supervised Learning ===<br /> <br /> Belief matching’s ability to allow neural networks to represent rich information in their predictions can be exploited to aid consistency based loss function for semi-supervised learning. Consistency-based loss functions use unlabelled samples to determine where to promote the robustness of predictions based on stochastic perturbations. This can be done by perturbing the inputs (which is the VAT model) or the networks (which is the pi-model). Both methods minimize the divergence between two categorical probabilities under some perturbations, thus belief matching can be used by the following replacements in the loss functions. The hope is that belief matching can provide better prediction consistencies using its Dirichlet distributions.<br /> <br /> [[File: being_bayesian_about_categorical_probability_semi_supervised_equation.png]]<br /> <br /> The results of training on ResNet28-2 with consistency based loss functions on CIFAR-10 are shown in this table. Belief matching does have lower classification error rates compared to using a softmax.<br /> <br /> [[File:being_bayesian_about_categorical_probability_semi_supervised_table.png]]<br /> <br /> == Conclusion ==<br /> <br /> Bayesian principles can be used to construct the target distribution by using the categorical probability as a random variable rather than a training label. This can be applied to neural network models by replacing only the softmax and cross-entropy loss, while improving the generalization performance and uncertainty estimation. <br /> <br /> In the future, the authors would like to allow for more expressive distributions in the belief matching framework, such as logistic normal distributions to capture strong semantic similarities among class labels. Furthermore, using input dependent priors would allow for interesting properties that would aid imbalanced datasets and multi-domain learning.<br /> <br /> == Citations ==<br /> <br />  Bridle, J. S. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pp. 227–236. Springer, 1990.<br /> <br />  Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural networks. In International Conference on Machine Learning, 2015.<br /> <br />  Gal, Y. and Ghahramani, Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.<br /> <br />  Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning, 2017. <br /> <br />  MacKay, D. J. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448– 472, 1992.<br /> <br />  Graves, A. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, 2011. <br /> <br />  Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate Bayesian inference. Journal of Machine Learning Research, 18(1):4873–4907, 2017.<br /> <br />  Zhang, G., Sun, S., Duvenaud, D., and Grosse, R. Noisy natural gradient as variational inference. In International Conference of Machine Learning, 2018.<br /> <br />  Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, 2019.<br /> <br />  Osawa, K., Swaroop, S., Jain, A., Eschenhagen, R., Turner, R. E., Yokota, R., and Khan, M. E. Practical deep learning with Bayesian principles. In Advances in Neural Information Processing Systems, 2019.<br /> <br />  Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.<br /> <br />  Neumann, L., Zisserman, A., and Vedaldi, A. Relaxed softmax: Efficient confidence auto-calibration for safe pedestrian detection. In NIPS Workshop on Machine Learning for Intelligent Transportation Systems, 2018.<br /> <br />  Xie, L., Wang, J., Wei, Z., Wang, M., and Tian, Q. Disturblabel: Regularizing cnn on the loss layer. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.<br /> <br />  Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., and Hinton, G. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Loss_Function_Search_for_Face_Recognition&diff=49048 Loss Function Search for Face Recognition 2020-12-03T23:53:00Z <p>Z42qin: /* Introduction */</p> <hr /> <div>== Presented by ==<br /> Jan Lau, Anas Mahdi, Will Thibault, Jiwon Yang<br /> <br /> == Introduction ==<br /> Face recognition is a technology that can label a face to a specific identity. The process involves two tasks: 1. Identifying and classifying a face to a certain identity and 2. Verifying if this face and another face map to the same identity. Loss functions play an important role in evaluating how well the prediction models the given data. In the application of face recognition, they are used for training convolutional neural networks (CNNs) with discriminative features. However, traditional softmax loss lacks the power of feature discrimination. To solve this problem, a center loss was developed to learn centers for each identity to enhance the intra-class compactness. Hence, the paper introduced a new loss function which can reduce the softmax probability. <br /> <br /> Margin-based (angular, additive, additive angular margins) soft-max loss functions are important in learning discriminative features in face recognition. There have been hand-crafted methods previously developed that require much efforts such as A-softmax, V-softmax, AM-Softmax, and Arc-softmax. Li et al. proposed an AutoML for loss function search method also known as AM-LFS from a hyper-parameter optimization perspective . It automatically determines the search space by leveraging reinforcement learning to the search loss functions during the training process, though the drawback is the complex and unstable search space.<br /> <br /> '''Soft Max'''<br /> Softmax probability is the probability for each class. It contains a vector of values that add up to 1 while ranging between 0 and 1. Cross-entropy loss is the negative log of the probabilities. When softmax probability is combined with cross-entropy loss in the last fully connected layer of the CNN, it yields the softmax loss function:<br /> <br /> &lt;center&gt;&lt;math&gt;L_1=-\log\frac{e^{w^T_yx}}{e^{w^T_yx} + \sum_{k≠y}^K{e^{w^T_yx}}}&lt;/math&gt;  &lt;/center&gt;<br /> <br /> <br /> Specifically for face recognition, &lt;math&gt;L_1&lt;/math&gt; is modified such that &lt;math&gt;w^T_yx&lt;/math&gt; is normalized and &lt;math&gt;s&lt;/math&gt; represents the magnitude of &lt;math&gt;w^T_yx&lt;/math&gt;:<br /> <br /> &lt;center&gt;&lt;math&gt;L_2=-\log\frac{e^{s \cos{(\theta_{{w_y},x})}}}{e^{s \cos{(\theta_{{w_y},x})}} + \sum_{k≠y}^K{e^{s \cos{(\theta_{{w_y},x})}}}}&lt;/math&gt;  &lt;/center&gt;<br /> <br /> Where &lt;math&gt; \cos{(\theta_{{w_k},x})} = w^T_y &lt;/math&gt; is cosine similarity and &lt;math&gt;\theta_{{w_k},x}&lt;/math&gt; is angle between &lt;math&gt; w_k&lt;/math&gt; and x. The learnt features with this soft max loss are prone to be separable (as desired).<br /> <br /> '''Margin-based Softmax'''<br /> <br /> This function is crucial in face recognition because it is used for enhancing feature discrimination. While there are different variations of the softmax loss function, they build upon the same structure as the equation above.<br /> <br /> The margin-based softmax function is:<br /> <br /> &lt;center&gt;&lt;math&gt;L_3=-\log\frac{e^{s f{(m,\theta_{{w_y},x})}}}{e^{s f{(m,\theta_{{w_y},x})}} + \sum_{k≠y}^K{e^{s \cos{(\theta_{{w_y},x})}}}} &lt;/math&gt; &lt;/center&gt;<br /> <br /> Here, &lt;math&gt;f{(m,\theta_{{w_y},x})} \leq \cos (\theta_{w_y,x})&lt;/math&gt; is a carefully chosen margin function.<br /> <br /> Some other variations of chosen functions:<br /> <br /> '''A-Softmax Loss:''' &lt;math&gt;f{(m_1,\theta_{{w_y},x})} = \cos (m_1\theta_{w_y,x})&lt;/math&gt; , where m1 &gt;= 1 and a integer.<br /> <br /> '''Arc-Softmax Loss:'''&lt;math&gt;f{(m_1,\theta_{{w_y},x})} = \cos (\theta_{w_y,x} + m_2)&lt;/math&gt;, where m2 &gt; 0<br /> <br /> '''AM-Softmax Loss:'''&lt;math&gt;f{(m,\theta_{{w_y},x})} = \cos (m_1\theta_{w_y,x} + m_2) - m_3&lt;/math&gt;, where m1 &gt;= 1 and a integer; m2,m3 &gt; 0<br /> <br /> <br /> <br /> In this paper, the authors first identified that reducing the softmax probability is a key contribution to feature discrimination and designed two design search spaces (random and reward-guided method). They then evaluated their Random-Softmax and Search-Softmax approaches by comparing the results against other face recognition algorithms using nine popular face recognition benchmarks.<br /> <br /> == Motivation ==<br /> Previous algorithms for facial recognition frequently rely on CNNs that may include metric learning loss functions such as contrastive loss or triplet loss. Without sensitive sample mining strategies, the computational cost for these functions is high. This drawback prompts the redesign of classical softmax loss that cannot discriminate features. Multiple softmax loss functions have since been developed, and including margin-based formulations, they often require fine-tuning of parameters and are susceptible to instability. Therefore, researchers need to put in a lot of effort in creating their method in the large design space. AM-LFS takes an optimization approach for selecting hyperparameters for the margin-based softmax functions, but its aforementioned drawbacks are caused by the lack of direction in designing the search space.<br /> <br /> To solve the issues associated with hand-tuned softmax loss functions and AM-LFS, the authors attempt to reduce the softmax probability to improve feature discrimination when using margin-based softmax loss functions. The development of margin-based softmax loss with only one required parameter and an improved search space using a reward-based method was determined by the authors to be the best option for their loss function.<br /> <br /> == Problem Formulation ==<br /> === Analysis of Margin-based Softmax Loss ===<br /> Based on the softmax probability and the margin-based softmax probability, the following function can be developed :<br /> <br /> &lt;center&gt;&lt;math&gt;p_m=\frac{1}{ap+(1-a)}*p&lt;/math&gt;&lt;/center&gt;<br /> &lt;center&gt; where &lt;math&gt;a=1-e^{s{cos{(\theta_{w_y},x)}-f{(m,\theta_{w_y},x)}}}&lt;/math&gt; and &lt;math&gt;a≤0&lt;/math&gt;&lt;/center&gt;<br /> <br /> &lt;math&gt;a&lt;/math&gt; is considered as a modulating factor and &lt;math&gt;h{(a,p)}=\frac{1}{ap+(1-a)} \in (0,1]&lt;/math&gt; is a modulating function . Therefore, regardless of the margin function (&lt;math&gt;f&lt;/math&gt;), the minimization of the softmax probability will ensure success.<br /> <br /> Compared to AM-LFS, this method involves only one parameter (&lt;math&gt;a&lt;/math&gt;) that is also constrained, versus AM-LFS which has 2M parameters without constraints that specify the piecewise linear functions the method requires. Also, the piecewise linear functions of AM-LFS (&lt;math&gt;p_m={a_i}p+b_i&lt;/math&gt;) may not be discriminative because it could be larger than the softmax probability.<br /> <br /> === Random Search ===<br /> Unified formulation &lt;math&gt;L_5&lt;/math&gt; is generated by inserting a simple modulating function &lt;math&gt;h{(a,p)}=\frac{1}{ap+(1-a)}&lt;/math&gt; into the original softmax loss. It can be written as below :<br /> <br /> &lt;center&gt;&lt;math&gt;L_5=-log{(h{(a,p)}*p)}&lt;/math&gt; where &lt;math&gt;h \in (0,1]&lt;/math&gt; and &lt;math&gt;a≤0&lt;/math&gt;&lt;/center&gt;<br /> <br /> This encourages the feature margin between different classes and has the capability of feature discrimination. This leads to defining the search space as the choice of &lt;math&gt;h{(a,p)}&lt;/math&gt; whose impacts on the training procedure are decided by the modulating factor &lt;math&gt;a&lt;/math&gt;. In order to validate the unified formulation, a modulating factor is randomly set at each training epoch. This is noted as Random-Softmax in this paper.<br /> <br /> === Reward-Guided Search ===<br /> Random search has no guidance for training. To solve this, the authors use reinforcement learning. Unlike supervised learning, reinforcement learning (RL) is a behavioral learning model. It does not need to have input/output labelled and it does not need a sub-optimal action to be explicitly corrected. The algorithm receives feedback from the data to achieve the best outcome. The system has an agent that guides the process by taking an action that maximizes the notion of cumulative reward . The process of RL is shown in figure 1. The equation of the cumulative reward function is: <br /> <br /> &lt;center&gt;&lt;math&gt;G_t \overset{\Delta}{=} R_t+R_{t+1}+R_{t+2}+⋯+R_T&lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;G_t&lt;/math&gt; = cumulative reward, &lt;math&gt;R_t&lt;/math&gt; = immediate reward, and &lt;math&gt;R_T&lt;/math&gt; = end of episode.<br /> <br /> &lt;math&gt;G_t&lt;/math&gt; is the sum of immediate rewards from arbitrary time &lt;math&gt;t&lt;/math&gt;. It is a random variable because it depends on the immediate reward which depends on the agent action and the environment's reaction to this action.<br /> <br /> &lt;center&gt;[[Image:G25_Figure1.png|300px |link=https://en.wikipedia.org/wiki/Reinforcement_learning#/media/File:Reinforcement_learning_diagram.svg |alt=Alt text|Title text]]&lt;/center&gt;<br /> &lt;center&gt;Figure 1: Reinforcement Learning scenario &lt;/center&gt;<br /> <br /> The reward function is what guides the agent to move in a certain direction. As mentioned above, the system receives feedback from the data to achieve the best outcome. This is caused by the reward being edited based on the feedback it receives when a task is completed . <br /> <br /> In this paper, RL is being used to generate a distribution of the hyperparameter &lt;math&gt;\mu&lt;/math&gt; for the SoftMax equation using the reward function. At each epoch, &lt;math&gt;B&lt;/math&gt; hyper-parameters &lt;math&gt;{a_1, a_2, ..., a_B }&lt;/math&gt; are sampled as &lt;math&gt;a \sim \mathcal{N}(\mu, \sigma)&lt;/math&gt;. In each epoch, &lt;math&gt;B&lt;/math&gt; models are generated with rewards &lt;math&gt;R(a_i), i \in [1, B]&lt;/math&gt;. &lt;math&gt;\mu&lt;/math&gt; updates after each epoch from the reward function. <br /> <br /> &lt;center&gt;&lt;math&gt;\mu_{e+1}=\mu_e + \eta \frac{1}{B} \sum_{i=1}^B R{(a_i)}{\nabla_a}log{(g(a_i;\mu,\sigma))}&lt;/math&gt;&lt;/center&gt;<br /> <br /> Where &lt;math&gt;{g(a_i; \mu, \sigma})&lt;/math&gt; is the PDF of a Gaussian distribution. The distributions of &lt;math&gt;{a}&lt;/math&gt; are updated and the best model if found from the &lt;math&gt;{B}&lt;/math&gt; candidates for the next epoch.<br /> <br /> === Optimization ===<br /> Calculating the reward involves a standard bi-level optimization problem. A standard bi-level optimization problem is a hierarchy of two optimization tasks, an upper-level or leader and lower-level or follower problems, which involves a hyperparameter ({&lt;math&gt;a_1,a_2,…,a_B&lt;/math&gt;}) that can be used for minimizing one objective function while maximizing another objective function simultaneously:<br /> <br /> &lt;center&gt;&lt;math&gt;max_a R(a)=r(M_{w^*(a)},S_v)&lt;/math&gt;&lt;/center&gt;<br /> &lt;center&gt;&lt;math&gt;w^*(a)=_w \sum_{(x,y) \in S_t} L^a (M_w(x),y)&lt;/math&gt;&lt;/center&gt;<br /> <br /> In this case, the loss function takes the training set &lt;math&gt;S_t&lt;/math&gt; and the reward function takes the validation set &lt;math&gt;S_v&lt;/math&gt;. The weights &lt;math&gt;w&lt;/math&gt; are trained such that the loss function is minimized while the reward function is maximized. The calculated reward for each model ({&lt;math&gt;M_{we1},M_{we2},…,M_{weB}&lt;/math&gt;}) yields the corresponding score, then the algorithm chooses the one with the highest score for model index selection. With the model containing the highest score being used in the next epoch, this process is repeated until the training reaches convergence. In the end, the algorithm takes the model with the highest score without retraining.<br /> <br /> == Results and Discussion ==<br /> === Data Preprocessing ===<br /> The training datasets consisted of cleaned versions of CASIA-WebFace and MS-Celeb-1M-v1c to remove the impact of noisy labels in the original sets.<br /> Furthermore, it is important to perform open-set evaluation for face recognition problem. That is, there shall be no overlapping identities between training and testing sets. As a result, there were a total of 15,414 identities removed from the testing sets. For fairness during comparison, all summarized results will be based on refined datasets.<br /> <br /> === Results on LFW, SLLFW, CALFW, CPLFW, AgeDB, DFP ===<br /> For LFW, there is not a noticeable difference between the algorithms proposed in this paper and the other algorithms, however, AM-Softmax achieved higher results than Search-Softmax. Random-Softmax achieved the highest results by 0.03%.<br /> <br /> Random-Softmax outperforms baseline Soft-max and is comparable to most of the margin-based softmax. Search-Softmax boost the performance and better most methods specifically when training CASIA-WebFace-R data set, it achieves 0.72% average improvement over AM-Softmax. The reason the model proposed by the paper gives better results is because of their optimization strategy which helps boost the discimination power. Also the sampled candidate from the paper’s proposed search space can well approximate the margin-based loss functions. More tests need to happen to more complicated protocols to test the performance further. Not a lot of improvement has been shown on those test sets, since they are relatively simple and the performance of all the methods on these test sets are near saturation. The following table gives a summary of the performance of each model.<br /> <br /> &lt;center&gt;Table 1.Verification performance (%) of different methods on the test sets LFW, SLLFW, CALFW, CPLFW, AgeDB and CFP. The training set is '''CASIA-WebFace-R''' .&lt;/center&gt;<br /> <br /> &lt;center&gt;[[Image:G25_Table1.png|900px |alt=Alt text|Title text]]&lt;/center&gt;<br /> <br /> === Results on RFW ===<br /> The RFW dataset measures racial bias which consists of Caucasian, Indian, Asian, and African. Using this as the test set, Random-softmax and Search-softmax performed better than the other methods. Random-softmax outperforms the baseline softmax by a large margin which means reducing the softmax probability will enhance the feature discrimination for face recognition. It is also observed that the reward guided search-softmax method is more likely to enhance the discriminative feature learning resulting in higher performance as shown in Table 2 and Table 3. <br /> <br /> &lt;center&gt;Table 2. Verification performance (%) of different methods on the test set RFW. The training set is '''CASIA-WebFace-R''' .&lt;/center&gt;<br /> &lt;center&gt;[[Image:G25_Table2.png|500px |alt=Alt text|Title text]]&lt;/center&gt;<br /> <br /> <br /> &lt;center&gt;Table 3. Verification performance (%) of different methods on the test set RFW. The training set is '''MS-Celeb-1M-v1c-R''' .&lt;/center&gt;<br /> &lt;center&gt;[[Image:G25_Table3.png|500px |alt=Alt text|Title text]]&lt;/center&gt;<br /> <br /> === Results on MegaFace and Trillion-Pairs ===<br /> The different loss functions are tested again with more complicated protocols. The identification (Id.) Rank-1 and the verification (Veri.) with the true positive rate (TPR) at low false acceptance rate (FAR) at &lt;math&gt;1e-3&lt;/math&gt; on MegaFace, the identification TPR@FAR = &lt;math&gt;1e-6&lt;/math&gt; and the verification TPR@FAR = &lt;math&gt;1e-9&lt;/math&gt; on Trillion-Pairs are reported on Table 4 and 5.<br /> <br /> On the test sets MegaFace and Trillion-Pairs, Search-softmax achieves the best performance over all other alternative methods. On MegaFace, Search-softmax beat the best competitor AM-softmax by a large margin. It also outperformed AM-LFS due to new designed search space. <br /> <br /> &lt;center&gt;Table 4. Performance (%) of different loss functions on the test sets MegaFace and Trillion-Pairs. The training set is '''CASIA-WebFace-R''' .&lt;/center&gt;<br /> &lt;center&gt;[[Image:G25_Table4.png|450px |alt=Alt text|Title text]]&lt;/center&gt;<br /> <br /> <br /> &lt;center&gt;Table 5. Performance (%) of different loss functions on the test sets MegaFace and Trillion-Pairs. The training set is '''MS-Celeb-1M-v1c-R''' .&lt;/center&gt;<br /> &lt;center&gt;[[Image:G25_Table5.png|450px |alt=Alt text|Title text]]&lt;/center&gt;<br /> <br /> From the CMC curves and ROC curves in Figure 2, similar trends are observed at other measures. There is a same trend on Trillion-Pairs where Search-softmax loss is found to be superior with 4% improvements with CASIA-WebFace-R and 1% improvements with MS-Celeb-1M-v1c-R at both the identification and verification. Based on these experiments, Search-Softmax loss can perform well, especially with a low false positive rate and it shows a strong generalization ability for face recognition.<br /> <br /> &lt;center&gt;[[Image:G25_Figure2_left.png|800px |alt=Alt text|Title text]] [[Image:G25_Figure2_right.png|800px |alt=Alt text|Title text]]&lt;/center&gt;<br /> &lt;center&gt;Figure 2. From Left to Right: CMC curves and ROC curves on MegaFace Set with training set CASIA-WebFace-R, CMC curves and ROC curves on MegaFace Set with training set MS-Celeb-1M-v1c-R .&lt;/center&gt;<br /> <br /> == Conclusion ==<br /> In this paper, it is discussed that in order to enhance feature discrimination for face recognition, it is key to know how to reduce the softmax probability. To achieve this goal, unified formulation for the margin-based softmax losses is designed. Two search methods have been developed using a random and a reward-guided loss function and they were validated to be effective over six other methods using nine different test data sets. While these developed methods were generally more effective in increasing accuracy versus previous methods, there is very little difference between the two. It can be seen that Search-Softmax performs slightly better than Random-Softmax most of the time.<br /> <br /> == Critiques ==<br /> * Thorough experimentation and comparison of results to state-of-the-art provided a convincing argument​.<br /> * Datasets used did require some preprocessing, which may have improved the results beyond what the method otherwise would​.<br /> * AM-LFS was created by the authors for experimentation (the code was not made public) so the comparison may not be accurate​.<br /> * The test data set they used to test Search-Softmax and Random-Softmax are simple and they saturate in other methods. So the results of their methods didn’t show many advantages since they produce very similar results. A more complicated data set needs to be tested to prove the method's reliability.<br /> * There is another paper Large-Margin Softmax Loss for Convolutional Neural Networks[https://arxiv.org/pdf/1612.02295.pdf] that provides a more detailed explanation about how to reduce margin-based softmax loss.<br /> * It is questionable when it comes to the accuracy of testing sets, as they only used the clean version of CASIA-WebFace and MS-Celeb-1M-vlc for training instead of these two training sets with noisy labels.<br /> * In a similar [https://arxiv.org/pdf/1905.09773.pdf?utm_source=thenewstack&amp;utm_medium=website&amp;utm_campaign=platform paper], written by Tae-Hyun Oh et al., they also discuss an optimal loss function for face recognition. However, since in the other paper, they were doing face recognition from voice audio, the loss function used was slightly different than the ones discussed in this paper.<br /> * This model has many applications such as identifying disguised prisoners for police. But we need to do a good data preprocessing otherwise we might not get a good predicted result. But authors did not mention about the data preprocessing which is a key part of this model.<br /> * It will be better if we can know what kind of noises was removed in the clean version. Also, simply removing the overlapping data is wasteful. It would be better to just put them into one of the train and test samples.<br /> * This paper indicate that the new searching method and loss function have induced more effective face recognition result than other six methods. But there is no mention of the increase or decrease in computational efficiency since only very little difference exist between those methods and the real time evaluation is often required at the face recognition application level.<br /> * There are some loss functions that receives more than 2 inputs. For example, the ''triplet loss'' function, developed by Google, takes 3 inputs: positive input, negative input and anchor input. This makes sense because for face recognition, we want to model to learn not only what it is supposed to predict but also what it is not supposed to predict. Typically, triplet loss handles false positives much better. This paper can extend its scope to such loss function that takes more than 2 inputs.<br /> * It would be good to also know what the training time is like for the method, specifically the &quot;Reward-Guided Search&quot; which uses RL. Also the authors mention some data preprocessing that was performed, was this same preprocessing also performed for the methods they compared against?<br /> <br /> == References ==<br />  X. Wang, S. Wang, C. Chi, S. Zhang and T. Mei, &quot;Loss Function Search for Face Recognition&quot;, in International Conference on Machine Learning, 2020, pp. 1-10.<br /> <br />  Li, C., Yuan, X., Lin, C., Guo, M., Wu, W., Yan, J., and Ouyang, W. Am-lfs: Automl for loss function search. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8410–8419, 2019.<br /> 2020].<br /> <br />  S. L. AI, “Reinforcement Learning algorithms - an intuitive overview,” Medium, 18-Feb-2019. [Online]. Available: https://medium.com/@SmartLabAI/reinforcement-learning-algorithms-an-intuitive-overview-904e2dff5bbc. [Accessed: 25-Nov-2020]. <br /> <br />  “Reinforcement learning,” Wikipedia, 17-Nov-2020. [Online]. Available: https://en.wikipedia.org/wiki/Reinforcement_learning. [Accessed: 24-Nov-2020].<br /> <br />  B. Osiński, “What is reinforcement learning? The complete guide,” deepsense.ai, 23-Jul-2020. [Online]. Available: https://deepsense.ai/what-is-reinforcement-learning-the-complete-guide/. [Accessed: 25-Nov-2020].</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Loss_Function_Search_for_Face_Recognition&diff=49047 Loss Function Search for Face Recognition 2020-12-03T23:45:54Z <p>Z42qin: </p> <hr /> <div>== Presented by ==<br /> Jan Lau, Anas Mahdi, Will Thibault, Jiwon Yang<br /> <br /> == Introduction ==<br /> Face recognition is a technology that can label a face to a specific identity. The process involves two tasks: 1. Identifying and classifying a face to a certain identity and 2. Verifying if this face and another face map to the same identity. Loss functions play an important role in evaluating how well the prediction models the given data. In the application of face recognition, they are used for training convolutional neural networks (CNNs) with discriminative features. However, traditional softmax loss lacks the power of feature discrimination. To solve this problem, a center loss was developed to learn centers for each identity to enhance the intra-class compactness. Hence, the paper introduced a new loss function which can reduce the softmax probability. <br /> <br /> Margin-based (angular, additive, additive angular margins) soft-max loss functions are important in learning discriminative features in face recognition. There have been hand-crafted methods previously developed that require much efforts such as A-softmax, V-softmax, AM-Softmax, and Arc-softmax. Li et al. proposed an AutoML for loss function search method also known as AM-LFS from a hyper-parameter optimization perspective . It automatically determines the search space by leveraging reinforcement learning to the search loss functions during the training process, though the drawback is the complex and unstable search space.<br /> <br /> '''Soft Max'''<br /> Softmax probability is the probability for each class. It contains a vector of values that add up to 1 while ranging between 0 and 1. Cross-entropy loss is the negative log of the probabilities. When softmax probability is combined with cross-entropy loss in the last fully connected layer of the CNN, it yields the softmax loss function:<br /> <br /> &lt;center&gt;&lt;math&gt;L_1=-\log\frac{e^{w^T_yx}}{e^{w^T_yx} + \sum_{k≠y}^K{e^{w^T_yx}}}&lt;/math&gt;  &lt;/center&gt;<br /> <br /> <br /> Specifically for face recognition, &lt;math&gt;L_1&lt;/math&gt; is modified such that &lt;math&gt;w^T_yx&lt;/math&gt; is normalized and &lt;math&gt;s&lt;/math&gt; represents the magnitude of &lt;math&gt;w^T_yx&lt;/math&gt;:<br /> <br /> &lt;center&gt;&lt;math&gt;L_2=-\log\frac{e^{s \cos{(\theta_{{w_y},x})}}}{e^{s \cos{(\theta_{{w_y},x})}} + \sum_{k≠y}^K{e^{s \cos{(\theta_{{w_y},x})}}}}&lt;/math&gt;  &lt;/center&gt;<br /> <br /> Where &lt;math&gt; \cos{(\theta_{{w_k},x})} = w^T_y &lt;/math&gt; is cosine similarity and &lt;math&gt;\theta_{{w_k},x}&lt;/math&gt; is angle between &lt;math&gt; w_k&lt;/math&gt; and x. The learnt features with this soft max loss are prone to be separable (as desired).<br /> <br /> '''Margin-based Softmax'''<br /> <br /> This function is crucial in face recognition because it is used for enhancing feature discrimination. While there are different variations of the softmax loss function, they build upon the same structure as the equation above.<br /> <br /> The margin-based softmax function is:<br /> <br /> &lt;center&gt;&lt;math&gt;L_3=-\log\frac{e^{s f{(m,\theta_{{w_y},x})}}}{e^{s f{(m,\theta_{{w_y},x})}} + \sum_{k≠y}^K{e^{s \cos{(\theta_{{w_y},x})}}}} &lt;/math&gt; &lt;/center&gt;<br /> <br /> Here, &lt;math&gt;f{(m,\theta_{{w_y},x})} \leq \cos (\theta_{w_y,x})&lt;/math&gt; is a carefully chosen margin function.<br /> <br /> Some other variations of softmax (including A-Softmax loss, Arc-Soft max loss, and AM-Softmax loss) will be discussed in detail in the later sections. <br /> <br /> In this paper, the authors first identified that reducing the softmax probability is a key contribution to feature discrimination and designed two design search spaces (random and reward-guided method). They then evaluated their Random-Softmax and Search-Softmax approaches by comparing the results against other face recognition algorithms using nine popular face recognition benchmarks.<br /> <br /> <br /> == Motivation ==<br /> Previous algorithms for facial recognition frequently rely on CNNs that may include metric learning loss functions such as contrastive loss or triplet loss. Without sensitive sample mining strategies, the computational cost for these functions is high. This drawback prompts the redesign of classical softmax loss that cannot discriminate features. Multiple softmax loss functions have since been developed, and including margin-based formulations, they often require fine-tuning of parameters and are susceptible to instability. Therefore, researchers need to put in a lot of effort in creating their method in the large design space. AM-LFS takes an optimization approach for selecting hyperparameters for the margin-based softmax functions, but its aforementioned drawbacks are caused by the lack of direction in designing the search space.<br /> <br /> To solve the issues associated with hand-tuned softmax loss functions and AM-LFS, the authors attempt to reduce the softmax probability to improve feature discrimination when using margin-based softmax loss functions. The development of margin-based softmax loss with only one required parameter and an improved search space using a reward-based method was determined by the authors to be the best option for their loss function.<br /> <br /> == Problem Formulation ==<br /> === Analysis of Margin-based Softmax Loss ===<br /> Based on the softmax probability and the margin-based softmax probability, the following function can be developed :<br /> <br /> &lt;center&gt;&lt;math&gt;p_m=\frac{1}{ap+(1-a)}*p&lt;/math&gt;&lt;/center&gt;<br /> &lt;center&gt; where &lt;math&gt;a=1-e^{s{cos{(\theta_{w_y},x)}-f{(m,\theta_{w_y},x)}}}&lt;/math&gt; and &lt;math&gt;a≤0&lt;/math&gt;&lt;/center&gt;<br /> <br /> &lt;math&gt;a&lt;/math&gt; is considered as a modulating factor and &lt;math&gt;h{(a,p)}=\frac{1}{ap+(1-a)} \in (0,1]&lt;/math&gt; is a modulating function . Therefore, regardless of the margin function (&lt;math&gt;f&lt;/math&gt;), the minimization of the softmax probability will ensure success.<br /> <br /> Compared to AM-LFS, this method involves only one parameter (&lt;math&gt;a&lt;/math&gt;) that is also constrained, versus AM-LFS which has 2M parameters without constraints that specify the piecewise linear functions the method requires. Also, the piecewise linear functions of AM-LFS (&lt;math&gt;p_m={a_i}p+b_i&lt;/math&gt;) may not be discriminative because it could be larger than the softmax probability.<br /> <br /> === Random Search ===<br /> Unified formulation &lt;math&gt;L_5&lt;/math&gt; is generated by inserting a simple modulating function &lt;math&gt;h{(a,p)}=\frac{1}{ap+(1-a)}&lt;/math&gt; into the original softmax loss. It can be written as below :<br /> <br /> &lt;center&gt;&lt;math&gt;L_5=-log{(h{(a,p)}*p)}&lt;/math&gt; where &lt;math&gt;h \in (0,1]&lt;/math&gt; and &lt;math&gt;a≤0&lt;/math&gt;&lt;/center&gt;<br /> <br /> This encourages the feature margin between different classes and has the capability of feature discrimination. This leads to defining the search space as the choice of &lt;math&gt;h{(a,p)}&lt;/math&gt; whose impacts on the training procedure are decided by the modulating factor &lt;math&gt;a&lt;/math&gt;. In order to validate the unified formulation, a modulating factor is randomly set at each training epoch. This is noted as Random-Softmax in this paper.<br /> <br /> === Reward-Guided Search ===<br /> Random search has no guidance for training. To solve this, the authors use reinforcement learning. Unlike supervised learning, reinforcement learning (RL) is a behavioral learning model. It does not need to have input/output labelled and it does not need a sub-optimal action to be explicitly corrected. The algorithm receives feedback from the data to achieve the best outcome. The system has an agent that guides the process by taking an action that maximizes the notion of cumulative reward . The process of RL is shown in figure 1. The equation of the cumulative reward function is: <br /> <br /> &lt;center&gt;&lt;math&gt;G_t \overset{\Delta}{=} R_t+R_{t+1}+R_{t+2}+⋯+R_T&lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;G_t&lt;/math&gt; = cumulative reward, &lt;math&gt;R_t&lt;/math&gt; = immediate reward, and &lt;math&gt;R_T&lt;/math&gt; = end of episode.<br /> <br /> &lt;math&gt;G_t&lt;/math&gt; is the sum of immediate rewards from arbitrary time &lt;math&gt;t&lt;/math&gt;. It is a random variable because it depends on the immediate reward which depends on the agent action and the environment's reaction to this action.<br /> <br /> &lt;center&gt;[[Image:G25_Figure1.png|300px |link=https://en.wikipedia.org/wiki/Reinforcement_learning#/media/File:Reinforcement_learning_diagram.svg |alt=Alt text|Title text]]&lt;/center&gt;<br /> &lt;center&gt;Figure 1: Reinforcement Learning scenario &lt;/center&gt;<br /> <br /> The reward function is what guides the agent to move in a certain direction. As mentioned above, the system receives feedback from the data to achieve the best outcome. This is caused by the reward being edited based on the feedback it receives when a task is completed . <br /> <br /> In this paper, RL is being used to generate a distribution of the hyperparameter &lt;math&gt;\mu&lt;/math&gt; for the SoftMax equation using the reward function. At each epoch, &lt;math&gt;B&lt;/math&gt; hyper-parameters &lt;math&gt;{a_1, a_2, ..., a_B }&lt;/math&gt; are sampled as &lt;math&gt;a \sim \mathcal{N}(\mu, \sigma)&lt;/math&gt;. In each epoch, &lt;math&gt;B&lt;/math&gt; models are generated with rewards &lt;math&gt;R(a_i), i \in [1, B]&lt;/math&gt;. &lt;math&gt;\mu&lt;/math&gt; updates after each epoch from the reward function. <br /> <br /> &lt;center&gt;&lt;math&gt;\mu_{e+1}=\mu_e + \eta \frac{1}{B} \sum_{i=1}^B R{(a_i)}{\nabla_a}log{(g(a_i;\mu,\sigma))}&lt;/math&gt;&lt;/center&gt;<br /> <br /> Where &lt;math&gt;{g(a_i; \mu, \sigma})&lt;/math&gt; is the PDF of a Gaussian distribution. The distributions of &lt;math&gt;{a}&lt;/math&gt; are updated and the best model if found from the &lt;math&gt;{B}&lt;/math&gt; candidates for the next epoch.<br /> <br /> === Optimization ===<br /> Calculating the reward involves a standard bi-level optimization problem. A standard bi-level optimization problem is a hierarchy of two optimization tasks, an upper-level or leader and lower-level or follower problems, which involves a hyperparameter ({&lt;math&gt;a_1,a_2,…,a_B&lt;/math&gt;}) that can be used for minimizing one objective function while maximizing another objective function simultaneously:<br /> <br /> &lt;center&gt;&lt;math&gt;max_a R(a)=r(M_{w^*(a)},S_v)&lt;/math&gt;&lt;/center&gt;<br /> &lt;center&gt;&lt;math&gt;w^*(a)=_w \sum_{(x,y) \in S_t} L^a (M_w(x),y)&lt;/math&gt;&lt;/center&gt;<br /> <br /> In this case, the loss function takes the training set &lt;math&gt;S_t&lt;/math&gt; and the reward function takes the validation set &lt;math&gt;S_v&lt;/math&gt;. The weights &lt;math&gt;w&lt;/math&gt; are trained such that the loss function is minimized while the reward function is maximized. The calculated reward for each model ({&lt;math&gt;M_{we1},M_{we2},…,M_{weB}&lt;/math&gt;}) yields the corresponding score, then the algorithm chooses the one with the highest score for model index selection. With the model containing the highest score being used in the next epoch, this process is repeated until the training reaches convergence. In the end, the algorithm takes the model with the highest score without retraining.<br /> <br /> == Results and Discussion ==<br /> === Data Preprocessing ===<br /> The training datasets consisted of cleaned versions of CASIA-WebFace and MS-Celeb-1M-v1c to remove the impact of noisy labels in the original sets.<br /> Furthermore, it is important to perform open-set evaluation for face recognition problem. That is, there shall be no overlapping identities between training and testing sets. As a result, there were a total of 15,414 identities removed from the testing sets. For fairness during comparison, all summarized results will be based on refined datasets.<br /> <br /> === Results on LFW, SLLFW, CALFW, CPLFW, AgeDB, DFP ===<br /> For LFW, there is not a noticeable difference between the algorithms proposed in this paper and the other algorithms, however, AM-Softmax achieved higher results than Search-Softmax. Random-Softmax achieved the highest results by 0.03%.<br /> <br /> Random-Softmax outperforms baseline Soft-max and is comparable to most of the margin-based softmax. Search-Softmax boost the performance and better most methods specifically when training CASIA-WebFace-R data set, it achieves 0.72% average improvement over AM-Softmax. The reason the model proposed by the paper gives better results is because of their optimization strategy which helps boost the discimination power. Also the sampled candidate from the paper’s proposed search space can well approximate the margin-based loss functions. More tests need to happen to more complicated protocols to test the performance further. Not a lot of improvement has been shown on those test sets, since they are relatively simple and the performance of all the methods on these test sets are near saturation. The following table gives a summary of the performance of each model.<br /> <br /> &lt;center&gt;Table 1.Verification performance (%) of different methods on the test sets LFW, SLLFW, CALFW, CPLFW, AgeDB and CFP. The training set is '''CASIA-WebFace-R''' .&lt;/center&gt;<br /> <br /> &lt;center&gt;[[Image:G25_Table1.png|900px |alt=Alt text|Title text]]&lt;/center&gt;<br /> <br /> === Results on RFW ===<br /> The RFW dataset measures racial bias which consists of Caucasian, Indian, Asian, and African. Using this as the test set, Random-softmax and Search-softmax performed better than the other methods. Random-softmax outperforms the baseline softmax by a large margin which means reducing the softmax probability will enhance the feature discrimination for face recognition. It is also observed that the reward guided search-softmax method is more likely to enhance the discriminative feature learning resulting in higher performance as shown in Table 2 and Table 3. <br /> <br /> &lt;center&gt;Table 2. Verification performance (%) of different methods on the test set RFW. The training set is '''CASIA-WebFace-R''' .&lt;/center&gt;<br /> &lt;center&gt;[[Image:G25_Table2.png|500px |alt=Alt text|Title text]]&lt;/center&gt;<br /> <br /> <br /> &lt;center&gt;Table 3. Verification performance (%) of different methods on the test set RFW. The training set is '''MS-Celeb-1M-v1c-R''' .&lt;/center&gt;<br /> &lt;center&gt;[[Image:G25_Table3.png|500px |alt=Alt text|Title text]]&lt;/center&gt;<br /> <br /> === Results on MegaFace and Trillion-Pairs ===<br /> The different loss functions are tested again with more complicated protocols. The identification (Id.) Rank-1 and the verification (Veri.) with the true positive rate (TPR) at low false acceptance rate (FAR) at &lt;math&gt;1e-3&lt;/math&gt; on MegaFace, the identification TPR@FAR = &lt;math&gt;1e-6&lt;/math&gt; and the verification TPR@FAR = &lt;math&gt;1e-9&lt;/math&gt; on Trillion-Pairs are reported on Table 4 and 5.<br /> <br /> On the test sets MegaFace and Trillion-Pairs, Search-softmax achieves the best performance over all other alternative methods. On MegaFace, Search-softmax beat the best competitor AM-softmax by a large margin. It also outperformed AM-LFS due to new designed search space. <br /> <br /> &lt;center&gt;Table 4. Performance (%) of different loss functions on the test sets MegaFace and Trillion-Pairs. The training set is '''CASIA-WebFace-R''' .&lt;/center&gt;<br /> &lt;center&gt;[[Image:G25_Table4.png|450px |alt=Alt text|Title text]]&lt;/center&gt;<br /> <br /> <br /> &lt;center&gt;Table 5. Performance (%) of different loss functions on the test sets MegaFace and Trillion-Pairs. The training set is '''MS-Celeb-1M-v1c-R''' .&lt;/center&gt;<br /> &lt;center&gt;[[Image:G25_Table5.png|450px |alt=Alt text|Title text]]&lt;/center&gt;<br /> <br /> From the CMC curves and ROC curves in Figure 2, similar trends are observed at other measures. There is a same trend on Trillion-Pairs where Search-softmax loss is found to be superior with 4% improvements with CASIA-WebFace-R and 1% improvements with MS-Celeb-1M-v1c-R at both the identification and verification. Based on these experiments, Search-Softmax loss can perform well, especially with a low false positive rate and it shows a strong generalization ability for face recognition.<br /> <br /> &lt;center&gt;[[Image:G25_Figure2_left.png|800px |alt=Alt text|Title text]] [[Image:G25_Figure2_right.png|800px |alt=Alt text|Title text]]&lt;/center&gt;<br /> &lt;center&gt;Figure 2. From Left to Right: CMC curves and ROC curves on MegaFace Set with training set CASIA-WebFace-R, CMC curves and ROC curves on MegaFace Set with training set MS-Celeb-1M-v1c-R .&lt;/center&gt;<br /> <br /> == Conclusion ==<br /> In this paper, it is discussed that in order to enhance feature discrimination for face recognition, it is key to know how to reduce the softmax probability. To achieve this goal, unified formulation for the margin-based softmax losses is designed. Two search methods have been developed using a random and a reward-guided loss function and they were validated to be effective over six other methods using nine different test data sets. While these developed methods were generally more effective in increasing accuracy versus previous methods, there is very little difference between the two. It can be seen that Search-Softmax performs slightly better than Random-Softmax most of the time.<br /> <br /> == Critiques ==<br /> * Thorough experimentation and comparison of results to state-of-the-art provided a convincing argument​.<br /> * Datasets used did require some preprocessing, which may have improved the results beyond what the method otherwise would​.<br /> * AM-LFS was created by the authors for experimentation (the code was not made public) so the comparison may not be accurate​.<br /> * The test data set they used to test Search-Softmax and Random-Softmax are simple and they saturate in other methods. So the results of their methods didn’t show many advantages since they produce very similar results. A more complicated data set needs to be tested to prove the method's reliability.<br /> * There is another paper Large-Margin Softmax Loss for Convolutional Neural Networks[https://arxiv.org/pdf/1612.02295.pdf] that provides a more detailed explanation about how to reduce margin-based softmax loss.<br /> * It is questionable when it comes to the accuracy of testing sets, as they only used the clean version of CASIA-WebFace and MS-Celeb-1M-vlc for training instead of these two training sets with noisy labels.<br /> * In a similar [https://arxiv.org/pdf/1905.09773.pdf?utm_source=thenewstack&amp;utm_medium=website&amp;utm_campaign=platform paper], written by Tae-Hyun Oh et al., they also discuss an optimal loss function for face recognition. However, since in the other paper, they were doing face recognition from voice audio, the loss function used was slightly different than the ones discussed in this paper.<br /> * This model has many applications such as identifying disguised prisoners for police. But we need to do a good data preprocessing otherwise we might not get a good predicted result. But authors did not mention about the data preprocessing which is a key part of this model.<br /> * It will be better if we can know what kind of noises was removed in the clean version. Also, simply removing the overlapping data is wasteful. It would be better to just put them into one of the train and test samples.<br /> * This paper indicate that the new searching method and loss function have induced more effective face recognition result than other six methods. But there is no mention of the increase or decrease in computational efficiency since only very little difference exist between those methods and the real time evaluation is often required at the face recognition application level.<br /> * There are some loss functions that receives more than 2 inputs. For example, the ''triplet loss'' function, developed by Google, takes 3 inputs: positive input, negative input and anchor input. This makes sense because for face recognition, we want to model to learn not only what it is supposed to predict but also what it is not supposed to predict. Typically, triplet loss handles false positives much better. This paper can extend its scope to such loss function that takes more than 2 inputs.<br /> * It would be good to also know what the training time is like for the method, specifically the &quot;Reward-Guided Search&quot; which uses RL. Also the authors mention some data preprocessing that was performed, was this same preprocessing also performed for the methods they compared against?<br /> <br /> == References ==<br />  X. Wang, S. Wang, C. Chi, S. Zhang and T. Mei, &quot;Loss Function Search for Face Recognition&quot;, in International Conference on Machine Learning, 2020, pp. 1-10.<br /> <br />  Li, C., Yuan, X., Lin, C., Guo, M., Wu, W., Yan, J., and Ouyang, W. Am-lfs: Automl for loss function search. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8410–8419, 2019.<br /> 2020].<br /> <br />  S. L. AI, “Reinforcement Learning algorithms - an intuitive overview,” Medium, 18-Feb-2019. [Online]. Available: https://medium.com/@SmartLabAI/reinforcement-learning-algorithms-an-intuitive-overview-904e2dff5bbc. [Accessed: 25-Nov-2020]. <br /> <br />  “Reinforcement learning,” Wikipedia, 17-Nov-2020. [Online]. Available: https://en.wikipedia.org/wiki/Reinforcement_learning. [Accessed: 24-Nov-2020].<br /> <br />  B. Osiński, “What is reinforcement learning? The complete guide,” deepsense.ai, 23-Jul-2020. [Online]. Available: https://deepsense.ai/what-is-reinforcement-learning-the-complete-guide/. [Accessed: 25-Nov-2020].</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Improving_neural_networks_by_preventing_co-adaption_of_feature_detectors&diff=49044 Improving neural networks by preventing co-adaption of feature detectors 2020-12-03T23:02:30Z <p>Z42qin: /* Improvement Intro */</p> <hr /> <div>== Presented by ==<br /> Stan Lee, Seokho Lim, Kyle Jung, Dae Hyun Kim<br /> <br /> = Improvement Intro =<br /> '''Drop Out Model'''<br /> <br /> In this paper, Hinton et al. introduce a novel way to improve neural networks’ performance, particularly in the case that a large feedforward neural network is trained on a small training set, which causes poor performance and leads to “overfitting” problem. This problem can be reduced by randomly omitting half of the feature detectors on each training case. In fact, By omitting neurons in hidden layers with a probability of 0.5, each hidden unit is prevented from relying on other hidden units being present during training. Hence there are fewer co-adaptations among them on the training data. Called “dropout,” this process is also an efficient alternative to train many separate networks and average their predictions on the test set. <br /> <br /> The intuition for dropout is that if neurons are randomly dropped during training, they can no longer rely on their neighbours, thus allowing each neutron to become more robust. Another interpretation is that dropout is similar to training an ensemble of models, since each epoch with randomly dropped neurons can be viewed as its own model. <br /> <br /> They used the standard, stochastic gradient descent algorithm and separated training data into mini-batches. An upper bound was set on the L2 norm of incoming weight vector for each hidden neuron, which was normalized if its size exceeds the bound. They found that using a constraint, instead of a penalty, forced model to do a more thorough search of the weight-space, when coupled with the very large learning rate that decays during training.<br /> <br /> '''Mean Network'''<br /> <br /> Their dropout models included all of the hidden neurons, and their outgoing weights were halved to account for the chances of omission. This is called an 'Mean Network'. This is similar to taking the geometric mean of the probability distribution predicted by all 2^N networks. Due to this cumulative addition, the correct answers will have higher log probability than an individual dropout network, which also lead to a lower square error of the network. <br /> <br /> <br /> The models were shown to result in lower test error rates on several datasets: MNIST; TIMIT; Reuters Corpus Volume; CIFAR-10; and ImageNet.<br /> <br /> = MNIST =<br /> The MNIST dataset contains 70,000 digit images of size 28 x 28. To see the impact of dropout, they used 4 different neural networks (784-800-800-10, 784-1200-1200-10, 784-2000-2000-10, 784-1200-1200-1200-10), using the same dropout rates as 50% for hidden neurons and 20% for visible neurons. Stochastic gradient descent was used with mini-batches of size 100 and a cross-entropy objective function as the loss function. Weights were updated after each minibatch, and training was done for 3000 epochs. An exponentially decaying learning rate &lt;math&gt;\epsilon&lt;/math&gt; was used, with the initial value set as 10.0, and it was multiplied by the decaying factor &lt;math&gt;f&lt;/math&gt; = 0.998 at the end of each epoch. At each hidden layer, the incoming weight vector for each hidden neuron was set an upper bound of its length, &lt;math&gt;l&lt;/math&gt;, and they found from cross-validation that the results were the best when &lt;math&gt;l&lt;/math&gt; = 15. Initial weights values were pooled from a normal distribution with mean 0 and standard deviation of 0.01. To update weights, an additional variable, ''p'', called momentum, was used to accelerate learning. The initial value of &lt;math&gt;p&lt;/math&gt; was 0.5, and it increased linearly to the final value 0.99 during the first 500 epochs, remaining unchanged after. Also, when updating weights, the learning rate was multiplied by &lt;math&gt;1 – p&lt;/math&gt;. &lt;math&gt;L&lt;/math&gt; denotes the gradient of loss function.<br /> <br /> [[File:weights_mnist2.png|center|400px]]<br /> <br /> The best published result for a standard feedforward neural network was 160 errors. This was reduced to about 130 errors with 0.5 dropout and different L2 constraints for each hidden unit input weight. By omitting a random 20% of the input pixels in addition to the aforementioned changes, the number of errors was further reduced to 110. The following figure visualizes the result.<br /> [[File:mnist_figure.png|center|500px]]<br /> A publicly available pre-trained deep belief net resulted in 118 errors, and it was reduced to 92 errors when the model was fine-tuned with dropout. Another publicly available model was a deep Boltzmann machine, and it resulted in 103, 97, 94, 93 and 88 when the model was fine-tuned using standard backpropagation and was unrolled. They were reduced to 83, 79, 78, 78, and 77 when the model was fine-tuned with dropout – the mean of 79 errors was a record for models that do not use prior knowledge or enhanced training sets.<br /> <br /> = TIMIT = <br /> <br /> TIMIT dataset includes voice samples of 630 American English speakers varying across 8 different dialects. It is often used to evaluate the performance of automatic speech recognition systems. Using Kaldi, the dataset was pre-processed to extract input features in the form of log filter bank responses.<br /> <br /> === Pre-training and Training ===<br /> <br /> For pretraining, they pretrained their neural network with a deep belief network and the first layer was built using Restricted Boltzmann Machine (RBM). Initializing visible biases with zero, weights were sampled from random numbers that followed normal distribution &lt;math&gt;N(0, 0.01)&lt;/math&gt;. Each visible neuron’s variance was set to 1.0 and remained unchanged.<br /> <br /> Minimizing Contrastive Divergence (CD) was used to facilitate learning. Since momentum is used to speed up learning, it was initially set to 0.5 and increased linearly to 0.9 over 20 epochs. The average gradient had 0.001 of a learning rate which was then multiplied by &lt;math&gt;(1-momentum)&lt;/math&gt; and L2 weight decay was set to 0.001. After setting up the hyperparameters, the model was done training after 100 epochs. Binary RBMs were used for training all subsequent layers with a learning rate of 0.01. Then, &lt;math&gt;p&lt;/math&gt; was set as the mean activation of a neuron in the data set and the visible bias of each neuron was initialized to &lt;math&gt;log(p/(1 − p))&lt;/math&gt;. Training each layer with 50 epochs, all remaining hyper-parameters were the same as those for the Gaussian RBM.<br /> <br /> === Dropout tuning ===<br /> <br /> The initial weights were set in a neural network from the pretrained RBMs. To finetune the network with dropout-backpropagation, momentum was initially set to 0.5 and increased linearly up to 0.9 over 10 epochs. The model had a small constant learning rate of 1.0 and it was used to apply to the average gradient on a minibatch. The model also retained all other hyperparameters the same as the model from MNIST dropout finetuning. The model required approximately 200 epochs to converge. For comparison purpose, they also finetuned the same network with standard backpropagation with a learning rate of 0.1 with the same hyperparameters.<br /> <br /> === Classification Test and Performance ===<br /> <br /> A Neural network was constructed to output the classification error rate on the test set of TIMIT dataset. They have built the neural network with four fully-connected hidden layers with 4000 neurons per layer. The output layer distinguishes distinct classes from 185 softmax output neurons that are merged into 39 classes. After constructing the neural network, 21 adjacent frames with an advance of 10ms per frame was given as an input.<br /> <br /> Comparing the performance of dropout with standard backpropagation on several network architectures and input representations, dropout consistently achieved lower error and cross-entropy. Results showed that it significantly controls overfitting, making the method robust to choices of network architecture. It also allowed much larger nets to be trained and removed the need for early stopping. Thus, neural network architectures with dropout are not very sensitive to the choice of learning rate and momentum.<br /> <br /> = Reuters Corpus Volume =<br /> Reuters Corpus Volume I archives 804,414 news documents that belong to 103 topics. Under four major themes - corporate/industrial, economics, government/social, and markets – they belonged to 63 classes. After removing 11 classes with no data and one class with insufficient data, they are left with 50 classes and 402,738 documents. The documents were divided into training and test sets equally and randomly, with each document representing the 2000 most frequent words in the dataset, excluding stopwords.<br /> <br /> They trained two neural networks, with size 2000-2000-1000-50, one using dropout and backpropagation, and the other using standard backpropagation. The training hyperparameters are the same as that in MNIST, but training was done for 500 epochs.<br /> <br /> In the following figure, we see the significant improvements by the model with dropout in the test set error. On the right side, we see that learning with dropout also proceeds smoother. <br /> <br /> [[File:reuters_figure.png|700px|center]]<br /> <br /> = CNN =<br /> <br /> Feed-forward neural networks consist of several layers of neurons where each neuron in a layer applies a linear filter to the input image data and is passed on to the neurons in the next layer. When calculating the neuron’s output, scalar bias a.k.a weights is applied to the filter with nonlinear activation function as parameters of the network that are learned by training data. [[File:cnnbigpicture.jpeg|thumb|upright=2|center|alt=text|Figure: Overview of Convolutional Neural Network]] There are several differences between Convolutional Neural networks and ordinary neural networks. The figure above gives a visual representation of a Convolutional Neural Network. First, CNN’s neurons are organized topographically into a bank and laid out on a 2D grid, so it reflects the organization of dimensions of the input data. Secondly, neurons in CNN apply filters which are local, and which are centered at the neuron’s location in the topographic organization. Meaning that useful metrics or clues to identify the object in an input image which can be found by examining local neighborhoods of the image. Next, all neurons in a bank apply the same filter at different locations in the input image. When looking at the image example, green is an input to one neuron bank, yellow is filter bank, and pink is the output of one neuron bank (convolved feature). A bank of neurons in a CNN applies a convolution operation, aka filters, to its input where a single layer in a CNN typically has multiple banks of neurons, each performing a convolution with a different filter. The resulting neuron banks become distinct input channels into the next layer. The whole process reduces the net’s representational capacity, but also reduces the capacity to overfit.<br /> [[File:bankofneurons.gif|thumb|upright=3|center|alt=text|Figure: Bank of neurons]]<br /> <br /> === Pooling ===<br /> <br /> Pooling layer summarizes the activities of local patches of neurons in the convolutional layer by subsampling the output of a convolutional layer. Pooling is useful for extracting dominant features, to decrease the computational power required to process the data through dimensionality reduction. The procedure of pooling goes on like this; output from convolutional layers is divided into sections called pooling units and they are laid out topographically, connected to a local neighborhood of other pooling units from the same convolutional output. Then, each pooling unit is computed with some function which could be maximum and average. Maximum pooling returns the maximum value from the section of the image covered by the pooling unit while average pooling returns the average of all the values inside the pooling unit (see example). In result, there are fewer total pooling units than convolutional unit outputs from the previous layer, this is due to larger spacing between pixels on pooling layers. Using the max-pooling function reduces the effect of outliers and improves generalization. Other than that, overlapping pooling makes this spacing between pixels smaller than the size of the neighborhood that the pooling units summarize (This spacing is usually referred as the stride between pooling units). With this variant, pooling layer can produce a coarse coding of the outputs which helps generalization. <br /> [[File:maxandavgpooling.jpeg|thumb|upright=2|center|alt=text|Figure: Max pooling and Average pooling]]<br /> <br /> === Local Response Normalization === <br /> <br /> This network includes local response normalization layers which are implemented in lateral form and used on neurons with unbounded activations and permits the detection of high-frequency features with a big neuron response. This regularizer encourages competition among neurons belonging to different banks. Normalization is done by dividing the activity of a neuron in bank &lt;math&gt;i&lt;/math&gt; at position &lt;math&gt;(x,y)&lt;/math&gt; by the equation:<br /> [[File:local response norm.png|upright=2|center|]] where the sum runs over &lt;math&gt;N&lt;/math&gt; ‘adjacent’ banks of neurons at the same position as in the topographic organization of neuron bank. The constants, &lt;math&gt;N&lt;/math&gt;, &lt;math&gt;alpha&lt;/math&gt; and &lt;math&gt;betas&lt;/math&gt; are hyper-parameters whose values are determined using a validation set. This technique is replaced by better techniques such as the combination of dropout and regularization methods (&lt;math&gt;L1&lt;/math&gt; and &lt;math&gt;L2&lt;/math&gt;)<br /> <br /> === Neuron nonlinearities ===<br /> <br /> All of the neurons for this model use the max-with-zero nonlinearity where output within a neuron is computed as &lt;math&gt; a^{i}_{x,y} = max(0, z^i_{x,y})&lt;/math&gt; where &lt;math&gt; z^i_{x,y} &lt;/math&gt; is the total input to the neuron. The reason they use nonlinearity is because it has several advantages over traditional saturating neuron models, such as significant reduction in training time required to reach a certain error rate. Another advantage is that nonlinearity reduces the need for contrast-normalization and data pre-processing since neurons do not saturate- meaning activities simply scale up little by little with usually large input values. For this model’s only pre-processing step, they subtract the mean activity from each pixel and the result is a centered data.<br /> <br /> === Objective function ===<br /> <br /> The objective function of their network maximizes the multinomial logistic regression objective which is the same as minimizing the average cross-entropy across training cases between the true label and the model’s predicted label.<br /> <br /> === Weight Initialization === <br /> <br /> It’s important to note that if a neuron always receives a negative value during training, it will not learn because its output is uniformly zero under the max-with-zero nonlinearity. Hence, the weights in their model were sampled from a zero-mean normal distribution with a high enough variance. High variance in weights will set a certain number of neurons with positive values for learning to happen, and in practice, it’s necessary to try out several candidates for variances until a working initialization is found. In their experiment, setting a positive constant, or 1, as biases of the neurons in the hidden layers was helpful in finding it.<br /> <br /> === Training ===<br /> <br /> In this model, a batch size of 128 samples and momentum of 0.9, we train our model using stochastic gradient descent. The update rule for weight &lt;math&gt;w&lt;/math&gt; is $$v_{i+1} = 0.9v_i + \epsilon &lt;\frac{dE}{dw_i}&gt; i$$ $$w_{i+1} = w_i + v_{i+1}$$ where &lt;math&gt;i&lt;/math&gt; is the iteration index, &lt;math&gt;v&lt;/math&gt; is a momentum variable, &lt;math&gt;\epsilon&lt;/math&gt; is the learning rate and &lt;math&gt;\frac{dE}{dw}&lt;/math&gt; is the average over the &lt;math&gt;i&lt;/math&gt;th batch of the derivative of the objective with respect to &lt;math&gt;w_i&lt;/math&gt;. The whole training process on CIFAR-10 takes roughly 90 minutes and ImageNet takes 4 days with dropout and two days without.<br /> <br /> === Learning ===<br /> To determine the learning rate for the network, it is a must to start with an equal learning rate for each layer which produces the largest reduction in the objective function with power of ten. Usually, it is in the order of &lt;math&gt;10^{-2}&lt;/math&gt; or &lt;math&gt;10^{-3}&lt;/math&gt;. In this case, they reduce the learning rate twice by a factor of ten before termination of training.<br /> <br /> = CIFAR-10 =<br /> <br /> === CIFAR-10 Dataset ===<br /> <br /> Removing incorrect labels, The CIFAR-10 dataset is a subset of the Tiny Images dataset with 10 classes. It contains 5000 training images and 1000 testing images for each class. The dataset has 32 x 32 color images searched from the web and the images are labeled with the noun used to search the image.<br /> <br /> [[File:CIFAR-10.png|thumb|upright=2|center|alt=text|Figure 4: CIFAR-10 Sample Dataset]]<br /> <br /> === Models for CIFAR-10 ===<br /> <br /> Two models, one with dropout and one without dropout, were built to test the performance of dropout on CIFAR-10. All models have CNN with three convolutional layers each with a pooling layer. All of the pooling payers use a stride=2 and summarize a 3*3 neighborhood. The max-pooling method is performed by the pooling layer which follows the first convolutional layer, and the average-pooling method is performed by remaining 2 pooling layers. The first and second pooling layers with &lt;math&gt;N = 9, α = 0.001&lt;/math&gt;, and &lt;math&gt;β = 0.75&lt;/math&gt; are followed by response normalization layers. A ten-unit softmax layer, which is used to output a probability distribution over class labels, is connected with the upper-most pooling layer. Using filter size of 5×5, all convolutional layers have 64 filter banks.<br /> <br /> Additional changes were made with the model with dropout. The model with dropout enables us to use more parameters because dropout forces a strong regularization on the network. Thus, a fourth weight layer is added to take the input from the previous pooling layer. This fourth weight layer is locally connected, but not convolutional, and contains 16 banks of filters of size 3 × 3 with 50% dropout. Lastly, the softmax layer takes its input from this fourth weight layer.<br /> <br /> Thus, with a neural network with 3 convolutional hidden layers with 3 max-pooling layers, the classification error achieved 16.6% to beat 18.5% from the best published error rate without using transformed data. The model with one additional locally-connected layer and dropout at the last hidden layer produced the error rate of 15.6%.<br /> <br /> = ImageNet =<br /> <br /> ===ImageNet Dataset===<br /> <br /> ImageNet is a dataset of millions of high-resolution images, and they are labeled among 1000 different categories. The data were collected from the web and manually labeled using MTerk tool, which is a crowd-sourcing tool provided by Amazon.<br /> Because this dataset has millions of labeled images in thousands of categories, it is very difficult to have perfect accuracy on this dataset even for humans because the ImageNet images may contain multiple objects and there are a large number of object classes. ImageNet and CIFAR-10 are very similar, but the scale of ImageNet is about 20 times bigger (1,300,000 vs 60,000). The size of ImageNet is about 1.3 million training images, 50,000 validation images, and 150,000 testing images. They used resized images of 256 x 256 pixels for their experiments.<br /> <br /> '''An ambiguous example to classify:'''<br /> <br /> [[File:imagenet1.png|200px|center]]<br /> <br /> When this paper was written, the best score on this dataset was the error rate of 45.7% by High-dimensional signature compression for large-scale image classification (J. Sanchez, F. Perronnin, CVPR11 (2011)). The authors of this paper could achieve a comparable performance of 48.6% error rate using a single neural network with five convolutional hidden layers with a max-pooling layer in between, followed by two globally connected layers and a final 1000-way softmax layer. When applying 50% dropout to the 6th layer, the error rate was brought down to 42.4%.<br /> <br /> '''ImageNet Dataset:'''<br /> <br /> [[File:imagenet2.png|400px|center]]<br /> <br /> ===Models for ImageNet===<br /> <br /> They mostly focused on the model with dropout because the one without dropout had a similar approach, but there was a serious issue with overfitting. They used a convolutional neural network trained by 224×224 patches randomly extracted from the 256 × 256 images. This could reduce the network’s capacity to overfit the training data and helped generalization as a form of data augmentation. The method of averaging the prediction of the net on ten 224 × 224 patches of the 256 × 256 input image was used for testing their model patched at the center, four corners, and their horizontal reflections. To maximize the performance on the validation set, this complicated network architecture was used and it was found that dropout was very effective. Also, it was demonstrated that using non-convolutional higher layers with the number of parameters worked well with dropout, but it had a negative impact to the performance without dropout.<br /> <br /> The network contains seven weight layers. The first five are convolutional, and the last two are globally-connected. Max-pooling layers follow the layer number 1,2, and 5. And then, the output of the last globally-connected layer was fed to a 1000-way softmax output layers. Using this architecture, the authors achieved the error rate of 48.6%. When applying 50% dropout to the 6th layer, the error rate was brought down to 42.4%.<br /> <br /> <br /> [[File:modelh2.png|700px|center]] <br /> <br /> [[File:layer2.png|600px|center]]<br /> <br /> Like the previous datasets, such as the MNIST, TIMIT, Reuters, and CIFAR-10, we also see a significant improvement for the ImageNet dataset. Including complicated architectures like this one, introducing dropout generalizes models better and gives lower test error rates.<br /> <br /> = Conclusion =<br /> <br /> The authors have shown a consistent improvement by the models trained with dropout in classifying objects in the following datasets: MNIST; TIMIT; Reuters Corpus Volume I; CIFAR-10; and ImageNet.<br /> <br /> = Critiques =<br /> It is a very brilliant idea to dropout half of the neurons to reduce co-adaptations. It is mentioned that for fully connected layers, dropout in all hidden layers works better than dropout in only one hidden layer. There is another paper Dropout: A Simple Way to Prevent Neural Networks from<br /> Overfitting[https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf] gives a more detailed explanation.<br /> <br /> It will be interesting to see how this paper could be used to prevent overfitting of LSTMs.<br /> <br /> Firstly, it is a very interested topic of classification by &quot;dropout&quot; CNN method(omitting neurons in hidden layers). If the author can briefly explain the advantages of this method in processing image data in theory, it will be easier for readers to understand. Also, how to deal with overfitting issue would be valuable.<br /> <br /> The authors mention that they tried various dropout probabilities and that the majority of them improved the model's generalization performance, but that more extreme probabilities tended to be worse which is why a dropout rate of 50% was used in the paper. The authors further develop this point to mention that the method can be improved by adapting individual dropout probabilities of each hidden or input unit using validation tests. This would be an interesting area to further develop and explore, as using a hardcoded 50% dropout for all layers might not be the optimal choice for all CNN applications. It would have been interesting to see the results of their investigations of differing dropout rates.<br /> <br /> The authors don't explain that during training, at each layer that we apply dropout, the values must be scaled by 1/p where p is dropout rate - this way the expected value of the layers is the same in both train and test time. They may have considered another solution for this discrepancy at the time (it is an old paper) but it doesn't seem like any solution was presented here. <br /> <br /> Despite the advantages of using dropout to prevent overfitting and reducing errors in testing, the authors did not discuss much about the effects on the length of training time. In another [https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf paper] published a few years later by the same authors, there was more discussion about this. It appears that dropout increases training time by 2-3 times compared to a standard NN with the same architecture, which is a drawback that might be worth mentioning. <br /> <br /> == Reference ==<br />  N. Srivastave, &quot;Dropout: a simple way to prevent neural networks from overfitting&quot;, The Journal of Machine Learning Research, Jan 2014.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Improving_neural_networks_by_preventing_co-adaption_of_feature_detectors&diff=49043 Improving neural networks by preventing co-adaption of feature detectors 2020-12-03T23:02:17Z <p>Z42qin: </p> <hr /> <div>== Presented by ==<br /> Stan Lee, Seokho Lim, Kyle Jung, Dae Hyun Kim<br /> <br /> = Improvement Intro =<br /> '''Drop Out Model'''<br /> In this paper, Hinton et al. introduce a novel way to improve neural networks’ performance, particularly in the case that a large feedforward neural network is trained on a small training set, which causes poor performance and leads to “overfitting” problem. This problem can be reduced by randomly omitting half of the feature detectors on each training case. In fact, By omitting neurons in hidden layers with a probability of 0.5, each hidden unit is prevented from relying on other hidden units being present during training. Hence there are fewer co-adaptations among them on the training data. Called “dropout,” this process is also an efficient alternative to train many separate networks and average their predictions on the test set. <br /> <br /> The intuition for dropout is that if neurons are randomly dropped during training, they can no longer rely on their neighbours, thus allowing each neutron to become more robust. Another interpretation is that dropout is similar to training an ensemble of models, since each epoch with randomly dropped neurons can be viewed as its own model. <br /> <br /> They used the standard, stochastic gradient descent algorithm and separated training data into mini-batches. An upper bound was set on the L2 norm of incoming weight vector for each hidden neuron, which was normalized if its size exceeds the bound. They found that using a constraint, instead of a penalty, forced model to do a more thorough search of the weight-space, when coupled with the very large learning rate that decays during training.<br /> <br /> '''Mean Network'''<br /> <br /> Their dropout models included all of the hidden neurons, and their outgoing weights were halved to account for the chances of omission. This is called an 'Mean Network'. This is similar to taking the geometric mean of the probability distribution predicted by all 2^N networks. Due to this cumulative addition, the correct answers will have higher log probability than an individual dropout network, which also lead to a lower square error of the network. <br /> <br /> <br /> The models were shown to result in lower test error rates on several datasets: MNIST; TIMIT; Reuters Corpus Volume; CIFAR-10; and ImageNet.<br /> <br /> = MNIST =<br /> The MNIST dataset contains 70,000 digit images of size 28 x 28. To see the impact of dropout, they used 4 different neural networks (784-800-800-10, 784-1200-1200-10, 784-2000-2000-10, 784-1200-1200-1200-10), using the same dropout rates as 50% for hidden neurons and 20% for visible neurons. Stochastic gradient descent was used with mini-batches of size 100 and a cross-entropy objective function as the loss function. Weights were updated after each minibatch, and training was done for 3000 epochs. An exponentially decaying learning rate &lt;math&gt;\epsilon&lt;/math&gt; was used, with the initial value set as 10.0, and it was multiplied by the decaying factor &lt;math&gt;f&lt;/math&gt; = 0.998 at the end of each epoch. At each hidden layer, the incoming weight vector for each hidden neuron was set an upper bound of its length, &lt;math&gt;l&lt;/math&gt;, and they found from cross-validation that the results were the best when &lt;math&gt;l&lt;/math&gt; = 15. Initial weights values were pooled from a normal distribution with mean 0 and standard deviation of 0.01. To update weights, an additional variable, ''p'', called momentum, was used to accelerate learning. The initial value of &lt;math&gt;p&lt;/math&gt; was 0.5, and it increased linearly to the final value 0.99 during the first 500 epochs, remaining unchanged after. Also, when updating weights, the learning rate was multiplied by &lt;math&gt;1 – p&lt;/math&gt;. &lt;math&gt;L&lt;/math&gt; denotes the gradient of loss function.<br /> <br /> [[File:weights_mnist2.png|center|400px]]<br /> <br /> The best published result for a standard feedforward neural network was 160 errors. This was reduced to about 130 errors with 0.5 dropout and different L2 constraints for each hidden unit input weight. By omitting a random 20% of the input pixels in addition to the aforementioned changes, the number of errors was further reduced to 110. The following figure visualizes the result.<br /> [[File:mnist_figure.png|center|500px]]<br /> A publicly available pre-trained deep belief net resulted in 118 errors, and it was reduced to 92 errors when the model was fine-tuned with dropout. Another publicly available model was a deep Boltzmann machine, and it resulted in 103, 97, 94, 93 and 88 when the model was fine-tuned using standard backpropagation and was unrolled. They were reduced to 83, 79, 78, 78, and 77 when the model was fine-tuned with dropout – the mean of 79 errors was a record for models that do not use prior knowledge or enhanced training sets.<br /> <br /> = TIMIT = <br /> <br /> TIMIT dataset includes voice samples of 630 American English speakers varying across 8 different dialects. It is often used to evaluate the performance of automatic speech recognition systems. Using Kaldi, the dataset was pre-processed to extract input features in the form of log filter bank responses.<br /> <br /> === Pre-training and Training ===<br /> <br /> For pretraining, they pretrained their neural network with a deep belief network and the first layer was built using Restricted Boltzmann Machine (RBM). Initializing visible biases with zero, weights were sampled from random numbers that followed normal distribution &lt;math&gt;N(0, 0.01)&lt;/math&gt;. Each visible neuron’s variance was set to 1.0 and remained unchanged.<br /> <br /> Minimizing Contrastive Divergence (CD) was used to facilitate learning. Since momentum is used to speed up learning, it was initially set to 0.5 and increased linearly to 0.9 over 20 epochs. The average gradient had 0.001 of a learning rate which was then multiplied by &lt;math&gt;(1-momentum)&lt;/math&gt; and L2 weight decay was set to 0.001. After setting up the hyperparameters, the model was done training after 100 epochs. Binary RBMs were used for training all subsequent layers with a learning rate of 0.01. Then, &lt;math&gt;p&lt;/math&gt; was set as the mean activation of a neuron in the data set and the visible bias of each neuron was initialized to &lt;math&gt;log(p/(1 − p))&lt;/math&gt;. Training each layer with 50 epochs, all remaining hyper-parameters were the same as those for the Gaussian RBM.<br /> <br /> === Dropout tuning ===<br /> <br /> The initial weights were set in a neural network from the pretrained RBMs. To finetune the network with dropout-backpropagation, momentum was initially set to 0.5 and increased linearly up to 0.9 over 10 epochs. The model had a small constant learning rate of 1.0 and it was used to apply to the average gradient on a minibatch. The model also retained all other hyperparameters the same as the model from MNIST dropout finetuning. The model required approximately 200 epochs to converge. For comparison purpose, they also finetuned the same network with standard backpropagation with a learning rate of 0.1 with the same hyperparameters.<br /> <br /> === Classification Test and Performance ===<br /> <br /> A Neural network was constructed to output the classification error rate on the test set of TIMIT dataset. They have built the neural network with four fully-connected hidden layers with 4000 neurons per layer. The output layer distinguishes distinct classes from 185 softmax output neurons that are merged into 39 classes. After constructing the neural network, 21 adjacent frames with an advance of 10ms per frame was given as an input.<br /> <br /> Comparing the performance of dropout with standard backpropagation on several network architectures and input representations, dropout consistently achieved lower error and cross-entropy. Results showed that it significantly controls overfitting, making the method robust to choices of network architecture. It also allowed much larger nets to be trained and removed the need for early stopping. Thus, neural network architectures with dropout are not very sensitive to the choice of learning rate and momentum.<br /> <br /> = Reuters Corpus Volume =<br /> Reuters Corpus Volume I archives 804,414 news documents that belong to 103 topics. Under four major themes - corporate/industrial, economics, government/social, and markets – they belonged to 63 classes. After removing 11 classes with no data and one class with insufficient data, they are left with 50 classes and 402,738 documents. The documents were divided into training and test sets equally and randomly, with each document representing the 2000 most frequent words in the dataset, excluding stopwords.<br /> <br /> They trained two neural networks, with size 2000-2000-1000-50, one using dropout and backpropagation, and the other using standard backpropagation. The training hyperparameters are the same as that in MNIST, but training was done for 500 epochs.<br /> <br /> In the following figure, we see the significant improvements by the model with dropout in the test set error. On the right side, we see that learning with dropout also proceeds smoother. <br /> <br /> [[File:reuters_figure.png|700px|center]]<br /> <br /> = CNN =<br /> <br /> Feed-forward neural networks consist of several layers of neurons where each neuron in a layer applies a linear filter to the input image data and is passed on to the neurons in the next layer. When calculating the neuron’s output, scalar bias a.k.a weights is applied to the filter with nonlinear activation function as parameters of the network that are learned by training data. [[File:cnnbigpicture.jpeg|thumb|upright=2|center|alt=text|Figure: Overview of Convolutional Neural Network]] There are several differences between Convolutional Neural networks and ordinary neural networks. The figure above gives a visual representation of a Convolutional Neural Network. First, CNN’s neurons are organized topographically into a bank and laid out on a 2D grid, so it reflects the organization of dimensions of the input data. Secondly, neurons in CNN apply filters which are local, and which are centered at the neuron’s location in the topographic organization. Meaning that useful metrics or clues to identify the object in an input image which can be found by examining local neighborhoods of the image. Next, all neurons in a bank apply the same filter at different locations in the input image. When looking at the image example, green is an input to one neuron bank, yellow is filter bank, and pink is the output of one neuron bank (convolved feature). A bank of neurons in a CNN applies a convolution operation, aka filters, to its input where a single layer in a CNN typically has multiple banks of neurons, each performing a convolution with a different filter. The resulting neuron banks become distinct input channels into the next layer. The whole process reduces the net’s representational capacity, but also reduces the capacity to overfit.<br /> [[File:bankofneurons.gif|thumb|upright=3|center|alt=text|Figure: Bank of neurons]]<br /> <br /> === Pooling ===<br /> <br /> Pooling layer summarizes the activities of local patches of neurons in the convolutional layer by subsampling the output of a convolutional layer. Pooling is useful for extracting dominant features, to decrease the computational power required to process the data through dimensionality reduction. The procedure of pooling goes on like this; output from convolutional layers is divided into sections called pooling units and they are laid out topographically, connected to a local neighborhood of other pooling units from the same convolutional output. Then, each pooling unit is computed with some function which could be maximum and average. Maximum pooling returns the maximum value from the section of the image covered by the pooling unit while average pooling returns the average of all the values inside the pooling unit (see example). In result, there are fewer total pooling units than convolutional unit outputs from the previous layer, this is due to larger spacing between pixels on pooling layers. Using the max-pooling function reduces the effect of outliers and improves generalization. Other than that, overlapping pooling makes this spacing between pixels smaller than the size of the neighborhood that the pooling units summarize (This spacing is usually referred as the stride between pooling units). With this variant, pooling layer can produce a coarse coding of the outputs which helps generalization. <br /> [[File:maxandavgpooling.jpeg|thumb|upright=2|center|alt=text|Figure: Max pooling and Average pooling]]<br /> <br /> === Local Response Normalization === <br /> <br /> This network includes local response normalization layers which are implemented in lateral form and used on neurons with unbounded activations and permits the detection of high-frequency features with a big neuron response. This regularizer encourages competition among neurons belonging to different banks. Normalization is done by dividing the activity of a neuron in bank &lt;math&gt;i&lt;/math&gt; at position &lt;math&gt;(x,y)&lt;/math&gt; by the equation:<br /> [[File:local response norm.png|upright=2|center|]] where the sum runs over &lt;math&gt;N&lt;/math&gt; ‘adjacent’ banks of neurons at the same position as in the topographic organization of neuron bank. The constants, &lt;math&gt;N&lt;/math&gt;, &lt;math&gt;alpha&lt;/math&gt; and &lt;math&gt;betas&lt;/math&gt; are hyper-parameters whose values are determined using a validation set. This technique is replaced by better techniques such as the combination of dropout and regularization methods (&lt;math&gt;L1&lt;/math&gt; and &lt;math&gt;L2&lt;/math&gt;)<br /> <br /> === Neuron nonlinearities ===<br /> <br /> All of the neurons for this model use the max-with-zero nonlinearity where output within a neuron is computed as &lt;math&gt; a^{i}_{x,y} = max(0, z^i_{x,y})&lt;/math&gt; where &lt;math&gt; z^i_{x,y} &lt;/math&gt; is the total input to the neuron. The reason they use nonlinearity is because it has several advantages over traditional saturating neuron models, such as significant reduction in training time required to reach a certain error rate. Another advantage is that nonlinearity reduces the need for contrast-normalization and data pre-processing since neurons do not saturate- meaning activities simply scale up little by little with usually large input values. For this model’s only pre-processing step, they subtract the mean activity from each pixel and the result is a centered data.<br /> <br /> === Objective function ===<br /> <br /> The objective function of their network maximizes the multinomial logistic regression objective which is the same as minimizing the average cross-entropy across training cases between the true label and the model’s predicted label.<br /> <br /> === Weight Initialization === <br /> <br /> It’s important to note that if a neuron always receives a negative value during training, it will not learn because its output is uniformly zero under the max-with-zero nonlinearity. Hence, the weights in their model were sampled from a zero-mean normal distribution with a high enough variance. High variance in weights will set a certain number of neurons with positive values for learning to happen, and in practice, it’s necessary to try out several candidates for variances until a working initialization is found. In their experiment, setting a positive constant, or 1, as biases of the neurons in the hidden layers was helpful in finding it.<br /> <br /> === Training ===<br /> <br /> In this model, a batch size of 128 samples and momentum of 0.9, we train our model using stochastic gradient descent. The update rule for weight &lt;math&gt;w&lt;/math&gt; is $$v_{i+1} = 0.9v_i + \epsilon &lt;\frac{dE}{dw_i}&gt; i$$ $$w_{i+1} = w_i + v_{i+1}$$ where &lt;math&gt;i&lt;/math&gt; is the iteration index, &lt;math&gt;v&lt;/math&gt; is a momentum variable, &lt;math&gt;\epsilon&lt;/math&gt; is the learning rate and &lt;math&gt;\frac{dE}{dw}&lt;/math&gt; is the average over the &lt;math&gt;i&lt;/math&gt;th batch of the derivative of the objective with respect to &lt;math&gt;w_i&lt;/math&gt;. The whole training process on CIFAR-10 takes roughly 90 minutes and ImageNet takes 4 days with dropout and two days without.<br /> <br /> === Learning ===<br /> To determine the learning rate for the network, it is a must to start with an equal learning rate for each layer which produces the largest reduction in the objective function with power of ten. Usually, it is in the order of &lt;math&gt;10^{-2}&lt;/math&gt; or &lt;math&gt;10^{-3}&lt;/math&gt;. In this case, they reduce the learning rate twice by a factor of ten before termination of training.<br /> <br /> = CIFAR-10 =<br /> <br /> === CIFAR-10 Dataset ===<br /> <br /> Removing incorrect labels, The CIFAR-10 dataset is a subset of the Tiny Images dataset with 10 classes. It contains 5000 training images and 1000 testing images for each class. The dataset has 32 x 32 color images searched from the web and the images are labeled with the noun used to search the image.<br /> <br /> [[File:CIFAR-10.png|thumb|upright=2|center|alt=text|Figure 4: CIFAR-10 Sample Dataset]]<br /> <br /> === Models for CIFAR-10 ===<br /> <br /> Two models, one with dropout and one without dropout, were built to test the performance of dropout on CIFAR-10. All models have CNN with three convolutional layers each with a pooling layer. All of the pooling payers use a stride=2 and summarize a 3*3 neighborhood. The max-pooling method is performed by the pooling layer which follows the first convolutional layer, and the average-pooling method is performed by remaining 2 pooling layers. The first and second pooling layers with &lt;math&gt;N = 9, α = 0.001&lt;/math&gt;, and &lt;math&gt;β = 0.75&lt;/math&gt; are followed by response normalization layers. A ten-unit softmax layer, which is used to output a probability distribution over class labels, is connected with the upper-most pooling layer. Using filter size of 5×5, all convolutional layers have 64 filter banks.<br /> <br /> Additional changes were made with the model with dropout. The model with dropout enables us to use more parameters because dropout forces a strong regularization on the network. Thus, a fourth weight layer is added to take the input from the previous pooling layer. This fourth weight layer is locally connected, but not convolutional, and contains 16 banks of filters of size 3 × 3 with 50% dropout. Lastly, the softmax layer takes its input from this fourth weight layer.<br /> <br /> Thus, with a neural network with 3 convolutional hidden layers with 3 max-pooling layers, the classification error achieved 16.6% to beat 18.5% from the best published error rate without using transformed data. The model with one additional locally-connected layer and dropout at the last hidden layer produced the error rate of 15.6%.<br /> <br /> = ImageNet =<br /> <br /> ===ImageNet Dataset===<br /> <br /> ImageNet is a dataset of millions of high-resolution images, and they are labeled among 1000 different categories. The data were collected from the web and manually labeled using MTerk tool, which is a crowd-sourcing tool provided by Amazon.<br /> Because this dataset has millions of labeled images in thousands of categories, it is very difficult to have perfect accuracy on this dataset even for humans because the ImageNet images may contain multiple objects and there are a large number of object classes. ImageNet and CIFAR-10 are very similar, but the scale of ImageNet is about 20 times bigger (1,300,000 vs 60,000). The size of ImageNet is about 1.3 million training images, 50,000 validation images, and 150,000 testing images. They used resized images of 256 x 256 pixels for their experiments.<br /> <br /> '''An ambiguous example to classify:'''<br /> <br /> [[File:imagenet1.png|200px|center]]<br /> <br /> When this paper was written, the best score on this dataset was the error rate of 45.7% by High-dimensional signature compression for large-scale image classification (J. Sanchez, F. Perronnin, CVPR11 (2011)). The authors of this paper could achieve a comparable performance of 48.6% error rate using a single neural network with five convolutional hidden layers with a max-pooling layer in between, followed by two globally connected layers and a final 1000-way softmax layer. When applying 50% dropout to the 6th layer, the error rate was brought down to 42.4%.<br /> <br /> '''ImageNet Dataset:'''<br /> <br /> [[File:imagenet2.png|400px|center]]<br /> <br /> ===Models for ImageNet===<br /> <br /> They mostly focused on the model with dropout because the one without dropout had a similar approach, but there was a serious issue with overfitting. They used a convolutional neural network trained by 224×224 patches randomly extracted from the 256 × 256 images. This could reduce the network’s capacity to overfit the training data and helped generalization as a form of data augmentation. The method of averaging the prediction of the net on ten 224 × 224 patches of the 256 × 256 input image was used for testing their model patched at the center, four corners, and their horizontal reflections. To maximize the performance on the validation set, this complicated network architecture was used and it was found that dropout was very effective. Also, it was demonstrated that using non-convolutional higher layers with the number of parameters worked well with dropout, but it had a negative impact to the performance without dropout.<br /> <br /> The network contains seven weight layers. The first five are convolutional, and the last two are globally-connected. Max-pooling layers follow the layer number 1,2, and 5. And then, the output of the last globally-connected layer was fed to a 1000-way softmax output layers. Using this architecture, the authors achieved the error rate of 48.6%. When applying 50% dropout to the 6th layer, the error rate was brought down to 42.4%.<br /> <br /> <br /> [[File:modelh2.png|700px|center]] <br /> <br /> [[File:layer2.png|600px|center]]<br /> <br /> Like the previous datasets, such as the MNIST, TIMIT, Reuters, and CIFAR-10, we also see a significant improvement for the ImageNet dataset. Including complicated architectures like this one, introducing dropout generalizes models better and gives lower test error rates.<br /> <br /> = Conclusion =<br /> <br /> The authors have shown a consistent improvement by the models trained with dropout in classifying objects in the following datasets: MNIST; TIMIT; Reuters Corpus Volume I; CIFAR-10; and ImageNet.<br /> <br /> = Critiques =<br /> It is a very brilliant idea to dropout half of the neurons to reduce co-adaptations. It is mentioned that for fully connected layers, dropout in all hidden layers works better than dropout in only one hidden layer. There is another paper Dropout: A Simple Way to Prevent Neural Networks from<br /> Overfitting[https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf] gives a more detailed explanation.<br /> <br /> It will be interesting to see how this paper could be used to prevent overfitting of LSTMs.<br /> <br /> Firstly, it is a very interested topic of classification by &quot;dropout&quot; CNN method(omitting neurons in hidden layers). If the author can briefly explain the advantages of this method in processing image data in theory, it will be easier for readers to understand. Also, how to deal with overfitting issue would be valuable.<br /> <br /> The authors mention that they tried various dropout probabilities and that the majority of them improved the model's generalization performance, but that more extreme probabilities tended to be worse which is why a dropout rate of 50% was used in the paper. The authors further develop this point to mention that the method can be improved by adapting individual dropout probabilities of each hidden or input unit using validation tests. This would be an interesting area to further develop and explore, as using a hardcoded 50% dropout for all layers might not be the optimal choice for all CNN applications. It would have been interesting to see the results of their investigations of differing dropout rates.<br /> <br /> The authors don't explain that during training, at each layer that we apply dropout, the values must be scaled by 1/p where p is dropout rate - this way the expected value of the layers is the same in both train and test time. They may have considered another solution for this discrepancy at the time (it is an old paper) but it doesn't seem like any solution was presented here. <br /> <br /> Despite the advantages of using dropout to prevent overfitting and reducing errors in testing, the authors did not discuss much about the effects on the length of training time. In another [https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf paper] published a few years later by the same authors, there was more discussion about this. It appears that dropout increases training time by 2-3 times compared to a standard NN with the same architecture, which is a drawback that might be worth mentioning. <br /> <br /> == Reference ==<br />  N. Srivastave, &quot;Dropout: a simple way to prevent neural networks from overfitting&quot;, The Journal of Machine Learning Research, Jan 2014.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Improving_neural_networks_by_preventing_co-adaption_of_feature_detectors&diff=49042 Improving neural networks by preventing co-adaption of feature detectors 2020-12-03T23:01:36Z <p>Z42qin: /* Mean Network */</p> <hr /> <div>== Presented by ==<br /> Stan Lee, Seokho Lim, Kyle Jung, Dae Hyun Kim<br /> <br /> = Drop Out Model =<br /> In this paper, Hinton et al. introduce a novel way to improve neural networks’ performance, particularly in the case that a large feedforward neural network is trained on a small training set, which causes poor performance and leads to “overfitting” problem. This problem can be reduced by randomly omitting half of the feature detectors on each training case. In fact, By omitting neurons in hidden layers with a probability of 0.5, each hidden unit is prevented from relying on other hidden units being present during training. Hence there are fewer co-adaptations among them on the training data. Called “dropout,” this process is also an efficient alternative to train many separate networks and average their predictions on the test set. <br /> <br /> The intuition for dropout is that if neurons are randomly dropped during training, they can no longer rely on their neighbours, thus allowing each neutron to become more robust. Another interpretation is that dropout is similar to training an ensemble of models, since each epoch with randomly dropped neurons can be viewed as its own model. <br /> <br /> They used the standard, stochastic gradient descent algorithm and separated training data into mini-batches. An upper bound was set on the L2 norm of incoming weight vector for each hidden neuron, which was normalized if its size exceeds the bound. They found that using a constraint, instead of a penalty, forced model to do a more thorough search of the weight-space, when coupled with the very large learning rate that decays during training.<br /> <br /> = Mean Network =<br /> <br /> Their dropout models included all of the hidden neurons, and their outgoing weights were halved to account for the chances of omission. This is called an 'Mean Network'. This is similar to taking the geometric mean of the probability distribution predicted by all 2^N networks. Due to this cumulative addition, the correct answers will have higher log probability than an individual dropout network, which also lead to a lower square error of the network. <br /> <br /> <br /> The models were shown to result in lower test error rates on several datasets: MNIST; TIMIT; Reuters Corpus Volume; CIFAR-10; and ImageNet.<br /> <br /> = MNIST =<br /> The MNIST dataset contains 70,000 digit images of size 28 x 28. To see the impact of dropout, they used 4 different neural networks (784-800-800-10, 784-1200-1200-10, 784-2000-2000-10, 784-1200-1200-1200-10), using the same dropout rates as 50% for hidden neurons and 20% for visible neurons. Stochastic gradient descent was used with mini-batches of size 100 and a cross-entropy objective function as the loss function. Weights were updated after each minibatch, and training was done for 3000 epochs. An exponentially decaying learning rate &lt;math&gt;\epsilon&lt;/math&gt; was used, with the initial value set as 10.0, and it was multiplied by the decaying factor &lt;math&gt;f&lt;/math&gt; = 0.998 at the end of each epoch. At each hidden layer, the incoming weight vector for each hidden neuron was set an upper bound of its length, &lt;math&gt;l&lt;/math&gt;, and they found from cross-validation that the results were the best when &lt;math&gt;l&lt;/math&gt; = 15. Initial weights values were pooled from a normal distribution with mean 0 and standard deviation of 0.01. To update weights, an additional variable, ''p'', called momentum, was used to accelerate learning. The initial value of &lt;math&gt;p&lt;/math&gt; was 0.5, and it increased linearly to the final value 0.99 during the first 500 epochs, remaining unchanged after. Also, when updating weights, the learning rate was multiplied by &lt;math&gt;1 – p&lt;/math&gt;. &lt;math&gt;L&lt;/math&gt; denotes the gradient of loss function.<br /> <br /> [[File:weights_mnist2.png|center|400px]]<br /> <br /> The best published result for a standard feedforward neural network was 160 errors. This was reduced to about 130 errors with 0.5 dropout and different L2 constraints for each hidden unit input weight. By omitting a random 20% of the input pixels in addition to the aforementioned changes, the number of errors was further reduced to 110. The following figure visualizes the result.<br /> [[File:mnist_figure.png|center|500px]]<br /> A publicly available pre-trained deep belief net resulted in 118 errors, and it was reduced to 92 errors when the model was fine-tuned with dropout. Another publicly available model was a deep Boltzmann machine, and it resulted in 103, 97, 94, 93 and 88 when the model was fine-tuned using standard backpropagation and was unrolled. They were reduced to 83, 79, 78, 78, and 77 when the model was fine-tuned with dropout – the mean of 79 errors was a record for models that do not use prior knowledge or enhanced training sets.<br /> <br /> = TIMIT = <br /> <br /> TIMIT dataset includes voice samples of 630 American English speakers varying across 8 different dialects. It is often used to evaluate the performance of automatic speech recognition systems. Using Kaldi, the dataset was pre-processed to extract input features in the form of log filter bank responses.<br /> <br /> === Pre-training and Training ===<br /> <br /> For pretraining, they pretrained their neural network with a deep belief network and the first layer was built using Restricted Boltzmann Machine (RBM). Initializing visible biases with zero, weights were sampled from random numbers that followed normal distribution &lt;math&gt;N(0, 0.01)&lt;/math&gt;. Each visible neuron’s variance was set to 1.0 and remained unchanged.<br /> <br /> Minimizing Contrastive Divergence (CD) was used to facilitate learning. Since momentum is used to speed up learning, it was initially set to 0.5 and increased linearly to 0.9 over 20 epochs. The average gradient had 0.001 of a learning rate which was then multiplied by &lt;math&gt;(1-momentum)&lt;/math&gt; and L2 weight decay was set to 0.001. After setting up the hyperparameters, the model was done training after 100 epochs. Binary RBMs were used for training all subsequent layers with a learning rate of 0.01. Then, &lt;math&gt;p&lt;/math&gt; was set as the mean activation of a neuron in the data set and the visible bias of each neuron was initialized to &lt;math&gt;log(p/(1 − p))&lt;/math&gt;. Training each layer with 50 epochs, all remaining hyper-parameters were the same as those for the Gaussian RBM.<br /> <br /> === Dropout tuning ===<br /> <br /> The initial weights were set in a neural network from the pretrained RBMs. To finetune the network with dropout-backpropagation, momentum was initially set to 0.5 and increased linearly up to 0.9 over 10 epochs. The model had a small constant learning rate of 1.0 and it was used to apply to the average gradient on a minibatch. The model also retained all other hyperparameters the same as the model from MNIST dropout finetuning. The model required approximately 200 epochs to converge. For comparison purpose, they also finetuned the same network with standard backpropagation with a learning rate of 0.1 with the same hyperparameters.<br /> <br /> === Classification Test and Performance ===<br /> <br /> A Neural network was constructed to output the classification error rate on the test set of TIMIT dataset. They have built the neural network with four fully-connected hidden layers with 4000 neurons per layer. The output layer distinguishes distinct classes from 185 softmax output neurons that are merged into 39 classes. After constructing the neural network, 21 adjacent frames with an advance of 10ms per frame was given as an input.<br /> <br /> Comparing the performance of dropout with standard backpropagation on several network architectures and input representations, dropout consistently achieved lower error and cross-entropy. Results showed that it significantly controls overfitting, making the method robust to choices of network architecture. It also allowed much larger nets to be trained and removed the need for early stopping. Thus, neural network architectures with dropout are not very sensitive to the choice of learning rate and momentum.<br /> <br /> = Reuters Corpus Volume =<br /> Reuters Corpus Volume I archives 804,414 news documents that belong to 103 topics. Under four major themes - corporate/industrial, economics, government/social, and markets – they belonged to 63 classes. After removing 11 classes with no data and one class with insufficient data, they are left with 50 classes and 402,738 documents. The documents were divided into training and test sets equally and randomly, with each document representing the 2000 most frequent words in the dataset, excluding stopwords.<br /> <br /> They trained two neural networks, with size 2000-2000-1000-50, one using dropout and backpropagation, and the other using standard backpropagation. The training hyperparameters are the same as that in MNIST, but training was done for 500 epochs.<br /> <br /> In the following figure, we see the significant improvements by the model with dropout in the test set error. On the right side, we see that learning with dropout also proceeds smoother. <br /> <br /> [[File:reuters_figure.png|700px|center]]<br /> <br /> = CNN =<br /> <br /> Feed-forward neural networks consist of several layers of neurons where each neuron in a layer applies a linear filter to the input image data and is passed on to the neurons in the next layer. When calculating the neuron’s output, scalar bias a.k.a weights is applied to the filter with nonlinear activation function as parameters of the network that are learned by training data. [[File:cnnbigpicture.jpeg|thumb|upright=2|center|alt=text|Figure: Overview of Convolutional Neural Network]] There are several differences between Convolutional Neural networks and ordinary neural networks. The figure above gives a visual representation of a Convolutional Neural Network. First, CNN’s neurons are organized topographically into a bank and laid out on a 2D grid, so it reflects the organization of dimensions of the input data. Secondly, neurons in CNN apply filters which are local, and which are centered at the neuron’s location in the topographic organization. Meaning that useful metrics or clues to identify the object in an input image which can be found by examining local neighborhoods of the image. Next, all neurons in a bank apply the same filter at different locations in the input image. When looking at the image example, green is an input to one neuron bank, yellow is filter bank, and pink is the output of one neuron bank (convolved feature). A bank of neurons in a CNN applies a convolution operation, aka filters, to its input where a single layer in a CNN typically has multiple banks of neurons, each performing a convolution with a different filter. The resulting neuron banks become distinct input channels into the next layer. The whole process reduces the net’s representational capacity, but also reduces the capacity to overfit.<br /> [[File:bankofneurons.gif|thumb|upright=3|center|alt=text|Figure: Bank of neurons]]<br /> <br /> === Pooling ===<br /> <br /> Pooling layer summarizes the activities of local patches of neurons in the convolutional layer by subsampling the output of a convolutional layer. Pooling is useful for extracting dominant features, to decrease the computational power required to process the data through dimensionality reduction. The procedure of pooling goes on like this; output from convolutional layers is divided into sections called pooling units and they are laid out topographically, connected to a local neighborhood of other pooling units from the same convolutional output. Then, each pooling unit is computed with some function which could be maximum and average. Maximum pooling returns the maximum value from the section of the image covered by the pooling unit while average pooling returns the average of all the values inside the pooling unit (see example). In result, there are fewer total pooling units than convolutional unit outputs from the previous layer, this is due to larger spacing between pixels on pooling layers. Using the max-pooling function reduces the effect of outliers and improves generalization. Other than that, overlapping pooling makes this spacing between pixels smaller than the size of the neighborhood that the pooling units summarize (This spacing is usually referred as the stride between pooling units). With this variant, pooling layer can produce a coarse coding of the outputs which helps generalization. <br /> [[File:maxandavgpooling.jpeg|thumb|upright=2|center|alt=text|Figure: Max pooling and Average pooling]]<br /> <br /> === Local Response Normalization === <br /> <br /> This network includes local response normalization layers which are implemented in lateral form and used on neurons with unbounded activations and permits the detection of high-frequency features with a big neuron response. This regularizer encourages competition among neurons belonging to different banks. Normalization is done by dividing the activity of a neuron in bank &lt;math&gt;i&lt;/math&gt; at position &lt;math&gt;(x,y)&lt;/math&gt; by the equation:<br /> [[File:local response norm.png|upright=2|center|]] where the sum runs over &lt;math&gt;N&lt;/math&gt; ‘adjacent’ banks of neurons at the same position as in the topographic organization of neuron bank. The constants, &lt;math&gt;N&lt;/math&gt;, &lt;math&gt;alpha&lt;/math&gt; and &lt;math&gt;betas&lt;/math&gt; are hyper-parameters whose values are determined using a validation set. This technique is replaced by better techniques such as the combination of dropout and regularization methods (&lt;math&gt;L1&lt;/math&gt; and &lt;math&gt;L2&lt;/math&gt;)<br /> <br /> === Neuron nonlinearities ===<br /> <br /> All of the neurons for this model use the max-with-zero nonlinearity where output within a neuron is computed as &lt;math&gt; a^{i}_{x,y} = max(0, z^i_{x,y})&lt;/math&gt; where &lt;math&gt; z^i_{x,y} &lt;/math&gt; is the total input to the neuron. The reason they use nonlinearity is because it has several advantages over traditional saturating neuron models, such as significant reduction in training time required to reach a certain error rate. Another advantage is that nonlinearity reduces the need for contrast-normalization and data pre-processing since neurons do not saturate- meaning activities simply scale up little by little with usually large input values. For this model’s only pre-processing step, they subtract the mean activity from each pixel and the result is a centered data.<br /> <br /> === Objective function ===<br /> <br /> The objective function of their network maximizes the multinomial logistic regression objective which is the same as minimizing the average cross-entropy across training cases between the true label and the model’s predicted label.<br /> <br /> === Weight Initialization === <br /> <br /> It’s important to note that if a neuron always receives a negative value during training, it will not learn because its output is uniformly zero under the max-with-zero nonlinearity. Hence, the weights in their model were sampled from a zero-mean normal distribution with a high enough variance. High variance in weights will set a certain number of neurons with positive values for learning to happen, and in practice, it’s necessary to try out several candidates for variances until a working initialization is found. In their experiment, setting a positive constant, or 1, as biases of the neurons in the hidden layers was helpful in finding it.<br /> <br /> === Training ===<br /> <br /> In this model, a batch size of 128 samples and momentum of 0.9, we train our model using stochastic gradient descent. The update rule for weight &lt;math&gt;w&lt;/math&gt; is $$v_{i+1} = 0.9v_i + \epsilon &lt;\frac{dE}{dw_i}&gt; i$$ $$w_{i+1} = w_i + v_{i+1}$$ where &lt;math&gt;i&lt;/math&gt; is the iteration index, &lt;math&gt;v&lt;/math&gt; is a momentum variable, &lt;math&gt;\epsilon&lt;/math&gt; is the learning rate and &lt;math&gt;\frac{dE}{dw}&lt;/math&gt; is the average over the &lt;math&gt;i&lt;/math&gt;th batch of the derivative of the objective with respect to &lt;math&gt;w_i&lt;/math&gt;. The whole training process on CIFAR-10 takes roughly 90 minutes and ImageNet takes 4 days with dropout and two days without.<br /> <br /> === Learning ===<br /> To determine the learning rate for the network, it is a must to start with an equal learning rate for each layer which produces the largest reduction in the objective function with power of ten. Usually, it is in the order of &lt;math&gt;10^{-2}&lt;/math&gt; or &lt;math&gt;10^{-3}&lt;/math&gt;. In this case, they reduce the learning rate twice by a factor of ten before termination of training.<br /> <br /> = CIFAR-10 =<br /> <br /> === CIFAR-10 Dataset ===<br /> <br /> Removing incorrect labels, The CIFAR-10 dataset is a subset of the Tiny Images dataset with 10 classes. It contains 5000 training images and 1000 testing images for each class. The dataset has 32 x 32 color images searched from the web and the images are labeled with the noun used to search the image.<br /> <br /> [[File:CIFAR-10.png|thumb|upright=2|center|alt=text|Figure 4: CIFAR-10 Sample Dataset]]<br /> <br /> === Models for CIFAR-10 ===<br /> <br /> Two models, one with dropout and one without dropout, were built to test the performance of dropout on CIFAR-10. All models have CNN with three convolutional layers each with a pooling layer. All of the pooling payers use a stride=2 and summarize a 3*3 neighborhood. The max-pooling method is performed by the pooling layer which follows the first convolutional layer, and the average-pooling method is performed by remaining 2 pooling layers. The first and second pooling layers with &lt;math&gt;N = 9, α = 0.001&lt;/math&gt;, and &lt;math&gt;β = 0.75&lt;/math&gt; are followed by response normalization layers. A ten-unit softmax layer, which is used to output a probability distribution over class labels, is connected with the upper-most pooling layer. Using filter size of 5×5, all convolutional layers have 64 filter banks.<br /> <br /> Additional changes were made with the model with dropout. The model with dropout enables us to use more parameters because dropout forces a strong regularization on the network. Thus, a fourth weight layer is added to take the input from the previous pooling layer. This fourth weight layer is locally connected, but not convolutional, and contains 16 banks of filters of size 3 × 3 with 50% dropout. Lastly, the softmax layer takes its input from this fourth weight layer.<br /> <br /> Thus, with a neural network with 3 convolutional hidden layers with 3 max-pooling layers, the classification error achieved 16.6% to beat 18.5% from the best published error rate without using transformed data. The model with one additional locally-connected layer and dropout at the last hidden layer produced the error rate of 15.6%.<br /> <br /> = ImageNet =<br /> <br /> ===ImageNet Dataset===<br /> <br /> ImageNet is a dataset of millions of high-resolution images, and they are labeled among 1000 different categories. The data were collected from the web and manually labeled using MTerk tool, which is a crowd-sourcing tool provided by Amazon.<br /> Because this dataset has millions of labeled images in thousands of categories, it is very difficult to have perfect accuracy on this dataset even for humans because the ImageNet images may contain multiple objects and there are a large number of object classes. ImageNet and CIFAR-10 are very similar, but the scale of ImageNet is about 20 times bigger (1,300,000 vs 60,000). The size of ImageNet is about 1.3 million training images, 50,000 validation images, and 150,000 testing images. They used resized images of 256 x 256 pixels for their experiments.<br /> <br /> '''An ambiguous example to classify:'''<br /> <br /> [[File:imagenet1.png|200px|center]]<br /> <br /> When this paper was written, the best score on this dataset was the error rate of 45.7% by High-dimensional signature compression for large-scale image classification (J. Sanchez, F. Perronnin, CVPR11 (2011)). The authors of this paper could achieve a comparable performance of 48.6% error rate using a single neural network with five convolutional hidden layers with a max-pooling layer in between, followed by two globally connected layers and a final 1000-way softmax layer. When applying 50% dropout to the 6th layer, the error rate was brought down to 42.4%.<br /> <br /> '''ImageNet Dataset:'''<br /> <br /> [[File:imagenet2.png|400px|center]]<br /> <br /> ===Models for ImageNet===<br /> <br /> They mostly focused on the model with dropout because the one without dropout had a similar approach, but there was a serious issue with overfitting. They used a convolutional neural network trained by 224×224 patches randomly extracted from the 256 × 256 images. This could reduce the network’s capacity to overfit the training data and helped generalization as a form of data augmentation. The method of averaging the prediction of the net on ten 224 × 224 patches of the 256 × 256 input image was used for testing their model patched at the center, four corners, and their horizontal reflections. To maximize the performance on the validation set, this complicated network architecture was used and it was found that dropout was very effective. Also, it was demonstrated that using non-convolutional higher layers with the number of parameters worked well with dropout, but it had a negative impact to the performance without dropout.<br /> <br /> The network contains seven weight layers. The first five are convolutional, and the last two are globally-connected. Max-pooling layers follow the layer number 1,2, and 5. And then, the output of the last globally-connected layer was fed to a 1000-way softmax output layers. Using this architecture, the authors achieved the error rate of 48.6%. When applying 50% dropout to the 6th layer, the error rate was brought down to 42.4%.<br /> <br /> <br /> [[File:modelh2.png|700px|center]] <br /> <br /> [[File:layer2.png|600px|center]]<br /> <br /> Like the previous datasets, such as the MNIST, TIMIT, Reuters, and CIFAR-10, we also see a significant improvement for the ImageNet dataset. Including complicated architectures like this one, introducing dropout generalizes models better and gives lower test error rates.<br /> <br /> = Conclusion =<br /> <br /> The authors have shown a consistent improvement by the models trained with dropout in classifying objects in the following datasets: MNIST; TIMIT; Reuters Corpus Volume I; CIFAR-10; and ImageNet.<br /> <br /> = Critiques =<br /> It is a very brilliant idea to dropout half of the neurons to reduce co-adaptations. It is mentioned that for fully connected layers, dropout in all hidden layers works better than dropout in only one hidden layer. There is another paper Dropout: A Simple Way to Prevent Neural Networks from<br /> Overfitting[https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf] gives a more detailed explanation.<br /> <br /> It will be interesting to see how this paper could be used to prevent overfitting of LSTMs.<br /> <br /> Firstly, it is a very interested topic of classification by &quot;dropout&quot; CNN method(omitting neurons in hidden layers). If the author can briefly explain the advantages of this method in processing image data in theory, it will be easier for readers to understand. Also, how to deal with overfitting issue would be valuable.<br /> <br /> The authors mention that they tried various dropout probabilities and that the majority of them improved the model's generalization performance, but that more extreme probabilities tended to be worse which is why a dropout rate of 50% was used in the paper. The authors further develop this point to mention that the method can be improved by adapting individual dropout probabilities of each hidden or input unit using validation tests. This would be an interesting area to further develop and explore, as using a hardcoded 50% dropout for all layers might not be the optimal choice for all CNN applications. It would have been interesting to see the results of their investigations of differing dropout rates.<br /> <br /> The authors don't explain that during training, at each layer that we apply dropout, the values must be scaled by 1/p where p is dropout rate - this way the expected value of the layers is the same in both train and test time. They may have considered another solution for this discrepancy at the time (it is an old paper) but it doesn't seem like any solution was presented here. <br /> <br /> Despite the advantages of using dropout to prevent overfitting and reducing errors in testing, the authors did not discuss much about the effects on the length of training time. In another [https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf paper] published a few years later by the same authors, there was more discussion about this. It appears that dropout increases training time by 2-3 times compared to a standard NN with the same architecture, which is a drawback that might be worth mentioning. <br /> <br /> == Reference ==<br />  N. Srivastave, &quot;Dropout: a simple way to prevent neural networks from overfitting&quot;, The Journal of Machine Learning Research, Jan 2014.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Improving_neural_networks_by_preventing_co-adaption_of_feature_detectors&diff=49041 Improving neural networks by preventing co-adaption of feature detectors 2020-12-03T23:01:19Z <p>Z42qin: /* Drop Out Model */</p> <hr /> <div>== Presented by ==<br /> Stan Lee, Seokho Lim, Kyle Jung, Dae Hyun Kim<br /> <br /> = Drop Out Model =<br /> In this paper, Hinton et al. introduce a novel way to improve neural networks’ performance, particularly in the case that a large feedforward neural network is trained on a small training set, which causes poor performance and leads to “overfitting” problem. This problem can be reduced by randomly omitting half of the feature detectors on each training case. In fact, By omitting neurons in hidden layers with a probability of 0.5, each hidden unit is prevented from relying on other hidden units being present during training. Hence there are fewer co-adaptations among them on the training data. Called “dropout,” this process is also an efficient alternative to train many separate networks and average their predictions on the test set. <br /> <br /> The intuition for dropout is that if neurons are randomly dropped during training, they can no longer rely on their neighbours, thus allowing each neutron to become more robust. Another interpretation is that dropout is similar to training an ensemble of models, since each epoch with randomly dropped neurons can be viewed as its own model. <br /> <br /> They used the standard, stochastic gradient descent algorithm and separated training data into mini-batches. An upper bound was set on the L2 norm of incoming weight vector for each hidden neuron, which was normalized if its size exceeds the bound. They found that using a constraint, instead of a penalty, forced model to do a more thorough search of the weight-space, when coupled with the very large learning rate that decays during training.<br /> <br /> = Mean Network =<br /> Their dropout models included all of the hidden neurons, and their outgoing weights were halved to account for the chances of omission. This is called an 'Mean Network'. This is similar to taking the geometric mean of the probability distribution predicted by all 2^N networks. Due to this cumulative addition, the correct answers will have higher log probability than an individual dropout network, which also lead to a lower square error of the network. <br /> <br /> <br /> The models were shown to result in lower test error rates on several datasets: MNIST; TIMIT; Reuters Corpus Volume; CIFAR-10; and ImageNet.<br /> <br /> = MNIST =<br /> The MNIST dataset contains 70,000 digit images of size 28 x 28. To see the impact of dropout, they used 4 different neural networks (784-800-800-10, 784-1200-1200-10, 784-2000-2000-10, 784-1200-1200-1200-10), using the same dropout rates as 50% for hidden neurons and 20% for visible neurons. Stochastic gradient descent was used with mini-batches of size 100 and a cross-entropy objective function as the loss function. Weights were updated after each minibatch, and training was done for 3000 epochs. An exponentially decaying learning rate &lt;math&gt;\epsilon&lt;/math&gt; was used, with the initial value set as 10.0, and it was multiplied by the decaying factor &lt;math&gt;f&lt;/math&gt; = 0.998 at the end of each epoch. At each hidden layer, the incoming weight vector for each hidden neuron was set an upper bound of its length, &lt;math&gt;l&lt;/math&gt;, and they found from cross-validation that the results were the best when &lt;math&gt;l&lt;/math&gt; = 15. Initial weights values were pooled from a normal distribution with mean 0 and standard deviation of 0.01. To update weights, an additional variable, ''p'', called momentum, was used to accelerate learning. The initial value of &lt;math&gt;p&lt;/math&gt; was 0.5, and it increased linearly to the final value 0.99 during the first 500 epochs, remaining unchanged after. Also, when updating weights, the learning rate was multiplied by &lt;math&gt;1 – p&lt;/math&gt;. &lt;math&gt;L&lt;/math&gt; denotes the gradient of loss function.<br /> <br /> [[File:weights_mnist2.png|center|400px]]<br /> <br /> The best published result for a standard feedforward neural network was 160 errors. This was reduced to about 130 errors with 0.5 dropout and different L2 constraints for each hidden unit input weight. By omitting a random 20% of the input pixels in addition to the aforementioned changes, the number of errors was further reduced to 110. The following figure visualizes the result.<br /> [[File:mnist_figure.png|center|500px]]<br /> A publicly available pre-trained deep belief net resulted in 118 errors, and it was reduced to 92 errors when the model was fine-tuned with dropout. Another publicly available model was a deep Boltzmann machine, and it resulted in 103, 97, 94, 93 and 88 when the model was fine-tuned using standard backpropagation and was unrolled. They were reduced to 83, 79, 78, 78, and 77 when the model was fine-tuned with dropout – the mean of 79 errors was a record for models that do not use prior knowledge or enhanced training sets.<br /> <br /> = TIMIT = <br /> <br /> TIMIT dataset includes voice samples of 630 American English speakers varying across 8 different dialects. It is often used to evaluate the performance of automatic speech recognition systems. Using Kaldi, the dataset was pre-processed to extract input features in the form of log filter bank responses.<br /> <br /> === Pre-training and Training ===<br /> <br /> For pretraining, they pretrained their neural network with a deep belief network and the first layer was built using Restricted Boltzmann Machine (RBM). Initializing visible biases with zero, weights were sampled from random numbers that followed normal distribution &lt;math&gt;N(0, 0.01)&lt;/math&gt;. Each visible neuron’s variance was set to 1.0 and remained unchanged.<br /> <br /> Minimizing Contrastive Divergence (CD) was used to facilitate learning. Since momentum is used to speed up learning, it was initially set to 0.5 and increased linearly to 0.9 over 20 epochs. The average gradient had 0.001 of a learning rate which was then multiplied by &lt;math&gt;(1-momentum)&lt;/math&gt; and L2 weight decay was set to 0.001. After setting up the hyperparameters, the model was done training after 100 epochs. Binary RBMs were used for training all subsequent layers with a learning rate of 0.01. Then, &lt;math&gt;p&lt;/math&gt; was set as the mean activation of a neuron in the data set and the visible bias of each neuron was initialized to &lt;math&gt;log(p/(1 − p))&lt;/math&gt;. Training each layer with 50 epochs, all remaining hyper-parameters were the same as those for the Gaussian RBM.<br /> <br /> === Dropout tuning ===<br /> <br /> The initial weights were set in a neural network from the pretrained RBMs. To finetune the network with dropout-backpropagation, momentum was initially set to 0.5 and increased linearly up to 0.9 over 10 epochs. The model had a small constant learning rate of 1.0 and it was used to apply to the average gradient on a minibatch. The model also retained all other hyperparameters the same as the model from MNIST dropout finetuning. The model required approximately 200 epochs to converge. For comparison purpose, they also finetuned the same network with standard backpropagation with a learning rate of 0.1 with the same hyperparameters.<br /> <br /> === Classification Test and Performance ===<br /> <br /> A Neural network was constructed to output the classification error rate on the test set of TIMIT dataset. They have built the neural network with four fully-connected hidden layers with 4000 neurons per layer. The output layer distinguishes distinct classes from 185 softmax output neurons that are merged into 39 classes. After constructing the neural network, 21 adjacent frames with an advance of 10ms per frame was given as an input.<br /> <br /> Comparing the performance of dropout with standard backpropagation on several network architectures and input representations, dropout consistently achieved lower error and cross-entropy. Results showed that it significantly controls overfitting, making the method robust to choices of network architecture. It also allowed much larger nets to be trained and removed the need for early stopping. Thus, neural network architectures with dropout are not very sensitive to the choice of learning rate and momentum.<br /> <br /> = Reuters Corpus Volume =<br /> Reuters Corpus Volume I archives 804,414 news documents that belong to 103 topics. Under four major themes - corporate/industrial, economics, government/social, and markets – they belonged to 63 classes. After removing 11 classes with no data and one class with insufficient data, they are left with 50 classes and 402,738 documents. The documents were divided into training and test sets equally and randomly, with each document representing the 2000 most frequent words in the dataset, excluding stopwords.<br /> <br /> They trained two neural networks, with size 2000-2000-1000-50, one using dropout and backpropagation, and the other using standard backpropagation. The training hyperparameters are the same as that in MNIST, but training was done for 500 epochs.<br /> <br /> In the following figure, we see the significant improvements by the model with dropout in the test set error. On the right side, we see that learning with dropout also proceeds smoother. <br /> <br /> [[File:reuters_figure.png|700px|center]]<br /> <br /> = CNN =<br /> <br /> Feed-forward neural networks consist of several layers of neurons where each neuron in a layer applies a linear filter to the input image data and is passed on to the neurons in the next layer. When calculating the neuron’s output, scalar bias a.k.a weights is applied to the filter with nonlinear activation function as parameters of the network that are learned by training data. [[File:cnnbigpicture.jpeg|thumb|upright=2|center|alt=text|Figure: Overview of Convolutional Neural Network]] There are several differences between Convolutional Neural networks and ordinary neural networks. The figure above gives a visual representation of a Convolutional Neural Network. First, CNN’s neurons are organized topographically into a bank and laid out on a 2D grid, so it reflects the organization of dimensions of the input data. Secondly, neurons in CNN apply filters which are local, and which are centered at the neuron’s location in the topographic organization. Meaning that useful metrics or clues to identify the object in an input image which can be found by examining local neighborhoods of the image. Next, all neurons in a bank apply the same filter at different locations in the input image. When looking at the image example, green is an input to one neuron bank, yellow is filter bank, and pink is the output of one neuron bank (convolved feature). A bank of neurons in a CNN applies a convolution operation, aka filters, to its input where a single layer in a CNN typically has multiple banks of neurons, each performing a convolution with a different filter. The resulting neuron banks become distinct input channels into the next layer. The whole process reduces the net’s representational capacity, but also reduces the capacity to overfit.<br /> [[File:bankofneurons.gif|thumb|upright=3|center|alt=text|Figure: Bank of neurons]]<br /> <br /> === Pooling ===<br /> <br /> Pooling layer summarizes the activities of local patches of neurons in the convolutional layer by subsampling the output of a convolutional layer. Pooling is useful for extracting dominant features, to decrease the computational power required to process the data through dimensionality reduction. The procedure of pooling goes on like this; output from convolutional layers is divided into sections called pooling units and they are laid out topographically, connected to a local neighborhood of other pooling units from the same convolutional output. Then, each pooling unit is computed with some function which could be maximum and average. Maximum pooling returns the maximum value from the section of the image covered by the pooling unit while average pooling returns the average of all the values inside the pooling unit (see example). In result, there are fewer total pooling units than convolutional unit outputs from the previous layer, this is due to larger spacing between pixels on pooling layers. Using the max-pooling function reduces the effect of outliers and improves generalization. Other than that, overlapping pooling makes this spacing between pixels smaller than the size of the neighborhood that the pooling units summarize (This spacing is usually referred as the stride between pooling units). With this variant, pooling layer can produce a coarse coding of the outputs which helps generalization. <br /> [[File:maxandavgpooling.jpeg|thumb|upright=2|center|alt=text|Figure: Max pooling and Average pooling]]<br /> <br /> === Local Response Normalization === <br /> <br /> This network includes local response normalization layers which are implemented in lateral form and used on neurons with unbounded activations and permits the detection of high-frequency features with a big neuron response. This regularizer encourages competition among neurons belonging to different banks. Normalization is done by dividing the activity of a neuron in bank &lt;math&gt;i&lt;/math&gt; at position &lt;math&gt;(x,y)&lt;/math&gt; by the equation:<br /> [[File:local response norm.png|upright=2|center|]] where the sum runs over &lt;math&gt;N&lt;/math&gt; ‘adjacent’ banks of neurons at the same position as in the topographic organization of neuron bank. The constants, &lt;math&gt;N&lt;/math&gt;, &lt;math&gt;alpha&lt;/math&gt; and &lt;math&gt;betas&lt;/math&gt; are hyper-parameters whose values are determined using a validation set. This technique is replaced by better techniques such as the combination of dropout and regularization methods (&lt;math&gt;L1&lt;/math&gt; and &lt;math&gt;L2&lt;/math&gt;)<br /> <br /> === Neuron nonlinearities ===<br /> <br /> All of the neurons for this model use the max-with-zero nonlinearity where output within a neuron is computed as &lt;math&gt; a^{i}_{x,y} = max(0, z^i_{x,y})&lt;/math&gt; where &lt;math&gt; z^i_{x,y} &lt;/math&gt; is the total input to the neuron. The reason they use nonlinearity is because it has several advantages over traditional saturating neuron models, such as significant reduction in training time required to reach a certain error rate. Another advantage is that nonlinearity reduces the need for contrast-normalization and data pre-processing since neurons do not saturate- meaning activities simply scale up little by little with usually large input values. For this model’s only pre-processing step, they subtract the mean activity from each pixel and the result is a centered data.<br /> <br /> === Objective function ===<br /> <br /> The objective function of their network maximizes the multinomial logistic regression objective which is the same as minimizing the average cross-entropy across training cases between the true label and the model’s predicted label.<br /> <br /> === Weight Initialization === <br /> <br /> It’s important to note that if a neuron always receives a negative value during training, it will not learn because its output is uniformly zero under the max-with-zero nonlinearity. Hence, the weights in their model were sampled from a zero-mean normal distribution with a high enough variance. High variance in weights will set a certain number of neurons with positive values for learning to happen, and in practice, it’s necessary to try out several candidates for variances until a working initialization is found. In their experiment, setting a positive constant, or 1, as biases of the neurons in the hidden layers was helpful in finding it.<br /> <br /> === Training ===<br /> <br /> In this model, a batch size of 128 samples and momentum of 0.9, we train our model using stochastic gradient descent. The update rule for weight &lt;math&gt;w&lt;/math&gt; is $$v_{i+1} = 0.9v_i + \epsilon &lt;\frac{dE}{dw_i}&gt; i$$ $$w_{i+1} = w_i + v_{i+1}$$ where &lt;math&gt;i&lt;/math&gt; is the iteration index, &lt;math&gt;v&lt;/math&gt; is a momentum variable, &lt;math&gt;\epsilon&lt;/math&gt; is the learning rate and &lt;math&gt;\frac{dE}{dw}&lt;/math&gt; is the average over the &lt;math&gt;i&lt;/math&gt;th batch of the derivative of the objective with respect to &lt;math&gt;w_i&lt;/math&gt;. The whole training process on CIFAR-10 takes roughly 90 minutes and ImageNet takes 4 days with dropout and two days without.<br /> <br /> === Learning ===<br /> To determine the learning rate for the network, it is a must to start with an equal learning rate for each layer which produces the largest reduction in the objective function with power of ten. Usually, it is in the order of &lt;math&gt;10^{-2}&lt;/math&gt; or &lt;math&gt;10^{-3}&lt;/math&gt;. In this case, they reduce the learning rate twice by a factor of ten before termination of training.<br /> <br /> = CIFAR-10 =<br /> <br /> === CIFAR-10 Dataset ===<br /> <br /> Removing incorrect labels, The CIFAR-10 dataset is a subset of the Tiny Images dataset with 10 classes. It contains 5000 training images and 1000 testing images for each class. The dataset has 32 x 32 color images searched from the web and the images are labeled with the noun used to search the image.<br /> <br /> [[File:CIFAR-10.png|thumb|upright=2|center|alt=text|Figure 4: CIFAR-10 Sample Dataset]]<br /> <br /> === Models for CIFAR-10 ===<br /> <br /> Two models, one with dropout and one without dropout, were built to test the performance of dropout on CIFAR-10. All models have CNN with three convolutional layers each with a pooling layer. All of the pooling payers use a stride=2 and summarize a 3*3 neighborhood. The max-pooling method is performed by the pooling layer which follows the first convolutional layer, and the average-pooling method is performed by remaining 2 pooling layers. The first and second pooling layers with &lt;math&gt;N = 9, α = 0.001&lt;/math&gt;, and &lt;math&gt;β = 0.75&lt;/math&gt; are followed by response normalization layers. A ten-unit softmax layer, which is used to output a probability distribution over class labels, is connected with the upper-most pooling layer. Using filter size of 5×5, all convolutional layers have 64 filter banks.<br /> <br /> Additional changes were made with the model with dropout. The model with dropout enables us to use more parameters because dropout forces a strong regularization on the network. Thus, a fourth weight layer is added to take the input from the previous pooling layer. This fourth weight layer is locally connected, but not convolutional, and contains 16 banks of filters of size 3 × 3 with 50% dropout. Lastly, the softmax layer takes its input from this fourth weight layer.<br /> <br /> Thus, with a neural network with 3 convolutional hidden layers with 3 max-pooling layers, the classification error achieved 16.6% to beat 18.5% from the best published error rate without using transformed data. The model with one additional locally-connected layer and dropout at the last hidden layer produced the error rate of 15.6%.<br /> <br /> = ImageNet =<br /> <br /> ===ImageNet Dataset===<br /> <br /> ImageNet is a dataset of millions of high-resolution images, and they are labeled among 1000 different categories. The data were collected from the web and manually labeled using MTerk tool, which is a crowd-sourcing tool provided by Amazon.<br /> Because this dataset has millions of labeled images in thousands of categories, it is very difficult to have perfect accuracy on this dataset even for humans because the ImageNet images may contain multiple objects and there are a large number of object classes. ImageNet and CIFAR-10 are very similar, but the scale of ImageNet is about 20 times bigger (1,300,000 vs 60,000). The size of ImageNet is about 1.3 million training images, 50,000 validation images, and 150,000 testing images. They used resized images of 256 x 256 pixels for their experiments.<br /> <br /> '''An ambiguous example to classify:'''<br /> <br /> [[File:imagenet1.png|200px|center]]<br /> <br /> When this paper was written, the best score on this dataset was the error rate of 45.7% by High-dimensional signature compression for large-scale image classification (J. Sanchez, F. Perronnin, CVPR11 (2011)). The authors of this paper could achieve a comparable performance of 48.6% error rate using a single neural network with five convolutional hidden layers with a max-pooling layer in between, followed by two globally connected layers and a final 1000-way softmax layer. When applying 50% dropout to the 6th layer, the error rate was brought down to 42.4%.<br /> <br /> '''ImageNet Dataset:'''<br /> <br /> [[File:imagenet2.png|400px|center]]<br /> <br /> ===Models for ImageNet===<br /> <br /> They mostly focused on the model with dropout because the one without dropout had a similar approach, but there was a serious issue with overfitting. They used a convolutional neural network trained by 224×224 patches randomly extracted from the 256 × 256 images. This could reduce the network’s capacity to overfit the training data and helped generalization as a form of data augmentation. The method of averaging the prediction of the net on ten 224 × 224 patches of the 256 × 256 input image was used for testing their model patched at the center, four corners, and their horizontal reflections. To maximize the performance on the validation set, this complicated network architecture was used and it was found that dropout was very effective. Also, it was demonstrated that using non-convolutional higher layers with the number of parameters worked well with dropout, but it had a negative impact to the performance without dropout.<br /> <br /> The network contains seven weight layers. The first five are convolutional, and the last two are globally-connected. Max-pooling layers follow the layer number 1,2, and 5. And then, the output of the last globally-connected layer was fed to a 1000-way softmax output layers. Using this architecture, the authors achieved the error rate of 48.6%. When applying 50% dropout to the 6th layer, the error rate was brought down to 42.4%.<br /> <br /> <br /> [[File:modelh2.png|700px|center]] <br /> <br /> [[File:layer2.png|600px|center]]<br /> <br /> Like the previous datasets, such as the MNIST, TIMIT, Reuters, and CIFAR-10, we also see a significant improvement for the ImageNet dataset. Including complicated architectures like this one, introducing dropout generalizes models better and gives lower test error rates.<br /> <br /> = Conclusion =<br /> <br /> The authors have shown a consistent improvement by the models trained with dropout in classifying objects in the following datasets: MNIST; TIMIT; Reuters Corpus Volume I; CIFAR-10; and ImageNet.<br /> <br /> = Critiques =<br /> It is a very brilliant idea to dropout half of the neurons to reduce co-adaptations. It is mentioned that for fully connected layers, dropout in all hidden layers works better than dropout in only one hidden layer. There is another paper Dropout: A Simple Way to Prevent Neural Networks from<br /> Overfitting[https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf] gives a more detailed explanation.<br /> <br /> It will be interesting to see how this paper could be used to prevent overfitting of LSTMs.<br /> <br /> Firstly, it is a very interested topic of classification by &quot;dropout&quot; CNN method(omitting neurons in hidden layers). If the author can briefly explain the advantages of this method in processing image data in theory, it will be easier for readers to understand. Also, how to deal with overfitting issue would be valuable.<br /> <br /> The authors mention that they tried various dropout probabilities and that the majority of them improved the model's generalization performance, but that more extreme probabilities tended to be worse which is why a dropout rate of 50% was used in the paper. The authors further develop this point to mention that the method can be improved by adapting individual dropout probabilities of each hidden or input unit using validation tests. This would be an interesting area to further develop and explore, as using a hardcoded 50% dropout for all layers might not be the optimal choice for all CNN applications. It would have been interesting to see the results of their investigations of differing dropout rates.<br /> <br /> The authors don't explain that during training, at each layer that we apply dropout, the values must be scaled by 1/p where p is dropout rate - this way the expected value of the layers is the same in both train and test time. They may have considered another solution for this discrepancy at the time (it is an old paper) but it doesn't seem like any solution was presented here. <br /> <br /> Despite the advantages of using dropout to prevent overfitting and reducing errors in testing, the authors did not discuss much about the effects on the length of training time. In another [https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf paper] published a few years later by the same authors, there was more discussion about this. It appears that dropout increases training time by 2-3 times compared to a standard NN with the same architecture, which is a drawback that might be worth mentioning. <br /> <br /> == Reference ==<br />  N. Srivastave, &quot;Dropout: a simple way to prevent neural networks from overfitting&quot;, The Journal of Machine Learning Research, Jan 2014.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Improving_neural_networks_by_preventing_co-adaption_of_feature_detectors&diff=49040 Improving neural networks by preventing co-adaption of feature detectors 2020-12-03T23:00:49Z <p>Z42qin: /* Introduction */</p> <hr /> <div>== Presented by ==<br /> Stan Lee, Seokho Lim, Kyle Jung, Dae Hyun Kim<br /> <br /> = Drop Out Model =<br /> In this paper, Hinton et al. introduce a novel way to improve neural networks’ performance, particularly in the case that a large feedforward neural network is trained on a small training set, which causes poor performance and leads to “overfitting” problem. This problem can be reduced by randomly omitting half of the feature detectors on each training case. In fact, By omitting neurons in hidden layers with a probability of 0.5, each hidden unit is prevented from relying on other hidden units being present during training. Hence there are fewer co-adaptations among them on the training data. Called “dropout,” this process is also an efficient alternative to train many separate networks and average their predictions on the test set. <br /> <br /> The intuition for dropout is that if neurons are randomly dropped during training, they can no longer rely on their neighbours, thus allowing each neutron to become more robust. Another interpretation is that dropout is similar to training an ensemble of models, since each epoch with randomly dropped neurons can be viewed as its own model. <br /> <br /> They used the standard, stochastic gradient descent algorithm and separated training data into mini-batches. An upper bound was set on the L2 norm of incoming weight vector for each hidden neuron, which was normalized if its size exceeds the bound. They found that using a constraint, instead of a penalty, forced model to do a more thorough search of the weight-space, when coupled with the very large learning rate that decays during training. <br /> <br /> = Mean Network =<br /> Their dropout models included all of the hidden neurons, and their outgoing weights were halved to account for the chances of omission. This is called an 'Mean Network'. This is similar to taking the geometric mean of the probability distribution predicted by all 2^N networks. Due to this cumulative addition, the correct answers will have higher log probability than an individual dropout network, which also lead to a lower square error of the network. <br /> <br /> <br /> The models were shown to result in lower test error rates on several datasets: MNIST; TIMIT; Reuters Corpus Volume; CIFAR-10; and ImageNet.<br /> <br /> = MNIST =<br /> The MNIST dataset contains 70,000 digit images of size 28 x 28. To see the impact of dropout, they used 4 different neural networks (784-800-800-10, 784-1200-1200-10, 784-2000-2000-10, 784-1200-1200-1200-10), using the same dropout rates as 50% for hidden neurons and 20% for visible neurons. Stochastic gradient descent was used with mini-batches of size 100 and a cross-entropy objective function as the loss function. Weights were updated after each minibatch, and training was done for 3000 epochs. An exponentially decaying learning rate &lt;math&gt;\epsilon&lt;/math&gt; was used, with the initial value set as 10.0, and it was multiplied by the decaying factor &lt;math&gt;f&lt;/math&gt; = 0.998 at the end of each epoch. At each hidden layer, the incoming weight vector for each hidden neuron was set an upper bound of its length, &lt;math&gt;l&lt;/math&gt;, and they found from cross-validation that the results were the best when &lt;math&gt;l&lt;/math&gt; = 15. Initial weights values were pooled from a normal distribution with mean 0 and standard deviation of 0.01. To update weights, an additional variable, ''p'', called momentum, was used to accelerate learning. The initial value of &lt;math&gt;p&lt;/math&gt; was 0.5, and it increased linearly to the final value 0.99 during the first 500 epochs, remaining unchanged after. Also, when updating weights, the learning rate was multiplied by &lt;math&gt;1 – p&lt;/math&gt;. &lt;math&gt;L&lt;/math&gt; denotes the gradient of loss function.<br /> <br /> [[File:weights_mnist2.png|center|400px]]<br /> <br /> The best published result for a standard feedforward neural network was 160 errors. This was reduced to about 130 errors with 0.5 dropout and different L2 constraints for each hidden unit input weight. By omitting a random 20% of the input pixels in addition to the aforementioned changes, the number of errors was further reduced to 110. The following figure visualizes the result.<br /> [[File:mnist_figure.png|center|500px]]<br /> A publicly available pre-trained deep belief net resulted in 118 errors, and it was reduced to 92 errors when the model was fine-tuned with dropout. Another publicly available model was a deep Boltzmann machine, and it resulted in 103, 97, 94, 93 and 88 when the model was fine-tuned using standard backpropagation and was unrolled. They were reduced to 83, 79, 78, 78, and 77 when the model was fine-tuned with dropout – the mean of 79 errors was a record for models that do not use prior knowledge or enhanced training sets.<br /> <br /> = TIMIT = <br /> <br /> TIMIT dataset includes voice samples of 630 American English speakers varying across 8 different dialects. It is often used to evaluate the performance of automatic speech recognition systems. Using Kaldi, the dataset was pre-processed to extract input features in the form of log filter bank responses.<br /> <br /> === Pre-training and Training ===<br /> <br /> For pretraining, they pretrained their neural network with a deep belief network and the first layer was built using Restricted Boltzmann Machine (RBM). Initializing visible biases with zero, weights were sampled from random numbers that followed normal distribution &lt;math&gt;N(0, 0.01)&lt;/math&gt;. Each visible neuron’s variance was set to 1.0 and remained unchanged.<br /> <br /> Minimizing Contrastive Divergence (CD) was used to facilitate learning. Since momentum is used to speed up learning, it was initially set to 0.5 and increased linearly to 0.9 over 20 epochs. The average gradient had 0.001 of a learning rate which was then multiplied by &lt;math&gt;(1-momentum)&lt;/math&gt; and L2 weight decay was set to 0.001. After setting up the hyperparameters, the model was done training after 100 epochs. Binary RBMs were used for training all subsequent layers with a learning rate of 0.01. Then, &lt;math&gt;p&lt;/math&gt; was set as the mean activation of a neuron in the data set and the visible bias of each neuron was initialized to &lt;math&gt;log(p/(1 − p))&lt;/math&gt;. Training each layer with 50 epochs, all remaining hyper-parameters were the same as those for the Gaussian RBM.<br /> <br /> === Dropout tuning ===<br /> <br /> The initial weights were set in a neural network from the pretrained RBMs. To finetune the network with dropout-backpropagation, momentum was initially set to 0.5 and increased linearly up to 0.9 over 10 epochs. The model had a small constant learning rate of 1.0 and it was used to apply to the average gradient on a minibatch. The model also retained all other hyperparameters the same as the model from MNIST dropout finetuning. The model required approximately 200 epochs to converge. For comparison purpose, they also finetuned the same network with standard backpropagation with a learning rate of 0.1 with the same hyperparameters.<br /> <br /> === Classification Test and Performance ===<br /> <br /> A Neural network was constructed to output the classification error rate on the test set of TIMIT dataset. They have built the neural network with four fully-connected hidden layers with 4000 neurons per layer. The output layer distinguishes distinct classes from 185 softmax output neurons that are merged into 39 classes. After constructing the neural network, 21 adjacent frames with an advance of 10ms per frame was given as an input.<br /> <br /> Comparing the performance of dropout with standard backpropagation on several network architectures and input representations, dropout consistently achieved lower error and cross-entropy. Results showed that it significantly controls overfitting, making the method robust to choices of network architecture. It also allowed much larger nets to be trained and removed the need for early stopping. Thus, neural network architectures with dropout are not very sensitive to the choice of learning rate and momentum.<br /> <br /> = Reuters Corpus Volume =<br /> Reuters Corpus Volume I archives 804,414 news documents that belong to 103 topics. Under four major themes - corporate/industrial, economics, government/social, and markets – they belonged to 63 classes. After removing 11 classes with no data and one class with insufficient data, they are left with 50 classes and 402,738 documents. The documents were divided into training and test sets equally and randomly, with each document representing the 2000 most frequent words in the dataset, excluding stopwords.<br /> <br /> They trained two neural networks, with size 2000-2000-1000-50, one using dropout and backpropagation, and the other using standard backpropagation. The training hyperparameters are the same as that in MNIST, but training was done for 500 epochs.<br /> <br /> In the following figure, we see the significant improvements by the model with dropout in the test set error. On the right side, we see that learning with dropout also proceeds smoother. <br /> <br /> [[File:reuters_figure.png|700px|center]]<br /> <br /> = CNN =<br /> <br /> Feed-forward neural networks consist of several layers of neurons where each neuron in a layer applies a linear filter to the input image data and is passed on to the neurons in the next layer. When calculating the neuron’s output, scalar bias a.k.a weights is applied to the filter with nonlinear activation function as parameters of the network that are learned by training data. [[File:cnnbigpicture.jpeg|thumb|upright=2|center|alt=text|Figure: Overview of Convolutional Neural Network]] There are several differences between Convolutional Neural networks and ordinary neural networks. The figure above gives a visual representation of a Convolutional Neural Network. First, CNN’s neurons are organized topographically into a bank and laid out on a 2D grid, so it reflects the organization of dimensions of the input data. Secondly, neurons in CNN apply filters which are local, and which are centered at the neuron’s location in the topographic organization. Meaning that useful metrics or clues to identify the object in an input image which can be found by examining local neighborhoods of the image. Next, all neurons in a bank apply the same filter at different locations in the input image. When looking at the image example, green is an input to one neuron bank, yellow is filter bank, and pink is the output of one neuron bank (convolved feature). A bank of neurons in a CNN applies a convolution operation, aka filters, to its input where a single layer in a CNN typically has multiple banks of neurons, each performing a convolution with a different filter. The resulting neuron banks become distinct input channels into the next layer. The whole process reduces the net’s representational capacity, but also reduces the capacity to overfit.<br /> [[File:bankofneurons.gif|thumb|upright=3|center|alt=text|Figure: Bank of neurons]]<br /> <br /> === Pooling ===<br /> <br /> Pooling layer summarizes the activities of local patches of neurons in the convolutional layer by subsampling the output of a convolutional layer. Pooling is useful for extracting dominant features, to decrease the computational power required to process the data through dimensionality reduction. The procedure of pooling goes on like this; output from convolutional layers is divided into sections called pooling units and they are laid out topographically, connected to a local neighborhood of other pooling units from the same convolutional output. Then, each pooling unit is computed with some function which could be maximum and average. Maximum pooling returns the maximum value from the section of the image covered by the pooling unit while average pooling returns the average of all the values inside the pooling unit (see example). In result, there are fewer total pooling units than convolutional unit outputs from the previous layer, this is due to larger spacing between pixels on pooling layers. Using the max-pooling function reduces the effect of outliers and improves generalization. Other than that, overlapping pooling makes this spacing between pixels smaller than the size of the neighborhood that the pooling units summarize (This spacing is usually referred as the stride between pooling units). With this variant, pooling layer can produce a coarse coding of the outputs which helps generalization. <br /> [[File:maxandavgpooling.jpeg|thumb|upright=2|center|alt=text|Figure: Max pooling and Average pooling]]<br /> <br /> === Local Response Normalization === <br /> <br /> This network includes local response normalization layers which are implemented in lateral form and used on neurons with unbounded activations and permits the detection of high-frequency features with a big neuron response. This regularizer encourages competition among neurons belonging to different banks. Normalization is done by dividing the activity of a neuron in bank &lt;math&gt;i&lt;/math&gt; at position &lt;math&gt;(x,y)&lt;/math&gt; by the equation:<br /> [[File:local response norm.png|upright=2|center|]] where the sum runs over &lt;math&gt;N&lt;/math&gt; ‘adjacent’ banks of neurons at the same position as in the topographic organization of neuron bank. The constants, &lt;math&gt;N&lt;/math&gt;, &lt;math&gt;alpha&lt;/math&gt; and &lt;math&gt;betas&lt;/math&gt; are hyper-parameters whose values are determined using a validation set. This technique is replaced by better techniques such as the combination of dropout and regularization methods (&lt;math&gt;L1&lt;/math&gt; and &lt;math&gt;L2&lt;/math&gt;)<br /> <br /> === Neuron nonlinearities ===<br /> <br /> All of the neurons for this model use the max-with-zero nonlinearity where output within a neuron is computed as &lt;math&gt; a^{i}_{x,y} = max(0, z^i_{x,y})&lt;/math&gt; where &lt;math&gt; z^i_{x,y} &lt;/math&gt; is the total input to the neuron. The reason they use nonlinearity is because it has several advantages over traditional saturating neuron models, such as significant reduction in training time required to reach a certain error rate. Another advantage is that nonlinearity reduces the need for contrast-normalization and data pre-processing since neurons do not saturate- meaning activities simply scale up little by little with usually large input values. For this model’s only pre-processing step, they subtract the mean activity from each pixel and the result is a centered data.<br /> <br /> === Objective function ===<br /> <br /> The objective function of their network maximizes the multinomial logistic regression objective which is the same as minimizing the average cross-entropy across training cases between the true label and the model’s predicted label.<br /> <br /> === Weight Initialization === <br /> <br /> It’s important to note that if a neuron always receives a negative value during training, it will not learn because its output is uniformly zero under the max-with-zero nonlinearity. Hence, the weights in their model were sampled from a zero-mean normal distribution with a high enough variance. High variance in weights will set a certain number of neurons with positive values for learning to happen, and in practice, it’s necessary to try out several candidates for variances until a working initialization is found. In their experiment, setting a positive constant, or 1, as biases of the neurons in the hidden layers was helpful in finding it.<br /> <br /> === Training ===<br /> <br /> In this model, a batch size of 128 samples and momentum of 0.9, we train our model using stochastic gradient descent. The update rule for weight &lt;math&gt;w&lt;/math&gt; is $$v_{i+1} = 0.9v_i + \epsilon &lt;\frac{dE}{dw_i}&gt; i$$ $$w_{i+1} = w_i + v_{i+1}$$ where &lt;math&gt;i&lt;/math&gt; is the iteration index, &lt;math&gt;v&lt;/math&gt; is a momentum variable, &lt;math&gt;\epsilon&lt;/math&gt; is the learning rate and &lt;math&gt;\frac{dE}{dw}&lt;/math&gt; is the average over the &lt;math&gt;i&lt;/math&gt;th batch of the derivative of the objective with respect to &lt;math&gt;w_i&lt;/math&gt;. The whole training process on CIFAR-10 takes roughly 90 minutes and ImageNet takes 4 days with dropout and two days without.<br /> <br /> === Learning ===<br /> To determine the learning rate for the network, it is a must to start with an equal learning rate for each layer which produces the largest reduction in the objective function with power of ten. Usually, it is in the order of &lt;math&gt;10^{-2}&lt;/math&gt; or &lt;math&gt;10^{-3}&lt;/math&gt;. In this case, they reduce the learning rate twice by a factor of ten before termination of training.<br /> <br /> = CIFAR-10 =<br /> <br /> === CIFAR-10 Dataset ===<br /> <br /> Removing incorrect labels, The CIFAR-10 dataset is a subset of the Tiny Images dataset with 10 classes. It contains 5000 training images and 1000 testing images for each class. The dataset has 32 x 32 color images searched from the web and the images are labeled with the noun used to search the image.<br /> <br /> [[File:CIFAR-10.png|thumb|upright=2|center|alt=text|Figure 4: CIFAR-10 Sample Dataset]]<br /> <br /> === Models for CIFAR-10 ===<br /> <br /> Two models, one with dropout and one without dropout, were built to test the performance of dropout on CIFAR-10. All models have CNN with three convolutional layers each with a pooling layer. All of the pooling payers use a stride=2 and summarize a 3*3 neighborhood. The max-pooling method is performed by the pooling layer which follows the first convolutional layer, and the average-pooling method is performed by remaining 2 pooling layers. The first and second pooling layers with &lt;math&gt;N = 9, α = 0.001&lt;/math&gt;, and &lt;math&gt;β = 0.75&lt;/math&gt; are followed by response normalization layers. A ten-unit softmax layer, which is used to output a probability distribution over class labels, is connected with the upper-most pooling layer. Using filter size of 5×5, all convolutional layers have 64 filter banks.<br /> <br /> Additional changes were made with the model with dropout. The model with dropout enables us to use more parameters because dropout forces a strong regularization on the network. Thus, a fourth weight layer is added to take the input from the previous pooling layer. This fourth weight layer is locally connected, but not convolutional, and contains 16 banks of filters of size 3 × 3 with 50% dropout. Lastly, the softmax layer takes its input from this fourth weight layer.<br /> <br /> Thus, with a neural network with 3 convolutional hidden layers with 3 max-pooling layers, the classification error achieved 16.6% to beat 18.5% from the best published error rate without using transformed data. The model with one additional locally-connected layer and dropout at the last hidden layer produced the error rate of 15.6%.<br /> <br /> = ImageNet =<br /> <br /> ===ImageNet Dataset===<br /> <br /> ImageNet is a dataset of millions of high-resolution images, and they are labeled among 1000 different categories. The data were collected from the web and manually labeled using MTerk tool, which is a crowd-sourcing tool provided by Amazon.<br /> Because this dataset has millions of labeled images in thousands of categories, it is very difficult to have perfect accuracy on this dataset even for humans because the ImageNet images may contain multiple objects and there are a large number of object classes. ImageNet and CIFAR-10 are very similar, but the scale of ImageNet is about 20 times bigger (1,300,000 vs 60,000). The size of ImageNet is about 1.3 million training images, 50,000 validation images, and 150,000 testing images. They used resized images of 256 x 256 pixels for their experiments.<br /> <br /> '''An ambiguous example to classify:'''<br /> <br /> [[File:imagenet1.png|200px|center]]<br /> <br /> When this paper was written, the best score on this dataset was the error rate of 45.7% by High-dimensional signature compression for large-scale image classification (J. Sanchez, F. Perronnin, CVPR11 (2011)). The authors of this paper could achieve a comparable performance of 48.6% error rate using a single neural network with five convolutional hidden layers with a max-pooling layer in between, followed by two globally connected layers and a final 1000-way softmax layer. When applying 50% dropout to the 6th layer, the error rate was brought down to 42.4%.<br /> <br /> '''ImageNet Dataset:'''<br /> <br /> [[File:imagenet2.png|400px|center]]<br /> <br /> ===Models for ImageNet===<br /> <br /> They mostly focused on the model with dropout because the one without dropout had a similar approach, but there was a serious issue with overfitting. They used a convolutional neural network trained by 224×224 patches randomly extracted from the 256 × 256 images. This could reduce the network’s capacity to overfit the training data and helped generalization as a form of data augmentation. The method of averaging the prediction of the net on ten 224 × 224 patches of the 256 × 256 input image was used for testing their model patched at the center, four corners, and their horizontal reflections. To maximize the performance on the validation set, this complicated network architecture was used and it was found that dropout was very effective. Also, it was demonstrated that using non-convolutional higher layers with the number of parameters worked well with dropout, but it had a negative impact to the performance without dropout.<br /> <br /> The network contains seven weight layers. The first five are convolutional, and the last two are globally-connected. Max-pooling layers follow the layer number 1,2, and 5. And then, the output of the last globally-connected layer was fed to a 1000-way softmax output layers. Using this architecture, the authors achieved the error rate of 48.6%. When applying 50% dropout to the 6th layer, the error rate was brought down to 42.4%.<br /> <br /> <br /> [[File:modelh2.png|700px|center]] <br /> <br /> [[File:layer2.png|600px|center]]<br /> <br /> Like the previous datasets, such as the MNIST, TIMIT, Reuters, and CIFAR-10, we also see a significant improvement for the ImageNet dataset. Including complicated architectures like this one, introducing dropout generalizes models better and gives lower test error rates.<br /> <br /> = Conclusion =<br /> <br /> The authors have shown a consistent improvement by the models trained with dropout in classifying objects in the following datasets: MNIST; TIMIT; Reuters Corpus Volume I; CIFAR-10; and ImageNet.<br /> <br /> = Critiques =<br /> It is a very brilliant idea to dropout half of the neurons to reduce co-adaptations. It is mentioned that for fully connected layers, dropout in all hidden layers works better than dropout in only one hidden layer. There is another paper Dropout: A Simple Way to Prevent Neural Networks from<br /> Overfitting[https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf] gives a more detailed explanation.<br /> <br /> It will be interesting to see how this paper could be used to prevent overfitting of LSTMs.<br /> <br /> Firstly, it is a very interested topic of classification by &quot;dropout&quot; CNN method(omitting neurons in hidden layers). If the author can briefly explain the advantages of this method in processing image data in theory, it will be easier for readers to understand. Also, how to deal with overfitting issue would be valuable.<br /> <br /> The authors mention that they tried various dropout probabilities and that the majority of them improved the model's generalization performance, but that more extreme probabilities tended to be worse which is why a dropout rate of 50% was used in the paper. The authors further develop this point to mention that the method can be improved by adapting individual dropout probabilities of each hidden or input unit using validation tests. This would be an interesting area to further develop and explore, as using a hardcoded 50% dropout for all layers might not be the optimal choice for all CNN applications. It would have been interesting to see the results of their investigations of differing dropout rates.<br /> <br /> The authors don't explain that during training, at each layer that we apply dropout, the values must be scaled by 1/p where p is dropout rate - this way the expected value of the layers is the same in both train and test time. They may have considered another solution for this discrepancy at the time (it is an old paper) but it doesn't seem like any solution was presented here. <br /> <br /> Despite the advantages of using dropout to prevent overfitting and reducing errors in testing, the authors did not discuss much about the effects on the length of training time. In another [https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf paper] published a few years later by the same authors, there was more discussion about this. It appears that dropout increases training time by 2-3 times compared to a standard NN with the same architecture, which is a drawback that might be worth mentioning. <br /> <br /> == Reference ==<br />  N. Srivastave, &quot;Dropout: a simple way to prevent neural networks from overfitting&quot;, The Journal of Machine Learning Research, Jan 2014.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Describtion_of_Text_Mining&diff=49035 Describtion of Text Mining 2020-12-03T22:33:17Z <p>Z42qin: </p> <hr /> <div>== Presented by == <br /> Yawen Wang, Danmeng Cui, Zijie Jiang, Mingkang Jiang, Haotian Ren, Haris Bin Zahid<br /> <br /> == Introduction ==<br /> This paper focuses on the different text mining techniques and the applications of text mining in the healthcare and biomedical domain. The text mining field has been popular as a result of the amount of text data that is available in different forms. The text data is bound to grow even more in 2020, indicating a 50 times growth since 2010. Text is unstructured information, which is easy for humans to construct and understand but difficult for machines. Hence, there is a need to design algorithms to effectively process this avalanche of text. To further explore the text mining field, the related text mining approaches can be considered. The different text mining approaches relate to two main methods: knowledge delivery and traditional data mining methods. <br /> <br /> The authors note that knowledge delivery methods involve the application of different steps to a specific data set to create specific patterns. Research in knowledge delivery methods has evolved over the years due to advances in hardware and software technology. On the other hand, data mining has experienced substantial development through the intersection of three fields: databases, machine learning, and statistics. As brought out by the authors, text mining approaches focus on the exploration of information from a specific text. The information explored is in the form of structured, semi-structured, and unstructured text. It is important to note that text mining covers different sets of algorithms and topics that include information retrieval. The topics and algorithms are used for analyzing different text forms.<br /> <br /> ==Text Representation and Encoding ==<br /> The authors review multiple methods of preprocessing text, including 4 methods to preprocess and recognize influence and frequency of individual group of words in a document. In many text mining algorithms, one of the key components is preprocessing. Preprocessing consists of different tasks that include filtering, tokenization, stemming, and lemmatization. The first step is tokenization, where a character sequence is broken down into different words or phrases. After the breakdown, filtering is carried out to remove some words. The various word inflected forms are grouped together through lemmatization, and later, the derived roots of the derived words are obtained through stemming.<br /> <br /> '''1. Tokenization'''<br /> <br /> This process splits text into units of words of phrases known as tokens while removing unnecessary characters. Characters such as punctuation are removed and the text is split at space characters. An example of this would be converting the string &quot;This is my string&quot; to &quot;This&quot;, &quot;is&quot;, &quot;my&quot;, &quot;string&quot;.<br /> <br /> '''2. Filtering'''<br /> <br /> Filtering is a process by which unnecessary words or characters are removed. Often these include punctuation, prepositions, and conjugations. The resulting corpus then contains words with maximal importance in distinguishing between classes.<br /> <br /> '''3. Lemmatization'''<br /> <br /> Lemmatization is a task where convert various inflected forms of a word to a single form. However, we must specify the POS of each words, which can prone to human error. <br /> <br /> '''4. Stemming'''<br /> <br /> Stemming extracts the roots of words. It is a language dependent process. The goal of both stemming is to reduce inflectional and related (definition wise) forms of a word to a common base form. An example of this would be changing &quot;am&quot;, &quot;are&quot;, or &quot;is&quot; to &quot;be&quot;.<br /> <br /> '''Vector Space Model'''<br /> In this section of the paper, the authors explore the different ways in which the text can be represented on a large collection of documents. One common way of representing the documents is in the form of a bag of words. The bag of words considers the occurrences of different terms.<br /> In different text mining applications, documents are ranked and represented as vectors so as to display the significance of any word. <br /> The authors note that the three basic models used are vector space, inference network, and the probabilistic models. The vector space model is used to represent documents by converting them into vectors. In the model, a variable is used to represent each model to indicate the importance of the word in the document. <br /> <br /> The weights have 2 main models used Boolean model and TF-IDF model: <br /> '''Boolean model'''<br /> terms are assignment with a positive wij if the term appears in the document. otherwise, it will be assigned a weight of 0. <br /> <br /> '''term frequency - inverse document frequency (TF-IDF)'''<br /> The words are weighted using the TF-IDF scheme computed as <br /> <br /> $$<br /> q(w)=f_d(w)*\log{\frac{|D|}{f_D(w)}}<br />$$<br /> <br /> The frequency of each term is normalized by the inverse of document frequency, which helps distinct words with low frequency is recognized its importance. Each document is represented by a vector of term weights, &lt;math&gt;\omega(d) = (\omega(d, w_1), \omega(d,w_2),...,\omega(d,w_v))&lt;/math&gt;. The similarity between two documents &lt;math&gt;d_1, d_2&lt;/math&gt; is commonly measured by cosine similarity:<br /> $$<br /> S(d_1,d_2) = \cos(\theta) = \frac{d_1\cdot d_2}{\sum_{i=1}^vw^2_{1i}\cdot\sum_{i=1}^vw^2_{2i}}<br />$$<br /> <br /> <br /> <br /> == Classification ==<br /> Classification in Text Mining aims to assign predefined classes to text documents. For a set &lt;math&gt;\mathcal{D} = {d_1, d_2, ... d_n}&lt;/math&gt; of documents, each &lt;math&gt;d_i&lt;/math&gt; is mapped to a label &lt;math&gt;l_i&lt;/math&gt; from the set &lt;math&gt;\mathcal{L} = {l_1, l_2, ... l_k}&lt;/math&gt;. The goal is to find a classification model &lt;math&gt;f&lt;/math&gt; such that: &lt;math&gt;\\&lt;/math&gt;<br /> $$<br /> f: \mathcal{D} \rightarrow \mathcal{L} \quad \quad \quad f(\mathcal{d}) = \mathcal{l}<br />$$<br /> The author illustrates 4 different classifiers that are commonly used in text mining.<br /> <br /> <br /> '''1. Naive Bayes Classifier''' <br /> <br /> Bayes rule is used to classify new examples and select the class that has the generated result that occurs most often. <br /> Naive Bayes Classifier models the distribution of documents in each class using a probabilistic model assuming that the distribution<br /> of different terms is independent of each other. The models commonly used in this classifier tried to find the posterior probability of a class based on the distribution and assumes that the documents generated are based on a mixture model parameterized by &lt;math&gt;\theta&lt;/math&gt; and compute the likelihood of a document using the sum of probabilities over all mixture component. In addition, the Naive Bayes Classifier can help get around the curse of dimensionality, which may arise with high-dimensional data, such as text. <br /> <br /> '''2. Nearest Neighbour Classifier'''<br /> <br /> Nearest Neighbour Classifier uses distance-based measures to perform the classification. The documents which belong to the same class are more likely &quot;similar&quot; or close to each other based on the similarity measure. The classification of the test documents is inferred from the class labels of similar documents in the training set. K-Nearest Neighbor classification is well known to suffer from the &quot;curse of dimensionality&quot;, as the proportional volume of each $d$-sphere surrounding each datapoint compared to the volume of the sample space shrinks exponentially in $d$. <br /> <br /> '''3. Decision Tree Classifier'''<br /> <br /> A hierarchical tree of the training instances, in which a condition on the attribute value is used to divide the data hierarchically. The decision tree recursively partitions the training data set into smaller subdivisions based on a set of tests defined at each node or branch. Each node of the tree is a test of some attribute of the training instance, and each branch descending from the node corresponds to one of the values of this attribute. The conditions on the nodes are commonly defined by the terms in the text documents.<br /> <br /> '''4. Support Vector Machines'''<br /> <br /> SVM is a form of Linear Classifiers which are models that makes a classification decision based on the value of the linear combinations of the documents features. The output of a linear predictor is defined to the &lt;math&gt; y=\vec{a} \cdot \vec{x} + b&lt;/math&gt; where &lt;math&gt;\vec{x}&lt;/math&gt; is the normalized document word frequency vector, &lt;math&gt;\vec{a}&lt;/math&gt; is a vector of coefficient and &lt;math&gt;b&lt;/math&gt; is a scalar. Support Vector Machines attempts to find a linear separators between various classes. An advantage of the SVM method is it is robust to high dimensionality.<br /> <br /> == Clustering ==<br /> Clustering has been extensively studied in the context of the text as it has a wide range of applications such as visualization and document organization.<br /> <br /> Clustering algorithms are used to group similar documents and thus aid in information retrieval. Text clustering can be in different levels of granularities, where clusters can be documents, paragraphs, sentences, or terms. Since text data has numerous distance characteristics that demand the design of text-specific algorithms for the task, using a binary vector to represent the text document is simply not enough. Here are some unique properties of text representation:<br /> <br /> 1. Text representation has a large dimensionality, in which the size of the vocabulary from which the documents are drawn is massive, but a document might only contain a small number of words.<br /> <br /> 2. The words in the documents are usually correlated with each other. Need to take the correlation into consideration when designing algorithms.<br /> <br /> 3. The number of words differs from one another of the document. Thus the document needs to be normalized first before the clustering process.<br /> <br /> Three most commonly used text clustering algorithms are presented below.<br /> <br /> <br /> '''1. Hierarchical Clustering algorithms''' <br /> <br /> Hierarchical Clustering algorithms builds a group of clusters that can be depicted as a hierarchy of clusters. The hierarchy can be constructed in top-down (divisive) or bottom-up (agglomeration). Hierarchical clustering algorithms are one of the Distanced-based clustering algorithms, i.e., using a similarity function to measure the closeness between text documents.<br /> <br /> In the top-down approach, the algorithm begins with one cluster which includes all the documents. we recursively split this cluster into sub-clusters.<br /> Here is an example of a Hierarchical Clustering algorithm, the data is to be clustered by the euclidean distance. This method builds the hierarchy from the individual elements by progressively merging clusters. In our example, we have six elements {a} {b} {c} {d} {e} and {f}. The first step determines which elements to merge in a cluster by taking the two closest elements, according to the chosen distance.<br /> <br /> <br /> [[File:418px-Hierarchical clustering simple diagram.svg.png| 300px | center]]<br /> <br /> <br /> &lt;div align=&quot;center&quot;&gt;Figure 1: Hierarchical Clustering Raw Data&lt;/div&gt;<br /> <br /> <br /> <br /> [[File:250px-Clusters.svg (1).png| 200px | center]]<br /> <br /> <br /> &lt;div align=&quot;center&quot;&gt;Figure 2: Hierarchical Clustering Clustered Data&lt;/div&gt;<br /> <br /> A main advantage of hierarchical clustering is that the algorithm only needs to be done once for any number of clusters (ie. if an individual wishes to use a different number of clusters than originally intended, they do not need to repeat the algorithm)<br /> <br /> '''2. k-means Clustering'''<br /> <br /> k-means clustering is a partitioning algorithm that partitions n documents in the context of text data into k clusters.<br /> <br /> Input: Document D, similarity measure S, number k of cluster<br /> Output: Set of k clusters<br /> Select randomly ''k'' datapoints as starting centroids<br /> While ''not converged'' do <br /> Assign documents to the centroids based on the closest similarity<br /> Calculate the cluster centroids for all clusters<br /> return ''k clusters''<br /> <br /> The main disadvantage of k-means clustering is that it is indeed very sensitive to the initial choice of the number of k. Also, since the function is run until clusters converges, k-means clustering tends to take longer to perform than hierarchical clustering. On the other hand, advantages of k-means clustering are that it is simple to implement, the algorithm scales well to large datasets, and the results are easily interpretable.<br /> <br /> <br /> '''3. Probabilistic Clustering and Topic Models'''<br /> <br /> Topic modeling is one of the most popular probabilistic clustering algorithms in recent studies. The main idea is to create a *probabilistic generative model* for the corpus of text documents. In topic models, documents are a mixture of topics, where each topic represents a probability distribution over words.<br /> <br /> There are two main topic models:<br /> * Probabilistic Latent Semantic Analysis (pLSA)<br /> * Latent Dirichlet Allocation (LDA)<br /> <br /> The paper covers LDA in more detail. LDA is a state-of-the-art unsupervised algorithm for extracting topics from a collection of documents.<br /> <br /> Given &lt;math&gt;\mathcal{D} = \{d_1, d_2, \cdots, d_{|\mathcal{D}|}\}&lt;/math&gt; is the corpus and &lt;math&gt;\mathcal{V} = \{w_1, w_2, \cdots, w_{|\mathcal{V}|}\}&lt;/math&gt; is the vocabulary of the corpus. <br /> <br /> A topic is &lt;math&gt;z_j, 1 \leq j \leq K&lt;/math&gt; is a multinomial probability distribution over &lt;math&gt;|\mathcal{V}|&lt;/math&gt; words. <br /> <br /> The distribution of words in a given document is:<br /> <br /> &lt;math&gt;p(w_i|d) = \Sigma_{j=1}^K p(w_i|z_j)p(z_j|d)&lt;/math&gt;<br /> <br /> The LDA assumes the following generative process for the corpus of &lt;math&gt;\mathcal{D}&lt;/math&gt;<br /> * For each topic &lt;math&gt;k\in \{1,2,\cdots, K\}&lt;/math&gt;, sample a word distribution &lt;math&gt;\phi_k \sim Dir(\beta)&lt;/math&gt;<br /> * For each document &lt;math&gt;d \in \{1,2,\cdots,D\}&lt;/math&gt;<br /> ** Sample a topic distribution &lt;math&gt;\theta_d \sim Dir(\alpha)&lt;/math&gt;<br /> ** For each word &lt;math&gt;w_n, n \in \{1,2,\cdots,N\}&lt;/math&gt; in document &lt;math&gt;d&lt;/math&gt;<br /> *** Sample a topic &lt;math&gt;z_i \sim Mult(\theta_d)&lt;/math&gt;<br /> *** Sample a word &lt;math&gt;w_n \sim Mult(\phi_{z_i})&lt;/math&gt;<br /> <br /> In practice, LDA is often used as a module in more complicated models and has already been applied to a wide variety of domains. In addition, many variations of LDA has been created, including supervised LDA (sLDA) and hierarchical LDA (hLDA)<br /> <br /> == Information Extraction ==<br /> Information Extraction (IE) is the process of extracting useful, structured information from unstructured or semi-structured text. It automatically extracts based on our command. <br /> <br /> For example, from the sentence “XYZ company was founded by Peter in the year of 1950”, we can identify the following information:<br /> <br /> Founderof(Peter, XYZ)<br /> Foundedin(1950, XYZ)<br /> <br /> The author mentioned 4 parts that are important for Information Extraction<br /> <br /> '''1. Named Entity Recognition(NER)'''<br /> <br /> This is the process of identifying real-world entity from free text, such as &quot;Apple Inc.&quot;, &quot;Donald Trump&quot;, &quot;PlayStation 5&quot; etc. Moreover, the task is to identify the category of these entities, such as &quot;Apple Inc.&quot; is in the category of the company, &quot;Donald Trump&quot; is in the category of the USA president, and &quot;PlayStation 5&quot; is in the category of the entertainment system. <br /> <br /> '''2. Hidden Markov Model'''<br /> <br /> Since traditional probabilistic classification does not consider the predicted labels of neighbor words, we use the Hidden Markov Model when doing Information Extraction. This model is different because it considers that the label of one word depends on the previous words that appeared. <br /> <br /> '''3. Conditional Random Fields'''<br /> <br /> This is a technique that is widely used in Information Extraction. The definition of it is related to graph theory. <br /> let G = (V, E) be a graph and Yv stands for the index of the vertices in G. Then (X, Y) is a conditional random field, when the random variables Yv, conditioned on X, obey Markov property with respect to the graph, and:<br /> p(Yv |X, Yw ,w , v) = p(Yv |X, Yw ,w ∼ v), where w ∼ v means w and v are neighbors in G.<br /> <br /> '''4. Relation Extraction'''<br /> <br /> This is a task of finding semantic relationships between word entities in text documents, for example in a sentence such as &quot;Seth Curry is the brother of Stephen Curry&quot;. If there is a document including these two names, the task is to identify the relationship of these two entities.<br /> <br /> == Biomedical Application ==<br /> <br /> Text mining has several applications in the domain of biomedical sciences. The explosion of academic literature in the field has made it quite hard for scientists to keep up with novel research. This is why text mining techniques are ever so important in making the knowledge digestible.<br /> <br /> The text mining techniques are able to extract meaningful information from large data by making use of biomedical ontology, which is a compilation of a common set of terms used in an area of knowledge. The Unified Medical Language System (UMLS) is the most comprehensive such resource, consisting of definitions of biomedical jargon. Several information extraction algorithms rely on the ontology to perform tasks such as Named Entity Recognition (NER) and Relation Extraction.<br /> <br /> NER involves locating and classifying biomedical entities into meaningful categories and assigning semantic representation to those entities. The NER methods can be broadly grouped into Dictionary-based, Rule-based and Statistical approaches. NER tasks are challenging in the biomedical domain due to three key reasons: (1) There is a continuously growing volume of semantically related entities in the biomedical domain due to continuous scientific progress, so NER systems depend on dictionaries of terms which can never be complete; (2) There are often numerous names for the same concept in the biomedical domain, such as &quot;heart attack&quot; and &quot;myocardial infarction&quot;; and (3) Acronyms and abbreviations are frequently used which makes it complicated to identify the concepts these terms express.<br /> <br /> Relation extraction, on the other hand, is the process of determining relationships between the entities. This is accomplished mainly by identifying the correlation between entities through analyzing the frequency of terms, as well as rules defined by domain experts. Moreover, modern algorithms are also able to summarize large documents and answer natural language questions posed by humans.<br /> <br /> Summarization is a common biomedical text mining task which largely utilizes information extraction tasks. The idea is the automatically identify significant aspects of documents and represent them in a coherent fashion. However, evaluating summarization methods becomes very difficult since deciding whether a summary is &quot;good&quot; is often subjective, although there are some automatic evaluation techniques for smamaries such as ROUGE (Recall-Oriented Understudy for Gisting Evaluation), which compares automatically generated summaries with those created by humans.<br /> <br /> == Conclusion ==<br /> <br /> This paper gave a holistic overview of the methods and applications of text mining, particularly its relevance in the biomedical domain. It highlights several popular algorithms and summarizes them along with their advantages, limitations and some potential situations where they could be used. Because of ever-growing data, for example, the very high volume of scientific literature being produced every year, the interest in this field is massive and is bound to grow in the future.<br /> <br /> == Critiques==<br /> <br /> This is a very detailed approach to introduce some different algorithms on text mining. Since many algorithms are given, it might be a good idea to compare their performances on text mining by training them on some text data and compare them to the former baselines, to see if there exists any improvement.<br /> <br /> it is a detailed summary of the techniques used in text mining. It would be more helpful if some dataset can be included for training and testing. The algorithms were grouped by different topics so that different datasets and measurements are required.<br /> <br /> It would be better for the paper to include test accuracy for testing and training sets to support text mining is a more efficient and effective algorithm compared to other techniques. Moreover, this paper mentioned Text Mining approach can be used to extract high-quality information from videos. It is to believe that extracting from videos is much more difficult than images and texts. How is it possible to retain its test accuracy at a good level for videos?<br /> <br /> Preprocessing an important step to analyze text, so it might be better to have the more details about that. For example, what types of words are usually removed and show we record the relative position of each word in the sentence. If one close related sentences were split into two sentences, how can we capture their relations?<br /> <br /> The authors could give more details on the applications of text mining in the healthcare and biomedical domain. For example, how could preprocessing, classification, clustering, and information extraction process be applied to this domain. Other than introduction of existing algorithms (e.g. NER), authors can provide more information about how they performs (with a sample dataset), what are their limitations, and comparisons among different algorithms.<br /> <br /> In the preprocessing section, it seems like the authors incorrectly describe what stemming is - stemming just removes the last few letters of a word (ex. studying -&gt; study, studies -&gt; studi). What the authors actually describe is lemmatization which is much more informative than stemming. The down side of lemmatization is that it takes more effort to build a lemmatizer than a stemmer and even once it is built it is slow in comparison with a stemmer.<br /> <br /> One of the challenges of text mining in the biomedical field is that a lot of patient data are still in the form of paper documents. Text mining can speed up the digitization of patient data and allow for the development of disease diagnosis algorithms. It'll be interesting to see how text mining can be integrated with healthcare AI such as the doppelganger algorithm to enhance question answering accuracy. (Cresswell et al, 2018)<br /> <br /> == References ==<br /> <br /> Allahyari, M., Pouriyeh, S., Assefi, M., Safaei, S., Trippe, E. D., Gutierrez, J. B., &amp; Kochut, K. (2017). A brief survey of text mining: Classification, clustering, and extraction techniques. arXiv preprint arXiv:1707.02919.<br /> <br /> Cresswell, Kathrin &amp; Cunningham-Burley, Sarah &amp; Sheikh, Aziz. (2018). Healthcare robotics � a qualitative exploration of key challenges and future directions (Preprint). Journal of Medical Internet Research. 20. 10.2196/10410.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Mask_RCNN&diff=49033 Mask RCNN 2020-12-03T22:06:54Z <p>Z42qin: /* Model Architecture */</p> <hr /> <div>== Presented by == <br /> Qing Guo, Xueguang Ma, James Ni, Yuanxin Wang<br /> <br /> == Introduction == <br /> Mask RCNN  is a deep neural network architecture that aims to solve instance segmentation problems in computer vision which is important when attempting to identify different objects within the same image. <br /> Mask R-CNN extends Faster R-CNN  by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework. Mask R-CNN achieved top results in all three tracks of the COCO suite of challenges , including instance segmentation, bounding-box object detection, and person keypoint detection.<br /> <br /> == Visual Perception Tasks == <br /> <br /> Figure 1 shows a visual representation of different types of visual perception tasks:<br /> <br /> - Image Classification: Predict a set of labels to characterize the contents of an input image<br /> <br /> - Object Detection: Build on image classification but localize each object in an image by placing bounding boxes around the objects<br /> <br /> - Semantic Segmentation: Associate every pixel in an input image with a class label<br /> <br /> - Instance Segmentation: Associate every pixel in an input image to a specific object. Instance segmentation combines image classification, object detection and semantic segmentation making it a complex task .<br /> <br /> [[File:instance segmentation.png | center]]<br /> &lt;div align=&quot;center&quot;&gt;Figure 1: Visual Perception tasks&lt;/div&gt;<br /> <br /> <br /> Mask RCNN is a deep neural network architecture for Instance Segmentation.<br /> <br /> == Related Work == <br /> Region Proposal Network: A Region Proposal Network (RPN) takes an image (of any size) as input and outputs a set of rectangular object proposals, each with an objectness score.<br /> <br /> ROI Pooling: The main use of ROI (Region of Interest) Pooling is to adjust the proposal to a uniform size. It’s better for the subsequent network to process. It maps the proposal to the corresponding position of the feature map, divide the mapped area into sections of the same size, and performs max pooling or average pooling operations on each section.<br /> <br /> Faster R-CNN: Faster R-CNN consists of two stages: Region Proposal Network and ROI Pooling. Region Proposal Network proposes candidate object bounding boxes. ROI Pooling, which is in essence Fast R-CNN, extracts features using RoIPool from each candidate box and performs classification and bounding-box regression. The features used by both stages can be shared for faster inference.<br /> <br /> [[File:FasterRCNN.png | center]]<br /> &lt;div align=&quot;center&quot;&gt;Figure 2: Faster RCNN architecture&lt;/div&gt;<br /> <br /> <br /> ResNet-FPN: FPN uses a top-down architecture with lateral connections to build an in-network feature pyramid from a single-scale input. FPN is a general architecture that can be used in conjunction with various networks, such as VGG, ResNet, etc. Faster R-CNN with an FPN backbone extracts RoI features from different levels of the feature pyramid according to their scale. Other than FPN, the rest of the approach is similar to vanilla ResNet. Using a ResNet-FPN backbone for feature extraction with Mask RCNN gives excellent gains in both accuracy and speed.<br /> <br /> [[File:ResNetFPN.png | center]]<br /> &lt;div align=&quot;center&quot;&gt;Figure 3: ResNetFPN architecture&lt;/div&gt;<br /> <br /> == Model Architecture == <br /> The structure of mask R-CNN is quite similar to the structure of faster R-CNN. <br /> Faster R-CNN has two stages, the RPN(Region Proposal Network) first proposes candidate object bounding boxes. Then RoIPool extracts the features from these boxes. After the features are extracted, these features data can be analyzed using classification and bounding-box regression. Mask R-CNN shares the identical first stage. But the second stage is adjusted to tackle the issue of simplifying the stages pipeline. Instead of only performing classification and bounding-box regression, it also outputs a binary mask for each RoI as &lt;math&gt;L=L_{cls}+L_{box}+L_{mask}&lt;/math&gt;, where &lt;math&gt;L_{cls}&lt;/math&gt;, &lt;math&gt;L_{box}&lt;/math&gt;, &lt;math&gt;L_{mask}&lt;/math&gt; represent the classification loss, bounding box loss and the average binary cross-entropy loss respectively.<br /> <br /> The important concept here is that, for most recent network systems, there's a certain order to follow when performing classification and regression, because classification depends on mask predictions. Mask R-CNN, on the other hand, applies bounding-box classification and regression in parallel, which effectively simplifies the multi-stage pipeline of the original R-CNN. And just for comparison, complete R-CNN pipeline stages involve 1. Make region proposals; 2. Feature extraction from region proposals; 3. SVM for object classification; 4. Bounding box regression. In conclusion, stages 3 and 4 are adjusted to simplify the network procedures.<br /> <br /> The system follows the multi-task loss, which by formula equals classification loss plus bounding-box loss plus the average binary cross-entropy loss.<br /> One thing worth noticing is that for other network systems, those masks across classes compete with each other, but in this particular case, with a <br /> per-pixel sigmoid and a binary loss the masks across classes no longer compete, which makes this formula the key for good instance segmentation results.<br /> <br /> '' RoIAlign''<br /> <br /> This concept is useful in stage 2 where the RoIPool extracts features from bounding-boxes. For each RoI as input, there will be a mask and a feature map as output. The mask is obtained using the FCN(Fully Convolutional Network) and the feature map is obtained using the RoIPool. The mask helps with spatial layout, which is crucial to the pixel-to-pixel correspondence. <br /> <br /> The two things we desire along the procedure are: pixel-to-pixel correspondence; no quantization is performed on any coordinates involved in the RoI, its bins, or the sampling points. Pixel-to-pixel correspondence makes sure that the input and output match in size. If there is a size difference, there will be information loss, and coordinates cannot be matched. Also, instead of quantization, the coordinates are computed using bilinear interpolation They uses bilinear interpolation to get the exact values of the inputs features at the 4 RoI bins and aggregate the result (using max or average). These results are robust to the sampling location and number of points and to guarantee spatial correspondence.<br /> <br /> The network architectures utilized are called ResNet and ResNeXt. The depth can be either 50 or 101.​ ResNet-FPN(Feature Pyramid Network) is used for feature extraction.​ <br /> <br /> Some implementation details should be mentioned: first, an RoI is considered positive if it has IoU with a ground-truth box of at least 0.5 and negative otherwise. It is important because the mask loss Lmask is defined only on positive RoIs. Second, image-centric training is used to rescale images so that pixel correspondence is achieved. An example complete structure is, the proposal number is 1000 for FPN, and then run the box prediction branch on these proposals. The mask branch is then applied to the highest scoring 100 detection boxes. The mask branch can predict K masks per RoI, but only the kth mask will be used, where k is the predicted class by the classification branch. The m-by-m floating-number mask output is then resized to the RoI size and binarized at a threshold of 0.5.<br /> <br /> == Results ==<br /> [[File:ExpInstanceSeg.png | center]]<br /> &lt;div align=&quot;center&quot;&gt;Figure 4: Instance Segmentation Experiments&lt;/div&gt;<br /> <br /> Instance Segmentation: Based on COCO dataset, Mask R-CNN outperforms all categories comparing to MNC and FCIS which are state of art model <br /> <br /> [[File:BoundingBoxExp.png | center]]<br /> &lt;div align=&quot;center&quot;&gt;Figure 5: Bounding Box Detection Experiments&lt;/div&gt;<br /> <br /> Bounding Box Detection: Mask R-CNN outperforms the base variants of all previous state-of-the-art models, including the winner of the COCO 2016 Detection Challenge.<br /> <br /> <br /> == Ablation Experiments ==<br /> [[File:BackboneExp.png | center]]<br /> &lt;div align=&quot;center&quot;&gt;Figure 6: Backbone Architecture Experiments&lt;/div&gt;<br /> <br /> (a) Backbone Architecture: Better backbones bring expected gains: deeper networks do better, FPN outperforms C4 features, and ResNeXt improves on ResNet. <br /> <br /> [[File:MultiVSInde.png | center]]<br /> &lt;div align=&quot;center&quot;&gt;Figure 7: Multinomial vs. Independent Masks Experiments&lt;/div&gt;<br /> <br /> (b) Multinomial vs. Independent Masks (ResNet-50-C4): Decoupling via perclass binary masks (sigmoid) gives large gains over multinomial masks (softmax).<br /> <br /> [[File: RoIAlign.png | center]]<br /> &lt;div align=&quot;center&quot;&gt;Figure 8: RoIAlign Experiments 1&lt;/div&gt;<br /> <br /> (c) RoIAlign (ResNet-50-C4): Mask results with various RoI layers. Our RoIAlign layer improves AP by ∼3 points and AP75 by ∼5 points. Using proper alignment is the only factor that contributes to the large gap between RoI layers. <br /> <br /> [[File: RoIAlignExp.png | center]]<br /> &lt;div align=&quot;center&quot;&gt;Figure 9: RoIAlign Experiments w Experiments&lt;/div&gt;<br /> <br /> (d) RoIAlign (ResNet-50-C5, stride 32): Mask-level and box-level AP using large-stride features. Misalignments are more severe than with stride-16 features, resulting in big accuracy gaps.<br /> <br /> [[File:MaskBranchExp.png | center]]<br /> &lt;div align=&quot;center&quot;&gt;Figure 10: Mask Branch Experiments&lt;/div&gt;<br /> <br /> (e) Mask Branch (ResNet-50-FPN): Fully convolutional networks (FCN) vs. multi-layer perceptrons (MLP, fully-connected) for mask prediction. FCNs improve results as they take advantage of explicitly encoding spatial layout.<br /> <br /> == Human Pose Estimation ==<br /> Mask RCNN can be extended to human pose estimation.<br /> <br /> The simple approach the paper presents is to model a keypoint’s location as a one-hot mask, and adopt Mask R-CNN to predict K masks, one for each of K keypoint types such as left shoulder, right elbow. <br /> <br /> [[File:HumanPose.png | center]]<br /> &lt;div align=&quot;center&quot;&gt;Figure 11: Keypoint Detection Results&lt;/div&gt;<br /> <br /> == Conclusion ==<br /> Mask RCNN is a deep neural network aimed to solve the instance segmentation problems in machine learning or computer vision. Mask R-CNN is a conceptually simple, flexible, and general framework for object instance segmentation. It can efficiently detect objects in an image while simultaneously generating a high-quality segmentation mask for each instance. It does object detection and instance segmentation, and can also be extended to human pose estimation.<br /> It extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small overhead to Faster R-CNN, running at 5 fps.<br /> <br /> == Critiques ==<br /> In Faster RCNN, the ROI boundary is quantized. However, mask RCNN avoids quantization and used the bilinear interpolation to compute exact values of features. By solving the misalignments due to quantization, the number and location of sampling points have no impact on the result.<br /> <br /> It may be better to compare the proposed model with other NN models or even non-NN methods like spectral clustering. Also, the applications can be further discussed like geometric mesh processing and motion analysis.<br /> <br /> The paper lacks the comparisons of different methods and Mask RNN on unlabeled data, as the paper only briefly mentioned that the authors found out that Mask R_CNN can benefit from extra data, even if the data is unlabelled.<br /> <br /> The Mask RCNN has many practical applications as well. A particular example, where Mask RCNNs are applied would be in autonomous vehicles. Namely, it would be able to help with isolating pedestrians, other vehicles, lights, etc.<br /> <br /> The Mask RCNN could be a candidate model to do short-term predictions on the physical behaviors of a person, which could be very useful at crime scenes.<br /> <br /> An interesting application of Mask RCNN would be on face recognition from CCTVs. Flurry pictures of crowded people could be obtained from CCTV, so that mask RCNN can be applied to distinguish each person.<br /> <br /> The main problem for CNN architectures like Mask RCNN is the running time. Due to slow running times, Single Shot Detector algorithms are preferred for applications like video or live stream detections, where a faster running time would mean a better response to changes in frames. It would be beneficial to have a graphical representation of the Mask RCNN running times against single shot detector algorithms such as YOLOv3.<br /> <br /> It is interesting to investigate a solution of embedding instance segmentation with semantic segmentation to improve time performance. Because in many situations, knowing the exact boundary of an object is not necessary.<br /> <br /> It will be better if we can have more comparisons with other models. It will also be nice if we can have more details about why Mask RCNN can perform better, and how about the efficiency of it?<br /> The authors mentioned that Mask R-CNN is a deep neural network architecture for Instance Segmentation. It's better to include more background information about this task. For example, challenges of this task (e.g. the model will need to take into account the overlapping of objects) and limitations of existing methods.<br /> <br /> It would be interesting to see how a postprocessing step with conditional random fields (CRF) might improve (or not?) segmentation. It would also have been interesting to see the performance of the method with lighter backbones since the backbones used to have a very large inference time which makes them unsuitable for many applications.<br /> <br /> An extension of the application of Mask RCNN in medical AI is to highlight areas of an MRI scan that correlate to certain behavioral/psychological patterns.<br /> <br /> == References ==<br />  Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick. Mask R-CNN. arXiv:1703.06870, 2017.<br /> <br />  Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, arXiv:1506.01497, 2015.<br /> <br />  Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, Piotr Dollár. Microsoft COCO: Common Objects in Context. arXiv:1405.0312, 2015</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Research_Papers_Classification_System&diff=49029 Research Papers Classification System 2020-12-03T21:19:53Z <p>Z42qin: /* Paper Classification Using K-means Clustering */</p> <hr /> <div>= Presented by =<br /> Jill Wang, Junyi (Jay) Yang, Yu Min (Chris) Wu, Chun Kit (Calvin) Li<br /> <br /> = Introduction =<br /> This paper introduces a paper classification system that utilizes the Term Frequency-Inverse Document Frequency (TF-IDF), Latent Dirichlet Allocation (LDA), and K-means clustering. The most important technology the system used to process big data is the Hadoop Distributed File System (HDFS). The system can handle quantitatively complex research paper classification problems efficiently and accurately.<br /> <br /> ===General Framework===<br /> <br /> The paper classification system classifies research papers based on the abstracts given that the core of most papers is presented in the abstracts. <br /> <br /> &lt;ol&gt;&lt;li&gt;Paper Crawling <br /> &lt;p&gt;Collects abstracts from research papers published during a given period&lt;/p&gt;&lt;/li&gt;<br /> &lt;li&gt;Preprocessing<br /> &lt;p&gt; &lt;ol style=&quot;list-style-type:lower-alpha&quot;&gt;&lt;li&gt;Removes stop words in the papers crawled, in which only nouns are extracted from the papers&lt;/li&gt;<br /> &lt;li&gt;generates a keyword dictionary, keeping only the top-N keywords with the highest frequencies&lt;/li&gt; &lt;/ol&gt;<br /> &lt;/p&gt;&lt;/li&gt; <br /> &lt;li&gt;Topic Modelling<br /> &lt;p&gt; Use the LDA to group the keywords into topics&lt;/p&gt;<br /> &lt;/li&gt;<br /> &lt;li&gt;Paper Length Calculation<br /> &lt;p&gt; Calculates the total number of occurrences of words to prevent an unbalanced TF values caused by the various length of abstracts using the map-reduce algorithm&lt;/p&gt;<br /> &lt;/li&gt;<br /> &lt;li&gt;Word Frequency Calculation<br /> &lt;p&gt; Calculates the Term Frequency (TF) values which represent the frequency of keywords in a research paper&lt;/p&gt;<br /> &lt;/li&gt;<br /> &lt;li&gt;Document Frequency Calculation<br /> &lt;p&gt; Calculates the Document Frequency (DF) values which represents the frequency of keywords in a collection of research papers. The higher the DF value, the lower the importance of a keyword.&lt;/p&gt;<br /> &lt;/li&gt;<br /> &lt;li&gt;TF-IDF calculation<br /> &lt;p&gt; Calculates the inverse of the DF which represents the importance of a keyword.&lt;/p&gt;<br /> &lt;/li&gt;<br /> &lt;li&gt;Paper Classification<br /> &lt;p&gt; Classify papers by topics using the K-means clustering algorithm.&lt;/p&gt;<br /> &lt;/li&gt;<br /> &lt;/ol&gt;<br /> <br /> ===Technologies===<br /> <br /> The HDFS with a Hadoop cluster composed of one master node, one sub-node, and four data nodes is what is used to process the massive paper data. Hadoop-2.6.5 version in Java is what is used to perform the TF-IDF calculation. Spark MLlib is what is used to perform the LDA. The Scikit-learn library is what is used to perform the K-means clustering.<br /> <br /> ===HDFS===<br /> <br /> Hadoop Distributed File System was used to process big data in this system. What Hadoop does is to break a big collection of data into different partitions and pass each partition to one individual processor. Each processor will only have information about the partition of data it has received.<br /> <br /> '''In this summary, we are going to focus on introducing the main algorithms of what this system uses, namely LDA, TF-IDF, and K-Means.'''<br /> <br /> =Data Preprocessing=<br /> ===Crawling of Abstract Data===<br /> <br /> Under the assumption that audiences tend to first read the abstract of a paper to gain an overall understanding of the material, it is reasonable to assume the abstract section includes “core words” that can be used to effectively classify a paper's subject.<br /> <br /> An abstract is crawled to have its stop words removed. Stop words are words that are usually ignored by search engines, such as “the”, “a”, and etc. Afterward, nouns are extracted, as a more condensed representation for efficient analysis.<br /> <br /> This is managed on HDFS. The TF-IDF value of each paper is calculated through map-reduce.<br /> <br /> ===Managing Paper Data===<br /> <br /> To construct an effective keyword dictionary using abstract data and keywords data in all of the crawled papers, the authors categorized keywords with similar meanings using a single representative keyword. The approach is called stemming, which is common in cleaning data. 1394 keyword categories are extracted, which is still too much to compute. Hence, only the top 30 keyword categories are used.<br /> <br /> &lt;div align=&quot;center&quot;&gt;[[File:table_1_kswf.JPG|700px]]&lt;/div&gt;<br /> <br /> =Topic Modeling Using LDA=<br /> <br /> Latent Dirichlet allocation (LDA) is a generative probabilistic model that views documents as random mixtures over latent topics. Each topic is a distribution over words, and the goal is to extract these topics from documents.<br /> <br /> LDA estimates the topic-word distribution &lt;math&gt;P\left(t | z\right)&lt;/math&gt; (probability of word &quot;z&quot; having topic &quot;t&quot;) and the document-topic distribution &lt;math&gt;P\left(z | d\right)&lt;/math&gt; (probability of finding word &quot;z&quot; within a given document &quot;d&quot;) using Dirichlet priors for the distributions with a fixed number of topics. For each document, obtain a feature vector:<br /> <br /> $F = \left( P\left(z_1 | d\right), P\left(z_2 | d\right), \cdots, P\left(z_k | d\right) \right)$<br /> <br /> In the paper, authors extract topics from preprocessed paper to generate three kinds of topic sets, each with 10, 20, and 30 topics respectively. The following is a table of the 10 topic sets of highest frequency keywords.<br /> <br /> &lt;div align=&quot;center&quot;&gt;[[File:table_2_tswtebls.JPG|700px]]&lt;/div&gt;<br /> <br /> <br /> ===LDA Intuition===<br /> <br /> LDA uses the Dirichlet priors of the Dirichlet distribution, which allows the algorithm to model a probability distribution ''over prior probability distributions of words and topics''. The following picture illustrates 2-simplex Dirichlet distributions with different alpha values, one for each corner of the triangles. <br /> <br /> &lt;div align=&quot;center&quot;&gt;[[File:dirichlet_dist.png|700px]]&lt;/div&gt;<br /> <br /> Simplex is a generalization of the notion of a triangle. In Dirichlet distribution, each parameter will be represented by a corner in simplex, so adding additional parameters implies increasing the dimensions of simplex. As illustrated, when alphas are smaller than 1 the distribution is dense at the corners. When the alphas are greater than 1 the distribution is dense at the centers.<br /> <br /> The following illustration shows an example LDA with 3 topics, 4 words and 7 documents.<br /> <br /> &lt;div align=&quot;center&quot;&gt;[[File:LDA_example.png|800px]]&lt;/div&gt;<br /> <br /> In the left diagram, there are three topics, hence it is a 2-simplex. In the right diagram there are four words, hence it is a 3-simplex. LDA essentially adjusts parameters in Dirichlet distributions and multinomial distributions (represented by the points), such that, in the left diagram, all the yellow points representing documents and, in the right diagram, all the points representing topics, are as close to a corner as possible. In other words, LDA finds topics for documents and also finds words for topics. At the end topic-word distribution &lt;math&gt;P\left(t | z\right)&lt;/math&gt; and the document-topic distribution &lt;math&gt;P\left(z | d\right)&lt;/math&gt; are produced.<br /> <br /> =Term Frequency Inverse Document Frequency (TF-IDF) Calculation=<br /> <br /> TF-IDF is widely used to evaluate the importance of a set of words in the fields of information retrieval and text mining. It is a combination of term frequency (TF) and inverse document frequency (IDF). The idea behind this combination is<br /> * It evaluates the importance of a word within a document, and<br /> * It evaluates the importance of the word among the collection of all documents<br /> <br /> The TF-IDF formula has the following form:<br /> <br /> $TF-IDF_{i,j} = TF_{i,j} \times IDF_{i}$<br /> <br /> where i stands for the &lt;math&gt;i^{th}&lt;/math&gt; word and j stands for the &lt;math&gt;j^{th}&lt;/math&gt; document.<br /> <br /> ===Term Frequency (TF)===<br /> <br /> TF evaluates the percentage of a given word in a document. Thus, TF value indicates the importance of a word. The TF has a positive relation with the importance.<br /> <br /> In this paper, we only calculate TF for words in the keyword dictionary obtained. For a given keyword i, &lt;math&gt;TF_{i,j}&lt;/math&gt; is the number of times word i appears in document j divided by the total number of words in document j.<br /> <br /> The formula for TF has the following form:<br /> <br /> $TF_{i,j} = \frac{n_{i,j} }{\sum_k n_{k,j} }$<br /> <br /> where i stands for the &lt;math&gt;i^{th}&lt;/math&gt; word, j stands for the &lt;math&gt;j^{th}&lt;/math&gt; document, &lt;math&gt;n_{i,j}&lt;/math&gt; stands for the number of times words &lt;math&gt;t_i&lt;/math&gt; appear in document &lt;math&gt;d_j&lt;/math&gt; and &lt;math&gt;\sum_k n_{k,j} &lt;/math&gt; stands for total number of occurence of words in document &lt;math&gt;d_j&lt;/math&gt;.<br /> <br /> Note that the denominator is the total number of words remaining in document j after crawling.<br /> <br /> ===Document Frequency (DF)===<br /> <br /> DF evaluates the percentage of documents that contain a given word over the entire collection of documents. Thus, the higher DF value is, the less important the word is.<br /> <br /> &lt;math&gt;DF_{i}&lt;/math&gt; is the number of documents in the collection with word i divided by the total number of documents in the collection. The formula for DF has the following form:<br /> <br /> $DF_{i} = \frac{|d_k \in D: n_{i,k} &gt; 0|}{|D|}$<br /> <br /> where &lt;math&gt;n_{i,k}&lt;/math&gt; is the number of times word i appears in document k, |D| is the total number of documents in the collection.<br /> <br /> Since DF and the importance of the word have an inverse relation, we use inverse document frequency (IDF) instead of DF.<br /> <br /> ===Inverse Document Frequency (IDF)===<br /> <br /> In this paper, IDF is calculated in a log scale. Since we will receive a large number of documents, i.e, we will have a large |D|<br /> <br /> The formula for IDF has the following form:<br /> <br /> $IDF_{i} = log\left(\frac{|D|}{|\{d_k \in D: n_{i,k} &gt; 0\}|}\right)$<br /> <br /> As mentioned before, we will use HDFS. The actual formula applied is:<br /> <br /> $IDF_{i} = log\left(\frac{|D|+1}{|\{d_k \in D: n_{i,k} &gt; 0\}|+1}\right)$<br /> <br /> The inverse document frequency gives a measure of how rare a certain term is in a given document corpus.<br /> <br /> =Paper Classification Using K-means Clustering=<br /> <br /> The K-means clustering is an unsupervised classification algorithm that groups similar data into the same class. It is an efficient and simple method that can work with different types of data attributes and is able to handle noise and outliers.<br /> &lt;br&gt;<br /> <br /> Given a set of &lt;math&gt;d&lt;/math&gt; by &lt;math&gt;n&lt;/math&gt; dataset &lt;math&gt;\mathbf{X} = \left[ \mathbf{x}_1 \cdots \mathbf{x}_n \right]&lt;/math&gt;, the algorithm will assign each &lt;math&gt;\mathbf{x}_j&lt;/math&gt; into &lt;math&gt;k&lt;/math&gt; different clusters based on the characteristics of &lt;math&gt;\mathbf{x}_j&lt;/math&gt; itself.<br /> &lt;br&gt;<br /> <br /> Moreover, when assigning data into a cluster, the algorithm will also try to minimise the distances between the data and the centre of the cluster which the data belongs to. That is, k-means clustering will minimize the sum of square error:<br /> <br /> \begin{align*}<br /> min \sum_{i=1}^{k} \sum_{j \in C_i} ||x_j - \mu_i||^2<br /> \end{align*}<br /> <br /> where<br /> &lt;ul&gt;<br /> &lt;li&gt;&lt;math&gt;k&lt;/math&gt;: the number of clusters&lt;/li&gt;<br /> &lt;li&gt;&lt;math&gt;C_i&lt;/math&gt;: the &lt;math&gt;i^th&lt;/math&gt; cluster&lt;/li&gt;<br /> &lt;li&gt;&lt;math&gt;x_j&lt;/math&gt;: the &lt;math&gt;j^th&lt;/math&gt; data in the &lt;math&gt;C_i&lt;/math&gt;&lt;/li&gt;<br /> &lt;li&gt;&lt;math&gt;mu_i&lt;/math&gt;: the centroid of &lt;math&gt;C_i&lt;/math&gt;&lt;/li&gt;<br /> &lt;li&gt;&lt;math&gt;||x_j - \mu_i||^2&lt;/math&gt;: the Euclidean distance between &lt;math&gt;x_j&lt;/math&gt; and &lt;math&gt;\mu_i&lt;/math&gt;&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;br&gt;<br /> <br /> K-means Clustering algorithm - an unsupervised algorithm- is chosen because of its advantages to deal with different types of attributes, to run with minimal requirement of domain knowledge, to deal with noise and outliers, to realize clusters with similarities. <br /> <br /> <br /> Since the goal for this paper is to classify research papers and group papers with similar topics based on keywords, the paper uses the K-means clustering algorithm. The algorithm first computes the cluster centroid for each group of papers with a specific topic. Then, it will assign a paper into a cluster based on the Euclidean distance between the cluster centroid and the paper’s TF-IDF value.<br /> &lt;br&gt;<br /> <br /> However, different values of &lt;math&gt;k&lt;/math&gt; (the number of clusters) will return different clustering results. Therefore, it is important to define the number of clusters before clustering. For example, in this paper, the authors choose to use the Elbow scheme to determine the value of &lt;math&gt;k&lt;/math&gt;. The Elbow scheme is a somewhat subjective way of choosing an optimal &lt;math&gt;k&lt;/math&gt; that involves plotting the average of the squared distances from the cluster centers of the respective clusters (distortion) as a function of &lt;math&gt;k&lt;/math&gt; and choosing a &lt;math&gt;k&lt;/math&gt; at which point the decrease in distortion is outweighed by the increase in complexity. Also, to measure the performance of clustering, the authors decide to use the Silhouette scheme. The results of clustering are validated if the Silhouette scheme returns a value greater than &lt;math&gt;0.5&lt;/math&gt;.<br /> <br /> =System Testing Results=<br /> <br /> In this paper, the dataset has 3264 research papers from the Future Generation Computer System (FGCS) journal between 1984 and 2017. For constructing keyword dictionaries for each paper, the authors have introduced three methods as shown below:<br /> <br /> &lt;div align=&quot;center&quot;&gt;[[File:table_3_tmtckd.JPG|700px]]&lt;/div&gt;<br /> <br /> <br /> Then, the authors use the Elbow scheme to define the number of clusters for each method with different numbers of keywords before running the K-means clustering algorithm. The results are shown below:<br /> <br /> &lt;div align=&quot;center&quot;&gt;[[File:table_4_nocobes.JPG|700px]]&lt;/div&gt;<br /> <br /> According to Table 4, there is a positive correlation between the number of keywords and the number of clusters. In addition, method 3 combines the advantages for both method 1 and method 2; thus, method 3 requires the least clusters in total. On the other hand, the wrong keywords might be presented in papers; hence, it might not be possible to group papers with similar subjects correctly by using method 1 and so method 1 needs the most number of clusters in total.<br /> <br /> <br /> Next, the Silhouette scheme had been used for measuring the performance for clustering. The average of the Silhouette values for each method with different numbers of keywords are shown below:<br /> <br /> &lt;div align=&quot;center&quot;&gt;[[File:table_5_asv.JPG|700px]]&lt;/div&gt;<br /> <br /> Since the clustering is validated if the Silhouette’s value is greater than 0.5, for methods with 10 and 30 keywords, the K-means clustering algorithm produces good results.<br /> <br /> <br /> To evaluate the accuracy of the classification system in this paper, the authors use the F-Score. The authors execute 5 times of experiment and use 500 randomly selected research papers for each trial. The following histogram shows the average value of F-Score for the three methods and different numbers of keywords:<br /> <br /> &lt;div align=&quot;center&quot;&gt;[[File:fig_16_fsvotm.JPG|700px]]&lt;/div&gt;<br /> <br /> Note that “TFIDF” means method 1, “LDA” means method 2, and “TFIDF-LDA” means method 3. The number 10, 20, and 30 after each method is the number of keywords the method has used.<br /> According to the histogram above, method 3 has the highest F-Score values than the other two methods with different numbers of keywords. Therefore, the classification system is most accurate when using method 3 as it combines the advantages for both method 1 and method 2.<br /> <br /> =Conclusion=<br /> <br /> This paper introduces a classification system that classifies research papers into different topics by using TF-IDF and LDA scheme with K-means clustering algorithm. The experimental results showed that the proposed system can classify the papers with similar subjects according to the keywords extracted from the abstracts of papers. This system allows users to search the papers they want quickly and with the most productivity.<br /> <br /> Furthermore, this classification system might be also used in different types of texts (e.g. documents, tweets, etc.) instead of only classifying research papers.<br /> <br /> =Critique=<br /> <br /> In this paper, DF values are calculated within each partition. This results that for each partition, DF value for a given word will vary and may have an inconsistent result for different partition methods. As mentioned above, there might be a divide by zero problem since some partitions do not have documents containing a given word, but this can be solved by introducing a dummy document as the authors did. Another method that might be better at solving inconsistent results and the divide by zero problems is to have all partitions to communicate with their DF value. Then pass the merged DF value to all partitions to do the final IDF and TF-IDF value. Having all partitions to communicate with the DF value will guarantee a consistent DF value across all partitions and helps avoid a divide by zero problem as words in the keyword dictionary must appear in some documents in the whole collection.<br /> <br /> This paper treated the words in the different parts of a document equivalently, it might perform better if it gives different weights to the same word in different parts. For example, if a word appears in the title of the document, it usually shows it's a main topic of this document so we can put more weight on it to categorize.<br /> <br /> When discussing the potential processing advantages of this classification system for other types of text samples, has the effect of processing mixed samples (text and image or text and video) taken into consideration? IF not, in terms of text classification only, does it have an overwhelming advantage over traditional classification models?<br /> <br /> The preprocessing should also include &lt;math&gt;n&lt;/math&gt;-gram tokenization for topic modelling because some topics are inherently two words, such as machine learning where if it is seen separately, it implies different topics.<br /> <br /> This system is very compute-intensive due to the large volumes of dictionaries that can be generated by processing large volumes of data. It would be nice to see how much data HDFS had to process and similarly how much time was saved by using Hadoop for data processing as opposed to centralized approach.<br /> <br /> This system can be improved further in terms of computation times by utilizing other big data framework MapReduce, that can also use HDFS, by parallelizing their computation across multiple nodes for K-means clustering as discussed in (Jin, et al) .<br /> <br /> It's not exactly clear what method 3 (TFIDF-LDA) is doing, how is it performing TF-IDF on the topics? Also it seems like the preprocessing step only keeps 10/20/30 top words? This seems like an extremely low number especially in comparison with the LDA which has 10/20/30 topics - what is the reason for so strongly limiting the number of words? It would also be interesting to see if both key words and topics are necessary - an ablation study showing the significance of both would be interesting.<br /> <br /> It is better if the paper has an example with some topics on some research papers. Also it is better if we can visualize the distance between each research paper and the topic names<br /> <br /> =References=<br /> <br /> Blei DM, el. (2003). Latent Dirichlet allocation. J Mach Learn Res 3:993–1022<br /> <br /> Gil, JM, Kim, SW. (2019). Research paper classification systems based on TF-IDF and LDA schemes. ''Human-centric Computing and Information Sciences'', 9, 30. https://doi.org/10.1186/s13673-019-0192-7<br /> <br /> Liu, S. (2019, January 11). Dirichlet distribution Motivating LDA. Retrieved November 2020, from https://towardsdatascience.com/dirichlet-distribution-a82ab942a879<br /> <br /> Serrano, L. (Director). (2020, March 18). Latent Dirichlet Allocation (Part 1 of 2) [Video file]. Retrieved 2020, from https://www.youtube.com/watch?v=T05t-SqKArY<br /> <br /> Jin, Cui, Yu. (2016). A New Parallelization Method for K-means. https://arxiv.org/ftp/arxiv/papers/1608/1608.06347.pdf</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Speech2Face:_Learning_the_Face_Behind_a_Voice&diff=49026 Speech2Face: Learning the Face Behind a Voice 2020-12-03T21:01:42Z <p>Z42qin: /* Model Architecture */</p> <hr /> <div>== Presented by == <br /> Ian Cheung, Russell Parco, Scholar Sun, Jacky Yao, Daniel Zhang<br /> <br /> == Introduction ==<br /> This paper presents a deep neural network architecture called Speech2Face. This architecture utilizes millions of Internet/Youtube videos of people speaking to learn the correlation between a voice and the respective face. The model learns the correlations, allowing it to produce facial reconstruction images that capture specific physical attributes, such as a person's age, gender, or ethnicity, through a self-supervised procedure. Namely, the model utilizes the simultaneous occurrence of faces and speech in videos and does not need to model the attributes explicitly. The model is evaluated and numerically quantifies how closely the reconstruction, done by the Speech2Face model, resembles the true face images of the respective speakers.<br /> <br /> == Previous Work ==<br /> With visual and audio signals being so dominant and accessible in our daily life, there has been huge interest in how visual and audio perceptions interact with each other. Arandjelovic and Zisserman  leveraged the existing database of mp4 files to learn a generic audio representation to classify whether a video frame and an audio clip correspond to each other. These learned audio-visual representations have been used in a variety of setting, including cross-modal retrieval, sound source localization and sound source separation. This also paved the path for specifically studying the association between faces and voices of agents in the field of computer vision. In particular, cross-modal signals extracted from faces and voices have been proposed as a binary or multi-task classification task and there have been some promising results. Studies have been able to identify active speakers of a video, separate speech from multiple concurrent sources, predict lip motion from speech, and even learn the emotion of the agents based on their voices. Aytar et al.  proposed a student-teacher training procedure in which a well established visual recognition model was used to transfer the knowledge obtained in the visual modality to the sound modality, using unlabeled videos.<br /> <br /> Recently, various methods have been suggested to use various audio signals to reconstruct visual information, where the reconstructed subject is subjected to a priori. Notably, Duarte et al.  were able to synthesize the exact face images and expression of an agent from speech using a GAN model. A generative adversarial network (GAN) model is one that uses a generator to produce seemingly possible data for training and a discriminator that identifies if the training data is fabricated by the generator or if it is real . This paper instead hopes to recover the dominant and generic facial structure from a speech.<br /> <br /> == Motivation ==<br /> It seems to be a common trait among humans to imagine what some people look like when we hear their voices before we have seen what they look lke. There is a strong connection between speech and appearance, which is a direct result of the factors that affect speech, including age, gender, and facial bone structure. In addition, other voice-appearance correlations stem from the way in which we talk: language, accent, speed, pronunciations, etc. These properties of speech are often common among many different nationalities and cultures, which can, in turn, translate to common physical features among different voices. Namely, from an input audio segment of a person speaking, the method would reconstruct an image of the person’s face in a canonical form (frontal-facing, neutral expression). The goal was to study to what extent people can infer how someone else looks from the way they talk. Rather than predicting a recognizable image of the exact face, the authors were more interested in capturing the dominant facial features.<br /> <br /> == Model Architecture == <br /> <br /> '''Speech2Face model and training pipeline'''<br /> <br /> [[File:ModelFramework.jpg|center]]<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt; Figure 1. '''Speech2Face model and training pipeline''' &lt;/div&gt;<br /> <br /> <br /> <br /> The Speech2Face Model used to achieve the desired result consists of two parts - a voice encoder which takes in a spectrogram of speech as input and outputs low dimensional face features, and a face decoder which takes in face features as input and outputs a normalized image of a face (neutral expression, looking forward). Figure 1 gives a visual representation of the pipeline of the entire model, from video input to a recognizable face. The combination of the voice encoder and face decoder results are combined to form an image. The variability in facial expressions, head positions and lighting conditions of the face images creates a challenge to both the design and training of the Speech2Face model. To avoid this problem the model is trained to first regress to a low dimensional intermediate representation of the face. <br /> <br /> '''Face Decoder''' <br /> The face decoder itself was taken from previous work The VGG-Face model by Cole et al  (a face recognition model that is pretrained on a largescale face database  is used to extract a 4069-D face feature from the penultimate layer of the network.) and will not be explored in great detail here, but in essence the facenet model is combined with a single multilayer perceptron layer, the result of which is passed through a convolutional neural network to determine the texture of the image, and a multilayer perception to determine the landmark locations. The face decoder kept the VGG-Face model's dimension and weights. The weights were also trained separately and remained fixed during the voice encoder training. <br /> <br /> '''Voice Encoder Architecture''' <br /> <br /> [[File:VoiceEncoderArch.JPG|center]]<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt; Table 1: '''Voice encoder architecture''' &lt;/div&gt;<br /> <br /> <br /> <br /> The voice encoder itself is a convolutional neural network, which transforms the input spectrogram into pseudo face features. The exact architecture is given in Table 1. The model alternates between convolution, ReLU, batch normalization layers, and layers of max-pooling. In each max-pooling layer, pooling is only done along the temporal dimension of the data. This is to ensure that the frequency, an important factor in determining vocal characteristics such as tone, is preserved. In the final pooling layer, an average pooling is applied along the temporal dimension. This allows the model to aggregate information over time and allows the model to be used for input speeches of varying lengths. Two fully connected layers at the end are used to return a 4096-dimensional facial feature output.<br /> <br /> '''Training'''<br /> <br /> The AVSSpeech dataset, a large-scale audio-visual dataset is used for the training. AVSSpeech dataset is comprised of millions of video segments from Youtube with over 100,000 different people. The training data is composed of educational videos and does not provide an accurate representation of the global population, which will clearly affect the model. Also note that facial features that are irrelevant to speech, like hair color, may be predicted by the model. From each video, a 224x224 pixels image of the face was passed through the face decoder to compute a facial feature vector. Combined with a spectrogram of the audio, a training and test set of 1.7 and 0.15 million entries respectively were constructed.<br /> <br /> The voice encoder is trained in a self-supervised manner. A frame that contains the face is extracted from each video and then inputted to the VGG-Face model to extract the feature vector &lt;math&gt;v_f&lt;/math&gt;, the 4096-dimensional facial feature vector given by the face decoder on a single frame from the input video. This provides the supervision signal for the voice-encoder. The feature &lt;math&gt;v_s&lt;/math&gt;, the 4096 dimensional facial feature vector from the voice encoder, is trained to predict &lt;math&gt;v_f&lt;/math&gt;.<br /> <br /> In order to train this model, a proper loss function must be defined. The L1 norm of the difference between &lt;math&gt;v_s&lt;/math&gt; and &lt;math&gt;v_f&lt;/math&gt;, given by &lt;math&gt;||v_f - v_s||_1&lt;/math&gt;, may seem like a suitable loss function, but in actuality results in unstable results and long training times. Figure 2, below, shows the difference in predicted facial features given by &lt;math&gt;||v_f - v_s||_1&lt;/math&gt; and the following loss. Based on the work of Castrejon et al. , a loss function is used which penalizes the differences in the last layer of the VGG-Face model &lt;math&gt;f_{VGG}&lt;/math&gt;: &lt;math&gt; \mathbb{R}^{4096} \to \mathbb{R}^{2622}&lt;/math&gt; and the first layer of face decoder &lt;math&gt;f_{dec}&lt;/math&gt; : &lt;math&gt; \mathbb{R}^{4096} \to \mathbb{R}^{1000}&lt;/math&gt;. The final loss function is given by: $$L_{total} = ||f_{dec}(v_f) - f_{dec}(v_s)|| + \lambda_1||\frac{v_f}{||v_f||} - \frac{v_s}{||v_s||}||^2_2 + \lambda_2 L_{distill}(f_{VGG}(v_f), f_{VGG}(v_s))$$<br /> This loss penalizes on both the normalized Euclidean distance between the 2 facial feature vectors and the knowledge distillation loss, which is given by: $$L_{distill}(a,b) = -\sum_ip_{(i)}(a)\text{log}p_{(i)}(b)$$ $$p_{(i)}(a) = \frac{\text{exp}(a_i/T)}{\sum_j \text{exp}(a_j/T)}$$ Knowledge distillation is used as an alternative to Cross-Entropy. By recommendation of Cole et al , &lt;math&gt; T = 2 &lt;/math&gt; was used to ensure a smooth activation. &lt;math&gt;\lambda_1 = 0.025&lt;/math&gt; and &lt;math&gt;\lambda_2 = 200&lt;/math&gt; were chosen so that magnitude of the gradient of each term with respect to &lt;math&gt;v_s&lt;/math&gt; are of similar scale at the &lt;math&gt;1000^{th}&lt;/math&gt; iteration.<br /> <br /> &lt;center&gt;<br /> [[File:L1vsTotalLoss.png | 700px]]<br /> &lt;/center&gt;<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt; Figure 2: '''Qualitative results on the AVSpeech test set''' &lt;/div&gt;<br /> <br /> == Results ==<br /> <br /> '''Confusion Matrix and Dataset statistics'''<br /> <br /> &lt;center&gt;<br /> [[File:Confusionmatrix.png| 600px]]<br /> &lt;/center&gt;<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt; Figure 3. '''Facial attribute evaluation''' &lt;/div&gt;<br /> <br /> <br /> <br /> In order to determine the similarity between the generated images and the ground truth, a commercial service known as Face++ which classifies faces for distinct attributes (such as gender, ethnicity, etc) was used. Figure 3 gives a confusion matrix based on gender, ethnicity, and age. By examining these matrices, it is seen that the Speech2Face model performs very well on gender, only misclassifying 6% of the time. Similarly, the model performs fairly well on ethnicities, especially with white or Asian faces. Although the model performs worse on black and Indian faces, that can be attributed to the vastly unbalanced data, where 50% of the data represented a white face, and 80% represented a white or Asian face. <br /> <br /> '''Feature Similarity'''<br /> <br /> &lt;center&gt;<br /> [[File:FeatSim.JPG]]<br /> &lt;/center&gt;<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt; Table 2. '''Feature similarity''' &lt;/div&gt;<br /> <br /> <br /> <br /> Another examination of the result is the similarity of features predicted by the Speech2Face model. The cosine, L1, and L2 distance between the facial feature vector produced by the model and the true facial feature vector from the face decoder were computed, and presented, above, in Table 2. A comparison of facial similarity was also done based on the length of audio input. From the table, it is evident that the 6-second audio produced a lower cosine, L1, and L2 distance, resulting in a facial feature vector that is closer to the ground truth. <br /> <br /> '''S2F -&gt; Face retrieval performance'''<br /> <br /> &lt;center&gt;<br /> [[File: Retrieval.JPG]]<br /> &lt;/center&gt;<br /> <br /> &lt;div style=&quot;text-align:center;&quot;&gt; Table 3. '''S2F -&gt; Face retrieval performance''' &lt;/div&gt;<br /> <br /> <br /> <br /> The performance of the model was also examined on how well it could produce the original image. The R@K metric, also known as retrieval performance by recall at K, measures the probability that the K closest images to the model output includes the correct image of the speaker's face. A higher R@K score indicates better performance. From Table 3, above, we see that both the 3-second and 6-second audio showed significant improvement over random chance, with the 6-second audio performing slightly better.<br /> <br /> '''Additional Observations''' <br /> <br /> Ablation studies were carried out to test the effect of audio duration and batch normalization. It was found that the duration of input audio during the training stage had little effect on convergence speed (comparing 3 and 6-second speech segments), while in the test stage longer input speech yields improvement in reconstruction quality. With respect to batch normalization (BN), it was found that without BN reconstructed faces would converge to an average face, while the inclusion of BN led to results which contained much richer facial features.<br /> <br /> == Conclusion ==<br /> The report presented a novel study of face reconstruction from audio recordings of a person speaking. The model was demonstrated to be able to predict plausible face reconstructions with similar facial features to real images of the person speaking. The problem was addressed by learning to align the feature space of speech to that of a pretrained face decoder. The model was trained on millions of videos of people speaking from YouTube. The model was then evaluated by comparing the reconstructed faces with a commercial facial detection service. The authors believe that facial reconstruction allows a more comprehensive view of voice-face correlation compared to predicting individual features, which may lead to new research opportunities and applications.<br /> <br /> == Discussion and Critiques ==<br /> <br /> There is evidence that the results of the model may be heavily influenced by external factors:<br /> <br /> 1. Their method of sampling random YouTube videos resulted in an unbalanced sample in terms of ethnicity. Over half of the samples were white. We also saw a large bias in the model's prediction of ethnicity towards white. The bias in the results shows that the model may be overfitting the training data and puts into question what the performance of the model would be when trained and tested on a balanced dataset. Figure (11) highlights this shortcoming: The same man heard speaking in either English or Chinese was predicted to have a &quot;white&quot; appearance or an &quot;asian&quot; appearance respectively.<br /> <br /> 2. The model was shown to infer different face features based on language. This puts into question how heavily the model depends on the spoken language. The paper mentioned the quality of face reconstruction may be affected by uncommon languages, where English is the most popular language on Youtube(training set). Testing a more controlled sample where all speech recording was of the same language may help address this concern to determine the model's reliance on spoken language.<br /> <br /> 3. The evaluation of the result is also highly dependent on the Face++ classifiers. Since they compare the age, gender, and ethnicity by running the Face++ classifiers on the original images and the reconstructions to evaluate their model, the model that they create can only be as good as the one they are using to evaluate it. Therefore, any limitations of the Face++ classifier may become a limitation of Speech2Face and may result in a compounding effect on the miss-classification rate.<br /> <br /> 4. Figure 4.b shows the AVSpeech dataset statistics. However, it doesn't show the statistics about speakers' ethnicity and the language of the video. If we train the model with a more comprehensive dataset that includes enough Asian/Indian English speakers and native language speakers will this increase the accuracy?<br /> <br /> 5. One concern about the source of the training data, i.e. the Youtube videos, is that resolution varies a lot since the videos are randomly selected. That may be the reason why the proposed model performs badly on some certain features. For example, it is hard to tell the age when the resolution is bad because the wrinkles on the face are neglected.<br /> <br /> 6. The topic of this project is very interesting, but I highly doubt this model will be practical in real-world problems. Because there are many factors to affect a person's sound in a real-world environment. Sounds such as phone clock, TV, car horn and so on. These sounds will decrease the accuracy of the predicted result of the model.<br /> <br /> 7. A lot of information can be obtained from someone's voice, this can potentially be useful for detective work and crime scene investigation. In our world of increasing surveillance, public voice recording is quite common and we can reconstruct images of potential suspects based on their voice. In order for this to be achieved, the model has to be thoroughly trained and tested to avoid false positives as it could have a highly destructive outcome for a falsely convicted suspect.<br /> <br /> 8. This is a very interesting topic, and this summary has a good structure for readers. Since this model uses Youtube to train model, but I think one problem is that most of the YouTubers are adult, and many additional reasons make this dataset highly unbalanced. What is more, some people may have a baby voice, this also could affect the performance of the model. But overall, this is a meaningful topic, it might help police to locate the suspects. So it might be interesting to apply this to the police.<br /> <br /> 9. In addition, it seems very unlikely that any results coming from this model would ever be held in regard even remotely close to being admissible in court to identify a person of interest until the results are improved and the model can be shown to work in real-world applications. Otherwise, there seems to be very little use for such technology and it could have negative impacts on people if they were to be depicted in an unflattering way by the model based on their voice.<br /> <br /> 10. Using voice as a factor of constructing the face is a good idea, but it seems like the data they have will have lots of noise and bias. The voice of a video might not come from the person in the video. There are so many YouTubers adjusting their voices before uploading their video and it's really hard to know whether they adjust their voice. Also, most YouTubers are adults so the model cannot have enough training samples about teenagers and kids.<br /> <br /> 11. It would be interesting to see how the performance changes with different face encoding sizes (instead of just 4096-D) and also difference face models (encoder/decoders) to see if better performance can be achieved. Also given that the dataset used was unbalanced, was the dataset used to train the face model the same dataset? or was a different dataset used (the model was pretrained). This could affect the performance of the model as well.<br /> <br /> 12. The audio input is transformed into a spectrogram before being used for training. They use STFT with a Hann window of 25 mm, a hop length of 10 ms, and 512 FFT frequency bands. They cite this method from a paper that focuses on speech separation, not speech classification. So, it would be interesting to see if there is a better way to do STFT, possibly with different hyperparameters (eg. different windowing, different number of bands), or if another type of transform (eg. wavelet transform) would have better results.<br /> <br /> 13. A easy way to get somewhat balanced data is to duplicate the data that are fewer.<br /> <br /> 14. This problem is interesting but is hard to generalize. This algorithm didn't account for other genders and mixed-race. In addition, the face recognition software Face++ introduces bias which can carry forward to Speech2Face algorithm. Face recognition algorithms are known to have higher error rates classifying darker-skinned individuals. Thus, it'll be tough to apply it to real-life scenarios like identifying suspects.<br /> <br /> == References ==<br />  R. Arandjelovic and A. Zisserman. Look, listen and learn. In<br /> IEEE International Conference on Computer Vision (ICCV),<br /> 2017.<br /> <br />  A. Duarte, F. Roldan, M. Tubau, J. Escur, S. Pascual, A. Salvador, E. Mohedano, K. McGuinness, J. Torres, and X. Giroi-Nieto. Wav2Pix: speech-conditioned face generation using generative adversarial networks. In IEEE International<br /> Conference on Acoustics, Speech and Signal Processing<br /> (ICASSP), 2019.<br /> <br />  F. Cole, D. Belanger, D. Krishnan, A. Sarna, I. Mosseri, and W. T. Freeman. Synthesizing normalized faces from facial identity features. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.<br /> <br />  L. Castrejon, Y. Aytar, C. Vondrick, H. Pirsiavash, and A. Torralba. Learning aligned cross-modal representations from weakly aligned data. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.<br /> <br />  O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. In British Machine Vision Conference (BMVC), 2015.<br /> <br />  “Overview of GAN Structure | Generative Adversarial Networks,” ''Google Developers'', 24-May-2019. [Online]. Available: https://developers.google.com/machine-learning/gan/gan_structure. [Accessed: 02-Dec-2020].</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Surround_Vehicle_Motion_Prediction&diff=48936 Surround Vehicle Motion Prediction 2020-12-02T22:04:39Z <p>Z42qin: /* Data Collection Devices and Outputs */</p> <hr /> <div>DROCC: '''Surround Vehicle Motion Prediction Using LSTM-RNN for Motion Planning of Autonomous Vehicles at Multi-Lane Turn Intersections'''<br /> == Presented by == <br /> Mushi Wang, Siyuan Qiu, Yan Yu<br /> <br /> == Introduction ==<br /> <br /> This paper presents a surrounding vehicle motion prediction algorithm for multi-lane turn intersections using a Long Short-Term Memory (LSTM)-based Recurrent Neural Network (RNN). More specifically, it focused on the improvement of in-lane target recognition and achieving human-like acceleration decisions at multi-lane turn intersections by introducing the learning-based target motion predictor and prediction-based motion predictor. A data-driven approach for predicting the trajectory and velocity of surrounding vehicles on urban roads at multi-lane turn intersections was described. LSTM architecture, a specific kind of RNN capable of learning long-term dependencies, is designed to manage complex vehicle motions in multi-lane turn intersections. The results show that the forecaster improves the recognition time of the leading vehicle and contributes to the improvement of prediction ability.<br /> <br /> == Previous Work ==<br /> The autonomous vehicle trajectory approaches previously used motion models like Constant Velocity and Constant Acceleration. These models are linear and are only able to handle straight motions. There are curvilinear models such as Constant Turn Rate and Velocity and Constant Turn Rate and Acceleration which handle rotations and more complex motions. Together with these models, Kalman Filter is used to predicting the vehicle trajectory. Kalman filtering is a common technique used in sensor fusion for state estimation that allows the vehicle's state to be predicted while taking into account the uncertainty associated with inputs and measurements. However, the performance of the Kalman Filter in predicting multi-step problem is not that good. Recurrent Neural Network performs significantly better than it. <br /> <br /> There are 3 main challenges to achieving fully autonomous driving on urban roads, which are scene awareness, inferring other drivers’ intentions, and predicting their future motions. Researchers are developing prediction algorithms that can simulate a driver’s intuition to improve safety when autonomous vehicles and human drivers drive together. To predict driver behavior on an urban road, there are 3 categories for the motion prediction model: (1) physics-based; (2) maneuver-based; and (3) interaction-aware. Physics-based models are simple and direct, which only consider the states of prediction vehicles kinematically. The advantage is that it has minimal computational burden among the three types. However, it is impossible to consider the interactions between vehicles. Maneuver-based models consider the driver’s intention and classified them. By predicting the driver maneuver, the future trajectory can be predicted. Identifying similar behaviors in driving is able to infer different drivers' intentions which are stated to improve the prediction accuracy. However, it still an assistant to improve physics-based models. <br /> <br /> Recurrent Neural Network (RNN) is a type of approach proposed to infer driver intention in this paper. Interaction-aware models can reflect interactions between surrounding vehicles, and predict future motions of detected vehicles simultaneously as a scene. While the prediction algorithm is more complex in computation which is often used in offline simulations. As Schulz et al. indicate, interaction models are very difficult to create as &quot;predicting complete trajectories at once is challenging, as one needs to account for multiple hypotheses and long-term interactions between multiple agents&quot; .<br /> <br /> == Motivation == <br /> Research results indicate that little research has been dedicated on predicting the trajectory of intersections. Moreover, public data sets for analyzing driver behaviour at intersections are not enough, and these data sets are not easy to collect. A model is needed to predict the various movements of the target around a multi-lane turning intersection. It is very necessary to design a motion predictor that can be used for real-time traffic.<br /> <br /> == Framework == <br /> The LSTM-RNN-based motion predictor comprises three parts: (1) a data encoder; (2) an LSTM-based RNN; and (3) a data decoder depicts the architecture of the surrounding target trajectory predictor. The proposed architecture uses a perception algorithm to estimate the state of surrounding vehicles, which relies on six scanners. The output predicts the state of the surrounding vehicles and is used to determine the expected longitudinal acceleration in the actual traffic at the intersection. The following image gives a visual representation of the model.<br /> <br /> &lt;center&gt;[[Image:Figure1_Yan.png|800px|]]&lt;/center&gt;<br /> <br /> == LSTM-RNN based motion predictor == <br /> <br /> === Sensor Outputs ===<br /> <br /> The input of the target perceptions are from the output of the sensors. The data collection in this article uses 6 different sensors with feature fusion to detect traffic in range up to 100m: 1) LiDAR system outputs: Relative position, heading, velocity and box size in local coordinates ; 2) Around0View Monitoring (AVM) and 3)GPS outputs: acquire lanes, road marker, global position; 4) Gateaway engine outputs: precise global position in urban road environment; 5) Micro-Autobox II and 6) a MDPS are used to control and actuate the subject. All data are stored in an industrial PC.<br /> <br /> === Data ===<br /> Multi-lane turn intersections are the target roads in this paper. The real dataset is captured on urban roads in Seoul. The training model is generated from 484 tracks collected when driving through intersections in real traffic. The previous and subsequent states of a vehicle at a particular time can be extracted. After post-processing, the collected data, a total of 16,660 data samples were generated, including 11,662 training data samples, and 4,998 evaluation data samples.<br /> <br /> === Motion predictor ===<br /> This article proposes a data-driven method to predict the future movement of surrounding vehicles based on their previous movement. The motion predictor based on the LSTM-RNN architecture in this work only uses information collected from sensors on autonomous vehicles, as shown in the figure below. The contribution of the network architecture of this study is that the future state of the target vehicle is used as the input feature for predicting the field of view. <br /> <br /> <br /> &lt;center&gt;[[Image:Figure7b_Yan.png|500px|]]&lt;/center&gt;<br /> <br /> <br /> ==== Network architecture ==== <br /> A RNN is an artificial neural network, suitable for use with sequential data. It can also be used for time-series data, where the pattern of the data depends on the time flow. Also, it can contain feedback loops that allow activations to flow alternately in the loop.<br /> An LSTM avoids the problem of vanishing gradients by making errors flow backward without a limit on the number of virtual layers. This property prevents errors from increasing or declining over time, which can make the network train improperly. The figure below shows the various layers of the LSTM-RNN and the number of units in each layer. This structure is determined by comparing the accuracy of 72 RNNs, which consist of a combination of four input sets and 18 network configurations.<br /> <br /> &lt;center&gt;[[Image:Figure8_Yan.png|800px|]]&lt;/center&gt;<br /> <br /> ==== Input and output features ==== <br /> In order to apply the motion predictor to the AV in motion, the speed of the data collection vehicle is added to the input sequence. The input sequence consists of relative X/Y position, relative heading angle, speed of surrounding target vehicles, and speed of data collection vehicles. The output sequence is the same as the input sequence, such as relative position, heading, and speed.<br /> <br /> ==== Encoder and decoder ==== <br /> In this study, the authors introduced an encoder and decoder that process the input from the sensor and the output from the RNN, respectively. The encoder normalizes each component of the input data to rescale the data to mean 0 and standard deviation 1, while the decoder denormalizes the output data to use the same parameters as in the encoder to scale it back to the actual unit. <br /> ==== Sequence length ==== <br /> The sequence length of RNN input and output is another important factor to improve prediction performance. In this study, 5, 10, 15, 20, 25, and 30 steps of 100 millisecond sampling times were compared, and 15 steps showed relatively accurate results, even among candidates The observation time is very short.<br /> <br /> == Motion planning based on surrounding vehicle motion prediction == <br /> In daily driving, experienced drivers will predict possible risks based on observations of surrounding vehicles, and ensure safety by changing behaviors before the risks occur. In order to achieve a human-like motion plan, based on the model predictive control (MPC) method, a prediction-based motion planner for autonomous vehicles is designed, which takes into account the driver’s future behavior. The cost function of the motion planner is determined as follows:<br /> \begin{equation*}<br /> \begin{split}<br /> J = &amp; \sum_{k=1}^{N_p} (x(k|t) - x_{ref}(k|t)^T) Q(x(k|t) - x_{ref}(k|t)) +\\<br /> &amp; R \sum_{k=0}^{N_p-1} u(k|t)^2 + R_{\Delta \mu}\sum_{k=0}^{N_p-2} (u(k+1|t) - u(k|t))^2 <br /> \end{split}<br /> \end{equation*}<br /> where &lt;math&gt;k&lt;/math&gt; and &lt;math&gt;t&lt;/math&gt; are the prediction step index and time index, respectively; &lt;math&gt;x(k|t)&lt;/math&gt; and &lt;math&gt;x_{ref} (k|t)&lt;/math&gt; are the states and reference of the MPC problem, respectively; &lt;math&gt;x(k|t)&lt;/math&gt; is composed of travel distance px and longitudinal velocity vx; &lt;math&gt;x_{ref} (k|t)&lt;/math&gt; consists of reference travel distance &lt;math&gt;p_{x,ref}&lt;/math&gt; and reference longitudinal velocity &lt;math&gt;v_{x,ref}&lt;/math&gt; ; &lt;math&gt;u(k|t)&lt;/math&gt; is the control input, which is the longitudinal acceleration command; &lt;math&gt;N_p&lt;/math&gt; is the prediction horizon; and Q, R, and &lt;math&gt;R_{\Delta \mu}&lt;/math&gt; are the weight matrices for states, input, and input derivative, respectively, and these weight matrices were tuned to obtain control inputs from the proposed controller that were as similar as possible to those of human-driven vehicles. <br /> The constraints of the control input are defined as follows:<br /> \begin{equation*}<br /> \begin{split}<br /> &amp;\mu_{min} \leq \mu(k|t) \leq \mu_{max} \\<br /> &amp;||\mu(k+1|t) - \mu(k|t)|| \leq S<br /> \end{split}<br /> \end{equation*}<br /> Where &lt;math&gt;u_{min}&lt;/math&gt;, &lt;math&gt;u_{max}&lt;/math&gt;and S are the minimum/maximum control input and maximum slew rate of input respectively.<br /> <br /> Determine the position and speed boundary based on the predicted state:<br /> \begin{equation*}<br /> \begin{split}<br /> &amp; p_{x,max}(k|t) = p_{x,tar}(k|t) - c_{des}(k|t) \quad p_{x,min}(k|t) = 0 \\<br /> &amp; v_{x,max}(k|t) = min(v_{x,ret}(k|t), v_{x,limit}) \quad v_{x,min}(k|t) = 0<br /> \end{split}<br /> \end{equation*}<br /> Where &lt;math&gt;v_{x, limit}&lt;/math&gt; are the speed limits of the target vehicle.<br /> <br /> == Prediction performance analysis and application to motion planning ==<br /> === Accuracy analysis ===<br /> The proposed algorithm was compared with the results from three base algorithms, a path-following model with <br /> constant velocity, a path-following model with traffic flow and a CTRV model.<br /> <br /> We compare those algorithms according to four sorts of errors, The &lt;math&gt;x&lt;/math&gt; position error &lt;math&gt;e_{x,T_p}&lt;/math&gt;, <br /> &lt;math&gt;y&lt;/math&gt; position error &lt;math&gt;e_{y,T_p}&lt;/math&gt;, heading error &lt;math&gt;e_{\theta,T_p}&lt;/math&gt;, and velocity error &lt;math&gt;e_{v,T_p}&lt;/math&gt; where &lt;math&gt;T_p&lt;/math&gt; denotes time &lt;math&gt;p&lt;/math&gt;. These four errors are defined as follows:<br /> <br /> \begin{equation*}<br /> \begin{split}<br /> e_{x,Tp}=&amp; p_{x,Tp} -\hat {p}_{x,Tp}\\ <br /> e_{y,Tp}=&amp; p_{y,Tp} -\hat {p}_{y,Tp}\\ <br /> e_{\theta,Tp}=&amp; \theta _{Tp} -\hat {\theta }_{Tp}\\ <br /> e_{v,Tp}=&amp; v_{Tp} -\hat {v}_{Tp}<br /> \end{split}<br /> \end{equation*}<br /> &lt;center&gt;[[Image:Figure10.1_YanYu.png|500px|]]&lt;/center&gt;<br /> <br /> The proposed model shows significantly fewer prediction errors compare to the based algorithms in terms of mean, <br /> standard deviation(STD), and root mean square error(RMSE). Meanwhile, the proposed model exhibits a bell-shaped <br /> curve with a close to zero mean, which indicates that the proposed algorithm's prediction of human divers' <br /> intensions are relatively precise. On the other hand, &lt;math&gt;e_{x,T_p}&lt;/math&gt;, &lt;math&gt;e_{y,T_p}&lt;/math&gt;, &lt;math&gt;e_{v,T_p}&lt;/math&gt; are bounded within <br /> reasonable levels. For instant, the three-sigma range of &lt;math&gt;e_{y,T_p}&lt;/math&gt; is within the width of a lane. Therefore, <br /> the proposed algorithm can be precise and maintain safety simultaneously.<br /> <br /> === Motion planning application ===<br /> ==== Case study of a multi-lane left turn scenario ====<br /> The proposed method mimics a human driver better, by simulating a human driver's decision-making process. <br /> In a multi-lane left turn scenario, the proposed algorithm correctly predicted the trajectory of a target <br /> the vehicle, even when the target vehicle was not following the intersection guideline.<br /> <br /> ==== Statistical analysis of motion planning application results ====<br /> The data is analyzed from two perspectives, the time to recognize the in-lane target and the similarity to <br /> human driver commands. In most of cases, the proposed algorithm detects the in-line target no late than based <br /> algorithm. In addition, the proposed algorithm only recognized cases later than the base algorithm did when <br /> the surrounding target vehicles first appeared beyond the sensors’ region of interest boundaries. This means <br /> that these cases took place sufficiently beyond the safety distance, and had little influence on determining <br /> the behaviour of the subject vehicle.<br /> <br /> &lt;center&gt;[[Image:Figure11_YanYu.png|500px|]]&lt;/center&gt;<br /> <br /> In order to compare the similarities between the results form the proposed algorithm and human driving decisions, <br /> we introduced another type of error, acceleration error &lt;math&gt;a_{x, error} = a_{x, human} - a_{x, cmd}&lt;/math&gt;. where &lt;math&gt;a_{x, human}&lt;/math&gt;<br /> and &lt;math&gt;a_{x, cmd}&lt;/math&gt; are the human driver’s acceleration history and the command from the proposed algorithm, <br /> respectively. The proposed algorithm showed more similar results to human drivers’ decisions than did the base <br /> algorithms. &lt;math&gt;91.97\%&lt;/math&gt; of the acceleration error lies in the region &lt;math&gt;\pm 1 m/s^2&lt;/math&gt;. Moreover, the base algorithm <br /> possesses a limited ability to respond to different in-lane target behaviours in traffic flow. Hence, the proposed <br /> model is efficient and safe.<br /> <br /> == Conclusion ==<br /> A surrounding vehicle motion predictor based on an LSTM-RNN at multi-lane turn intersections was developed, and its application in an autonomous vehicle was evaluated. The model was trained by using the data captured on the urban road in Seoul in MPC. The evaluation results showed precise prediction accuracy and so the algorithm is safe to be applied on an autonomous vehicle. Also, the comparison with the other three base algorithms (CV/Path, V_flow/Path, and CTRV) revealed the superiority of the proposed algorithm. The evaluation results showed precise prediction accuracy. In addition, the time-to-recognize in-lane targets within the intersection improved significantly over the performance of the base algorithms. The proposed algorithm was compared with human driving data, and it showed similar longitudinal acceleration. The motion predictor can be applied to path planners when AVs travel in unconstructed environments, such as multi-lane turn intersections.<br /> <br /> == Future works ==<br /> 1.Developing trajectory prediction algorithms using other machine learning algorithms, such as attention-aware neural networks.<br /> <br /> 2.Applying the machine learning-based approach to infer lane change intention at motorways and main roads of urban environments.<br /> <br /> 3.Extending the target road of the trajectory predictor, such as roundabouts or uncontrolled intersections, to infer yield intention.<br /> <br /> 4.Learning the behavior of surrounding vehicles in real time while automated vehicles drive with real traffic.<br /> <br /> == Critiques ==<br /> The literature review is not sufficient. It should focus more on LSTM, RNN, and the study in different types of roads. Why the LSTM-RNN is used, and the background of the method is not stated clearly. There is a lack of concept so that it is difficult to distinguish between LSTM-RNN based motion predictor and motion planning.<br /> <br /> This is an interesting topic to discuss. This is a major topic for some famous vehicle company such as Tesla, Tesla nows already have a good service called Autopilot to give self-driving and Motion Prediction. This summary can include more diagrams in architecture in the model to give readers a whole view of how the model looks like. Since it is using LSTM-RNN, include some pictures of the LSTM-RNN will be great. I think it will be interesting to discuss more applications by using this method, such as Airplane, boats.<br /> <br /> Autonomous driving is a hot very topic, and training the model with LSTM-RNN is also a meaningful topic to discuss. By the way, it would be an interesting approach to compare the performance of different algorithms or some other traditional motion planning algorithms like KF.<br /> <br /> There are some papers that discussed the accuracy of different models in vehicle predictions, such as Deep Kinematic Models for Kinematically Feasible Vehicle Trajectory Predictions[https://arxiv.org/pdf/1908.00219.pdf.] The LSTM didn't show good performance. They increased the accuracy by combing LSTM with an unconstrained model(UM) by adding an additional LSTM layer of size 128 that is used to recursively output positions instead of simultaneously outputting positions for all horizons.<br /> <br /> It may be better to provide the results of experiments to support the efficiency of LSTM-RNN, talk about the prediction of training and test sets, and compared it with other autonomous driving systems that exist in the world.<br /> <br /> The topic of surround vehicle motion prediction is analogous to the topic of autonomous vehicles. An example of an application of these frameworks would be the transportation services industry. Many companies, such as Lyft and Uber, have started testing their own commercial autonomous vehicles.<br /> <br /> It would be really helpful if some visualization or data summary can be provided to understand the content, such as the track of the car movement.<br /> <br /> The model should have been tested in other regions besides just Seoul, as driving behaviors can vary drastically from region to region.<br /> <br /> Understandably, a supervised learning problem should be evaluated on some test dataset. However, supervised learning techniques are inherently ill-suited for general planning problems. The test dataset was obtained from human driving data which is known to be extremely noisy as well as unpredictable when it comes to motion planning. It would be crucial to determine the successes of this paper based on the state-of-the-art reinforcement learning techniques.<br /> <br /> It would be better if the authors compared their method against other SOTA methods. Also one of the reasons motion planning is done using interpretable methods rather than black boxes (such as this model) is because it is hard to see where things go wrong and fix problems with the black box when they occur - this is something the authors should have also discussed.<br /> <br /> A future area of study is to combine other source of information such as signals from Lidar or car side cameras to make a better prediction model.<br /> <br /> == Reference ==<br />  E. Choi, Crash Factors in Intersection-Related Crashes: An On-Scene Perspective (No. Dot HS 811 366), U.S. DOT Nat. Highway Traffic Safety Admin., Washington, DC, USA, 2010.<br /> <br />  D. J. Phillips, T. A. Wheeler, and M. J. Kochenderfer, “Generalizable intention prediction of human drivers at intersections,” in Proc. IEEE Intell. Veh. Symp. (IV), Los Angeles, CA, USA, 2017, pp. 1665–1670.<br /> <br />  B. Kim, C. M. Kang, J. Kim, S. H. Lee, C. C. Chung, and J. W. Choi, “Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network,” in Proc. IEEE 20th Int. Conf. Intell. Transp. Syst. (ITSC), Yokohama, Japan, 2017, pp. 399–404.<br /> <br />  E. Strigel, D. Meissner, F. Seeliger, B. Wilking, and K. Dietmayer, “The Ko-PER intersection laserscanner and video dataset,” in Proc. 17th Int. IEEE Conf. Intell. Transp. Syst. (ITSC), Qingdao, China, 2014, pp. 1900–1901.<br /> <br />  Henggang Cui, Thi Nguyen, Fang-Chieh Chou, Tsung-Han Lin, Jeff Schneider, David Bradley, Nemanja Djuric: “Deep Kinematic Models for Kinematically Feasible Vehicle Trajectory Predictions”, 2019; [http://arxiv.org/abs/1908.00219 arXiv:1908.00219].<br /> <br /> Schulz, Jens &amp; Hubmann, Constantin &amp; Morin, Nikolai &amp; Löchner, Julian &amp; Burschka, Darius. (2019). Learning Interaction-Aware Probabilistic Driver Behavior Models from Urban Scenarios. 10.1109/IVS.2019.8814080.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Surround_Vehicle_Motion_Prediction&diff=48935 Surround Vehicle Motion Prediction 2020-12-02T22:04:05Z <p>Z42qin: /* Data Collection Devices and Outputs */</p> <hr /> <div>DROCC: '''Surround Vehicle Motion Prediction Using LSTM-RNN for Motion Planning of Autonomous Vehicles at Multi-Lane Turn Intersections'''<br /> == Presented by == <br /> Mushi Wang, Siyuan Qiu, Yan Yu<br /> <br /> == Introduction ==<br /> <br /> This paper presents a surrounding vehicle motion prediction algorithm for multi-lane turn intersections using a Long Short-Term Memory (LSTM)-based Recurrent Neural Network (RNN). More specifically, it focused on the improvement of in-lane target recognition and achieving human-like acceleration decisions at multi-lane turn intersections by introducing the learning-based target motion predictor and prediction-based motion predictor. A data-driven approach for predicting the trajectory and velocity of surrounding vehicles on urban roads at multi-lane turn intersections was described. LSTM architecture, a specific kind of RNN capable of learning long-term dependencies, is designed to manage complex vehicle motions in multi-lane turn intersections. The results show that the forecaster improves the recognition time of the leading vehicle and contributes to the improvement of prediction ability.<br /> <br /> == Previous Work ==<br /> The autonomous vehicle trajectory approaches previously used motion models like Constant Velocity and Constant Acceleration. These models are linear and are only able to handle straight motions. There are curvilinear models such as Constant Turn Rate and Velocity and Constant Turn Rate and Acceleration which handle rotations and more complex motions. Together with these models, Kalman Filter is used to predicting the vehicle trajectory. Kalman filtering is a common technique used in sensor fusion for state estimation that allows the vehicle's state to be predicted while taking into account the uncertainty associated with inputs and measurements. However, the performance of the Kalman Filter in predicting multi-step problem is not that good. Recurrent Neural Network performs significantly better than it. <br /> <br /> There are 3 main challenges to achieving fully autonomous driving on urban roads, which are scene awareness, inferring other drivers’ intentions, and predicting their future motions. Researchers are developing prediction algorithms that can simulate a driver’s intuition to improve safety when autonomous vehicles and human drivers drive together. To predict driver behavior on an urban road, there are 3 categories for the motion prediction model: (1) physics-based; (2) maneuver-based; and (3) interaction-aware. Physics-based models are simple and direct, which only consider the states of prediction vehicles kinematically. The advantage is that it has minimal computational burden among the three types. However, it is impossible to consider the interactions between vehicles. Maneuver-based models consider the driver’s intention and classified them. By predicting the driver maneuver, the future trajectory can be predicted. Identifying similar behaviors in driving is able to infer different drivers' intentions which are stated to improve the prediction accuracy. However, it still an assistant to improve physics-based models. <br /> <br /> Recurrent Neural Network (RNN) is a type of approach proposed to infer driver intention in this paper. Interaction-aware models can reflect interactions between surrounding vehicles, and predict future motions of detected vehicles simultaneously as a scene. While the prediction algorithm is more complex in computation which is often used in offline simulations. As Schulz et al. indicate, interaction models are very difficult to create as &quot;predicting complete trajectories at once is challenging, as one needs to account for multiple hypotheses and long-term interactions between multiple agents&quot; .<br /> <br /> == Motivation == <br /> Research results indicate that little research has been dedicated on predicting the trajectory of intersections. Moreover, public data sets for analyzing driver behaviour at intersections are not enough, and these data sets are not easy to collect. A model is needed to predict the various movements of the target around a multi-lane turning intersection. It is very necessary to design a motion predictor that can be used for real-time traffic.<br /> <br /> == Framework == <br /> The LSTM-RNN-based motion predictor comprises three parts: (1) a data encoder; (2) an LSTM-based RNN; and (3) a data decoder depicts the architecture of the surrounding target trajectory predictor. The proposed architecture uses a perception algorithm to estimate the state of surrounding vehicles, which relies on six scanners. The output predicts the state of the surrounding vehicles and is used to determine the expected longitudinal acceleration in the actual traffic at the intersection. The following image gives a visual representation of the model.<br /> <br /> &lt;center&gt;[[Image:Figure1_Yan.png|800px|]]&lt;/center&gt;<br /> <br /> == LSTM-RNN based motion predictor == <br /> <br /> === Data Collection Devices and Outputs ===<br /> <br /> The input of the target perceptions are from the output of the sensors. The data collection in this article uses 6 different sensors with feature fusion to detect traffic in range up to 100m: 1) LiDAR system outputs: Relative position, heading, velocity and box size in local coordinates ; 2) Around0View Monitoring (AVM) and 3)GPS outputs: acquire lanes, road marker, global position; 4) Gateaway engine outputs: precise global position in urban road environment; 5) Micro-Autobox II and 6) a MDPS are used to control and actuate the subject. All data are stored in an industrial PC.<br /> <br /> === Data ===<br /> Multi-lane turn intersections are the target roads in this paper. The real dataset is captured on urban roads in Seoul. The training model is generated from 484 tracks collected when driving through intersections in real traffic. The previous and subsequent states of a vehicle at a particular time can be extracted. After post-processing, the collected data, a total of 16,660 data samples were generated, including 11,662 training data samples, and 4,998 evaluation data samples.<br /> <br /> === Motion predictor ===<br /> This article proposes a data-driven method to predict the future movement of surrounding vehicles based on their previous movement. The motion predictor based on the LSTM-RNN architecture in this work only uses information collected from sensors on autonomous vehicles, as shown in the figure below. The contribution of the network architecture of this study is that the future state of the target vehicle is used as the input feature for predicting the field of view. <br /> <br /> <br /> &lt;center&gt;[[Image:Figure7b_Yan.png|500px|]]&lt;/center&gt;<br /> <br /> <br /> ==== Network architecture ==== <br /> A RNN is an artificial neural network, suitable for use with sequential data. It can also be used for time-series data, where the pattern of the data depends on the time flow. Also, it can contain feedback loops that allow activations to flow alternately in the loop.<br /> An LSTM avoids the problem of vanishing gradients by making errors flow backward without a limit on the number of virtual layers. This property prevents errors from increasing or declining over time, which can make the network train improperly. The figure below shows the various layers of the LSTM-RNN and the number of units in each layer. This structure is determined by comparing the accuracy of 72 RNNs, which consist of a combination of four input sets and 18 network configurations.<br /> <br /> &lt;center&gt;[[Image:Figure8_Yan.png|800px|]]&lt;/center&gt;<br /> <br /> ==== Input and output features ==== <br /> In order to apply the motion predictor to the AV in motion, the speed of the data collection vehicle is added to the input sequence. The input sequence consists of relative X/Y position, relative heading angle, speed of surrounding target vehicles, and speed of data collection vehicles. The output sequence is the same as the input sequence, such as relative position, heading, and speed.<br /> <br /> ==== Encoder and decoder ==== <br /> In this study, the authors introduced an encoder and decoder that process the input from the sensor and the output from the RNN, respectively. The encoder normalizes each component of the input data to rescale the data to mean 0 and standard deviation 1, while the decoder denormalizes the output data to use the same parameters as in the encoder to scale it back to the actual unit. <br /> ==== Sequence length ==== <br /> The sequence length of RNN input and output is another important factor to improve prediction performance. In this study, 5, 10, 15, 20, 25, and 30 steps of 100 millisecond sampling times were compared, and 15 steps showed relatively accurate results, even among candidates The observation time is very short.<br /> <br /> == Motion planning based on surrounding vehicle motion prediction == <br /> In daily driving, experienced drivers will predict possible risks based on observations of surrounding vehicles, and ensure safety by changing behaviors before the risks occur. In order to achieve a human-like motion plan, based on the model predictive control (MPC) method, a prediction-based motion planner for autonomous vehicles is designed, which takes into account the driver’s future behavior. The cost function of the motion planner is determined as follows:<br /> \begin{equation*}<br /> \begin{split}<br /> J = &amp; \sum_{k=1}^{N_p} (x(k|t) - x_{ref}(k|t)^T) Q(x(k|t) - x_{ref}(k|t)) +\\<br /> &amp; R \sum_{k=0}^{N_p-1} u(k|t)^2 + R_{\Delta \mu}\sum_{k=0}^{N_p-2} (u(k+1|t) - u(k|t))^2 <br /> \end{split}<br /> \end{equation*}<br /> where &lt;math&gt;k&lt;/math&gt; and &lt;math&gt;t&lt;/math&gt; are the prediction step index and time index, respectively; &lt;math&gt;x(k|t)&lt;/math&gt; and &lt;math&gt;x_{ref} (k|t)&lt;/math&gt; are the states and reference of the MPC problem, respectively; &lt;math&gt;x(k|t)&lt;/math&gt; is composed of travel distance px and longitudinal velocity vx; &lt;math&gt;x_{ref} (k|t)&lt;/math&gt; consists of reference travel distance &lt;math&gt;p_{x,ref}&lt;/math&gt; and reference longitudinal velocity &lt;math&gt;v_{x,ref}&lt;/math&gt; ; &lt;math&gt;u(k|t)&lt;/math&gt; is the control input, which is the longitudinal acceleration command; &lt;math&gt;N_p&lt;/math&gt; is the prediction horizon; and Q, R, and &lt;math&gt;R_{\Delta \mu}&lt;/math&gt; are the weight matrices for states, input, and input derivative, respectively, and these weight matrices were tuned to obtain control inputs from the proposed controller that were as similar as possible to those of human-driven vehicles. <br /> The constraints of the control input are defined as follows:<br /> \begin{equation*}<br /> \begin{split}<br /> &amp;\mu_{min} \leq \mu(k|t) \leq \mu_{max} \\<br /> &amp;||\mu(k+1|t) - \mu(k|t)|| \leq S<br /> \end{split}<br /> \end{equation*}<br /> Where &lt;math&gt;u_{min}&lt;/math&gt;, &lt;math&gt;u_{max}&lt;/math&gt;and S are the minimum/maximum control input and maximum slew rate of input respectively.<br /> <br /> Determine the position and speed boundary based on the predicted state:<br /> \begin{equation*}<br /> \begin{split}<br /> &amp; p_{x,max}(k|t) = p_{x,tar}(k|t) - c_{des}(k|t) \quad p_{x,min}(k|t) = 0 \\<br /> &amp; v_{x,max}(k|t) = min(v_{x,ret}(k|t), v_{x,limit}) \quad v_{x,min}(k|t) = 0<br /> \end{split}<br /> \end{equation*}<br /> Where &lt;math&gt;v_{x, limit}&lt;/math&gt; are the speed limits of the target vehicle.<br /> <br /> == Prediction performance analysis and application to motion planning ==<br /> === Accuracy analysis ===<br /> The proposed algorithm was compared with the results from three base algorithms, a path-following model with <br /> constant velocity, a path-following model with traffic flow and a CTRV model.<br /> <br /> We compare those algorithms according to four sorts of errors, The &lt;math&gt;x&lt;/math&gt; position error &lt;math&gt;e_{x,T_p}&lt;/math&gt;, <br /> &lt;math&gt;y&lt;/math&gt; position error &lt;math&gt;e_{y,T_p}&lt;/math&gt;, heading error &lt;math&gt;e_{\theta,T_p}&lt;/math&gt;, and velocity error &lt;math&gt;e_{v,T_p}&lt;/math&gt; where &lt;math&gt;T_p&lt;/math&gt; denotes time &lt;math&gt;p&lt;/math&gt;. These four errors are defined as follows:<br /> <br /> \begin{equation*}<br /> \begin{split}<br /> e_{x,Tp}=&amp; p_{x,Tp} -\hat {p}_{x,Tp}\\ <br /> e_{y,Tp}=&amp; p_{y,Tp} -\hat {p}_{y,Tp}\\ <br /> e_{\theta,Tp}=&amp; \theta _{Tp} -\hat {\theta }_{Tp}\\ <br /> e_{v,Tp}=&amp; v_{Tp} -\hat {v}_{Tp}<br /> \end{split}<br /> \end{equation*}<br /> &lt;center&gt;[[Image:Figure10.1_YanYu.png|500px|]]&lt;/center&gt;<br /> <br /> The proposed model shows significantly fewer prediction errors compare to the based algorithms in terms of mean, <br /> standard deviation(STD), and root mean square error(RMSE). Meanwhile, the proposed model exhibits a bell-shaped <br /> curve with a close to zero mean, which indicates that the proposed algorithm's prediction of human divers' <br /> intensions are relatively precise. On the other hand, &lt;math&gt;e_{x,T_p}&lt;/math&gt;, &lt;math&gt;e_{y,T_p}&lt;/math&gt;, &lt;math&gt;e_{v,T_p}&lt;/math&gt; are bounded within <br /> reasonable levels. For instant, the three-sigma range of &lt;math&gt;e_{y,T_p}&lt;/math&gt; is within the width of a lane. Therefore, <br /> the proposed algorithm can be precise and maintain safety simultaneously.<br /> <br /> === Motion planning application ===<br /> ==== Case study of a multi-lane left turn scenario ====<br /> The proposed method mimics a human driver better, by simulating a human driver's decision-making process. <br /> In a multi-lane left turn scenario, the proposed algorithm correctly predicted the trajectory of a target <br /> the vehicle, even when the target vehicle was not following the intersection guideline.<br /> <br /> ==== Statistical analysis of motion planning application results ====<br /> The data is analyzed from two perspectives, the time to recognize the in-lane target and the similarity to <br /> human driver commands. In most of cases, the proposed algorithm detects the in-line target no late than based <br /> algorithm. In addition, the proposed algorithm only recognized cases later than the base algorithm did when <br /> the surrounding target vehicles first appeared beyond the sensors’ region of interest boundaries. This means <br /> that these cases took place sufficiently beyond the safety distance, and had little influence on determining <br /> the behaviour of the subject vehicle.<br /> <br /> &lt;center&gt;[[Image:Figure11_YanYu.png|500px|]]&lt;/center&gt;<br /> <br /> In order to compare the similarities between the results form the proposed algorithm and human driving decisions, <br /> we introduced another type of error, acceleration error &lt;math&gt;a_{x, error} = a_{x, human} - a_{x, cmd}&lt;/math&gt;. where &lt;math&gt;a_{x, human}&lt;/math&gt;<br /> and &lt;math&gt;a_{x, cmd}&lt;/math&gt; are the human driver’s acceleration history and the command from the proposed algorithm, <br /> respectively. The proposed algorithm showed more similar results to human drivers’ decisions than did the base <br /> algorithms. &lt;math&gt;91.97\%&lt;/math&gt; of the acceleration error lies in the region &lt;math&gt;\pm 1 m/s^2&lt;/math&gt;. Moreover, the base algorithm <br /> possesses a limited ability to respond to different in-lane target behaviours in traffic flow. Hence, the proposed <br /> model is efficient and safe.<br /> <br /> == Conclusion ==<br /> A surrounding vehicle motion predictor based on an LSTM-RNN at multi-lane turn intersections was developed, and its application in an autonomous vehicle was evaluated. The model was trained by using the data captured on the urban road in Seoul in MPC. The evaluation results showed precise prediction accuracy and so the algorithm is safe to be applied on an autonomous vehicle. Also, the comparison with the other three base algorithms (CV/Path, V_flow/Path, and CTRV) revealed the superiority of the proposed algorithm. The evaluation results showed precise prediction accuracy. In addition, the time-to-recognize in-lane targets within the intersection improved significantly over the performance of the base algorithms. The proposed algorithm was compared with human driving data, and it showed similar longitudinal acceleration. The motion predictor can be applied to path planners when AVs travel in unconstructed environments, such as multi-lane turn intersections.<br /> <br /> == Future works ==<br /> 1.Developing trajectory prediction algorithms using other machine learning algorithms, such as attention-aware neural networks.<br /> <br /> 2.Applying the machine learning-based approach to infer lane change intention at motorways and main roads of urban environments.<br /> <br /> 3.Extending the target road of the trajectory predictor, such as roundabouts or uncontrolled intersections, to infer yield intention.<br /> <br /> 4.Learning the behavior of surrounding vehicles in real time while automated vehicles drive with real traffic.<br /> <br /> == Critiques ==<br /> The literature review is not sufficient. It should focus more on LSTM, RNN, and the study in different types of roads. Why the LSTM-RNN is used, and the background of the method is not stated clearly. There is a lack of concept so that it is difficult to distinguish between LSTM-RNN based motion predictor and motion planning.<br /> <br /> This is an interesting topic to discuss. This is a major topic for some famous vehicle company such as Tesla, Tesla nows already have a good service called Autopilot to give self-driving and Motion Prediction. This summary can include more diagrams in architecture in the model to give readers a whole view of how the model looks like. Since it is using LSTM-RNN, include some pictures of the LSTM-RNN will be great. I think it will be interesting to discuss more applications by using this method, such as Airplane, boats.<br /> <br /> Autonomous driving is a hot very topic, and training the model with LSTM-RNN is also a meaningful topic to discuss. By the way, it would be an interesting approach to compare the performance of different algorithms or some other traditional motion planning algorithms like KF.<br /> <br /> There are some papers that discussed the accuracy of different models in vehicle predictions, such as Deep Kinematic Models for Kinematically Feasible Vehicle Trajectory Predictions[https://arxiv.org/pdf/1908.00219.pdf.] The LSTM didn't show good performance. They increased the accuracy by combing LSTM with an unconstrained model(UM) by adding an additional LSTM layer of size 128 that is used to recursively output positions instead of simultaneously outputting positions for all horizons.<br /> <br /> It may be better to provide the results of experiments to support the efficiency of LSTM-RNN, talk about the prediction of training and test sets, and compared it with other autonomous driving systems that exist in the world.<br /> <br /> The topic of surround vehicle motion prediction is analogous to the topic of autonomous vehicles. An example of an application of these frameworks would be the transportation services industry. Many companies, such as Lyft and Uber, have started testing their own commercial autonomous vehicles.<br /> <br /> It would be really helpful if some visualization or data summary can be provided to understand the content, such as the track of the car movement.<br /> <br /> The model should have been tested in other regions besides just Seoul, as driving behaviors can vary drastically from region to region.<br /> <br /> Understandably, a supervised learning problem should be evaluated on some test dataset. However, supervised learning techniques are inherently ill-suited for general planning problems. The test dataset was obtained from human driving data which is known to be extremely noisy as well as unpredictable when it comes to motion planning. It would be crucial to determine the successes of this paper based on the state-of-the-art reinforcement learning techniques.<br /> <br /> It would be better if the authors compared their method against other SOTA methods. Also one of the reasons motion planning is done using interpretable methods rather than black boxes (such as this model) is because it is hard to see where things go wrong and fix problems with the black box when they occur - this is something the authors should have also discussed.<br /> <br /> A future area of study is to combine other source of information such as signals from Lidar or car side cameras to make a better prediction model.<br /> <br /> == Reference ==<br />  E. Choi, Crash Factors in Intersection-Related Crashes: An On-Scene Perspective (No. Dot HS 811 366), U.S. DOT Nat. Highway Traffic Safety Admin., Washington, DC, USA, 2010.<br /> <br />  D. J. Phillips, T. A. Wheeler, and M. J. Kochenderfer, “Generalizable intention prediction of human drivers at intersections,” in Proc. IEEE Intell. Veh. Symp. (IV), Los Angeles, CA, USA, 2017, pp. 1665–1670.<br /> <br />  B. Kim, C. M. Kang, J. Kim, S. H. Lee, C. C. Chung, and J. W. Choi, “Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network,” in Proc. IEEE 20th Int. Conf. Intell. Transp. Syst. (ITSC), Yokohama, Japan, 2017, pp. 399–404.<br /> <br />  E. Strigel, D. Meissner, F. Seeliger, B. Wilking, and K. Dietmayer, “The Ko-PER intersection laserscanner and video dataset,” in Proc. 17th Int. IEEE Conf. Intell. Transp. Syst. (ITSC), Qingdao, China, 2014, pp. 1900–1901.<br /> <br />  Henggang Cui, Thi Nguyen, Fang-Chieh Chou, Tsung-Han Lin, Jeff Schneider, David Bradley, Nemanja Djuric: “Deep Kinematic Models for Kinematically Feasible Vehicle Trajectory Predictions”, 2019; [http://arxiv.org/abs/1908.00219 arXiv:1908.00219].<br /> <br /> Schulz, Jens &amp; Hubmann, Constantin &amp; Morin, Nikolai &amp; Löchner, Julian &amp; Burschka, Darius. (2019). Learning Interaction-Aware Probabilistic Driver Behavior Models from Urban Scenarios. 10.1109/IVS.2019.8814080.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Surround_Vehicle_Motion_Prediction&diff=48918 Surround Vehicle Motion Prediction 2020-12-02T21:38:17Z <p>Z42qin: /* LSTM-RNN based motion predictor */</p> <hr /> <div>DROCC: '''Surround Vehicle Motion Prediction Using LSTM-RNN for Motion Planning of Autonomous Vehicles at Multi-Lane Turn Intersections'''<br /> == Presented by == <br /> Mushi Wang, Siyuan Qiu, Yan Yu<br /> <br /> == Introduction ==<br /> <br /> This paper presents a surrounding vehicle motion prediction algorithm for multi-lane turn intersections using a Long Short-Term Memory (LSTM)-based Recurrent Neural Network (RNN). More specifically, it focused on the improvement of in-lane target recognition and achieving human-like acceleration decisions at multi-lane turn intersections by introducing the learning-based target motion predictor and prediction-based motion predictor. A data-driven approach for predicting the trajectory and velocity of surrounding vehicles on urban roads at multi-lane turn intersections was described. LSTM architecture, a specific kind of RNN capable of learning long-term dependencies, is designed to manage complex vehicle motions in multi-lane turn intersections. The results show that the forecaster improves the recognition time of the leading vehicle and contributes to the improvement of prediction ability.<br /> <br /> == Previous Work ==<br /> The autonomous vehicle trajectory approaches previously used motion models like Constant Velocity and Constant Acceleration. These models are linear and are only able to handle straight motions. There are curvilinear models such as Constant Turn Rate and Velocity and Constant Turn Rate and Acceleration which handle rotations and more complex motions. Together with these models, Kalman Filter is used to predicting the vehicle trajectory. Kalman filtering is a common technique used in sensor fusion for state estimation that allows the vehicle's state to be predicted while taking into account the uncertainty associated with inputs and measurements. However, the performance of the Kalman Filter in predicting multi-step problem is not that good. Recurrent Neural Network performs significantly better than it. <br /> <br /> There are 3 main challenges to achieving fully autonomous driving on urban roads, which are scene awareness, inferring other drivers’ intentions, and predicting their future motions. Researchers are developing prediction algorithms that can simulate a driver’s intuition to improve safety when autonomous vehicles and human drivers drive together. To predict driver behavior on an urban road, there are 3 categories for the motion prediction model: (1) physics-based; (2) maneuver-based; and (3) interaction-aware. Physics-based models are simple and direct, which only consider the states of prediction vehicles kinematically. The advantage is that it has minimal computational burden among the three types. However, it is impossible to consider the interactions between vehicles. Maneuver-based models consider the driver’s intention and classified them. By predicting the driver maneuver, the future trajectory can be predicted. Identifying similar behaviors in driving is able to infer different drivers' intentions which are stated to improve the prediction accuracy. However, it still an assistant to improve physics-based models. <br /> <br /> Recurrent Neural Network (RNN) is a type of approach proposed to infer driver intention in this paper. Interaction-aware models can reflect interactions between surrounding vehicles, and predict future motions of detected vehicles simultaneously as a scene. While the prediction algorithm is more complex in computation which is often used in offline simulations. As Schulz et al. indicate, interaction models are very difficult to create as &quot;predicting complete trajectories at once is challenging, as one needs to account for multiple hypotheses and long-term interactions between multiple agents&quot; .<br /> <br /> == Motivation == <br /> Research results indicate that little research has been dedicated on predicting the trajectory of intersections. Moreover, public data sets for analyzing driver behaviour at intersections are not enough, and these data sets are not easy to collect. A model is needed to predict the various movements of the target around a multi-lane turning intersection. It is very necessary to design a motion predictor that can be used for real-time traffic.<br /> <br /> == Framework == <br /> The LSTM-RNN-based motion predictor comprises three parts: (1) a data encoder; (2) an LSTM-based RNN; and (3) a data decoder depicts the architecture of the surrounding target trajectory predictor. The proposed architecture uses a perception algorithm to estimate the state of surrounding vehicles, which relies on six scanners. The output predicts the state of the surrounding vehicles and is used to determine the expected longitudinal acceleration in the actual traffic at the intersection. The following image gives a visual representation of the model.<br /> <br /> &lt;center&gt;[[Image:Figure1_Yan.png|800px|]]&lt;/center&gt;<br /> <br /> == LSTM-RNN based motion predictor == <br /> <br /> === Data Collection Devices and Outputs ===<br /> <br /> <br /> === Data ===<br /> Multi-lane turn intersections are the target roads in this paper. The real dataset is captured on urban roads in Seoul. The training model is generated from 484 tracks collected when driving through intersections in real traffic. The previous and subsequent states of a vehicle at a particular time can be extracted. After post-processing, the collected data, a total of 16,660 data samples were generated, including 11,662 training data samples, and 4,998 evaluation data samples.<br /> <br /> === Motion predictor ===<br /> This article proposes a data-driven method to predict the future movement of surrounding vehicles based on their previous movement. The motion predictor based on the LSTM-RNN architecture in this work only uses information collected from sensors on autonomous vehicles, as shown in the figure below. The contribution of the network architecture of this study is that the future state of the target vehicle is used as the input feature for predicting the field of view. <br /> <br /> <br /> &lt;center&gt;[[Image:Figure7b_Yan.png|500px|]]&lt;/center&gt;<br /> <br /> <br /> ==== Network architecture ==== <br /> A RNN is an artificial neural network, suitable for use with sequential data. It can also be used for time-series data, where the pattern of the data depends on the time flow. Also, it can contain feedback loops that allow activations to flow alternately in the loop.<br /> An LSTM avoids the problem of vanishing gradients by making errors flow backward without a limit on the number of virtual layers. This property prevents errors from increasing or declining over time, which can make the network train improperly. The figure below shows the various layers of the LSTM-RNN and the number of units in each layer. This structure is determined by comparing the accuracy of 72 RNNs, which consist of a combination of four input sets and 18 network configurations.<br /> <br /> &lt;center&gt;[[Image:Figure8_Yan.png|800px|]]&lt;/center&gt;<br /> <br /> ==== Input and output features ==== <br /> In order to apply the motion predictor to the AV in motion, the speed of the data collection vehicle is added to the input sequence. The input sequence consists of relative X/Y position, relative heading angle, speed of surrounding target vehicles, and speed of data collection vehicles. The output sequence is the same as the input sequence, such as relative position, heading, and speed.<br /> <br /> ==== Encoder and decoder ==== <br /> In this study, the authors introduced an encoder and decoder that process the input from the sensor and the output from the RNN, respectively. The encoder normalizes each component of the input data to rescale the data to mean 0 and standard deviation 1, while the decoder denormalizes the output data to use the same parameters as in the encoder to scale it back to the actual unit. <br /> ==== Sequence length ==== <br /> The sequence length of RNN input and output is another important factor to improve prediction performance. In this study, 5, 10, 15, 20, 25, and 30 steps of 100 millisecond sampling times were compared, and 15 steps showed relatively accurate results, even among candidates The observation time is very short.<br /> <br /> == Motion planning based on surrounding vehicle motion prediction == <br /> In daily driving, experienced drivers will predict possible risks based on observations of surrounding vehicles, and ensure safety by changing behaviors before the risks occur. In order to achieve a human-like motion plan, based on the model predictive control (MPC) method, a prediction-based motion planner for autonomous vehicles is designed, which takes into account the driver’s future behavior. The cost function of the motion planner is determined as follows:<br /> \begin{equation*}<br /> \begin{split}<br /> J = &amp; \sum_{k=1}^{N_p} (x(k|t) - x_{ref}(k|t)^T) Q(x(k|t) - x_{ref}(k|t)) +\\<br /> &amp; R \sum_{k=0}^{N_p-1} u(k|t)^2 + R_{\Delta \mu}\sum_{k=0}^{N_p-2} (u(k+1|t) - u(k|t))^2 <br /> \end{split}<br /> \end{equation*}<br /> where &lt;math&gt;k&lt;/math&gt; and &lt;math&gt;t&lt;/math&gt; are the prediction step index and time index, respectively; &lt;math&gt;x(k|t)&lt;/math&gt; and &lt;math&gt;x_{ref} (k|t)&lt;/math&gt; are the states and reference of the MPC problem, respectively; &lt;math&gt;x(k|t)&lt;/math&gt; is composed of travel distance px and longitudinal velocity vx; &lt;math&gt;x_{ref} (k|t)&lt;/math&gt; consists of reference travel distance &lt;math&gt;p_{x,ref}&lt;/math&gt; and reference longitudinal velocity &lt;math&gt;v_{x,ref}&lt;/math&gt; ; &lt;math&gt;u(k|t)&lt;/math&gt; is the control input, which is the longitudinal acceleration command; &lt;math&gt;N_p&lt;/math&gt; is the prediction horizon; and Q, R, and &lt;math&gt;R_{\Delta \mu}&lt;/math&gt; are the weight matrices for states, input, and input derivative, respectively, and these weight matrices were tuned to obtain control inputs from the proposed controller that were as similar as possible to those of human-driven vehicles. <br /> The constraints of the control input are defined as follows:<br /> \begin{equation*}<br /> \begin{split}<br /> &amp;\mu_{min} \leq \mu(k|t) \leq \mu_{max} \\<br /> &amp;||\mu(k+1|t) - \mu(k|t)|| \leq S<br /> \end{split}<br /> \end{equation*}<br /> Where &lt;math&gt;u_{min}&lt;/math&gt;, &lt;math&gt;u_{max}&lt;/math&gt;and S are the minimum/maximum control input and maximum slew rate of input respectively.<br /> <br /> Determine the position and speed boundary based on the predicted state:<br /> \begin{equation*}<br /> \begin{split}<br /> &amp; p_{x,max}(k|t) = p_{x,tar}(k|t) - c_{des}(k|t) \quad p_{x,min}(k|t) = 0 \\<br /> &amp; v_{x,max}(k|t) = min(v_{x,ret}(k|t), v_{x,limit}) \quad v_{x,min}(k|t) = 0<br /> \end{split}<br /> \end{equation*}<br /> Where &lt;math&gt;v_{x, limit}&lt;/math&gt; are the speed limits of the target vehicle.<br /> <br /> == Prediction performance analysis and application to motion planning ==<br /> === Accuracy analysis ===<br /> The proposed algorithm was compared with the results from three base algorithms, a path-following model with <br /> constant velocity, a path-following model with traffic flow and a CTRV model.<br /> <br /> We compare those algorithms according to four sorts of errors, The &lt;math&gt;x&lt;/math&gt; position error &lt;math&gt;e_{x,T_p}&lt;/math&gt;, <br /> &lt;math&gt;y&lt;/math&gt; position error &lt;math&gt;e_{y,T_p}&lt;/math&gt;, heading error &lt;math&gt;e_{\theta,T_p}&lt;/math&gt;, and velocity error &lt;math&gt;e_{v,T_p}&lt;/math&gt; where &lt;math&gt;T_p&lt;/math&gt; denotes time &lt;math&gt;p&lt;/math&gt;. These four errors are defined as follows:<br /> <br /> \begin{equation*}<br /> \begin{split}<br /> e_{x,Tp}=&amp; p_{x,Tp} -\hat {p}_{x,Tp}\\ <br /> e_{y,Tp}=&amp; p_{y,Tp} -\hat {p}_{y,Tp}\\ <br /> e_{\theta,Tp}=&amp; \theta _{Tp} -\hat {\theta }_{Tp}\\ <br /> e_{v,Tp}=&amp; v_{Tp} -\hat {v}_{Tp}<br /> \end{split}<br /> \end{equation*}<br /> &lt;center&gt;[[Image:Figure10.1_YanYu.png|500px|]]&lt;/center&gt;<br /> <br /> The proposed model shows significantly fewer prediction errors compare to the based algorithms in terms of mean, <br /> standard deviation(STD), and root mean square error(RMSE). Meanwhile, the proposed model exhibits a bell-shaped <br /> curve with a close to zero mean, which indicates that the proposed algorithm's prediction of human divers' <br /> intensions are relatively precise. On the other hand, &lt;math&gt;e_{x,T_p}&lt;/math&gt;, &lt;math&gt;e_{y,T_p}&lt;/math&gt;, &lt;math&gt;e_{v,T_p}&lt;/math&gt; are bounded within <br /> reasonable levels. For instant, the three-sigma range of &lt;math&gt;e_{y,T_p}&lt;/math&gt; is within the width of a lane. Therefore, <br /> the proposed algorithm can be precise and maintain safety simultaneously.<br /> <br /> === Motion planning application ===<br /> ==== Case study of a multi-lane left turn scenario ====<br /> The proposed method mimics a human driver better, by simulating a human driver's decision-making process. <br /> In a multi-lane left turn scenario, the proposed algorithm correctly predicted the trajectory of a target <br /> the vehicle, even when the target vehicle was not following the intersection guideline.<br /> <br /> ==== Statistical analysis of motion planning application results ====<br /> The data is analyzed from two perspectives, the time to recognize the in-lane target and the similarity to <br /> human driver commands. In most of cases, the proposed algorithm detects the in-line target no late than based <br /> algorithm. In addition, the proposed algorithm only recognized cases later than the base algorithm did when <br /> the surrounding target vehicles first appeared beyond the sensors’ region of interest boundaries. This means <br /> that these cases took place sufficiently beyond the safety distance, and had little influence on determining <br /> the behaviour of the subject vehicle.<br /> <br /> &lt;center&gt;[[Image:Figure11_YanYu.png|500px|]]&lt;/center&gt;<br /> <br /> In order to compare the similarities between the results form the proposed algorithm and human driving decisions, <br /> we introduced another type of error, acceleration error &lt;math&gt;a_{x, error} = a_{x, human} - a_{x, cmd}&lt;/math&gt;. where &lt;math&gt;a_{x, human}&lt;/math&gt;<br /> and &lt;math&gt;a_{x, cmd}&lt;/math&gt; are the human driver’s acceleration history and the command from the proposed algorithm, <br /> respectively. The proposed algorithm showed more similar results to human drivers’ decisions than did the base <br /> algorithms. &lt;math&gt;91.97\%&lt;/math&gt; of the acceleration error lies in the region &lt;math&gt;\pm 1 m/s^2&lt;/math&gt;. Moreover, the base algorithm <br /> possesses a limited ability to respond to different in-lane target behaviours in traffic flow. Hence, the proposed <br /> model is efficient and safe.<br /> <br /> == Conclusion ==<br /> A surrounding vehicle motion predictor based on an LSTM-RNN at multi-lane turn intersections was developed, and its application in an autonomous vehicle was evaluated. The model was trained by using the data captured on the urban road in Seoul in MPC. The evaluation results showed precise prediction accuracy and so the algorithm is safe to be applied on an autonomous vehicle. Also, the comparison with the other three base algorithms (CV/Path, V_flow/Path, and CTRV) revealed the superiority of the proposed algorithm. The evaluation results showed precise prediction accuracy. In addition, the time-to-recognize in-lane targets within the intersection improved significantly over the performance of the base algorithms. The proposed algorithm was compared with human driving data, and it showed similar longitudinal acceleration. The motion predictor can be applied to path planners when AVs travel in unconstructed environments, such as multi-lane turn intersections.<br /> <br /> == Future works ==<br /> 1.Developing trajectory prediction algorithms using other machine learning algorithms, such as attention-aware neural networks.<br /> <br /> 2.Applying the machine learning-based approach to infer lane change intention at motorways and main roads of urban environments.<br /> <br /> 3.Extending the target road of the trajectory predictor, such as roundabouts or uncontrolled intersections, to infer yield intention.<br /> <br /> 4.Learning the behavior of surrounding vehicles in real time while automated vehicles drive with real traffic.<br /> <br /> == Critiques ==<br /> The literature review is not sufficient. It should focus more on LSTM, RNN, and the study in different types of roads. Why the LSTM-RNN is used, and the background of the method is not stated clearly. There is a lack of concept so that it is difficult to distinguish between LSTM-RNN based motion predictor and motion planning.<br /> <br /> This is an interesting topic to discuss. This is a major topic for some famous vehicle company such as Tesla, Tesla nows already have a good service called Autopilot to give self-driving and Motion Prediction. This summary can include more diagrams in architecture in the model to give readers a whole view of how the model looks like. Since it is using LSTM-RNN, include some pictures of the LSTM-RNN will be great. I think it will be interesting to discuss more applications by using this method, such as Airplane, boats.<br /> <br /> Autonomous driving is a hot very topic, and training the model with LSTM-RNN is also a meaningful topic to discuss. By the way, it would be an interesting approach to compare the performance of different algorithms or some other traditional motion planning algorithms like KF.<br /> <br /> There are some papers that discussed the accuracy of different models in vehicle predictions, such as Deep Kinematic Models for Kinematically Feasible Vehicle Trajectory Predictions[https://arxiv.org/pdf/1908.00219.pdf.] The LSTM didn't show good performance. They increased the accuracy by combing LSTM with an unconstrained model(UM) by adding an additional LSTM layer of size 128 that is used to recursively output positions instead of simultaneously outputting positions for all horizons.<br /> <br /> It may be better to provide the results of experiments to support the efficiency of LSTM-RNN, talk about the prediction of training and test sets, and compared it with other autonomous driving systems that exist in the world.<br /> <br /> The topic of surround vehicle motion prediction is analogous to the topic of autonomous vehicles. An example of an application of these frameworks would be the transportation services industry. Many companies, such as Lyft and Uber, have started testing their own commercial autonomous vehicles.<br /> <br /> It would be really helpful if some visualization or data summary can be provided to understand the content, such as the track of the car movement.<br /> <br /> The model should have been tested in other regions besides just Seoul, as driving behaviors can vary drastically from region to region.<br /> <br /> Understandably, a supervised learning problem should be evaluated on some test dataset. However, supervised learning techniques are inherently ill-suited for general planning problems. The test dataset was obtained from human driving data which is known to be extremely noisy as well as unpredictable when it comes to motion planning. It would be crucial to determine the successes of this paper based on the state-of-the-art reinforcement learning techniques.<br /> <br /> It would be better if the authors compared their method against other SOTA methods. Also one of the reasons motion planning is done using interpretable methods rather than black boxes (such as this model) is because it is hard to see where things go wrong and fix problems with the black box when they occur - this is something the authors should have also discussed.<br /> <br /> == Reference ==<br />  E. Choi, Crash Factors in Intersection-Related Crashes: An On-Scene Perspective (No. Dot HS 811 366), U.S. DOT Nat. Highway Traffic Safety Admin., Washington, DC, USA, 2010.<br /> <br />  D. J. Phillips, T. A. Wheeler, and M. J. Kochenderfer, “Generalizable intention prediction of human drivers at intersections,” in Proc. IEEE Intell. Veh. Symp. (IV), Los Angeles, CA, USA, 2017, pp. 1665–1670.<br /> <br />  B. Kim, C. M. Kang, J. Kim, S. H. Lee, C. C. Chung, and J. W. Choi, “Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network,” in Proc. IEEE 20th Int. Conf. Intell. Transp. Syst. (ITSC), Yokohama, Japan, 2017, pp. 399–404.<br /> <br />  E. Strigel, D. Meissner, F. Seeliger, B. Wilking, and K. Dietmayer, “The Ko-PER intersection laserscanner and video dataset,” in Proc. 17th Int. IEEE Conf. Intell. Transp. Syst. (ITSC), Qingdao, China, 2014, pp. 1900–1901.<br /> <br />  Henggang Cui, Thi Nguyen, Fang-Chieh Chou, Tsung-Han Lin, Jeff Schneider, David Bradley, Nemanja Djuric: “Deep Kinematic Models for Kinematically Feasible Vehicle Trajectory Predictions”, 2019; [http://arxiv.org/abs/1908.00219 arXiv:1908.00219].<br /> <br /> Schulz, Jens &amp; Hubmann, Constantin &amp; Morin, Nikolai &amp; Löchner, Julian &amp; Burschka, Darius. (2019). Learning Interaction-Aware Probabilistic Driver Behavior Models from Urban Scenarios. 10.1109/IVS.2019.8814080.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Learning_for_Cardiologist-level_Myocardial_Infarction_Detection_in_Electrocardiograms&diff=43520 Deep Learning for Cardiologist-level Myocardial Infarction Detection in Electrocardiograms 2020-11-09T01:41:34Z <p>Z42qin: </p> <hr /> <div><br /> == Presented by ==<br /> <br /> Zihui (Betty) Qin, Wenqi (Maggie) Zhao, Muyuan Yang, Amartya (Marty) Mukherjee<br /> <br /> == Introduction ==<br /> <br /> This paper presents an approach to the detection of heart disease from ECG signals by fine-tuning the deep learning neural network, ConvNetQuake, in the area of scientific machine learning. A deep learning approach was used due to the model’s ability to be trained using multiple GPUs and terabyte-sized datasets. This, in turn, creates a model that is robust against noise. The purpose of this paper is to provide detailed analyses of the contributions of the ECG leads on identifying heart disease, to show the use of multiple channels in ConvNetQuake enhances prediction accuracy, and to show that feature engineering is not necessary for any of the training, validation, or testing processes.<br /> <br /> == Previous Work and Motivation ==<br /> <br /> The database used in previous works is the Physikalisch-Technische Bundesanstalt (PTB) database, which consists of ECG records. Previous papers used techniques, such as CNN, SVM, K-nearest neighbours, naïve Bayes classification, and ANN. From these instances, the paper observes several faults in the previous papers. The first being the issue that most papers use feature selection on the raw ECG data before training the model. Dabanloo, and Attarodi  used various techniques such as ANN, K-nearest neighbours, and Naïve Bayes. However, they extracted two features, the T-wave integral and the total integral, to aid in localizing and detecting heart disease. Sharma and Sunkaria  used SVM and K-nearest neighbours as their classifier, but extracted various features using stationary wavelet transforms to decompose the ECG signal into sub-bands. The second issue is that papers that do not use feature selection would arbitrarily pick ECG leads for classification without rationale. For example, Liu et al.  used a deep CNN that uses 3 seconds of ECG signal from lead II at a time as input. The decision for using lead II compared to the other leads was not explained. <br /> <br /> The issue with feature selection is that it can be time-consuming and impractical with large volumes of data. The second issue with the arbitrary selection of leads is that it does not offer insight into why the lead was chosen and the contributions of each lead in the identification of heart disease. Thus, this paper addresses these two issues through implementing a deep learning model that does not rely on feature selection of ECG data and to quantify the contributions of each ECG and Frank lead in identifying heart disease.<br /> <br /> == Model Architecture ==<br /> <br /> The dataset consists of 549 ECG records taken from 290 unique patients. Each ECG record has a mean length of over 100 seconds.<br /> <br /> This Deep Neural Network model was created by modifying the ConvNetQuake model by adding 1D batch normalization layers.<br /> <br /> The input layer is a 10-second long ECG signal. There are 8 hidden layers in this model, each of which consists of a 1D convolution layer with the ReLu activation function followed by a batch normalization layer. The output layer is a one-dimensional layer that uses the Sigmoid activation function.<br /> <br /> This model is trained by using batches of size 10. The learning rate is 10^-4. The ADAM optimizer is used. In training the model, the dataset is split into a train set, validation set and test set with ratios 80-10-10.<br /> <br /> ==Result== <br /> <br /> The paper first uses quantification of accuracies for single channels with 20-fold cross-validation, resulting highest individual accuracies: v5, v6, vx, vz, and ii. The researcher further investigated the accuracies for pairs of top 5 highest individual channels using 20-fold cross-validation. The arrived at the conclusion of highest pairs accuracies to fed into a the neural network is lead v6 and lead vz. They then use 100-fold cross validation on v6 and vz pair of channels, then compare outliers based on top 20, top 50 and total 100 performing models, finding that standard deviation is non-trivial and there are few models performed very poorly. <br /> <br /> Next, they discussed 2 factors effecting model performance evaluation: 1） Random train-val-test split might have effects of the performance of the model, but it can be improved by access with a larger data set and further discussion; and 2） random initialization of the weights of neural network shows little effects on the performance of the model performance evaluation, because of showing a high average results with a fixed train-val-test split. <br /> <br /> Comparing with other models in other 12 papers, the model in this article has the highest accuracy, specificity, and precision. With concerns of patients' records effecting the training accuracy, they used 290 fold patient-wise split, resulting the same highest accuracy of the pair v6 and vz same as record-wise split. Even though the patient-wise split might result lower accuracy evaluation, however, it still maintain an high average of 97.83%. <br /> <br /> ==Discussion &amp; Conclusion== <br /> <br /> The paper introduced a new architecture for heart condition classification based on raw ECG signals using multiple leads. It outperformed the state-of-art model by a large margin of 1 percent. This study finds that out of the 15 ECG channels(12 conventional ECG leads and 3 Frank Leads), channel v6, vz and ii contain the most meaningful information for detecting myocardial infraction. Also, recent advances in machine learning can be leveraged to produce a model capable of classifying myocardial infraction with a cardiologist-level success rate. To further improve the performance of the models, access to larger labelled data set is needed. The PTB database is small so it is difficult to test the true robustness of the model with a relatively small test set. If a larger data set can be found to help correctly identify other heart conditions beyond myocardial infraction, the research group plans to share the deep learning models and develop an open source, computationally efficient app that can be readily used by cardiologists.<br /> <br /> A detailed analysis of the relative importance of each of the standard 15 ECG channels indicates that deep learning can identify myocardial infraction by processing only ten seconds of raw ECG data from the v6, vz and ii leads and reaches cardiologist-level success rate. Deep learning algorithms may be readily used as commodity software. Neural network model that was originally designed to identify earthquakes may be re-designed and tuned to identify myocardial infraction. Feature engineering of ECG data is not required to identify myocardial infraction in the PTB database. This model only required ten seconds of raw ECG data to identify this heart condition with cardiologist-level performance. Access to larger database should be provided to deep learning researchers so they can work on detecting different types of heart conditions. Deep learning researchers and cardiology community can work together to develop deep learning algorithms that provides trustworthy, real-time information regarding heart conditions with minimal computational resources.</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=stat441F21&diff=43502 stat441F21 2020-11-08T23:44:21Z <p>Z42qin: /* Paper presentation */</p> <hr /> <div><br /> <br /> == [[F20-STAT 441/841 CM 763-Proposal| Project Proposal ]] ==<br /> <br /> &lt;!--[https://goo.gl/forms/apurag4dr9kSR76X2 Your feedback on presentations]--&gt;<br /> <br /> = Record your contributions here [https://docs.google.com/spreadsheets/d/10CHiJpAylR6kB9QLqN7lZHN79D9YEEW6CDTH27eAhbQ/edit?usp=sharing]=<br /> <br /> Use the following notations:<br /> <br /> P: You have written a summary/critique on the paper.<br /> <br /> T: You had a technical contribution on a paper (excluding the paper that you present).<br /> <br /> E: You had an editorial contribution on a paper (excluding the paper that you present).<br /> <br /> =Paper presentation=<br /> {| class=&quot;wikitable&quot;<br /> <br /> {| border=&quot;1&quot; cellpadding=&quot;3&quot;<br /> |-<br /> |width=&quot;60pt&quot;|Date<br /> |width=&quot;250pt&quot;|Name <br /> |width=&quot;15pt&quot;|Paper number <br /> |width=&quot;700pt&quot;|Title<br /> |width=&quot;15pt&quot;|Link to the paper<br /> |width=&quot;30pt&quot;|Link to the summary<br /> |width=&quot;30pt&quot;|Link to the video<br /> |-<br /> |Sep 15 (example)||Ri Wang || ||Sequence to sequence learning with neural networks.||[http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf Paper] || [https://wiki.math.uwaterloo.ca/statwiki/index.php?title=Going_Deeper_with_Convolutions Summary] || [https://youtu.be/JWozRg_X-Vg?list=PLehuLRPyt1HzXDemu7K4ETcF0Ld_B5adG&amp;t=539]<br /> |-<br /> |Week of Nov 16 ||Sharman Bharat, Li Dylan,Lu Leonie, Li Mingdao || 1|| Risk prediction in life insurance industry using supervised learning algorithms || [https://rdcu.be/b780J Paper] ||[https://wiki.math.uwaterloo.ca/statwiki/index.php?title=User:Bsharman Summary] ||<br /> [https://www.youtube.com/watch?v=TVLpSFYgF0c&amp;feature=youtu.be]<br /> |-<br /> |Week of Nov 16 || Delaney Smith, Mohammad Assem Mahmoud || 2|| Influenza Forecasting Framework based on Gaussian Processes || [https://proceedings.icml.cc/static/paper_files/icml/2020/1239-Paper.pdf] paper || ||<br /> |-<br /> |Week of Nov 16 || Tatianna Krikella, Swaleh Hussain, Grace Tompkins || 3|| Processing of Missing Data by Neural Networks || [http://papers.nips.cc/paper/7537-processing-of-missing-data-by-neural-networks.pdf Paper] || [https://wiki.math.uwaterloo.ca/statwiki/index.php?title=User:Gtompkin Summary] ||<br /> |-<br /> |Week of Nov 16 ||Jonathan Chow, Nyle Dharani, Ildar Nasirov ||4 ||Streaming Bayesian Inference for Crowdsourced Classification ||[https://papers.nips.cc/paper/9439-streaming-bayesian-inference-for-crowdsourced-classification.pdf Paper] || ||<br /> |-<br /> |Week of Nov 16 || Matthew Hall, Johnathan Chalaturnyk || 5|| Neural Ordinary Differential Equations || [https://papers.nips.cc/paper/7892-neural-ordinary-differential-equations.pdf] || ||<br /> |-<br /> |Week of Nov 16 || Luwen Chang, Qingyang Yu, Tao Kong, Tianrong Sun || 6|| Adversarial Attacks on Copyright Detection Systems || Paper [https://proceedings.icml.cc/static/paper_files/icml/2020/1894-Paper.pdf] || ||<br /> |-<br /> |Week of Nov 16 || Casey De Vera, Solaiman Jawad, Jihoon Han || 7|| || || ||<br /> |-<br /> |Week of Nov 16 || Yuxin Wang, Evan Peters, Cynthia Mou, Sangeeth Kalaichanthiran || 8|| Uniform convergence may be unable to explain generalization in deep learning || [https://papers.nips.cc/paper/9336-uniform-convergence-may-be-unable-to-explain-generalization-in-deep-learning.pdf] || ||<br /> |-<br /> |Week of Nov 16 || Yuchuan Wu || 9|| || || ||<br /> |-<br /> |Week of Nov 16 || Zhou Zeping, Siqi Li, Yuqin Fang, Fu Rao || 10|| The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network || [http://people.cs.uchicago.edu/~pworah/rmt2.pdf] || ||<br /> |-<br /> |Week of Nov 23 ||Jinjiang Lian, Jiawen Hou, Yisheng Zhu, Mingzhe Huang || 11|| DROCC: Deep Robust One-Class Classification || [https://proceedings.icml.cc/static/paper_files/icml/2020/6556-Paper.pdf paper] || ||<br /> |-<br /> |Week of Nov 23 || Bushra Haque, Hayden Jones, Michael Leung, Cristian Mustatea || 12|| Combine Convolution with Recurrent Networks for Text Classification || [https://arxiv.org/pdf/2006.15795.pdf Paper] || ||<br /> |-<br /> |Week of Nov 23 || Taohao Wang, Zeren Shen, Zihao Guo, Rui Chen || 13|| Deep multiple instance learning for image classification and auto-annotation || [https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Wu_Deep_Multiple_Instance_2015_CVPR_paper.pdf paper] || ||<br /> |-<br /> |Week of Nov 23 || Qianlin Song, William Loh, Junyue Bai, Phoebe Choi || 14|| Task Understanding from Confusing Multi-task Data || [https://proceedings.icml.cc/static/paper_files/icml/2020/578-Paper.pdf Paper] || [https://wiki.math.uwaterloo.ca/statwiki/index.php?title=Task_Understanding_from_Confusing_Multi-task_Data Summary] ||<br /> |-<br /> |Week of Nov 23 || Rui Gong, Xuetong Wang, Xinqi Ling, Di Ma || 15|| Semantic Relation Classification via Convolution Neural Network|| [https://www.aclweb.org/anthology/S18-1127.pdf paper] || ||<br /> |-<br /> |Week of Nov 23 || Xiaolan Xu, Robin Wen, Yue Weng, Beizhen Chang || 16|| Graph Structure of Neural Networks || [https://proceedings.icml.cc/paper/2020/file/757b505cfd34c64c85ca5b5690ee5293-Paper.pdf Paper] || ||<br /> |-<br /> |Week of Nov 23 ||Hansa Halim, Sanjana Rajendra Naik, Samka Marfua, Shawrupa Proshasty || 17|| Superhuman AI for multiplayer poker || [https://www.cs.cmu.edu/~noamb/papers/19-Science-Superhuman.pdf Paper] || ||<br /> |-<br /> |Week of Nov 23 ||Guanting Pan, Haocheng Chang, Zaiwei Zhang || 18|| Point-of-Interest Recommendation: Exploiting Self-Attentive Autoencoders with Neighbor-Aware Influence || [https://arxiv.org/pdf/1809.10770.pdf Paper] || ||<br /> |-<br /> |Week of Nov 23 || Jerry Huang, Daniel Jiang, Minyan Dai || 19|| Neural Speed Reading Via Skim-RNN ||[https://arxiv.org/pdf/1711.02085.pdf?fbclid=IwAR3EeFsKM_b5p9Ox7X9mH-1oI3U3oOKPBy3xUOBN0XvJa7QW2ZeJJ9ypQVo Paper] || [https://wiki.math.uwaterloo.ca/statwiki/index.php?title=Neural_Speed_Reading_via_Skim-RNN Summary]||<br /> |-<br /> |Week of Nov 23 ||Ruixian Chin, Yan Kai Tan, Jason Ong, Wen Cheen Chiew || 20|| DivideMix: Learning with Noisy Labels as Semi-supervised Learning || [https://openreview.net/pdf?id=HJgExaVtwr] || ||<br /> |-<br /> |Week of Nov 30 || Banno Dion, Battista Joseph, Kahn Solomon || 21|| Music Recommender System Based on Genre using Convolutional Recurrent Neural Networks || [https://www.sciencedirect.com/science/article/pii/S1877050919310646] || ||<br /> |-<br /> |Week of Nov 30 || Sai Arvind Budaraju, Isaac Ellmen, Dorsa Mohammadrezaei, Emilee Carson || 22|| A universal SNP and small-indel variant caller using deep neural networks||[https://www.nature.com/articles/nbt.4235.epdf?author_access_token=q4ZmzqvvcGBqTuKyKgYrQ9RgN0jAjWel9jnR3ZoTv0NuM3saQzpZk8yexjfPUhdFj4zyaA4Yvq0LWBoCYQ4B9vqPuv8e2HHy4vShDgEs8YxI_hLs9ov6Y1f_4fyS7kGZ Paper] || ||<br /> |-<br /> |Week of Nov 30 || Daniel Fagan, Cooper Brooke, Maya Perelman || 23|| Efficient kNN Classification With Different Number of Nearest Neighbors || [https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=7898482 Paper] || ||<br /> |-<br /> |Week of Nov 30 || Karam Abuaisha, Evan Li, Jason Pu, Nicholas Vadivelu || 24|| Being Bayesian about Categorical Probability || [https://proceedings.icml.cc/static/paper_files/icml/2020/3560-Paper.pdf Paper] || ||<br /> |-<br /> |Week of Nov 30 || Anas Mahdi Will Thibault Jan Lau Jiwon Yang || 25|| Loss Function Search for Face Recognition<br /> || [https://proceedings.icml.cc/static/paper_files/icml/2020/245-Paper.pdf] paper || ||<br /> |-<br /> |Week of Nov 30 ||Zihui (Betty) Qin, Wenqi (Maggie) Zhao, Muyuan Yang, Amartya (Marty) Mukherjee || 26|| Deep Learning for Cardiologist-level Myocardial Infarction Detection in Electrocardiograms || [https://arxiv.org/pdf/1912.07618.pdf?fbclid=IwAR0RwATSn4CiT3qD9LuywYAbJVw8YB3nbex8Kl19OCExIa4jzWaUut3oVB0 Paper] || Summary [https://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Learning_for_Cardiologist-level_Myocardial_Infarction_Detection_in_Electrocardiograms&amp;fbclid=IwAR1Tad2DAM7LT6NXXuSYDZtHHBvN0mjZtDdCOiUFFq_XwVcQxG3hU-3XcaE] ||<br /> |-<br /> |Week of Nov 30 || Stan Lee, Seokho Lim, Kyle Jung, Daehyun Kim || 27|| Bag of Tricks for Efficient Text Classification || [https://arxiv.org/pdf/1607.01759.pdf paper] || ||<br /> |-<br /> |Week of Nov 30 || Yawen Wang, Danmeng Cui, ZiJie Jiang, Mingkang Jiang, Haotian Ren, Haris Bin Zahid || 28|| A Brief Survey of Text Mining: Classification, Clustering and Extraction Techniques || [https://arxiv.org/pdf/1707.02919.pdf Paper] || ||<br /> |-<br /> |Week of Nov 30 || Qing Guo, XueGuang Ma, James Ni, Yuanxin Wang || 29|| Mask R-CNN || [https://arxiv.org/pdf/1703.06870.pdf Paper] || ||<br /> |-<br /> |Week of Nov 30 || Bertrand Sodjahin, Junyi Yang, Jill Yu Chieh Wang, Yu Min Wu, Calvin Li || 30|| Research paper classifcation systems based on TF‑IDF and LDA schemes || [https://hcis-journal.springeropen.com/articles/10.1186/s13673-019-0192-7?fbclid=IwAR3swO-eFrEbj1BUQfmomJazxxeFR6SPgr6gKayhs38Y7aBG-zX1G3XWYRM Paper] || ||<br /> |-<br /> |Week of Nov 30 || Daniel Zhang, Jacky Yao, Scholar Sun, Russell Parco, Ian Cheung || 31 || Speech2Face: Learning the Face Behind a Voice || [https://arxiv.org/pdf/1905.09773.pdf?utm_source=thenewstack&amp;utm_medium=website&amp;utm_campaign=platform Paper] || ||<br /> |-<br /> |Week of Nov 30 || Siyuan Xia, Jiaxiang Liu, Jiabao Dong, Yipeng Du || 32 || Evaluating Machine Accuracy on ImageNet || [https://proceedings.icml.cc/static/paper_files/icml/2020/6173-Paper.pdf] || ||<br /> |-<br /> |Week of Nov 30 || Msuhi Wang, Siyuan Qiu, Yan Yu || 33 || Surround Vehicle Motion Prediction Using LSTM-RNN for Motion Planning of Autonomous Vehicles at Multi-Lane Turn Intersections || [https://ieeexplore.ieee.org/abstract/document/8957421 paper] || ||</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=stat441F21&diff=43501 stat441F21 2020-11-08T23:43:16Z <p>Z42qin: /* Paper presentation */</p> <hr /> <div><br /> <br /> == [[F20-STAT 441/841 CM 763-Proposal| Project Proposal ]] ==<br /> <br /> &lt;!--[https://goo.gl/forms/apurag4dr9kSR76X2 Your feedback on presentations]--&gt;<br /> <br /> = Record your contributions here [https://docs.google.com/spreadsheets/d/10CHiJpAylR6kB9QLqN7lZHN79D9YEEW6CDTH27eAhbQ/edit?usp=sharing]=<br /> <br /> Use the following notations:<br /> <br /> P: You have written a summary/critique on the paper.<br /> <br /> T: You had a technical contribution on a paper (excluding the paper that you present).<br /> <br /> E: You had an editorial contribution on a paper (excluding the paper that you present).<br /> <br /> =Paper presentation=<br /> {| class=&quot;wikitable&quot;<br /> <br /> {| border=&quot;1&quot; cellpadding=&quot;3&quot;<br /> |-<br /> |width=&quot;60pt&quot;|Date<br /> |width=&quot;250pt&quot;|Name <br /> |width=&quot;15pt&quot;|Paper number <br /> |width=&quot;700pt&quot;|Title<br /> |width=&quot;15pt&quot;|Link to the paper<br /> |width=&quot;30pt&quot;|Link to the summary<br /> |width=&quot;30pt&quot;|Link to the video<br /> |-<br /> |Sep 15 (example)||Ri Wang || ||Sequence to sequence learning with neural networks.||[http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf Paper] || [https://wiki.math.uwaterloo.ca/statwiki/index.php?title=Going_Deeper_with_Convolutions Summary] || [https://youtu.be/JWozRg_X-Vg?list=PLehuLRPyt1HzXDemu7K4ETcF0Ld_B5adG&amp;t=539]<br /> |-<br /> |Week of Nov 16 ||Sharman Bharat, Li Dylan,Lu Leonie, Li Mingdao || 1|| Risk prediction in life insurance industry using supervised learning algorithms || [https://rdcu.be/b780J Paper] ||[https://wiki.math.uwaterloo.ca/statwiki/index.php?title=User:Bsharman Summary] ||<br /> [https://www.youtube.com/watch?v=TVLpSFYgF0c&amp;feature=youtu.be]<br /> |-<br /> |Week of Nov 16 || Delaney Smith, Mohammad Assem Mahmoud || 2|| Influenza Forecasting Framework based on Gaussian Processes || [https://proceedings.icml.cc/static/paper_files/icml/2020/1239-Paper.pdf] paper || ||<br /> |-<br /> |Week of Nov 16 || Tatianna Krikella, Swaleh Hussain, Grace Tompkins || 3|| Processing of Missing Data by Neural Networks || [http://papers.nips.cc/paper/7537-processing-of-missing-data-by-neural-networks.pdf Paper] || [https://wiki.math.uwaterloo.ca/statwiki/index.php?title=User:Gtompkin Summary] ||<br /> |-<br /> |Week of Nov 16 ||Jonathan Chow, Nyle Dharani, Ildar Nasirov ||4 ||Streaming Bayesian Inference for Crowdsourced Classification ||[https://papers.nips.cc/paper/9439-streaming-bayesian-inference-for-crowdsourced-classification.pdf Paper] || ||<br /> |-<br /> |Week of Nov 16 || Matthew Hall, Johnathan Chalaturnyk || 5|| Neural Ordinary Differential Equations || [https://papers.nips.cc/paper/7892-neural-ordinary-differential-equations.pdf] || ||<br /> |-<br /> |Week of Nov 16 || Luwen Chang, Qingyang Yu, Tao Kong, Tianrong Sun || 6|| Adversarial Attacks on Copyright Detection Systems || Paper [https://proceedings.icml.cc/static/paper_files/icml/2020/1894-Paper.pdf] || ||<br /> |-<br /> |Week of Nov 16 || Casey De Vera, Solaiman Jawad, Jihoon Han || 7|| || || ||<br /> |-<br /> |Week of Nov 16 || Yuxin Wang, Evan Peters, Cynthia Mou, Sangeeth Kalaichanthiran || 8|| Uniform convergence may be unable to explain generalization in deep learning || [https://papers.nips.cc/paper/9336-uniform-convergence-may-be-unable-to-explain-generalization-in-deep-learning.pdf] || ||<br /> |-<br /> |Week of Nov 16 || Yuchuan Wu || 9|| || || ||<br /> |-<br /> |Week of Nov 16 || Zhou Zeping, Siqi Li, Yuqin Fang, Fu Rao || 10|| The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network || [http://people.cs.uchicago.edu/~pworah/rmt2.pdf] || ||<br /> |-<br /> |Week of Nov 23 ||Jinjiang Lian, Jiawen Hou, Yisheng Zhu, Mingzhe Huang || 11|| DROCC: Deep Robust One-Class Classification || [https://proceedings.icml.cc/static/paper_files/icml/2020/6556-Paper.pdf paper] || ||<br /> |-<br /> |Week of Nov 23 || Bushra Haque, Hayden Jones, Michael Leung, Cristian Mustatea || 12|| Combine Convolution with Recurrent Networks for Text Classification || [https://arxiv.org/pdf/2006.15795.pdf Paper] || ||<br /> |-<br /> |Week of Nov 23 || Taohao Wang, Zeren Shen, Zihao Guo, Rui Chen || 13|| Deep multiple instance learning for image classification and auto-annotation || [https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Wu_Deep_Multiple_Instance_2015_CVPR_paper.pdf paper] || ||<br /> |-<br /> |Week of Nov 23 || Qianlin Song, William Loh, Junyue Bai, Phoebe Choi || 14|| Task Understanding from Confusing Multi-task Data || [https://proceedings.icml.cc/static/paper_files/icml/2020/578-Paper.pdf Paper] || [https://wiki.math.uwaterloo.ca/statwiki/index.php?title=Task_Understanding_from_Confusing_Multi-task_Data Summary] ||<br /> |-<br /> |Week of Nov 23 || Rui Gong, Xuetong Wang, Xinqi Ling, Di Ma || 15|| Semantic Relation Classification via Convolution Neural Network|| [https://www.aclweb.org/anthology/S18-1127.pdf paper] || ||<br /> |-<br /> |Week of Nov 23 || Xiaolan Xu, Robin Wen, Yue Weng, Beizhen Chang || 16|| Graph Structure of Neural Networks || [https://proceedings.icml.cc/paper/2020/file/757b505cfd34c64c85ca5b5690ee5293-Paper.pdf Paper] || ||<br /> |-<br /> |Week of Nov 23 ||Hansa Halim, Sanjana Rajendra Naik, Samka Marfua, Shawrupa Proshasty || 17|| Superhuman AI for multiplayer poker || [https://www.cs.cmu.edu/~noamb/papers/19-Science-Superhuman.pdf Paper] || ||<br /> |-<br /> |Week of Nov 23 ||Guanting Pan, Haocheng Chang, Zaiwei Zhang || 18|| Point-of-Interest Recommendation: Exploiting Self-Attentive Autoencoders with Neighbor-Aware Influence || [https://arxiv.org/pdf/1809.10770.pdf Paper] || ||<br /> |-<br /> |Week of Nov 23 || Jerry Huang, Daniel Jiang, Minyan Dai || 19|| Neural Speed Reading Via Skim-RNN ||[https://arxiv.org/pdf/1711.02085.pdf?fbclid=IwAR3EeFsKM_b5p9Ox7X9mH-1oI3U3oOKPBy3xUOBN0XvJa7QW2ZeJJ9ypQVo Paper] || [https://wiki.math.uwaterloo.ca/statwiki/index.php?title=Neural_Speed_Reading_via_Skim-RNN Summary]||<br /> |-<br /> |Week of Nov 23 ||Ruixian Chin, Yan Kai Tan, Jason Ong, Wen Cheen Chiew || 20|| DivideMix: Learning with Noisy Labels as Semi-supervised Learning || [https://openreview.net/pdf?id=HJgExaVtwr] || ||<br /> |-<br /> |Week of Nov 30 || Banno Dion, Battista Joseph, Kahn Solomon || 21|| Music Recommender System Based on Genre using Convolutional Recurrent Neural Networks || [https://www.sciencedirect.com/science/article/pii/S1877050919310646] || ||<br /> |-<br /> |Week of Nov 30 || Sai Arvind Budaraju, Isaac Ellmen, Dorsa Mohammadrezaei, Emilee Carson || 22|| A universal SNP and small-indel variant caller using deep neural networks||[https://www.nature.com/articles/nbt.4235.epdf?author_access_token=q4ZmzqvvcGBqTuKyKgYrQ9RgN0jAjWel9jnR3ZoTv0NuM3saQzpZk8yexjfPUhdFj4zyaA4Yvq0LWBoCYQ4B9vqPuv8e2HHy4vShDgEs8YxI_hLs9ov6Y1f_4fyS7kGZ Paper] || ||<br /> |-<br /> |Week of Nov 30 || Daniel Fagan, Cooper Brooke, Maya Perelman || 23|| Efficient kNN Classification With Different Number of Nearest Neighbors || [https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=7898482 Paper] || ||<br /> |-<br /> |Week of Nov 30 || Karam Abuaisha, Evan Li, Jason Pu, Nicholas Vadivelu || 24|| Being Bayesian about Categorical Probability || [https://proceedings.icml.cc/static/paper_files/icml/2020/3560-Paper.pdf Paper] || ||<br /> |-<br /> |Week of Nov 30 || Anas Mahdi Will Thibault Jan Lau Jiwon Yang || 25|| Loss Function Search for Face Recognition<br /> || [https://proceedings.icml.cc/static/paper_files/icml/2020/245-Paper.pdf] paper || ||<br /> |-<br /> |Week of Nov 30 ||Zihui (Betty) Qin, Wenqi (Maggie) Zhao, Muyuan Yang, Amartya (Marty) Mukherjee || 26|| Deep Learning for Cardiologist-level Myocardial Infarction Detection in Electrocardiograms || [https://arxiv.org/pdf/1912.07618.pdf?fbclid=IwAR0RwATSn4CiT3qD9LuywYAbJVw8YB3nbex8Kl19OCExIa4jzWaUut3oVB0 Paper] || [https://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Learning_for_Cardiologist-level_Myocardial_Infarction_Detection_in_Electrocardiograms&amp;fbclid=IwAR1Tad2DAM7LT6NXXuSYDZtHHBvN0mjZtDdCOiUFFq_XwVcQxG3hU-3XcaE] ||<br /> |-<br /> |Week of Nov 30 || Stan Lee, Seokho Lim, Kyle Jung, Daehyun Kim || 27|| Bag of Tricks for Efficient Text Classification || [https://arxiv.org/pdf/1607.01759.pdf paper] || ||<br /> |-<br /> |Week of Nov 30 || Yawen Wang, Danmeng Cui, ZiJie Jiang, Mingkang Jiang, Haotian Ren, Haris Bin Zahid || 28|| A Brief Survey of Text Mining: Classification, Clustering and Extraction Techniques || [https://arxiv.org/pdf/1707.02919.pdf Paper] || ||<br /> |-<br /> |Week of Nov 30 || Qing Guo, XueGuang Ma, James Ni, Yuanxin Wang || 29|| Mask R-CNN || [https://arxiv.org/pdf/1703.06870.pdf Paper] || ||<br /> |-<br /> |Week of Nov 30 || Bertrand Sodjahin, Junyi Yang, Jill Yu Chieh Wang, Yu Min Wu, Calvin Li || 30|| Research paper classifcation systems based on TF‑IDF and LDA schemes || [https://hcis-journal.springeropen.com/articles/10.1186/s13673-019-0192-7?fbclid=IwAR3swO-eFrEbj1BUQfmomJazxxeFR6SPgr6gKayhs38Y7aBG-zX1G3XWYRM Paper] || ||<br /> |-<br /> |Week of Nov 30 || Daniel Zhang, Jacky Yao, Scholar Sun, Russell Parco, Ian Cheung || 31 || Speech2Face: Learning the Face Behind a Voice || [https://arxiv.org/pdf/1905.09773.pdf?utm_source=thenewstack&amp;utm_medium=website&amp;utm_campaign=platform Paper] || ||<br /> |-<br /> |Week of Nov 30 || Siyuan Xia, Jiaxiang Liu, Jiabao Dong, Yipeng Du || 32 || Evaluating Machine Accuracy on ImageNet || [https://proceedings.icml.cc/static/paper_files/icml/2020/6173-Paper.pdf] || ||<br /> |-<br /> |Week of Nov 30 || Msuhi Wang, Siyuan Qiu, Yan Yu || 33 || Surround Vehicle Motion Prediction Using LSTM-RNN for Motion Planning of Autonomous Vehicles at Multi-Lane Turn Intersections || [https://ieeexplore.ieee.org/abstract/document/8957421 paper] || ||</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Learning_for_Cardiologist-level_Myocardial_Infarction_Detection_in_Electrocardiograms&diff=42830 Deep Learning for Cardiologist-level Myocardial Infarction Detection in Electrocardiograms 2020-10-26T00:26:32Z <p>Z42qin: </p> <hr /> <div><br /> == Presented by ==<br /> <br /> Zihui (Betty) Qin, Wenqi (Maggie) Zhao, Muyuan Zhao, Amartya (Marty) Mukherjee<br /> <br /> == Maggie ==<br /> <br /> Introduction<br /> <br />  Problem: To detect risk of heart disease from ECG signals.<br /> <br />  Importance: To provide a correct diagnosis so that proper health care can be given to patients.<br /> <br />  Area: Deep learning – scientific machine learning<br /> <br /> o Concerned about design, training, and use of ML algorithm in an optimal matter towards a certain problem.<br /> <br />  Deep learning model benefits:<br /> <br /> o Uses multiple GPUs to construct complicated NN that can be trained using TB-sized datasets, which are robust against noise.<br /> <br /> o After training is complete, it takes little computational power to conduct statistical inference.<br /> <br />  Question that need to be answered:<br /> <br /> o What AI architecture and datasets can provide evidence of symptoms, ECG changes, imaging evidence that shows loss of viable myocardium and wall motion abnormality?<br /> <br />  Purpose of the article:<br /> <br /> o To provide a detailed analysis on the contribution of each ECG lead in identifying heart disease in the model. This is because the selection of data in previous studies regarding heart disease made data selection arbitrary in identifying heart conditions.<br /> <br /> o To show the use of using multiple data channels of information to enhance prediction accuracy in deep learning, i.e. processing the top three ECG leads simultaneously in the neural network.<br /> <br /> o Show that feature engineering is not necessary in the training, validation, or testing process for ECG data in neural networks.<br /> <br /> Related work<br /> <br />  Database used: PTB database<br /> <br /> 1. CNN Network<br /> <br /> a. Used both noisy and denoised ECGs without feature engineering.<br /> <br /> 2. Used artificial neural network, probabilistic neural network, KNN, multi-layer perceptron, and Naïve Bayes Classification<br /> <br /> a. Extracted two features: T-wave integral and total integral to identify heart disease.<br /> <br /> 3. Developed two different ANN: RBF and MLP<br /> <br /> 4. Supervised learning techniques have limited success to the problem and used multiple instance learning.<br /> <br /> a. Demonstrate proposed algorithm LTMIL surpasses supervised approaches.<br /> <br /> 5. Create new feature by approximating ECG signal using a 20th order polynomial, which achieved 94.4% accuracy.<br /> <br /> 6. Stationary wavelet transforms to decompose ECG into sub-bands.<br /> <br /> a. SVM and KNN used to classify.<br /> <br /> b. Features used: sample entropy, normalized sub-bands, log energy entropy, median slope from sub-bands.<br /> <br /> 7. Transfer learning – used deep CNN model for arrhythmia and developed it to detect heart disease.<br /> <br /> 8. Simple adaptive threshold (SAT)<br /> <br /> a. Multiresolution approach, adaptive thresholding used to extract features: depth of Q peak and elevation in ST segment<br /> <br /> 9. Subject-oriented approach using CNN to take in leads II, III, AVF<br /> <br /> 10. Model ECG using 2nd order ODE and feed the best-fitting coefficients of the ECG signal into a SVM.<br /> <br /> 11. Multi-channel CNN (16 layers) with long-short memory units<br /> <br /> 12. Deep CNN that takes 3 seconds at a time of lead II as input<br /> <br />  Most use feature extraction/selection from the raw ECG data before training.<br /> <br /> o Problem with feature selection is that it is not practical for large volumes of data.<br /> <br />  Other papers that do not use feature selection arbitrarily picks ECG leads for classification and does not provide rationale.<br /> <br /> ==Result== <br /> <br /> 1. Quantification of accuracies for single channels with 20-fold cross-validation, resulting highest individual accuracies: v5, v6, vx, vz, and ii<br /> <br /> 2. Quantification of accuracies for pairs of top 5 highest individual channels with 20-fold cross-validation, resulting highest pairs accuracies to fed into a the neural network: lead v6 and lead vz <br /> <br /> 3. Use 100-fold cross validation on v6 and vz pair of channels, then compare outliers based on top 20, top 50 and total 100 performing models, finding that standard deviation is non-trivial and there are few models performed very poorly.<br /> <br /> 4. Discussing 2 factors effecting model performance evaluation:<br /> <br /> 1） Random train-val-test split might have effects of the performance of the model, but it can be improved by access with a larger data set and further discussion <br /> <br /> 2） random initialization of the weights of neural network shows little effects on the performance of the model performance evaluation, because of showing a high average results with a fixed train-val-test split <br /> <br /> 5. Comparing with other models in other 12 papers, the model in this article has the highest accuracy, specificity, and precision<br /> <br /> 6. Further using 290 fold patient-wise split, resulting the same highest accuracy of the pair v6 and vz as record-wise split <br /> <br /> 1） Discuss patient-wise split might result lower accuracy evaluation, however, it still maintain an average of 97.83%<br /> <br /> == Marty ==<br /> <br /> 3.1: Data curation<br /> <br /> Dataset: 549 ECG records total<br /> <br /> 290 unique patients<br /> <br /> Each ECG record has a mean length of over 100s<br /> <br /> 3.2: ANN model<br /> <br /> ConvNetQuake model + 1D batch normalization + Label-smoothing<br /> <br /> Model (PyTorch):<br /> <br /> - Input layer: 10-second long ECG signal<br /> <br /> - Hidden layers: 8 * (1D convolution layer, Activation function: RELU, 1D batch normalization layer)<br /> <br /> - Output layer: 1280 dimensions -&gt; 1 dimension, Activation function: Sigmoid<br /> <br /> Batch size = 10<br /> <br /> Learning rate = 10^-4<br /> <br /> Optimizer = ADAM<br /> <br /> 80-10-10: Train-Validation-Test</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Learning_for_Cardiologist-level_Myocardial_Infarction_Detection_in_Electrocardiograms&diff=42828 Deep Learning for Cardiologist-level Myocardial Infarction Detection in Electrocardiograms 2020-10-26T00:24:17Z <p>Z42qin: /* Betty */</p> <hr /> <div><br /> == Maggie ==<br /> <br /> Introduction<br /> <br />  Problem: To detect risk of heart disease from ECG signals.<br /> <br />  Importance: To provide a correct diagnosis so that proper health care can be given to patients.<br /> <br />  Area: Deep learning – scientific machine learning<br /> <br /> o Concerned about design, training, and use of ML algorithm in an optimal matter towards a certain problem.<br /> <br />  Deep learning model benefits:<br /> <br /> o Uses multiple GPUs to construct complicated NN that can be trained using TB-sized datasets, which are robust against noise.<br /> <br /> o After training is complete, it takes little computational power to conduct statistical inference.<br /> <br />  Question that need to be answered:<br /> <br /> o What AI architecture and datasets can provide evidence of symptoms, ECG changes, imaging evidence that shows loss of viable myocardium and wall motion abnormality?<br /> <br />  Purpose of the article:<br /> <br /> o To provide a detailed analysis on the contribution of each ECG lead in identifying heart disease in the model. This is because the selection of data in previous studies regarding heart disease made data selection arbitrary in identifying heart conditions.<br /> <br /> o To show the use of using multiple data channels of information to enhance prediction accuracy in deep learning, i.e. processing the top three ECG leads simultaneously in the neural network.<br /> <br /> o Show that feature engineering is not necessary in the training, validation, or testing process for ECG data in neural networks.<br /> <br /> Related work<br /> <br />  Database used: PTB database<br /> <br /> 1. CNN Network<br /> <br /> a. Used both noisy and denoised ECGs without feature engineering.<br /> <br /> 2. Used artificial neural network, probabilistic neural network, KNN, multi-layer perceptron, and Naïve Bayes Classification<br /> <br /> a. Extracted two features: T-wave integral and total integral to identify heart disease.<br /> <br /> 3. Developed two different ANN: RBF and MLP<br /> <br /> 4. Supervised learning techniques have limited success to the problem and used multiple instance learning.<br /> <br /> a. Demonstrate proposed algorithm LTMIL surpasses supervised approaches.<br /> <br /> 5. Create new feature by approximating ECG signal using a 20th order polynomial, which achieved 94.4% accuracy.<br /> <br /> 6. Stationary wavelet transforms to decompose ECG into sub-bands.<br /> <br /> a. SVM and KNN used to classify.<br /> <br /> b. Features used: sample entropy, normalized sub-bands, log energy entropy, median slope from sub-bands.<br /> <br /> 7. Transfer learning – used deep CNN model for arrhythmia and developed it to detect heart disease.<br /> <br /> 8. Simple adaptive threshold (SAT)<br /> <br /> a. Multiresolution approach, adaptive thresholding used to extract features: depth of Q peak and elevation in ST segment<br /> <br /> 9. Subject-oriented approach using CNN to take in leads II, III, AVF<br /> <br /> 10. Model ECG using 2nd order ODE and feed the best-fitting coefficients of the ECG signal into a SVM.<br /> <br /> 11. Multi-channel CNN (16 layers) with long-short memory units<br /> <br /> 12. Deep CNN that takes 3 seconds at a time of lead II as input<br /> <br />  Most use feature extraction/selection from the raw ECG data before training.<br /> <br /> o Problem with feature selection is that it is not practical for large volumes of data.<br /> <br />  Other papers that do not use feature selection arbitrarily picks ECG leads for classification and does not provide rationale.<br /> <br /> ==Betty==<br /> <br /> Result: <br /> <br /> 1. Quantification of accuracies for single channels with 20-fold cross-validation, resulting highest individual accuracies: v5, v6, vx, vz, and ii<br /> <br /> 2. Quantification of accuracies for pairs of top 5 highest individual channels with 20-fold cross-validation, resulting highest pairs accuracies to fed into a the neural network: lead v6 and lead vz <br /> <br /> 3. Use 100-fold cross validation on v6 and vz pair of channels, then compare outliers based on top 20, top 50 and total 100 performing models, finding that standard deviation is non-trivial and there are few models performed very poorly.<br /> <br /> 4. Discussing 2 factors effecting model performance evaluation:<br /> <br /> 1） Random train-val-test split might have effects of the performance of the model, but it can be improved by access with a larger data set and further discussion <br /> <br /> 2） random initialization of the weights of neural network shows little effects on the performance of the model performance evaluation, because of showing a high average results with a fixed train-val-test split <br /> <br /> 5. Comparing with other models in other 12 papers, the model in this article has the highest accuracy, specificity, and precision<br /> <br /> 6. Further using 290 fold patient-wise split, resulting the same highest accuracy of the pair v6 and vz as record-wise split <br /> <br /> 1） Discuss patient-wise split might result lower accuracy evaluation, however, it still maintain an average of 97.83%<br /> <br /> == Marty ==<br /> <br /> 3.1: Data curation<br /> <br /> Dataset: 549 ECG records total<br /> <br /> 290 unique patients<br /> <br /> Each ECG record has a mean length of over 100s<br /> <br /> 3.2: ANN model<br /> <br /> ConvNetQuake model + 1D batch normalization + Label-smoothing<br /> <br /> Model (PyTorch):<br /> <br /> - Input layer: 10-second long ECG signal<br /> <br /> - Hidden layers: 8 * (1D convolution layer, Activation function: RELU, 1D batch normalization layer)<br /> <br /> - Output layer: 1280 dimensions -&gt; 1 dimension, Activation function: Sigmoid<br /> <br /> Batch size = 10<br /> <br /> Learning rate = 10^-4<br /> <br /> Optimizer = ADAM<br /> <br /> 80-10-10: Train-Validation-Test</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Learning_for_Cardiologist-level_Myocardial_Infarction_Detection_in_Electrocardiograms&diff=42827 Deep Learning for Cardiologist-level Myocardial Infarction Detection in Electrocardiograms 2020-10-26T00:23:37Z <p>Z42qin: /* Maggie */</p> <hr /> <div><br /> == Maggie ==<br /> <br /> Introduction<br /> <br />  Problem: To detect risk of heart disease from ECG signals.<br /> <br />  Importance: To provide a correct diagnosis so that proper health care can be given to patients.<br /> <br />  Area: Deep learning – scientific machine learning<br /> <br /> o Concerned about design, training, and use of ML algorithm in an optimal matter towards a certain problem.<br /> <br />  Deep learning model benefits:<br /> <br /> o Uses multiple GPUs to construct complicated NN that can be trained using TB-sized datasets, which are robust against noise.<br /> <br /> o After training is complete, it takes little computational power to conduct statistical inference.<br /> <br />  Question that need to be answered:<br /> <br /> o What AI architecture and datasets can provide evidence of symptoms, ECG changes, imaging evidence that shows loss of viable myocardium and wall motion abnormality?<br /> <br />  Purpose of the article:<br /> <br /> o To provide a detailed analysis on the contribution of each ECG lead in identifying heart disease in the model. This is because the selection of data in previous studies regarding heart disease made data selection arbitrary in identifying heart conditions.<br /> <br /> o To show the use of using multiple data channels of information to enhance prediction accuracy in deep learning, i.e. processing the top three ECG leads simultaneously in the neural network.<br /> <br /> o Show that feature engineering is not necessary in the training, validation, or testing process for ECG data in neural networks.<br /> <br /> Related work<br /> <br />  Database used: PTB database<br /> <br /> 1. CNN Network<br /> <br /> a. Used both noisy and denoised ECGs without feature engineering.<br /> <br /> 2. Used artificial neural network, probabilistic neural network, KNN, multi-layer perceptron, and Naïve Bayes Classification<br /> <br /> a. Extracted two features: T-wave integral and total integral to identify heart disease.<br /> <br /> 3. Developed two different ANN: RBF and MLP<br /> <br /> 4. Supervised learning techniques have limited success to the problem and used multiple instance learning.<br /> <br /> a. Demonstrate proposed algorithm LTMIL surpasses supervised approaches.<br /> <br /> 5. Create new feature by approximating ECG signal using a 20th order polynomial, which achieved 94.4% accuracy.<br /> <br /> 6. Stationary wavelet transforms to decompose ECG into sub-bands.<br /> <br /> a. SVM and KNN used to classify.<br /> <br /> b. Features used: sample entropy, normalized sub-bands, log energy entropy, median slope from sub-bands.<br /> <br /> 7. Transfer learning – used deep CNN model for arrhythmia and developed it to detect heart disease.<br /> <br /> 8. Simple adaptive threshold (SAT)<br /> <br /> a. Multiresolution approach, adaptive thresholding used to extract features: depth of Q peak and elevation in ST segment<br /> <br /> 9. Subject-oriented approach using CNN to take in leads II, III, AVF<br /> <br /> 10. Model ECG using 2nd order ODE and feed the best-fitting coefficients of the ECG signal into a SVM.<br /> <br /> 11. Multi-channel CNN (16 layers) with long-short memory units<br /> <br /> 12. Deep CNN that takes 3 seconds at a time of lead II as input<br /> <br />  Most use feature extraction/selection from the raw ECG data before training.<br /> <br /> o Problem with feature selection is that it is not practical for large volumes of data.<br /> <br />  Other papers that do not use feature selection arbitrarily picks ECG leads for classification and does not provide rationale.<br /> <br /> ==Betty==<br /> <br /> Result: <br /> <br /> 1. Quantification of accuracies for single channels with 20-fold cross-validation, resulting highest individual accuracies: v5, v6, vx, vz, and ii<br /> <br /> 2. Quantification of accuracies for pairs of top 5 highest individual channels with 20-fold cross-validation, resulting highest pairs accuracies to fed into a the neural network: lead v6 and lead vz <br /> <br /> 3. Use 100-fold cross validation on v6 and vz pair of channels, then compare outliers based on top 20, top 50 and total 100 performing models, finding that standard deviation is non-trivial and there are few models performed very poorly.<br /> <br /> 4. Discussing 2 factors effecting model performance evaluation:<br /> <br /> 1） Random train-val-test split might have effects of the performance of the model, but it can be improved by access with a larger data set and further discussion <br /> <br /> 2） random initialization of the weights of neural network shows little effects on the performance of the model performance evaluation, because of showing a high average results with a fixed train-val-test split <br /> <br /> 5. Comparing with other models, the model in this article has the highest accuracy, specificity, and precision<br /> <br /> 6. Further using 290 fold patient-wise split, resulting the same highest accuracy of the pair v6 and vz as record-wise split <br /> <br /> 1） Discuss patient-wise split might result lower accuracy evaluation, however, it still maintain an average of 97.83%<br /> <br /> == Marty ==<br /> <br /> 3.1: Data curation<br /> <br /> Dataset: 549 ECG records total<br /> <br /> 290 unique patients<br /> <br /> Each ECG record has a mean length of over 100s<br /> <br /> 3.2: ANN model<br /> <br /> ConvNetQuake model + 1D batch normalization + Label-smoothing<br /> <br /> Model (PyTorch):<br /> <br /> - Input layer: 10-second long ECG signal<br /> <br /> - Hidden layers: 8 * (1D convolution layer, Activation function: RELU, 1D batch normalization layer)<br /> <br /> - Output layer: 1280 dimensions -&gt; 1 dimension, Activation function: Sigmoid<br /> <br /> Batch size = 10<br /> <br /> Learning rate = 10^-4<br /> <br /> Optimizer = ADAM<br /> <br /> 80-10-10: Train-Validation-Test</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Learning_for_Cardiologist-level_Myocardial_Infarction_Detection_in_Electrocardiograms&diff=42826 Deep Learning for Cardiologist-level Myocardial Infarction Detection in Electrocardiograms 2020-10-26T00:23:17Z <p>Z42qin: /* Maggie */</p> <hr /> <div><br /> == Maggie ==<br /> <br /> Introduction<br /> <br />  Problem: To detect risk of heart disease from ECG signals.<br /> <br />  Importance: To provide a correct diagnosis so that proper health care can be given to patients.<br /> <br />  Area: Deep learning – scientific machine learning<br /> <br /> o Concerned about design, training, and use of ML algorithm in an optimal matter towards a certain problem.<br /> <br />  Deep learning model benefits:<br /> <br /> o Uses multiple GPUs to construct complicated NN that can be trained using TB-sized datasets, which are robust against noise.<br /> <br /> o After training is complete, it takes little computational power to conduct statistical inference.<br /> <br />  Question that need to be answered:<br /> <br /> o What AI architecture and datasets can provide evidence of symptoms, ECG changes, imaging evidence that shows loss of viable myocardium and wall motion abnormality?<br /> <br />  Purpose of the article:<br /> <br /> o To provide a detailed analysis on the contribution of each ECG lead in identifying heart disease in the model. This is because the selection of data in previous studies regarding heart disease made data selection arbitrary in identifying heart conditions.<br /> <br /> o To show the use of using multiple data channels of information to enhance prediction accuracy in deep learning, i.e. processing the top three ECG leads simultaneously in the neural network.<br /> <br /> o Show that feature engineering is not necessary in the training, validation, or testing process for ECG data in neural networks.<br /> <br /> Related work<br /> <br />  Database used: PTB database<br /> <br /> 1. CNN Network<br /> <br /> a. Used both noisy and denoised ECGs without feature engineering.<br /> <br /> 2. Used artificial neural network, probabilistic neural network, KNN, multi-layer perceptron, and Naïve Bayes Classification<br /> <br /> a. Extracted two features: T-wave integral and total integral to identify heart disease.<br /> <br /> 3. Developed two different ANN: RBF and MLP<br /> <br /> 4. Supervised learning techniques have limited success to the problem and used multiple instance learning.<br /> <br /> a. Demonstrate proposed algorithm LTMIL surpasses supervised approaches.<br /> <br /> 5. Create new feature by approximating ECG signal using a 20th order polynomial, which achieved 94.4% accuracy.<br /> <br /> 6. Stationary wavelet transforms to decompose ECG into sub-bands.<br /> <br /> a. SVM and KNN used to classify.<br /> <br /> b. Features used: sample entropy, normalized sub-bands, log energy entropy, median slope from sub-bands.<br /> <br /> 7. Transfer learning – used deep CNN model for arrhythmia and developed it to detect heart disease.<br /> <br /> 8. Simple adaptive threshold (SAT)<br /> <br /> a. Multiresolution approach, adaptive thresholding used to extract features: depth of Q peak and elevation in ST segment<br /> <br /> 9. Subject-oriented approach using CNN to take in leads II, III, AVF<br /> <br /> 10. Model ECG using 2nd order ODE and feed the best-fitting coefficients of the ECG signal into a SVM.<br /> <br /> 11. Multi-channel CNN (16 layers) with long-short memory units<br /> <br /> 12. Deep CNN that takes 3 seconds at a time of lead II as input<br /> <br />  Most use feature extraction/selection from the raw ECG data before training.<br /> <br /> o Problem with feature selection is that it is not practical for large volumes of data.<br /> <br />  Other papers that do not use feature selection arbitrarily picks ECG leads for classification and does not provide rationale.<br /> <br /> ==Betty==<br /> <br /> Result: <br /> <br /> 1. Quantification of accuracies for single channels with 20-fold cross-validation, resulting highest individual accuracies: v5, v6, vx, vz, and ii<br /> <br /> 2. Quantification of accuracies for pairs of top 5 highest individual channels with 20-fold cross-validation, resulting highest pairs accuracies to fed into a the neural network: lead v6 and lead vz <br /> <br /> 3. Use 100-fold cross validation on v6 and vz pair of channels, then compare outliers based on top 20, top 50 and total 100 performing models, finding that standard deviation is non-trivial and there are few models performed very poorly.<br /> <br /> 4. Discussing 2 factors effecting model performance evaluation:<br /> <br /> 1） Random train-val-test split might have effects of the performance of the model, but it can be improved by access with a larger data set and further discussion <br /> <br /> 2） random initialization of the weights of neural network shows little effects on the performance of the model performance evaluation, because of showing a high average results with a fixed train-val-test split <br /> <br /> 5. Comparing with other models, the model in this article has the highest accuracy, specificity, and precision<br /> <br /> 6. Further using 290 fold patient-wise split, resulting the same highest accuracy of the pair v6 and vz as record-wise split <br /> <br /> 1） Discuss patient-wise split might result lower accuracy evaluation, however, it still maintain an average of 97.83%<br /> <br /> == Marty ==<br /> <br /> 3.1: Data curation<br /> <br /> Dataset: 549 ECG records total<br /> <br /> 290 unique patients<br /> <br /> Each ECG record has a mean length of over 100s<br /> <br /> 3.2: ANN model<br /> <br /> ConvNetQuake model + 1D batch normalization + Label-smoothing<br /> <br /> Model (PyTorch):<br /> <br /> - Input layer: 10-second long ECG signal<br /> <br /> - Hidden layers: 8 * (1D convolution layer, Activation function: RELU, 1D batch normalization layer)<br /> <br /> - Output layer: 1280 dimensions -&gt; 1 dimension, Activation function: Sigmoid<br /> <br /> Batch size = 10<br /> <br /> Learning rate = 10^-4<br /> <br /> Optimizer = ADAM<br /> <br /> 80-10-10: Train-Validation-Test</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Learning_for_Cardiologist-level_Myocardial_Infarction_Detection_in_Electrocardiograms&diff=42825 Deep Learning for Cardiologist-level Myocardial Infarction Detection in Electrocardiograms 2020-10-26T00:22:55Z <p>Z42qin: /* Maggie */</p> <hr /> <div><br /> == Maggie ==<br /> <br /> Introduction<br /> <br />  Problem: To detect risk of heart disease from ECG signals.<br /> <br />  Importance: To provide a correct diagnosis so that proper health care can be given to patients.<br /> <br />  Area: Deep learning – scientific machine learning<br /> <br /> o Concerned about design, training, and use of ML algorithm in an optimal matter towards a certain problem.<br /> <br />  Deep learning model benefits:<br /> <br /> o Uses multiple GPUs to construct complicated NN that can be trained using TB-sized datasets, which are robust against noise.<br /> <br /> o After training is complete, it takes little computational power to conduct statistical inference.<br /> <br />  Question that need to be answered:<br /> <br /> o What AI architecture and datasets can provide evidence of symptoms, ECG changes, imaging evidence that shows loss of viable myocardium and wall motion abnormality?<br /> <br />  Purpose of the article:<br /> <br /> o To provide a detailed analysis on the contribution of each ECG lead in identifying heart disease in the model. This is because the selection of data in previous studies regarding heart disease made data selection arbitrary in identifying heart conditions.<br /> <br /> o To show the use of using multiple data channels of information to enhance prediction accuracy in deep learning, i.e. processing the top three ECG leads simultaneously in the neural network.<br /> <br /> o Show that feature engineering is not necessary in the training, validation, or testing process for ECG data in neural networks.<br /> <br /> Related work<br /> <br />  Database used: PTB database<br /> <br /> 1. CNN Network<br /> <br /> a. Used both noisy and denoised ECGs without feature engineering.<br /> <br /> 2. Used artificial neural network, probabilistic neural network, KNN, multi-layer perceptron, and Naïve Bayes Classification<br /> <br /> a. Extracted two features: T-wave integral and total integral to identify heart disease.<br /> <br /> 3. Developed two different ANN: RBF and MLP<br /> <br /> 4. Supervised learning techniques have limited success to the problem and used multiple instance learning.<br /> <br /> a. Demonstrate proposed algorithm LTMIL surpasses supervised approaches.<br /> <br /> 5. Create new feature by approximating ECG signal using a 20th order polynomial, which achieved 94.4% accuracy.<br /> <br /> 6. Stationary wavelet transforms to decompose ECG into sub-bands.<br /> <br /> a. SVM and KNN used to classify.<br /> <br /> b. Features used: sample entropy, normalized sub-bands, log energy entropy, median slope from sub-bands.<br /> <br /> 7. Transfer learning – used deep CNN model for arrhythmia and developed it to detect heart disease.<br /> <br /> 8. Simple adaptive threshold (SAT)<br /> <br /> a. Multiresolution approach, adaptive thresholding used to extract features: depth of Q peak and elevation in ST segment<br /> <br /> 9. Subject-oriented approach using CNN to take in leads II, III, AVF<br /> <br /> 10. Model ECG using 2nd order ODE and feed the best-fitting coefficients of the ECG signal into a SVM.<br /> <br /> 11. Multi-channel CNN (16 layers) with long-short memory units<br /> <br /> 12. Deep CNN that takes 3 seconds at a time of lead II as input<br /> <br />  Most use feature extraction/selection from the raw ECG data before training.<br /> <br /> o Problem with feature selection is that it is not practical for large volumes of data.<br /> <br />  Other papers that do not use feature selection arbitrarily picks ECG leads for classification and does not provide rationale.<br /> <br /> ==Betty==<br /> <br /> Result: <br /> <br /> 1. Quantification of accuracies for single channels with 20-fold cross-validation, resulting highest individual accuracies: v5, v6, vx, vz, and ii<br /> <br /> 2. Quantification of accuracies for pairs of top 5 highest individual channels with 20-fold cross-validation, resulting highest pairs accuracies to fed into a the neural network: lead v6 and lead vz <br /> <br /> 3. Use 100-fold cross validation on v6 and vz pair of channels, then compare outliers based on top 20, top 50 and total 100 performing models, finding that standard deviation is non-trivial and there are few models performed very poorly.<br /> <br /> 4. Discussing 2 factors effecting model performance evaluation:<br /> <br /> 1） Random train-val-test split might have effects of the performance of the model, but it can be improved by access with a larger data set and further discussion <br /> <br /> 2） random initialization of the weights of neural network shows little effects on the performance of the model performance evaluation, because of showing a high average results with a fixed train-val-test split <br /> <br /> 5. Comparing with other models, the model in this article has the highest accuracy, specificity, and precision<br /> <br /> 6. Further using 290 fold patient-wise split, resulting the same highest accuracy of the pair v6 and vz as record-wise split <br /> <br /> 1） Discuss patient-wise split might result lower accuracy evaluation, however, it still maintain an average of 97.83%<br /> <br /> == Marty ==<br /> <br /> 3.1: Data curation<br /> <br /> Dataset: 549 ECG records total<br /> <br /> 290 unique patients<br /> <br /> Each ECG record has a mean length of over 100s<br /> <br /> 3.2: ANN model<br /> <br /> ConvNetQuake model + 1D batch normalization + Label-smoothing<br /> <br /> Model (PyTorch):<br /> <br /> - Input layer: 10-second long ECG signal<br /> <br /> - Hidden layers: 8 * (1D convolution layer, Activation function: RELU, 1D batch normalization layer)<br /> <br /> - Output layer: 1280 dimensions -&gt; 1 dimension, Activation function: Sigmoid<br /> <br /> Batch size = 10<br /> <br /> Learning rate = 10^-4<br /> <br /> Optimizer = ADAM<br /> <br /> 80-10-10: Train-Validation-Test</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Learning_for_Cardiologist-level_Myocardial_Infarction_Detection_in_Electrocardiograms&diff=42824 Deep Learning for Cardiologist-level Myocardial Infarction Detection in Electrocardiograms 2020-10-26T00:22:46Z <p>Z42qin: /* Maggie */</p> <hr /> <div><br /> == Maggie ==<br /> <br /> Introduction<br /> <br />  Problem: To detect risk of heart disease from ECG signals.<br /> <br />  Importance: To provide a correct diagnosis so that proper health care can be given to patients.<br /> <br />  Area: Deep learning – scientific machine learning<br /> <br /> o Concerned about design, training, and use of ML algorithm in an optimal matter towards a certain problem.<br /> <br />  Deep learning model benefits:<br /> <br /> o Uses multiple GPUs to construct complicated NN that can be trained using TB-sized datasets, which are robust against noise.<br /> <br /> o After training is complete, it takes little computational power to conduct statistical inference.<br /> <br />  Question that need to be answered:<br /> <br /> o What AI architecture and datasets can provide evidence of symptoms, ECG changes, imaging evidence that shows loss of viable myocardium and wall motion abnormality?<br /> <br />  Purpose of the article:<br /> <br /> o To provide a detailed analysis on the contribution of each ECG lead in identifying heart disease in the model. This is because the selection of data in previous studies regarding heart disease made data selection arbitrary in identifying heart conditions.<br /> <br /> o To show the use of using multiple data channels of information to enhance prediction accuracy in deep learning, i.e. processing the top three ECG leads simultaneously in the neural network.<br /> <br /> o Show that feature engineering is not necessary in the training, validation, or testing process for ECG data in neural networks.<br /> <br /> Related work<br /> <br />  Database used: PTB database<br /> <br /> 1. CNN Network<br /> <br /> a. Used both noisy and denoised ECGs without feature engineering.<br /> <br /> 2. Used artificial neural network, probabilistic neural network, KNN, multi-layer perceptron, and Naïve Bayes Classification<br /> <br /> a. Extracted two features: T-wave integral and total integral to identify heart disease.<br /> <br /> 3. Developed two different ANN: RBF and MLP<br /> <br /> 4. Supervised learning techniques have limited success to the problem and used multiple instance learning.<br /> <br /> a. Demonstrate proposed algorithm LTMIL surpasses supervised approaches.<br /> <br /> 5. Create new feature by approximating ECG signal using a 20th order polynomial, which achieved 94.4% accuracy.<br /> <br /> 6. Stationary wavelet transforms to decompose ECG into sub-bands.<br /> <br /> a. SVM and KNN used to classify.<br /> <br /> b. Features used: sample entropy, normalized sub-bands, log energy entropy, median slope from sub-bands.<br /> <br /> 7. Transfer learning – used deep CNN model for arrhythmia and developed it to detect heart disease.<br /> <br /> 8. Simple adaptive threshold (SAT)<br /> <br /> a. Multiresolution approach, adaptive thresholding used to extract features: depth of Q peak and elevation in ST segment<br /> <br /> 9. Subject-oriented approach using CNN to take in leads II, III, AVF<br /> <br /> 10. Model ECG using 2nd order ODE and feed the best-fitting coefficients of the ECG signal into a SVM.<br /> <br /> 11. Multi-channel CNN (16 layers) with long-short memory units<br /> <br /> 12. Deep CNN that takes 3 seconds at a time of lead II as input<br /> <br />  Most use feature extraction/selection from the raw ECG data before training.<br /> <br /> o Problem with feature selection is that it is not practical for large volumes of data.<br /> <br />  Other papers that do not use feature selection arbitrarily picks ECG leads for classification and does not provide rationale.<br /> <br /> ==Betty==<br /> <br /> Result: <br /> <br /> 1. Quantification of accuracies for single channels with 20-fold cross-validation, resulting highest individual accuracies: v5, v6, vx, vz, and ii<br /> <br /> 2. Quantification of accuracies for pairs of top 5 highest individual channels with 20-fold cross-validation, resulting highest pairs accuracies to fed into a the neural network: lead v6 and lead vz <br /> <br /> 3. Use 100-fold cross validation on v6 and vz pair of channels, then compare outliers based on top 20, top 50 and total 100 performing models, finding that standard deviation is non-trivial and there are few models performed very poorly.<br /> <br /> 4. Discussing 2 factors effecting model performance evaluation:<br /> <br /> 1） Random train-val-test split might have effects of the performance of the model, but it can be improved by access with a larger data set and further discussion <br /> <br /> 2） random initialization of the weights of neural network shows little effects on the performance of the model performance evaluation, because of showing a high average results with a fixed train-val-test split <br /> <br /> 5. Comparing with other models, the model in this article has the highest accuracy, specificity, and precision<br /> <br /> 6. Further using 290 fold patient-wise split, resulting the same highest accuracy of the pair v6 and vz as record-wise split <br /> <br /> 1） Discuss patient-wise split might result lower accuracy evaluation, however, it still maintain an average of 97.83%<br /> <br /> == Marty ==<br /> <br /> 3.1: Data curation<br /> <br /> Dataset: 549 ECG records total<br /> <br /> 290 unique patients<br /> <br /> Each ECG record has a mean length of over 100s<br /> <br /> 3.2: ANN model<br /> <br /> ConvNetQuake model + 1D batch normalization + Label-smoothing<br /> <br /> Model (PyTorch):<br /> <br /> - Input layer: 10-second long ECG signal<br /> <br /> - Hidden layers: 8 * (1D convolution layer, Activation function: RELU, 1D batch normalization layer)<br /> <br /> - Output layer: 1280 dimensions -&gt; 1 dimension, Activation function: Sigmoid<br /> <br /> Batch size = 10<br /> <br /> Learning rate = 10^-4<br /> <br /> Optimizer = ADAM<br /> <br /> 80-10-10: Train-Validation-Test</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Learning_for_Cardiologist-level_Myocardial_Infarction_Detection_in_Electrocardiograms&diff=42823 Deep Learning for Cardiologist-level Myocardial Infarction Detection in Electrocardiograms 2020-10-26T00:21:02Z <p>Z42qin: </p> <hr /> <div><br /> == Maggie ==<br /> <br /> Introduction<br /> <br />  Problem: To detect risk of heart disease from ECG signals.<br /> <br />  Importance: To provide a correct diagnosis so that proper health care can be given to patients.<br /> <br />  Area: Deep learning – scientific machine learning<br /> <br /> o Concerned about design, training, and use of ML algorithm in an optimal matter towards a certain problem.<br /> <br />  Deep learning model benefits:<br /> <br /> o Uses multiple GPUs to construct complicated NN that can be trained using TB-sized datasets, which are robust against noise.<br /> <br /> o After training is complete, it takes little computational power to conduct statistical inference.<br /> <br />  Question that need to be answered:<br /> <br /> o What AI architecture and datasets can provide evidence of symptoms, ECG changes, imaging evidence that shows loss of viable myocardium and wall motion abnormality?<br /> <br />  Purpose of the article:<br /> <br /> o To provide a detailed analysis on the contribution of each ECG lead in identifying heart disease in the model. This is because the selection of data in previous studies regarding heart disease made data selection arbitrary in identifying heart conditions.<br /> <br /> o To show the use of using multiple data channels of information to enhance prediction accuracy in deep learning, i.e. processing the top three ECG leads simultaneously in the neural network.<br /> <br /> o Show that feature engineering is not necessary in the training, validation, or testing process for ECG data in neural networks.<br /> <br /> Related work<br /> <br />  Database used: PTB database<br /> <br /> 1. CNN Network<br /> <br /> a. Used both noisy and denoised ECGs without feature engineering.<br /> <br /> 2. Used artificial neural network, probabilistic neural network, KNN, multi-layer perceptron, and Naïve Bayes Classification<br /> <br /> a. Extracted two features: T-wave integral and total integral to identify heart disease.<br /> <br /> 3. Developed two different ANN: RBF and MLP<br /> <br /> 4. Supervised learning techniques have limited success to the problem and used multiple instance learning.<br /> <br /> a. Demonstrate proposed algorithm LTMIL surpasses supervised approaches.<br /> <br /> 5. Create new feature by approximating ECG signal using a 20th order polynomial, which achieved 94.4% accuracy.<br /> <br /> 6. Stationary wavelet transforms to decompose ECG into sub-bands.<br /> <br /> a. SVM and KNN used to classify.<br /> <br /> b. Features used: sample entropy, normalized sub-bands, log energy entropy, median slope from sub-bands.<br /> <br /> 7. Transfer learning – used deep CNN model for arrhythmia and developed it to detect heart disease.<br /> <br /> 8. Simple adaptive threshold (SAT)<br /> <br /> a. Multiresolution approach, adaptive thresholding used to extract features: depth of Q peak and elevation in ST segment<br /> <br /> 9. Subject-oriented approach using CNN to take in leads II, III, AVF<br /> <br /> 10. Model ECG using 2nd order ODE and feed the best-fitting coefficients of the ECG signal into a SVM.<br /> <br /> 11. Multi-channel CNN (16 layers) with long-short memory units<br /> <br /> 12. Deep CNN that takes 3 seconds at a time of lead II as input<br /> <br />  Most use feature extraction/selection from the raw ECG data before training.<br /> <br /> o Problem with feature selection is that it is not practical for large volumes of data.<br /> <br />  Other papers that do not use feature selection arbitrarily picks ECG leads for classification and does not provide rationale.<br /> <br /> <br /> ==Betty==<br /> <br /> Result: <br /> <br /> 1. Quantification of accuracies for single channels with 20-fold cross-validation, resulting highest individual accuracies: v5, v6, vx, vz, and ii<br /> <br /> 2. Quantification of accuracies for pairs of top 5 highest individual channels with 20-fold cross-validation, resulting highest pairs accuracies to fed into a the neural network: lead v6 and lead vz <br /> <br /> 3. Use 100-fold cross validation on v6 and vz pair of channels, then compare outliers based on top 20, top 50 and total 100 performing models, finding that standard deviation is non-trivial and there are few models performed very poorly.<br /> <br /> 4. Discussing 2 factors effecting model performance evaluation:<br /> <br /> 1） Random train-val-test split might have effects of the performance of the model, but it can be improved by access with a larger data set and further discussion <br /> <br /> 2） random initialization of the weights of neural network shows little effects on the performance of the model performance evaluation, because of showing a high average results with a fixed train-val-test split <br /> <br /> 5. Comparing with other models, the model in this article has the highest accuracy, specificity, and precision<br /> <br /> 6. Further using 290 fold patient-wise split, resulting the same highest accuracy of the pair v6 and vz as record-wise split <br /> <br /> 1） Discuss patient-wise split might result lower accuracy evaluation, however, it still maintain an average of 97.83%<br /> <br /> == Marty ==<br /> <br /> 3.1: Data curation<br /> <br /> Dataset: 549 ECG records total<br /> <br /> 290 unique patients<br /> <br /> Each ECG record has a mean length of over 100s<br /> <br /> 3.2: ANN model<br /> <br /> ConvNetQuake model + 1D batch normalization + Label-smoothing<br /> <br /> Model (PyTorch):<br /> <br /> - Input layer: 10-second long ECG signal<br /> <br /> - Hidden layers: 8 * (1D convolution layer, Activation function: RELU, 1D batch normalization layer)<br /> <br /> - Output layer: 1280 dimensions -&gt; 1 dimension, Activation function: Sigmoid<br /> <br /> Batch size = 10<br /> <br /> Learning rate = 10^-4<br /> <br /> Optimizer = ADAM<br /> <br /> 80-10-10: Train-Validation-Test</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Learning_for_Cardiologist-level_Myocardial_Infarction_Detection_in_Electrocardiograms&diff=42820 Deep Learning for Cardiologist-level Myocardial Infarction Detection in Electrocardiograms 2020-10-26T00:18:06Z <p>Z42qin: </p> <hr /> <div><br /> == Maggie ==<br /> <br /> Introduction<br /> <br />  Problem: To detect risk of heart disease from ECG signals.<br /> <br />  Importance: To provide a correct diagnosis so that proper health care can be given to patients.<br /> <br />  Area: Deep learning – scientific machine learning<br /> <br /> o Concerned about design, training, and use of ML algorithm in an optimal matter towards a certain problem.<br /> <br />  Deep learning model benefits:<br /> <br /> o Uses multiple GPUs to construct complicated NN that can be trained using TB-sized datasets, which are robust against noise.<br /> <br /> o After training is complete, it takes little computational power to conduct statistical inference.<br /> <br />  Question that need to be answered:<br /> <br /> o What AI architecture and datasets can provide evidence of symptoms, ECG changes, imaging evidence that shows loss of viable myocardium and wall motion abnormality?<br /> <br />  Purpose of the article:<br /> <br /> o To provide a detailed analysis on the contribution of each ECG lead in identifying heart disease in the model. This is because the selection of data in previous studies regarding heart disease made data selection arbitrary in identifying heart conditions.<br /> <br /> o To show the use of using multiple data channels of information to enhance prediction accuracy in deep learning, i.e. processing the top three ECG leads simultaneously in the neural network.<br /> <br /> o Show that feature engineering is not necessary in the training, validation, or testing process for ECG data in neural networks.<br /> <br /> Related work<br /> <br />  Database used: PTB database<br /> <br /> 1. CNN Network<br /> <br /> a. Used both noisy and denoised ECGs without feature engineering.<br /> <br /> 2. Used artificial neural network, probabilistic neural network, KNN, multi-layer perceptron, and Naïve Bayes Classification<br /> <br /> a. Extracted two features: T-wave integral and total integral to identify heart disease.<br /> <br /> 3. Developed two different ANN: RBF and MLP<br /> <br /> 4. Supervised learning techniques have limited success to the problem and used multiple instance learning.<br /> <br /> a. Demonstrate proposed algorithm LTMIL surpasses supervised approaches.<br /> <br /> 5. Create new feature by approximating ECG signal using a 20th order polynomial, which achieved 94.4% accuracy.<br /> <br /> 6. Stationary wavelet transforms to decompose ECG into sub-bands.<br /> <br /> a. SVM and KNN used to classify.<br /> <br /> b. Features used: sample entropy, normalized sub-bands, log energy entropy, median slope from sub-bands.<br /> <br /> 7. Transfer learning – used deep CNN model for arrhythmia and developed it to detect heart disease.<br /> <br /> 8. Simple adaptive threshold (SAT)<br /> <br /> a. Multiresolution approach, adaptive thresholding used to extract features: depth of Q peak and elevation in ST segment<br /> <br /> 9. Subject-oriented approach using CNN to take in leads II, III, AVF<br /> <br /> 10. Model ECG using 2nd order ODE and feed the best-fitting coefficients of the ECG signal into a SVM.<br /> <br /> 11. Multi-channel CNN (16 layers) with long-short memory units<br /> <br /> 12. Deep CNN that takes 3 seconds at a time of lead II as input<br /> <br />  Most use feature extraction/selection from the raw ECG data before training.<br /> <br /> o Problem with feature selection is that it is not practical for large volumes of data.<br /> <br />  Other papers that do not use feature selection arbitrarily picks ECG leads for classification and does not provide rationale.<br /> <br /> <br /> ==Betty==<br /> <br /> Result: <br /> <br /> 1. Quantification of accuracies for single channels with 20-fold cross-validation, resulting highest individual accuracies: v5, v6, vx, vz, and ii<br /> <br /> 2. Quantification of accuracies for pairs of top 5 highest individual channels with 20-fold cross-validation, resulting highest pairs accuracies to fed into a the neural network: lead v6 and lead vz <br /> <br /> 3. Use 100-fold cross validation on v6 and vz pair of channels, then compare outliers based on top 20, top 50 and total 100 performing models, finding that standard deviation is non-trivial and there are few models performed very poorly.<br /> <br /> 4. Discussing 2 factors effecting model performance evaluation:<br /> <br /> 1） Random train-val-test split might have effects of the performance of the model, but it can be improved by access with a larger data set and further discussion <br /> <br /> 2） random initialization of the weights of neural network shows little effects on the performance of the model performance evaluation, because of showing a high average results with a fixed train-val-test split <br /> <br /> 5. Comparing with other models, the model in this article has the highest accuracy, specificity, and precision<br /> <br /> 6. Further using 290 fold patient-wise split, resulting the same highest accuracy of the pair v6 and vz as record-wise split <br /> <br /> 1） Discuss patient-wise split might result lower accuracy evaluation, however, it still maintain an average of 97.83%<br /> <br /> == Marty ==<br /> <br /> 3.1: Data curation<br /> <br /> Dataset: 549 ECG records total 290 unique patients Each ECG record has a mean length of over 100s<br /> <br /> 3.2: ANN model<br /> <br /> ConvNetQuake model + 1D batch normalization + Label-smoothing<br /> <br /> Model (PyTorch): - Input layer: 10-second long ECG signal - Hidden layers: 8 * (1D convolution layer, Activation function: RELU, 1D batch normalization layer) - Output layer: 1280 dimensions -&gt; 1 dimension, Activation function: Sigmoid<br /> <br /> Batch size = 10 Learning rate = 10^-4 Optimizer = ADAM<br /> <br /> 80-10-10: Train-Validation-Test</div> Z42qin http://wiki.math.uwaterloo.ca/statwiki/index.php?title=Deep_Learning_for_Cardiologist-level_Myocardial_Infarction_Detection_in_Electrocardiograms&diff=42817 Deep Learning for Cardiologist-level Myocardial Infarction Detection in Electrocardiograms 2020-10-26T00:15:45Z <p>Z42qin: </p> <hr /> <div><br /> <br /> ==Betty==<br /> <br /> Result: <br /> <br /> 1. Quantification of accuracies for single channels with 20-fold cross-validation, resulting highest individual accuracies: v5, v6, vx, vz, and ii<br /> <br /> 2. Quantification of accuracies for pairs of top 5 highest individual channels with 20-fold cross-validation, resulting highest pairs accuracies to fed into a the neural network: lead v6 and lead vz <br /> <br /> 3. Use 100-fold cross validation on v6 and vz pair of channels, then compare outliers based on top 20, top 50 and total 100 performing models, finding that standard deviation is non-trivial and there are few models performed very poorly.<br /> <br /> 4. Discussing 2 factors effecting model performance evaluation:<br /> <br /> 1） Random train-val-test split might have effects of the performance of the model, but it can be improved by access with a larger data set and further discussion <br /> <br /> 2） random initialization of the weights of neural network shows little effects on the performance of the model performance evaluation, because of showing a high average results with a fixed train-val-test split <br /> <br /> 5. Comparing with other models, the model in this article has the highest accuracy, specificity, and precision<br /> <br /> 6. Further using 290 fold patient-wise split, resulting the same highest accuracy of the pair v6 and vz as record-wise split <br /> <br /> 1） Discuss patient-wise split might result lower accuracy evaluation, however, it still maintain an average of 97.83%</div> Z42qin