Difference between revisions of "stat841f10"

From statwiki
Jump to: navigation, search
(Reference articles on decision tree method)
m (Conversion script moved page Stat841f10 to stat841f10: Converting page titles to lowercase)
 
(331 intermediate revisions by 19 users not shown)
Line 41: Line 41:
  
 
As another example, suppose we wish to classify newly-given fruits into apples and oranges by considering three features of a fruit that comprise its color, its diameter, and its weight. After selecting a classifier and constructing a model using training data <math>\,\{(X_{color, 1}, X_{diameter, 1}, X_{weight, 1}, Y_{1}), \dots , (X_{color, n}, X_{diameter, n}, X_{weight, n}, Y_{n})\}</math>, we could then use the classifier's classification rule <math>\,h</math> to assign any newly-given fruit having known feature values <math>\,x = (\,x_{color}, x_{diameter} , x_{weight})</math> the label <math>\, \hat{Y}=h(x) \in \mathcal{Y}= \{apple,orange\}</math>.
 
As another example, suppose we wish to classify newly-given fruits into apples and oranges by considering three features of a fruit that comprise its color, its diameter, and its weight. After selecting a classifier and constructing a model using training data <math>\,\{(X_{color, 1}, X_{diameter, 1}, X_{weight, 1}, Y_{1}), \dots , (X_{color, n}, X_{diameter, n}, X_{weight, n}, Y_{n})\}</math>, we could then use the classifier's classification rule <math>\,h</math> to assign any newly-given fruit having known feature values <math>\,x = (\,x_{color}, x_{diameter} , x_{weight})</math> the label <math>\, \hat{Y}=h(x) \in \mathcal{Y}= \{apple,orange\}</math>.
 +
 +
=== Examples of Classification ===
 +
 +
• Email spam filtering (spam vs not spam).
 +
 +
• Detecting credit card fraud (fraudulent or legitimate).
 +
 +
• Face detection in images (face or background).
 +
 +
• Web page classification (sports vs politics vs entertainment etc).
 +
 +
• Steering an autonomous car across the US (turn left, right, or go straight).
 +
 +
• Medical diagnosis (classification of disease based on observed symptoms).
 +
 +
=== Independent and Identically Distributed (iid) Data Assumption ===
 +
 +
Suppose that we have training data X which contains N data points.  The Independent and Identically Distributed (IID) assumption declares that the datapoints are drawn independently from identical distributions.  This assumption implies that ordering of the data points does not matter, and the assumption is used in many classification problems.  For an example of data that is not IID, consider daily temperature: today's temperature is not independent of the yesterday's temperature -- rather, there is a strong correlation between the temperatures of the two days.
  
 
=== Error rate ===
 
=== Error rate ===
Line 53: Line 71:
  
 
In practice, the empirical error rate is obtained to estimate the true error rate, whose value is impossible to be known because the parameter values of the underlying process cannot be known but can only be estimated using available data. The empirical error rate, in practice, estimates the true error rate quite well in that, as mentioned [http://www.liebertonline.com/doi/pdf/10.1089/106652703321825928 here], it is an unbiased estimator of the true error rate.
 
In practice, the empirical error rate is obtained to estimate the true error rate, whose value is impossible to be known because the parameter values of the underlying process cannot be known but can only be estimated using available data. The empirical error rate, in practice, estimates the true error rate quite well in that, as mentioned [http://www.liebertonline.com/doi/pdf/10.1089/106652703321825928 here], it is an unbiased estimator of the true error rate.
 +
 +
An Error Rate Comparison of Classification Methods [http://pdfserve.informaworld.com/311525_770885140_713826662.pdf]
 +
 +
=== Decision Theory ===
 +
we can identify three distinct approaches to solving decision problems, all of which have been used in practical applications. These are given, in decreasing order of complexity, by:
 +
 +
a. First solve the inference problem of determining the class-conditional densities <math>\ p(x|C_k)</math> for each class <math>\ C_k</math> individually. Also separately infer the prior class probabilities <math>\ p(C_k)</math>. Then use Bayes’ theorem in the form
 +
 +
<math>\begin{align}p(C_k|x)=\frac{p(x|C_k)p(C_k)}{p(x)} \end{align}</math>
 +
 +
to find the posterior class probabilities <math>\ p(C_k|x)</math>. As usual, the denominator in Bayes’ theorem can be found in terms of the quantities appearing in the numerator, because
 +
 +
<math>\begin{align}p(x)=\sum_{k} p(x|C_k)p(C_k) \end{align}</math>
 +
 +
Equivalently, we can model the joint distribution <math>\ p(x, C_k)</math> directly and then normalize to obtain the posterior probabilities. Having found the posterior probabilities, we use decision theory to determine class membership for each new input <math>\ x</math>. Approaches that explicitly or implicitly model the distribution of inputs as well as outputs are known as "generative models", because by sampling from them it is possible to generate synthetic data points in the input space.
 +
 +
b. First solve the inference problem of determining the posterior class probabilities <math>\ p(C_k|x)</math>, and then subsequently use decision theory to assign each new <math>\ x</math> to one of the classes. Approaches that model the posterior probabilities directly
 +
are called "discriminative models".
 +
 +
c. Find a function <math>\ f(x)</math>, called a discriminant function, which maps each input <math>\ x</math> directly onto a class label. For instance, in the case of two-class problems, <math>\ f(.)</math> might be binary valued and such that <math>\ f = 0</math> represents class <math>\ C_1</math> and <math>\ f = 1</math> represents class <math>\ C_2</math>. In this case, probabilities play no role.
 +
 +
This topic has brought to you from Pattern Recognition and Machine Learning by Christopher M. Bishop (Chapter 1)
  
 
=== Bayes Classifier ===
 
=== Bayes Classifier ===
Line 200: Line 240:
  
 
More information regarding the Bayesian and the frequentist schools of thought are available [http://www.statisticalengineering.com/frequentists_and_bayesians.htm here]. Furthermore, an interesting and informative youtube video that explains the Bayesian and frequentist views of probability is available [http://www.youtube.com/watch?v=hLKOKdAircA here].
 
More information regarding the Bayesian and the frequentist schools of thought are available [http://www.statisticalengineering.com/frequentists_and_bayesians.htm here]. Furthermore, an interesting and informative youtube video that explains the Bayesian and frequentist views of probability is available [http://www.youtube.com/watch?v=hLKOKdAircA here].
 +
 +
There is useful information about Machine Learning, Neural and Statistical Classification in this link [http://www.amsta.leeds.ac.uk/~charles/statlog/] Machine Learning, Neural and Statistical Classification; there is some description of Classification in chapter 2 Classical Statistical Methods in chapter 3 and  Modern Statistical Techniques in chapter 4.
 +
 +
=== Extension: Statistical Classification Framework ===
 +
 +
In statistical classification, each object is represented by 'd' (a set of features) a measurement vector, and the goal of classifier becomes finding compact and disjoint regions for classes in a d-dimensional feature space. Such decision regions are defined by decision rules that are known or can be trained. The simplest configuration of a classification consists of a decision rule and multiple membership functions; each membership function represents a class. The following figures illustrate this general framework.
 +
 +
[[File:cs1.png]]
 +
 +
Simple Conceptual Classifier.
 +
 +
[[File:cs2.png]]
 +
 +
[http://www.orfeo-toolbox.org/SoftwareGuide/SoftwareGuidech17.html#x44-2480011 Statistical Classification Framework]
 +
 +
 +
The classification process can be described as follows:
 +
 +
A measurement vector is input to each membership function.
 +
Membership functions feed the membership scores to the decision rule.
 +
A decision rule compares the membership scores and returns a class label.
  
 
== '''Linear and Quadratic Discriminant Analysis'''  ==
 
== '''Linear and Quadratic Discriminant Analysis'''  ==
 +
 +
===Introduction===
 +
'''Linear discriminant analysis''' ([http://en.wikipedia.org/wiki/Linear_discriminant_analysis LDA]) and the related '''Fisher's linear discriminant''' are methods used in statistics, pattern recognition and machine learning to find a linear combination of features which characterize or separate two or more classes of objects or events. The resulting combination may be used as a linear classifier, or, more commonly, for dimensionality reduction before later classification.
 +
 +
LDA is also closely related to principal component analysis ([http://en.wikipedia.org/wiki/Principal_component_analysis PCA]) and [http://en.wikipedia.org/wiki/Factor_analysis factor analysis] in that both look for linear combinations of variables which best explain the data. LDA explicitly attempts to model the difference between the classes of data. PCA on the other hand does not take into account any difference in class, and factor analysis builds the feature combinations based on differences rather than similarities. Discriminant analysis is also different from factor analysis in that it is not an interdependence technique: a distinction between independent variables and dependent variables (also called criterion variables) must be made.
 +
 +
LDA works when the measurements made on independent variables for each observation are continuous quantities. When dealing with categorical independent variables, the equivalent technique is '''discriminant correspondence analysis'''.
 +
 +
=== Content ===
 
First, we shall limit ourselves to the case where there are two classes, i.e. <math>\, \mathcal{Y}=\{0, 1\}</math>. In the above discussion, we introduced the Bayes classifier's ''decision boundary'' <math>\,D(h^*)=\{x: P(Y=1|X=x)=P(Y=0|X=x)\}</math>, which represents a [http://en.wikipedia.org/wiki/Hyperplane hyperplane] that determines the class of any new data input depending on which side of the hyperplane the input lies in. Now, we shall look at how to derive the Bayes classifier's decision boundary under certain assumptions of the data. [http://en.wikipedia.org/wiki/Linear_discriminant_analysis Linear discriminant analysis (LDA)] and [http://en.wikipedia.org/wiki/Quadratic_classifier#Quadratic_discriminant_analysis quadratic discriminant analysis (QDA)] are two of the most well-known ways for deriving the Bayes classifier's decision boundary, and we shall look at each of them in turn.
 
First, we shall limit ourselves to the case where there are two classes, i.e. <math>\, \mathcal{Y}=\{0, 1\}</math>. In the above discussion, we introduced the Bayes classifier's ''decision boundary'' <math>\,D(h^*)=\{x: P(Y=1|X=x)=P(Y=0|X=x)\}</math>, which represents a [http://en.wikipedia.org/wiki/Hyperplane hyperplane] that determines the class of any new data input depending on which side of the hyperplane the input lies in. Now, we shall look at how to derive the Bayes classifier's decision boundary under certain assumptions of the data. [http://en.wikipedia.org/wiki/Linear_discriminant_analysis Linear discriminant analysis (LDA)] and [http://en.wikipedia.org/wiki/Quadratic_classifier#Quadratic_discriminant_analysis quadratic discriminant analysis (QDA)] are two of the most well-known ways for deriving the Bayes classifier's decision boundary, and we shall look at each of them in turn.
  
Line 238: Line 308:
 
* LDA implicitly assumes that the mean rather than the variance is the discriminating factor.
 
* LDA implicitly assumes that the mean rather than the variance is the discriminating factor.
 
* LDA may over-fit the training data.
 
* LDA may over-fit the training data.
 +
 +
The following link provides a comparison of discriminant analysis and artificial neural networks [http://www.jstor.org/stable/2584434?seq=4]
 +
 +
====Different Approaches to LDA====
 +
Data sets can be transformed and test vectors can be classified in the transformed space by two
 +
different approaches.
 +
 +
*Class-dependent transformation: This type of approach involves maximizing the ratio of between
 +
class variance to within class variance. The main objective is to maximize this ratio so that adequate
 +
class separability is obtained. The class-specific type approach involves using two optimizing criteria
 +
for transforming the data sets independently.
 +
 +
*Class-independent transformation: This approach involves maximizing the ratio of overall variance
 +
to within class variance. This approach uses only one optimizing criterion to transform the data sets
 +
and hence all data points irrespective of their class identity are transformed using this transform. In
 +
this type of LDA, each class is considered as a separate class against all other classes.
 +
 +
== Further reading  ==
 +
The following are some applications that use LDA and QDA:
 +
 +
1- Linear discriminant analysis for improved large vocabulary continuous speech recognition [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=225984 here]
 +
 +
2- 2D-LDA: A statistical linear discriminant analysis for image matrix  [http://www.sciencedirect.com/science?_ob=MImg&_imagekey=B6V15-4DK6B5P-4-1&_cdi=5665&_user=1067412&_pii=S0167865504002272&_origin=search&_coverDate=04%2F01%2F2005&_sk=999739994&view=c&wchp=dGLzVlz-zSkzV&md5=60ea1cf7ff045f76421f5bde64bf855a&ie=/sdarticle.pdf here]
 +
 +
3- Regularization studies of linear discriminant analysis in small sample size scenarios with application to face recognition  [http://www.sciencedirect.com/science?_ob=MImg&_imagekey=B6V15-4DTJVF4-2-9&_cdi=5665&_user=1067412&_pii=S0167865504002260&_origin=search&_coverDate=01%2F15%2F2005&_sk=999739997&view=c&wchp=dGLzVtb-zSkzk&md5=1bba55e357b1c79579987638dcbf6828&ie=/sdarticle.pdf here]
 +
 +
4- The sparse discriminant vectors are useful for supervised dimension reduction for high dimensional data.
 +
Naive application of classical Fisher’s LDA to high dimensional, low sample size settings suffers from the data piling problem.  In [http://www.iaeng.org/IJAM/issues_v39/issue_1/IJAM_39_1_06.pdf] they have use sparse LDA method which selects important variables for discriminant analysis and thereby
 +
yield improved classification. Introducing sparsity in the discriminant vectors is very effective in eliminating data piling and the associated overfitting
 +
problem.
  
 
== '''Linear and Quadratic Discriminant Analysis cont'd - September 23, 2010''' ==
 
== '''Linear and Quadratic Discriminant Analysis cont'd - September 23, 2010''' ==
Line 347: Line 447:
 
A similar transformation of all the centers can be done from <math>\,\mu_k</math> to <math>\,\mu_k^*</math> where <math> \, \mu_k^* \leftarrow S_k^{-\frac{1}{2}}U_k^\top \mu_k </math>.
 
A similar transformation of all the centers can be done from <math>\,\mu_k</math> to <math>\,\mu_k^*</math> where <math> \, \mu_k^* \leftarrow S_k^{-\frac{1}{2}}U_k^\top \mu_k </math>.
  
It is now possible to do classification with <math>\,x^*</math> and <math>\,\mu_k^*</math>, treating them as in Case 1 above.
+
It is now possible to do classification with <math>\,x^*</math> and <math>\,\mu_k^*</math>, treating them as in Case 1 above. This strategy is correct because by transforming <math>\, x</math> to <math>\,x^*</math> where <math> \, x^* \leftarrow S_k^{-\frac{1}{2}}U_k^\top x </math>, the new variable variance is <math>I</math>
 
 
{{Cleanup|date=October 18 2010|reason=The sentence above may cause some misleading. In general case, <math>\,\Sigma_k </math> may not be the same . So you can't treat them completely the same as in Case 1 above. You need to compute <math>\, log{|\Sigma_k |} </math> differently. Here is a detailed discussion below:}}
 
{{Cleanup|date=October 18 2010|reason=The sentence above is right since by transforming<math>\,x</math> to <math>\,x^*</math> where <math> \, x^* \leftarrow S_k^{-\frac{1}{2}}U_k^\top x </math>, the new variable variance is <math>I</math>}}
 
 
 
  
 
Note that when we have multiple classes, we also need to compute <math>\, log{|\Sigma_k|}</math> respectively. Then we compute <math> \,\delta_k </math> for QDA .
 
Note that when we have multiple classes, we also need to compute <math>\, log{|\Sigma_k|}</math> respectively. Then we compute <math> \,\delta_k </math> for QDA .
Line 358: Line 454:
  
 
If the classes have different shapes, in another word, have different covariance <math>\,\Sigma_k</math>, can we use the same method to transform all data points <math>\,x</math> to <math>\,x^*</math>?
 
If the classes have different shapes, in another word, have different covariance <math>\,\Sigma_k</math>, can we use the same method to transform all data points <math>\,x</math> to <math>\,x^*</math>?
 
The answer is NO. Consider that you have two classes with different shapes, then consider transforming them to the same shape. Given a data point, justify which class this point belongs to. The question is, which transformation can you use? For example, if you use the transformation of class A, then you have assumed that this data point belongs to class A.
 
 
{{Cleanup|date=October 18 2010|reason=The statement above may not be true, because in assignment 1, we did do the QDA computation using this approach although the corresponding three covarience matrices are different, the reason why the answer is Yes is as below }}
 
  
 
The answer is Yes. Consider that you have two classes with different shapes. Given a data point, justify which class this point belongs to. You just do the transformations corresponding to the 2 classes respectively, then you get <math>\,\delta_1 ,\delta_2 </math> ,then you determine which class the data point belongs to by comparing <math> \,\delta_1 </math>  and <math> \,\delta_2 </math> .
 
The answer is Yes. Consider that you have two classes with different shapes. Given a data point, justify which class this point belongs to. You just do the transformations corresponding to the 2 classes respectively, then you get <math>\,\delta_1 ,\delta_2 </math> ,then you determine which class the data point belongs to by comparing <math> \,\delta_1 </math>  and <math> \,\delta_2 </math> .
Line 369: Line 461:
 
:: Step 1: For each class <math>\,k</math>, apply singular value decomposition on <math>\,X_k</math> to obtain <math>\,S_k</math> and <math>\,U_k</math>.
 
:: Step 1: For each class <math>\,k</math>, apply singular value decomposition on <math>\,X_k</math> to obtain <math>\,S_k</math> and <math>\,U_k</math>.
  
:: Step 2: For each class <math>\,k</math>, transform each <math>\,x</math> belonging to that class to <math>\,x^* = S_k^{-\frac{1}{2}}U_k^\top x</math>, and transform its center <math>\,\mu_k</math> to <math>\,\mu_k^* = S_k^{-\frac{1}{2}}U_k^\top \mu_k</math>.
+
:: Step 2: For each class <math>\,k</math>, transform each <math>\,x</math> belonging to that class to <math>\,x_k^* = S_k^{-\frac{1}{2}}U_k^\top x</math>, and transform its center <math>\,\mu_k</math> to <math>\,\mu_k^* = S_k^{-\frac{1}{2}}U_k^\top \mu_k</math>.
  
:: Step 3: For each data point <math>\,x \in X</math>, find the squared Euclidean distance between the transformed data point <math>\,x^*</math> and the transformed center <math>\,\mu^*</math> of each class, and assign <math>\,x</math> to the class such that the squared Euclidean distance between <math>\,x^*</math> and <math>\,\mu^*</math> is the least over all of the classes.
+
:: Step 3: For each data point <math>\,x \in X</math>, find the squared Euclidean distance between the transformed data point <math>\,x_k^*</math> and the transformed center <math>\,\mu_k^*</math> of each class <math>\,k</math>, and assign <math>\,x</math> to class <math>\,k</math> such that the squared Euclidean distance between <math>\,x_k^*</math> and <math>\,\mu_k^*</math> is the least for all possible <math>\,k</math>'s.
  
  
Line 402: Line 494:
 
Discriminant analysis (DA) is widely used in classification problems. Except LDA and QDA, there is also an intermediate method between LDA and QDA, a regularized version of discriminant analysis (RDA) proposed by Friedman [1989], and it has been shown to be more flexible in dealing with various class distributions. RDA applies the regularization techniques by using two regularization parameters, which are selected to jointly maximize the classification performance. The optimal pair of parameters is commonly estimated via cross-validation from a set of candidate pairs. More detail about this method can be found in the book by Hastie et al. [2001]. On the other hand, the time of computing last long for high dimensional data, especially when the candidate set is large, which limits the applications of RDA to low dimensional data. In 2006, Ye Jieping and Wang Tie develop a novel algorithm for RDA for high dimensional data. It can estimate the optimal regularization parameters from a large set of parameter candidates efficiently. Experiments on a variety of datasets confirm the claimed theoretical estimate of the efficiency, and also show that, for a properly chosen pair of regularization parameters, RDA performs favourably in classification, in comparison with other existing classification methods. For more details, see Ye, Jieping; Wang, Tie  
 
Discriminant analysis (DA) is widely used in classification problems. Except LDA and QDA, there is also an intermediate method between LDA and QDA, a regularized version of discriminant analysis (RDA) proposed by Friedman [1989], and it has been shown to be more flexible in dealing with various class distributions. RDA applies the regularization techniques by using two regularization parameters, which are selected to jointly maximize the classification performance. The optimal pair of parameters is commonly estimated via cross-validation from a set of candidate pairs. More detail about this method can be found in the book by Hastie et al. [2001]. On the other hand, the time of computing last long for high dimensional data, especially when the candidate set is large, which limits the applications of RDA to low dimensional data. In 2006, Ye Jieping and Wang Tie develop a novel algorithm for RDA for high dimensional data. It can estimate the optimal regularization parameters from a large set of parameter candidates efficiently. Experiments on a variety of datasets confirm the claimed theoretical estimate of the efficiency, and also show that, for a properly chosen pair of regularization parameters, RDA performs favourably in classification, in comparison with other existing classification methods. For more details, see Ye, Jieping; Wang, Tie  
 
Regularized discriminant analysis for high dimensional, low sample size data Conference on Knowledge Discovery in Data: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining; 20-23 Aug. 2006
 
Regularized discriminant analysis for high dimensional, low sample size data Conference on Knowledge Discovery in Data: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining; 20-23 Aug. 2006
 +
 +
===Further Reading for Regularized Discriminant Analysis (RDA)===
 +
 +
1. Regularized Discriminant Analysis and Reduced-Rank LDA
 +
[http://www.stat.psu.edu/~jiali/course/stat597e/notes2/lda2.pdf]
 +
 +
2. Regularized discriminant analysis for the small sample size in face recognition
 +
[http://www.google.ca/url?sa=t&source=web&cd=2&sqi=2&ved=0CCQQFjAB&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.84.6960%26rep%3Drep1%26type%3Dpdf&rct=j&q=Regularized%20Discriminant%20Analysis&ei=IPr2TJ_2MKWV4gaP5eH-Bg&usg=AFQjCNHB3fk6eVe5fSjlQCMfK44kU1-lug&sig2=5EJv_AV3W_ngSVFIa1nfRg&cad=rja.pdf]
 +
 +
3. Regularized Discriminant Analysis and Its Application in Microarrays
 +
[http://www-stat.stanford.edu/~hastie/Papers/RDA-6.pdf]
  
 
== Trick: Using LDA to do QDA  - September 28, 2010==
 
== Trick: Using LDA to do QDA  - September 28, 2010==
Line 418: Line 521:
 
Suppose we can estimate some vector <math>\underline{w}^T</math> such that
 
Suppose we can estimate some vector <math>\underline{w}^T</math> such that
  
<math>y = \underline{w}^Tx</math>
+
<math>y = \underline{w}^T\underline{x}</math>
  
where <math>\underline{w}</math> is a d-dimensional column vector, and <math style="vertical-align:0%;">x\ \epsilon\ \mathbb{R}^d</math> (vector in d dimensions).
+
where <math>\underline{w}</math> is a d-dimensional column vector, and <math style="vertical-align:0%;">\underline{x}\ \epsilon\ \mathbb{R}^d</math> (vector in d dimensions).
  
We also have a non-linear function <math>g(x) = y = x^Tvx + \underline{w}^Tx</math> that we cannot estimate.
+
We also have a non-linear function <math>g(x) = y = \underline{x}^Tv\underline{x} + \underline{w}^T\underline{x}</math> that we cannot estimate.
  
Using our trick, we create two new vectors, <math>\,\underline{w}^*</math> and <math>\,x^*</math> such that:
+
Using our trick, we create two new vectors, <math>\,\underline{w}^*</math> and <math>\,\underline{x}^*</math> such that:
  
 
<math>\underline{w}^{*T} = [w_1,w_2,...,w_d,v_1,v_2,...,v_d]</math>
 
<math>\underline{w}^{*T} = [w_1,w_2,...,w_d,v_1,v_2,...,v_d]</math>
Line 430: Line 533:
 
and
 
and
  
<math>x^{*T} = [x_1,x_2,...,x_d,{x_1}^2,{x_2}^2,...,{x_d}^2]</math>
+
<math>\underline{x}^{*T} = [x_1,x_2,...,x_d,{x_1}^2,{x_2}^2,...,{x_d}^2]</math>
  
We can then estimate a new function, <math>g^*(x,x^2) = y^* = \underline{w}^{*T}x^*</math>.
+
We can then estimate a new function, <math>g^*(\underline{x},\underline{x}^2) = y^* = \underline{w}^{*T}\underline{x}^*</math>.
  
Note that we can do this for any <math>x</math> and in any dimension; we could extend a <math>D \times n</math> matrix to a quadratic dimension by appending another <math>D \times n</math> matrix with the original matrix squared, to a cubic dimension with the original matrix cubed, or even with a different function altogether, such as a <math>\,sin(x)</math> dimension. Pay attention, We don't do QDA with LDA. If we try QDA directly on this problem the resulting decision boundary will be different. Here we try to find a nonlinear boundary for a better possible boundary but it is different with general QDA method. We can call it nonlinear LDA.
+
Note that we can do this for any <math>\, x</math> and in any dimension; we could extend a <math>D \times n</math> matrix to a quadratic dimension by appending another <math>D \times n</math> matrix with the original matrix squared, to a cubic dimension with the original matrix cubed, or even with a different function altogether, such as a <math>\,sin(x)</math> dimension. Pay attention, We don't do QDA with LDA. If we try QDA directly on this problem the resulting decision boundary will be different. Here we try to find a nonlinear boundary for a better possible boundary but it is different with general QDA method. We can call it nonlinear LDA.
  
 
=== By Example ===
 
=== By Example ===
Line 473: Line 576:
  
 
:Not only does LDA give us a better result than it did previously, it actually beats QDA, which only correctly classified 371 data points for this data set. Continuing this procedure by adding another two dimensions with <math>x^4</math> (i.e. we set <code>X_star(i,j+2) = X_star(i,j)^4</code>) we can correctly classify 376 points.
 
:Not only does LDA give us a better result than it did previously, it actually beats QDA, which only correctly classified 371 data points for this data set. Continuing this procedure by adding another two dimensions with <math>x^4</math> (i.e. we set <code>X_star(i,j+2) = X_star(i,j)^4</code>) we can correctly classify 376 points.
 +
 +
===Working Example - Diabetes Data Set===
 +
 +
Let's take a look at specific data set. This is a [http://archive.ics.uci.edu/ml/datasets/Diabetes diabetes data set] from the UC Irvine Machine Learning Repository. It is a fairly small data set by today's standards. The original data had eight variable dimensions. What I did here was to obtain the two prominent principal components from these eight variables. Instead of using the original eight dimensions we will just use these two principal components for this example.
 +
 +
The Diabetes data set has two types of samples in it. One sample type are healthy individuals the other are individuals with a higher risk of diabetes. Here are the prior probabilities estimated for both of the sample types, first for the healthy individuals and second for those individuals at risk:
 +
 +
[[File:eq1.png]]
 +
 +
The first type has a prior probability estimated at 0.651. This means that among the data set, (250 to 300 data points), about 65% of these belong to class one and the other 35% belong to class two. Next, we computed the mean vector for the two classes separately:[[File:eq2.png]]
 +
 +
Then we computed [[File:eq3.jpg]] using the formulas discussed earlier.
 +
 +
Once we have done all of this, we compute the linear discriminant function and found the classification rule. Classification rule:[[File:eq4.jpg]]
 +
 +
In this example, if you give me an <math>\, x</math>, I then plug this value into the above linear function. If the result is greater than or equal to zero, I claim that it is in class one. Otherwise, it is in class two.
 +
Below is a scatter plot of the dominant principle components. The two classes are represented. The first, without diabetes, is shown with red stars (class 1), and the second class, with diabetes, is shown with blue circles (class 2). The solid line represents the classification boundary obtained by LDA. It appears the two classes are not that well separated. The dashed or dotted line is the boundary obtained by linear regression of indicator matrix. In this case, the results of the two different linear boundaries are very close.
 +
 +
[[File:eq5.jpg]]
 +
 +
It is always good practice to visualize the scheme to check for any obvious mistakes.
 +
 +
• Within training data classification error rate: 28.26%.
 +
• Sensitivity: 45.90%.
 +
• Specificity: 85.60%.
 +
 +
Below is the contour plot for the density of the diabetes data (the marginal density for <math>\, x</math> is a mixture of two Gaussians, 2 classes). It looks like a single Gaussian distribution. The reason for this is that the two classes are so close together that they merge into a single mode.
 +
 +
[[File:eq6.jpg]]
  
 
=== LDA and QDA in Matlab ===
 
=== LDA and QDA in Matlab ===
Line 542: Line 674:
  
 
'''Recall: An analysis of the function of <code>princomp</code> in matlab.'''
 
'''Recall: An analysis of the function of <code>princomp</code> in matlab.'''
<br />In our assignment 1, we have learnt that how to perform Principal Component Analysis using SVD method. In fact, the matlab offers us a function called [http://www.mathworks.com/access/helpdesk/help/toolbox/stats/index.html?/access/helpdesk/help/toolbox/stats/princomp.html&http://www.google.cn/search?hl=zh-CN&q=mathwork+princomp&btnG=Google+%E6%90%9C%E7%B4%A2&aq=f&oq= <code>princomp</code>] which can perform PCA conveniently. From the matlab help file on <code>princomp</code>, you can find the details about this function. But here we will analyze the code of the function of <code>princomp()</code> in matlab to find something different when comparing with SVD method. The following is the code of princomp and explanations to some emphasized steps.
+
<br />In our assignment 1, we learned how to perform Principal Component Analysis using the SVD (Singular Value Decomposition) method. In fact, matlab offers a built-in function called [http://www.mathworks.com/access/helpdesk/help/toolbox/stats/index.html?/access/helpdesk/help/toolbox/stats/princomp.html&http://www.google.cn/search?hl=zh-CN&q=mathwork+princomp&btnG=Google+%E6%90%9C%E7%B4%A2&aq=f&oq= <code>princomp</code>] which performs PCA. From the matlab help file on <code>princomp</code>, you can find the details about this function. Here we will analyze Matlab's <code>princomp()</code> code. We find something different than the SVD method we used on our first assignment. The following is Matlab's code for princomp followed by some explanations to emphasize some key steps.
  
 
     function [pc, score, latent, tsquare] = princomp(x);
 
     function [pc, score, latent, tsquare] = princomp(x);
Line 577: Line 709:
 
     tsquare = sum(tmp.*tmp)';
 
     tsquare = sum(tmp.*tmp)';
  
From the above code, we should pay attention to the following aspects when comparing with SVD method:
+
We should compare the following aspects of the above code with the SVD method:
  
 
First, Rows of <math>\,X</math> correspond to observations, columns to variables. When using princomp on 2_3 data in assignment 1, note that we take the transpose of <math>\,X</math>.
 
First, Rows of <math>\,X</math> correspond to observations, columns to variables. When using princomp on 2_3 data in assignment 1, note that we take the transpose of <math>\,X</math>.
Line 587: Line 719:
 
The third, when <math>\,X=UdV'</math>, princomp uses <math>\,V</math> as coefficients for principal components, rather than <math>\,U</math>.
 
The third, when <math>\,X=UdV'</math>, princomp uses <math>\,V</math> as coefficients for principal components, rather than <math>\,U</math>.
  
The following is an example to perform PCA using princomp and SVD respectively to get the same results.
+
The following is an example to perform PCA using princomp and SVD respectively to get the same result.
 
:SVD method
 
:SVD method
 
   >> load 2_3
 
   >> load 2_3
Line 600: Line 732:
 
Then we can see that y=score, v=U.
 
Then we can see that y=score, v=U.
  
'''useful resouces:'''
+
'''useful resources:'''
 
LDA and QDA in Matlab[http://www.mathworks.com/products/statistics/demos.html?file=/products/demos/shipping/stats/classdemo.html],[http://www.mathworks.com/matlabcentral/fileexchange/189],[http://seed.ucsd.edu/~cse190/media07/MatlabClassificationDemo.pdf]
 
LDA and QDA in Matlab[http://www.mathworks.com/products/statistics/demos.html?file=/products/demos/shipping/stats/classdemo.html],[http://www.mathworks.com/matlabcentral/fileexchange/189],[http://seed.ucsd.edu/~cse190/media07/MatlabClassificationDemo.pdf]
  
Line 625: Line 757:
  
 
[http://www.uni-leipzig.de/~strimmer/lab/courses/ss06/seminar/slides/daniela-2x4.pdf LDA & QDA]
 
[http://www.uni-leipzig.de/~strimmer/lab/courses/ss06/seminar/slides/daniela-2x4.pdf LDA & QDA]
 +
 +
Using discriminant analysis for multi-class classification: an experimental investigation [http://www.springerlink.com/content/6851416084227k8p/fulltext.pdf]
  
 
===Reference articles on solving a small sample size problem when LDA is applied===
 
===Reference articles on solving a small sample size problem when LDA is applied===
 
( Based on Li-Fen Chen, Hong-Yuan Mark Liao, Ming-Tat Ko, Ja-Chen Lin, Gwo-Jong Yu  A new LDA-based face recognition system which can solve the small sample size problem Pattern Recognition 33 (2000) 1713-1726 )
 
( Based on Li-Fen Chen, Hong-Yuan Mark Liao, Ming-Tat Ko, Ja-Chen Lin, Gwo-Jong Yu  A new LDA-based face recognition system which can solve the small sample size problem Pattern Recognition 33 (2000) 1713-1726 )
  
Small sample size indicates that the number of samples is smaller than the dimension of each sample. In this case, the within-class covariance we stated in class could be a singular matrix and naturally we cannot find its inverse matrix for further analysis.
+
Small sample size indicates that the number of samples is smaller than the dimension of each sample. In this case, the within-class covariance we stated in class could be a singular matrix and naturally we cannot find its inverse matrix for further analysis.However, many researchers tried to solve it by different techniques:<br />
 
+
1.Goudail et al. proposed a technique which calculated 25 local autocorrelation coefficients from each sample image to achieve dimensionality reduction. (Referenced by F. Goudail, E. Lange, T. Iwamoto, K. Kyuma, N. Otsu, Face recognition system using local autocorrelations and multiscale integration, IEEE Trans. Pattern Anal. Mach. Intell. 18 (10) (1996) 1024-1028.)<br />
However, many researchers tried to solve it by different techniques:
+
2.Swets and Weng applied the PCA approach to accomplish reduction of image dimensionality. (Referenced by D. Swets, J. Weng, Using discriminant eigen features for image retrieval, IEEE Trans. Pattern Anal. Mach. Intell.18 (8) (1996) 831-836.)<br />
 
+
3.Fukunaga proposed a more efficient algorithm and calculated eigenvalues and eigenvectors from an m*m matrix, where n is the dimensionality of the samples and m is the rank of the within-class scatter matrix Sw. (Referenced by K. Fukunaga, Introduction to Statistical Pattern Recognition, Academic Press, New York, 1990.)<br />
1.Goudail et al. proposed a technique which calculated 25 local autocorrelation coefficients from each sample image to achieve dimensionality reduction. (Referenced by F. Goudail, E. Lange, T. Iwamoto, K. Kyuma, N. Otsu, Face recognition system using local autocorrelations and multiscale integration, IEEE Trans. Pattern Anal. Mach. Intell. 18 (10) (1996) 1024-1028.)
+
4.Tian et al. used a positive pseudoinverse matrix instead of calculating the inverse matrix Sw. (Referenced by Q. Tian, M. Barbero, Z.H. Gu, S.H. Lee, Image classification by the Foley-Sammon transform, Opt. Eng. 25 (7) (1986) 834-840.)<br />
 
+
5.Hong and Yang tried to add the singular value perturbation in Sw and made Sw a nonsingular matrix. (Referenced by Zi-Quan Hong, Jing-Yu Yang, Optimal discriminant plane for a small number of samples and design method of classifier on the plane, Pattern Recognition 24 (4) (1991) 317-324)<br />
2.Swets and Weng applied the PCA approach to accomplish reduction of image dimensionality. (Referenced by D. Swets, J. Weng, Using discriminant eigen features for image retrieval, IEEE Trans. Pattern Anal. Mach. Intell.18 (8) (1996) 831-836.)
+
6.Cheng et al. proposed another method based on the principle of rank decomposition of matrices. The above three methods are all based on the conventional Fisher's criterion function. (Referenced by Y.Q. Cheng, Y.M. Zhuang, J.Y. Yang, Optimal fisher discriminant analysis using the rank decomposition, Pattern Recognition 25 (1) (1992) 101-111.)<br />
 
 
3.Fukunaga proposed a more efficient algorithm and calculated eigenvalues and eigenvectors from an m*m matrix, where n is the dimensionality of the samples and m is the rank of the within-class scatter matrix Sw. (Referenced by K. Fukunaga, Introduction to Statistical Pattern Recognition, Academic Press, New York, 1990.)
 
 
 
4.Tian et al. used a positive pseudoinverse matrix instead of calculating the inverse matrix Sw. (Referenced by Q. Tian, M. Barbero, Z.H. Gu, S.H. Lee, Image classification by the Foley-Sammon transform, Opt. Eng. 25 (7) (1986) 834-840.)
 
 
 
5.Hong and Yang tried to add the singular value perturbation in Sw and made Sw a nonsingular matrix. (Referenced by Zi-Quan Hong, Jing-Yu Yang, Optimal discriminant plane for a small number of samples and design method of classifier on the plane, Pattern Recognition 24 (4) (1991) 317-324)
 
 
 
6.Cheng et al. proposed another method based on the principle of rank decomposition of matrices. The above three methods are all based on the conventional Fisher's criterion function. (Referenced by Y.Q. Cheng, Y.M. Zhuang, J.Y. Yang, Optimal fisher discriminant analysis using the rank decomposition, Pattern Recognition 25 (1) (1992) 101-111.)
 
 
 
 
7.Liu et al. modified the conventional Fisher's criterion function and conducted a number of researches based on the new criterion function. They used the total scatter matrix as the divisor of the original Fisher's function instead of merely using the within-class scatter matrix. (Referenced by K. Liu, Y. Cheng, J. Yang, A generalized optimal set of discriminant vectors, Pattern Recognition 25 (7) (1992) 731-739.)
 
7.Liu et al. modified the conventional Fisher's criterion function and conducted a number of researches based on the new criterion function. They used the total scatter matrix as the divisor of the original Fisher's function instead of merely using the within-class scatter matrix. (Referenced by K. Liu, Y. Cheng, J. Yang, A generalized optimal set of discriminant vectors, Pattern Recognition 25 (7) (1992) 731-739.)
  
Line 651: Line 776:
 
===Brief introduction on dimension reduction method===
 
===Brief introduction on dimension reduction method===
  
Dimension reduction is a process to reduce the number of variables of the data by some techniques. Principal components analysis (PCA) and factor analysis are two primary classical methods on dimension reduction. PCA is a method to create some new variables by a linear combination of the variables in the data and the number of new variables depends on what proportion of the variance the new ones contribute. On the contrary, factor analysis method tries to express the old variables by the linear combination of new variables. So before creating the expressions, a certain number of factors should be determined firstly by analysis on the features of old variables. In general, the idea of both PCA and factor analysis is to use as less as possible mixed variables to reflect as more as possible information.
+
Dimension reduction is a process to reduce the number of variables of the data by some techniques. [http://en.wikipedia.org/wiki/Principal_component_analysis Principal components analysis] (PCA) and factor analysis are two primary classical methods on dimension reduction. PCA is a method to create some new variables by a linear combination of the variables in the data and the number of new variables depends on what proportion of the variance the new ones contribute. On the contrary, factor analysis method tries to express the old variables by the linear combination of new variables. So before creating the expressions, a certain number of factors should be determined firstly by analysis on the features of old variables. In general, the idea of both PCA and factor analysis is to use as less as possible mixed variables to reflect as more as possible information.
  
 
===Rough definition===
 
===Rough definition===
Line 663: Line 788:
  
  
Furthermore, if one considers the lower dimensional representation produced by PCA as a least squares fit of our original data, then it can also be easily shown that this representation is the one that minimizes the reconstruction error of our data. It should be noted however, that one usually does not have control over which dimensions PCA deems to be the most informative for a given set of data, and thus one usually does not know which dimensions PCA selects to be the most informative dimensions in order to create the lower-dimensional representation.  
+
Furthermore, if one considers the lower dimensional representation produced by PCA as a least square fit of our original data, then it can also be easily shown that this representation is the one that minimizes the reconstruction error of our data. It should be noted, however, that one usually does not have control over which dimensions PCA deems to be the most informative for a given set of data, and thus one usually does not know which dimensions PCA should be selected to be the most informative dimensions in order to create the lower-dimensional representation.  
  
  
Line 682: Line 807:
  
  
PCA takes a sample of ''d'' - dimensional vectors and produces an orthogonal(zero covariance) set of ''d'' 'Principal Components'. The first Principal Component is the direction of greatest variance in the sample. The second principal component is the direction of second greatest variance (orthogonal to the first component), etc.
+
PCA takes a sample of <math>\, d</math> - dimensional vectors and produces an orthogonal(zero covariance) set of <math>\, d</math> 'Principal Components'. The first Principal Component is the direction of greatest variance in the sample. The second principal component is the direction of second greatest variance (orthogonal to the first component), etc.
  
Then we can preserve most of the variance in the sample in a lower dimension by choosing the first ''k'' Principle Components and approximating the data in ''k'' - dimensional space, which is easier to analyze and plot.
+
Then we can preserve most of the variance in the sample in a lower dimension by choosing the first <math>\, k</math> Principle Components and approximating the data in <math>\, k</math> - dimensional space, which is easier to analyze and plot.
  
 
===Principal Components of handwritten digits===
 
===Principal Components of handwritten digits===
 
Suppose that we have a set of 130 images (28 by 23 pixels) of handwritten threes.  
 
Suppose that we have a set of 130 images (28 by 23 pixels) of handwritten threes.  
{{Cleanup|date=September 6 2010|reason=If anyone can tell me where I can find the 2-3 data set, I would create the new image. In the mean time, I found a non-copyrighted image of different looking 3s online, but as you can see, it is not as nice as one we could make.}}
 
{{Cleanup|date=September 6 2010|reason=I think you can find it on your UW-ACE account for this course.}}
 
{{Cleanup|date=Nov 6 2010|reason=I can't find it on my UW-ACE account for this course, can you tell me the specific directory?}}
 
{{Cleanup|date=Nov 6 2010|reason=You can use the data set of the first assignment}}
 
[[File:Handwritten 3s.gif]]
 
  
  
Line 765: Line 885:
 
====Lagrange Multiplier====
 
====Lagrange Multiplier====
  
Before we can proceed, we must review Lagrange Multipliers.
+
Before we can proceed, we must review Lagrange multipliers.
  
 
[[Image:LagrangeMultipliers2D.svg.png|right|thumb|200px|"The red line shows the constraint g(x,y) = c. The blue lines are contours of f(x,y). The point where the red line tangentially touches a blue contour is our solution." [Lagrange Multipliers, Wikipedia]]]
 
[[Image:LagrangeMultipliers2D.svg.png|right|thumb|200px|"The red line shows the constraint g(x,y) = c. The blue lines are contours of f(x,y). The point where the red line tangentially touches a blue contour is our solution." [Lagrange Multipliers, Wikipedia]]]
Line 784: Line 904:
  
 
====Example====
 
====Example====
Suppose we wish to maximise the function <math>\displaystyle f(x,y)=x-y</math> subject to the constraint <math>\displaystyle x^{2}+y^{2}=1</math>. We can apply the Lagrange multiplier method on this example; the lagrangian is:
+
Suppose we wish to maximize the function <math>\displaystyle f(x,y)=x-y</math> subject to the constraint <math>\displaystyle x^{2}+y^{2}=1</math>. We can apply the Lagrange multiplier method on this example; the lagrangian is:
  
 
<math>\displaystyle L(x,y,\lambda) = x-y - \lambda (x^{2}+y^{2}-1)</math>
 
<math>\displaystyle L(x,y,\lambda) = x-y - \lambda (x^{2}+y^{2}-1)</math>
Line 820: Line 940:
 
<math>\displaystyle S\textbf{w}^* = \lambda^*\textbf{w}^* </math>
 
<math>\displaystyle S\textbf{w}^* = \lambda^*\textbf{w}^* </math>
 
<br><br />
 
<br><br />
{{Cleanup|date=October 2010|reason=It is good discussion, what will happen if we don't have distinct eigenvalues and eigenvectors? What does this situation mean? }}
 
{{Cleanup|date=October 2010|reason=If the eigenvalues are not distinct, I suppose we could still take the leftmost eigenvector by default. Not sure if this is the correct approach, so can anyone please explain further? Thanks }}
 
{{Cleanup|date=October 2010|reason= As U is the eigenvector of a symetric matrix, is it possible that we have 2 similar eigen vector? }}
 
{{Cleanup|date=Nov 6 2010|reason= <math>\lambda_1 \geq \lambda_2 \geq ... \geq \lambda_D </math>, I think it has no difference when eigenvalues are equal }}
 
{{Cleanup|date=Nov 13 2010|reason= Since we're only interested in the largest (left-most) <math>n</math> eigenvalues, if other smaller eigenvalues are equal it does not really matter in PCA. If two eigenvalues are equal, this means that the data has the same amount of variation in each of the two directions that they correspond to. Thus, either one of the directions could be chosen without loss of generality to reduce the variation, unless there is some other consideration that would give an advantage to choosing one direction over another. Note also, that if ALL of the eigenvalues are the same then this means that the data is on the surface of a d-dimensional sphere (all directions have the same amount of variation). Is this correct? }}
 
 
  
 
From the eigenvalue equation <math>\, \textbf{w}^*</math> is an eigenvector of '''S''' and <math>\, \lambda^*</math> is the corresponding eigenvalue of '''S'''. If we substitute <math>\displaystyle\textbf{w}^*</math> in <math>\displaystyle \textbf{w}^T S\textbf{w}</math> we obtain, <br /><br />
 
From the eigenvalue equation <math>\, \textbf{w}^*</math> is an eigenvector of '''S''' and <math>\, \lambda^*</math> is the corresponding eigenvalue of '''S'''. If we substitute <math>\displaystyle\textbf{w}^*</math> in <math>\displaystyle \textbf{w}^T S\textbf{w}</math> we obtain, <br /><br />
Line 831: Line 945:
 
<br><br />
 
<br><br />
 
In order to maximize the objective function we choose the eigenvector corresponding to the largest eigenvalue. We choose the first PC, '''u<sub>1</sub>''' to have the maximum variance<br /> (i.e. capturing as much variability in <math>\displaystyle x_1, x_2,...,x_D </math> as possible.) Subsequent principal components will take up successively smaller parts of the total variability.
 
In order to maximize the objective function we choose the eigenvector corresponding to the largest eigenvalue. We choose the first PC, '''u<sub>1</sub>''' to have the maximum variance<br /> (i.e. capturing as much variability in <math>\displaystyle x_1, x_2,...,x_D </math> as possible.) Subsequent principal components will take up successively smaller parts of the total variability.
 
  
 
D dimensional data will have D eigenvectors
 
D dimensional data will have D eigenvectors
Line 841: Line 954:
 
<math>Var(u_1) \geq Var(u_2) \geq ... \geq Var(u_D)</math>
 
<math>Var(u_1) \geq Var(u_2) \geq ... \geq Var(u_D)</math>
  
 +
If two eigenvalues happen to be equal, then the data has the same amount of variation in each of the two directions that they correspond to with. If only one of the two equal eigenvalues are to be chosen for dimensionality reduction, then either will do. Note that if ALL of the eigenvalues are the same then this means that the data is on the surface of a d-dimensional sphere (all directions have the same amount of variation).
  
 
Note that the Principal Components decompose the total variance in the data:
 
Note that the Principal Components decompose the total variance in the data:
Line 913: Line 1,027:
  
  
==== References ====
+
==== Feature Extraction Uses and Discussion ====
 +
 
 +
PCA, as well as other feature extraction methods not within the scope of the course [http://en.wikipedia.org/wiki/Feature_extraction] are used as a first step to classification in enhancing generalization capability: one of the classification aspects that will be discussed later in the course is model complexity. As a classification model becomes more complex over its training set, classification error over test data tends to increase. By performing feature extraction prior to attempting classification, we restrict model inputs to only the most important variables, thus decreasing complexity and potentially improving test results.
 +
 
 +
Feature ''selection'' methods, that are used to select subsets of relevant features for building robust learning models, differ from extraction methods, where features are transformed. Feature selection has the added benefit of improving model interpretability.
 +
 
 +
 
 +
 
 +
=== Independent Component Analysis ===
 +
As we have already seen, the Principal Component Analysis (PCA) performed by the Karhunen-Lokve transform produces features <math>\ y ( i ) ; i = 0, 1, . . . , N - 1</math>, that are mutually uncorrelated. The obtained by the KL transform solution is optimal when dimensionality reduction is the goal and one wishes to minimize the approximation mean square error. However, for certain applications, the obtained solution falls short of the expectations. In contrast, the more recently developed Independent Component Analysis (ICA) theory, tries to achieve much more than simple decorrelation of the data. The ICA task is casted as follows: Given the set of input samples <math>\ x</math>, determine an <math>\ N \times N</math> invertible matrix <math>\ W</math> such that the entries <math>\ y(i), i = 0, 1, . . . , N - 1</math>, of the transformed vector
 +
 
 +
<math>\ y = W.x</math>
 +
 
 +
are mutually independent. The goal of statistical independence is a stronger condition than the uncorrelatedness required by the PCA. The two conditions are equivalent only for Gaussian random variables. Searching for independent rather than uncorrelated features gives us the means of exploiting a lot more of information, hidden in the higher order statistics of the data.
 +
 
 +
This topic has brought to you from Pattern Recognition by Sergios Theodoridis and Konstantinos Koutroumbas. (Chapter 6) For further details on the ICA and its varieties, refer to this book.
 +
 
 +
=== References ===
 
1. Probabilistic Principal Component Analysis
 
1. Probabilistic Principal Component Analysis
 
[http://onlinelibrary.wiley.com/doi/10.1111/1467-9868.00196/abstract]
 
[http://onlinelibrary.wiley.com/doi/10.1111/1467-9868.00196/abstract]
Line 924: Line 1,055:
  
 
4. Principal Component Analysis
 
4. Principal Component Analysis
[http://onlinelibrary.wiley.com/doi/10.1002/0470013192.bsa501/full]
+
[http://onlinelibrary.wiley.com/doi/10.1002/0470013192.bsa501/full] and [http://support.sas.com/publishing/pubcat/chaps/55129.pdf]
 +
 
 +
=== Further Readings ===
 +
1. I. T. Jolliffe "Principal component analysis" Available [http://books.google.ca/books?id=_olByCrhjwIC&printsec=frontcover&dq=principal+component+analysis&hl=en&ei=TooCTaesN42YnweR843lDQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CC4Q6AEwAA#v=onepage&q&f=false here].
 +
 
 +
2. James V. Stone "Independent component analysis: a tutorial introduction" Available [http://books.google.ca/books?id=P0rROE-WFCwC&pg=PA129&dq=principal+component+analysis&hl=en&ei=TooCTaesN42YnweR843lDQ&sa=X&oi=book_result&ct=result&resnum=7&ved=0CEYQ6AEwBg#v=onepage&q=principal%20component%20analysis&f=false here].
 +
 
 +
3. Aapo Hyvärinen, Juha Karhunen, Erkki Oja "Independent component analysis" Available [http://books.google.ca/books?id=96D0ypDwAkkC&printsec=frontcover&dq=independent+component+analysis&hl=en&ei=F4wCTZqjJY2RnAew6pnlDQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CCoQ6AEwAA#v=onepage&q&f=false here].
  
 
== Fisher's (Linear) Discriminant Analysis (FDA) - Two Class Problem  - October 5, 2010 ==
 
== Fisher's (Linear) Discriminant Analysis (FDA) - Two Class Problem  - October 5, 2010 ==
  
 
===Sir Ronald A. Fisher===
 
===Sir Ronald A. Fisher===
Fisher's Discriminant Analysis (FDA), also known as Fisher's Linear Discriminant Analysis (LDA) in some sources, is a classical feature extraction technique. It was originally described in 1936 by Sir [http://en.wikipedia.org/wiki/Ronald_A._Fisher Ronald Aylmer Fisher], an English statistician and eugenicist who has been described as one of the founders of modern statistical science. His original paper describing FDA can be found [http://digital.library.adelaide.edu.au/dspace/handle/2440/15227 here]; a Wikipedia article summarizing the algorithm can be found [http://en.wikipedia.org/wiki/Linear_discriminant_analysis#Fisher.27s_linear_discriminant  here].  
+
Fisher's Discriminant Analysis (FDA), also known as Fisher's Linear Discriminant Analysis ([http://en.wikipedia.org/wiki/Linear_discriminant_analysis LDA]) in some sources, is a classical [http://en.wikipedia.org/wiki/Feature_extraction feature extraction] technique. It was originally described in 1936 by Sir [http://en.wikipedia.org/wiki/Ronald_A._Fisher Ronald Aylmer Fisher], an English statistician and eugenicist who has been described as one of the founders of modern statistical science. His original paper describing FDA can be found [http://digital.library.adelaide.edu.au/dspace/handle/2440/15227 here]; a Wikipedia article summarizing the algorithm can be found [http://en.wikipedia.org/wiki/Linear_discriminant_analysis#Fisher.27s_linear_discriminant  here].  
 
In this paper Fisher used for the first time the term DISCRIMINANT FUNCTION. The term DISCRIMINANT ANALYSIS was introduced later by Fisher himself in a subsequent paper which can be found [http://digital.library.adelaide.edu.au/coll/special//fisher/155.pdf here].
 
In this paper Fisher used for the first time the term DISCRIMINANT FUNCTION. The term DISCRIMINANT ANALYSIS was introduced later by Fisher himself in a subsequent paper which can be found [http://digital.library.adelaide.edu.au/coll/special//fisher/155.pdf here].
 +
 +
===Introduction===
 +
'''Linear discriminant analysis''' ([http://en.wikipedia.org/wiki/Linear_discriminant_analysis LDA]) and the related '''Fisher's linear discriminant''' are methods used in statistics, pattern recognition and machine learning to find a linear combination of features which characterize or separate two or more classes of objects or events. The resulting combination may be used as a linear classifier, or, more commonly, for dimensionality reduction before later classification.
 +
 +
LDA is also closely related to principal component analysis ([http://en.wikipedia.org/wiki/Principal_component_analysis PCA]) and [http://en.wikipedia.org/wiki/Factor_analysis factor analysis] in that both look for linear combinations of variables which best explain the data. LDA explicitly attempts to model the difference between the classes of data. PCA on the other hand does not take into account any difference in class, and factor analysis builds the feature combinations based on differences rather than similarities. Discriminant analysis is also different from factor analysis in that it is not an interdependence technique: a distinction between independent variables and dependent variables (also called criterion variables) must be made.
 +
 +
LDA works when the measurements made on independent variables for each observation are continuous quantities. When dealing with categorical independent variables, the equivalent technique is '''discriminant correspondence analysis'''.
  
 
=== Contrasting FDA with PCA ===
 
=== Contrasting FDA with PCA ===
Line 938: Line 1,083:
 
Suppose we have 2-dimensional data, then FDA would attempt to project the data of each class onto a point in such a way that the resulting two points would be as far apart from each other as possible. Intuitively, this basic idea behind FDA is the optimal way for separating each pair of classes along a certain direction.  
 
Suppose we have 2-dimensional data, then FDA would attempt to project the data of each class onto a point in such a way that the resulting two points would be as far apart from each other as possible. Intuitively, this basic idea behind FDA is the optimal way for separating each pair of classes along a certain direction.  
  
 +
Please note dimention reduction in PCA is different from subspace cluster , see the details about the subspace cluser [http://en.wikipedia.org/wiki/Clustering_high-dimensional_data]
 
{{Cleanup|date=October 2010|reason= Just a thought: how relevant is "Dimensionality reduction techniques" to the concept of "subspace clustering"? As in subspace clustering, the goal is to find a set of features (relevant features, the concept is referred to as local feature relevance in the literature) in the high dimensional space, where potential subspaces accommodating different classes of data points can be defined. This means; the data points are dense when they are considered in a subset of dimensions (features).}}
 
{{Cleanup|date=October 2010|reason= Just a thought: how relevant is "Dimensionality reduction techniques" to the concept of "subspace clustering"? As in subspace clustering, the goal is to find a set of features (relevant features, the concept is referred to as local feature relevance in the literature) in the high dimensional space, where potential subspaces accommodating different classes of data points can be defined. This means; the data points are dense when they are considered in a subset of dimensions (features).}}
 
{{Cleanup|date=October 2010|reason=If I'm not mistaken, classification techniques like FDA use labeled training data whereas clustering techniques use unlabeled training data instead. Any other input regarding this would be much appreciated. Thanks}}
 
{{Cleanup|date=October 2010|reason=If I'm not mistaken, classification techniques like FDA use labeled training data whereas clustering techniques use unlabeled training data instead. Any other input regarding this would be much appreciated. Thanks}}
Line 1,009: Line 1,155:
  
 
=== Two-class problem  ===
 
=== Two-class problem  ===
In the two-class problem, we have the pre-knowledge that data points belong to two classes. Intuitively speaking points of each class form a cloud around the mean of the class, with each class having possibly different size. To be able to separate the two classes we must determine the class whose mean is closest to a given point while also accounting for the different size of each class, which is represented by the covariance of each class.
+
In the two-class problem, we have prior knowledge that the data points belong to two classes. Conceptually, points of each class form a cloud around the class mean, and each class has an distinct size. To divide points among the two classes, we must determine the class whose mean is closest to each point, and we must also account for the different size of each class given by the covariance of each class.
  
 
Assume <math>\underline{\mu_{1}}=\frac{1}{n_{1}}\displaystyle\sum_{i:y_{i}=1}\underline{x_{i}}</math> and <math>\displaystyle\Sigma_{1}</math>,
 
Assume <math>\underline{\mu_{1}}=\frac{1}{n_{1}}\displaystyle\sum_{i:y_{i}=1}\underline{x_{i}}</math> and <math>\displaystyle\Sigma_{1}</math>,
Line 1,023: Line 1,169:
 
{{Cleanup|date=October 2010|reason=In 2. above, I wonder if the computation would be much more complex if we instead find a weighted sum of the covariances of the two classes where the weights are the sizes of the two classes?}}
 
{{Cleanup|date=October 2010|reason=In 2. above, I wonder if the computation would be much more complex if we instead find a weighted sum of the covariances of the two classes where the weights are the sizes of the two classes?}}
  
 +
{{Cleanup|date=December 2010|reason= If using the weighted sum of two covariances, you will need to use the shared mean of the two classes, and the weighted sum will be the shared covariance. Doing this will result in collapsing the two classes into one point, which contradicts the purpose of using FDA}}
  
 
As is demonstrated below, both of these goals can be accomplished simultaneously.
 
As is demonstrated below, both of these goals can be accomplished simultaneously.
Line 1,189: Line 1,336:
  
 
8. Kodipaka, S.; Vemuri, B.C.; Rangarajan, A.; Leonard, C.M.; Schmallfuss, I.; Eisenschenk, S.; "Kernel Fisher discriminant for shape-based classification in epilepsy" Hournal Medical Image Analysis, 2007. [http://www.sciencedirect.com/science/article/B6W6Y-4MH8BS0-1/2/055fb314828d785a5c3ca3a6bf3c24e9 8]
 
8. Kodipaka, S.; Vemuri, B.C.; Rangarajan, A.; Leonard, C.M.; Schmallfuss, I.; Eisenschenk, S.; "Kernel Fisher discriminant for shape-based classification in epilepsy" Hournal Medical Image Analysis, 2007. [http://www.sciencedirect.com/science/article/B6W6Y-4MH8BS0-1/2/055fb314828d785a5c3ca3a6bf3c24e9 8]
 +
 +
9. Fisher LDA and Kernel Fisher LDA [http://www.ics.uci.edu/~welling/classnotes/papers_class/Fisher-LDA.pdf]
  
 
==Fisher's (Linear) Discriminant Analysis (FDA) - Multi-Class Problem  - October 7, 2010==
 
==Fisher's (Linear) Discriminant Analysis (FDA) - Multi-Class Problem  - October 7, 2010==
  
====Obtaining Covariance Matrices====
+
===Obtaining Covariance Matrices===
  
  
Line 1,360: Line 1,509:
 
eigenvalues with respect to
 
eigenvalues with respect to
  
{{Cleanup|date=What if we encounter complex eigenvalues? Then concept of being large does not dense. What is the solution in that case? }}
+
{{Cleanup|reason=What if we encounter complex eigenvalues? Then concept of being large does not dense. What is the solution in that case? }}
 +
{{Cleanup|date=December 2010|reason=Covariance matrices are positive semi-definite. The inverse of a positive semi-definite matrix is positive semi-definite.  The product of positive semi-definite matrices is positive semi-definite. The eigenvalues of a positive semi-definite matrix are all real, non-negative values. As a result, the eigenvalues of \mathbf{S}_{W}^{-1}\mathbf{S}_{B} will always be real, non-negative values.}}
  
 
:<math>
 
:<math>
Line 1,462: Line 1,612:
 
{{Cleanup|date=October 2010|reason=Would you please show how could we reconstruct our original data from the data that its dimentionality is reduced by FDA.}}
 
{{Cleanup|date=October 2010|reason=Would you please show how could we reconstruct our original data from the data that its dimentionality is reduced by FDA.}}
 
{{Cleanup|date=October 2010|reason= When you reduce the dimensionality of data in most general form you lose some features of the data and you cannot reconstruct the data from redacted space unless the data have special features that help you in reconstruction like sparsity. In FDA it seems that we cannot reconstruct data in general form using reducted version of data  }}
 
{{Cleanup|date=October 2010|reason= When you reduce the dimensionality of data in most general form you lose some features of the data and you cannot reconstruct the data from redacted space unless the data have special features that help you in reconstruction like sparsity. In FDA it seems that we cannot reconstruct data in general form using reducted version of data  }}
 +
 +
====Advantages of FDA compared with PCA====
 +
 +
-PCA find components which are useful for representing data.
 +
 +
-While there is no reason to assume that components are useful to discriminate data between classes.
 +
 +
-In FDA , we try to use labels to find the components which are useful for discriminating data.
  
 
===Generalization of Fisher's Linear Discriminant Analysis ===
 
===Generalization of Fisher's Linear Discriminant Analysis ===
Line 1,473: Line 1,631:
  
 
(MDA) is also termed Discriminant Factor Analysis and Canonical Discriminant Analysis. It adopts a similar perspective to PCA: the rows of the data matrix to be examined constitute points in a multidimensional space, as also do the group mean vectors. Discriminating axes are determined in this space, in such a way that optimal separation of the predefined groups is attained. As with PCA, the problem becomes mathematically the eigenreduction of a real, symmetric matrix. The eigenvalues represent the discriminating power of the associated eigenvectors. The nYgroups lie in a space of dimension at most <math>n_{y-1}</math>. This will be the number of discriminant axes or factors obtainable in the most common practical case when n > m > nY (where n is the number of rows, and m the number of columns of the input data matrix.
 
(MDA) is also termed Discriminant Factor Analysis and Canonical Discriminant Analysis. It adopts a similar perspective to PCA: the rows of the data matrix to be examined constitute points in a multidimensional space, as also do the group mean vectors. Discriminating axes are determined in this space, in such a way that optimal separation of the predefined groups is attained. As with PCA, the problem becomes mathematically the eigenreduction of a real, symmetric matrix. The eigenvalues represent the discriminating power of the associated eigenvectors. The nYgroups lie in a space of dimension at most <math>n_{y-1}</math>. This will be the number of discriminant axes or factors obtainable in the most common practical case when n > m > nY (where n is the number of rows, and m the number of columns of the input data matrix.
 +
 +
===Matlab Example: Multiple Discriminant Analysis for Face Recognition===
 +
 +
% The following MATLAB code is an example of using MDA in face recognition. The used dataset can be    % found be found [http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html here]. IT contains % a set of face images taken between April 1992 and April 1994 at the lab. The database was used in the % context of a face recognition project carried out in collaboration with the Speech, Vision and      % Robotics Group of the Cambridge University Engineering Department.
 +
 +
load orl_faces_112x92.mat
 +
u=(mean(faces'))';
 +
stfaces=faces-u*ones(1,400);
 +
S=stfaces'*stfaces;
 +
[V,E] = eig(S);
 +
U=zeros(length(stfaces),150);%%%%%%
 +
for i=400:-1:251
 +
    U(:,401-i)=stfaces*V(:,i)/sqrt(E(i,i));
 +
end
 +
 +
defaces=U'*stfaces;
 +
for i=1:40
 +
    for j=1:5
 +
        lsamfaces(:,j+5*i-5)=defaces(:,j+10*i-10);
 +
        ltesfaces(:,j+5*i-5)=defaces(:,j+10*i-5);
 +
    end
 +
end
 +
stlsamfaces=lsamfaces-lsamfaces*wdiag(ones(5,5),40)/5;
 +
Sw=stlsamfaces*stlsamfaces';
 +
zstlsamfaces=lsamfaces-(mean(lsamfaces'))'*ones(1,200);
 +
St=zstlsamfaces*zstlsamfaces';
 +
Sb=St-Sw;
 +
[V D]=eig(Sw\Sb);
 +
U=V(:,1:39);
 +
desamfaces=U'*lsamfaces;
 +
detesfaces=U'*ltesfaces;
 +
rightnum=0;
 +
for i=1:200
 +
    mindis=10^10;minplace=1;
 +
    for j=1:200
 +
        distan=norm(desamfaces(:,i)-detesfaces(:,j));
 +
        if mindis>distan
 +
            mindis=distan;
 +
            minplace=j;
 +
        end
 +
    end
 +
    if floor(minplace/5-0.2)==floor(i/5-0.2)
 +
        rightnum=rightnum+1;
 +
    end
 +
end
 +
rightrate=rightnum/200
  
 
===K-NNs Discriminant Analysis===
 
===K-NNs Discriminant Analysis===
Line 1,485: Line 1,689:
 
:4.The k-NN rule is simply defined and implemented, especially if there is insufficient data to adequately define sample means and covariance matrices.
 
:4.The k-NN rule is simply defined and implemented, especially if there is insufficient data to adequately define sample means and covariance matrices.
 
:5.MDA is most appropriately used for feature selection. As in  the case of PCA, we may want to focus on the variables used in order to investigate the differences between groups; to create synthetic variables which improve the grouping ability of the data; to arrive at a similar objective by discarding irrelevant variables; or to determine the most parsimonious variables for graphical representational purposes.
 
:5.MDA is most appropriately used for feature selection. As in  the case of PCA, we may want to focus on the variables used in order to investigate the differences between groups; to create synthetic variables which improve the grouping ability of the data; to arrive at a similar objective by discarding irrelevant variables; or to determine the most parsimonious variables for graphical representational purposes.
 +
 +
=== Fisher Score ===
 +
Fisher Discriminant Analysis should be distinguished from Fisher Score. Feature score is a means, by which we can evaluate the importance of each of the features in a binary classification task. Here is the Fisher score, or in brief <math>\ FS</math>.
 +
 +
<math>FS_i=\frac{(\mu_i^1-\mu_i)^2+(\mu_i^2-\mu_i)^2}{var_i^1+var_i^2}</math>
 +
 +
Where <math>\ \mu_i^1</math>, and <math>\ \mu_i^2</math> are the average of the feature <math>\ i</math> for the class 1 and 2 respectively and <math>\ \mu_i</math> is the average of the feature <math>\ i</math> over both of the classes. And <math>\ var_i^1</math>, and <math>\ var_i^2</math> are the variances of the feature <math>\ i</math> in the two classes of 1 and 2 respectively.
 +
 +
We can estimate the FS over all of the features and then select those features with the highest FS. We want features to discriminate as much as possible between two classes and describe each of the classes as dense as possible; this is exactly the criterion that has been taken into consideration for defining the Fisher Score.
  
  
Line 1,497: Line 1,710:
 
3. Fisher discriminant analysis with kernels
 
3. Fisher discriminant analysis with kernels
 
[http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=788121]
 
[http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=788121]
 +
 +
4. Fisher LDA and Kernel Fisher LDA [http://www.ics.uci.edu/~welling/classnotes/papers_class/Fisher-LDA.pdf]
 +
 +
5. Previous STAT 841 notes. [http://www.math.uwaterloo.ca/~aghodsib/courses/f07stat841/notes/lecture7.pdf]
 +
 +
6. Another useful pdf introducing FDA [http://www.cedar.buffalo.edu/~srihari/CSE555/Chap3.Part6.pdf]
  
 
==Random Projection==
 
==Random Projection==
Line 1,641: Line 1,860:
 
This model does not classify Y between 0 and 1, so it is not good but at times it can lead to a decent classifier. <math>\ y_i=\frac{1}{n_1} </math>  <math>\ \frac{-1}{n_2} </math>
 
This model does not classify Y between 0 and 1, so it is not good but at times it can lead to a decent classifier. <math>\ y_i=\frac{1}{n_1} </math>  <math>\ \frac{-1}{n_2} </math>
 
[[File:Example.jpg]]
 
[[File:Example.jpg]]
 +
 +
==== Recursive Linear Regression ====
 +
In some applications, we need to estimate the weights of the linear regression in an online scheme. For real-time applications, efficiency of computations comes to be very important. In cases like this, we have a batch data and more data samples are still being observed and according to the whole observed data points, we need for example to predict the class label of the upcoming samples. To be able to do that in real-time we should better take advantage of the computations that we have done up to any given sample point and estimate the new weights -having seen the new sample point- using the previous weights -before observing the new sample point. So we want to update the weights, like this:
 +
 +
<math>\ W_{new}=h(W_{old},x_{new},y_{new})</math>
 +
 +
In which <math>\ W_{new}</math> and <math>\ W_{old}</math> are the linear regression weights after and before observation of the new sample pair, <math>\ (x_{new},y_{new})</math>. The function <math>\ h</math> could be obtained using the following procedure.
 +
 +
<math>\begin{align}
 +
W_{old}&=(XX^T-x_{new}x_{new}^T)^{-1}(XY-x_{new}y_{new}) \\
 +
\rightarrow (XX^T-x_{new}x_{new}^T)W_{old}&=XY-x_{new}y_{new}  \\
 +
\rightarrow XX^TW_{old}&=XY-x_{new}y_{new}+x_{new}x_{new}^TW_{old}  \\
 +
\rightarrow W_{old}&=(XX^T)^{-1}(XY-x_{new}y_{new}+x_{new}x_{new}^TW_{old})  \\
 +
\rightarrow W_{old}&=W_{new}+(XX^T)^{-1}(-x_{new}y_{new}+x_{new}x_{new}^TW_{old}) \\
 +
\rightarrow W_{new}&=W_{old}-(XX^T)^{-1}(-x_{new}y_{new}+x_{new}x_{new}^TW_{old})
 +
\end{align}</math>
 +
 +
Where <math>\ X</math>, and <math>\ Y</math> represent the whole set of sample points pairs, including the recently seen sample pair, <math>\ (x_{new},y_{new})</math>.
  
 
====Comments about Linear regression model====
 
====Comments about Linear regression model====
Line 1,649: Line 1,886:
  
 
''Disadvantages'': The main disadvantages of linear least squares are limitations in the shapes that linear models can assume over long ranges, possibly poor extrapolation properties, and sensitivity to outliers. Linear models with nonlinear terms in the predictor variables curve relatively slowly, so for inherently nonlinear processes it becomes increasingly difficult to find a linear model that fits the data well as the range of the data increases. As the explanatory variables become extreme, the output of the linear model will also always more extreme. This means that linear models may not be effective for extrapolating the results of a process for which data cannot be collected in the region of interest. Of course extrapolation is potentially dangerous regardless of the model type. Finally, while the method of least squares often gives optimal estimates of the unknown parameters, it is very sensitive to the presence of unusual data points in the data used to fit a model. One or two outliers can sometimes seriously skew the results of a least squares analysis. This makes model validation, especially with respect to outliers, critical to obtaining sound answers to the questions motivating the construction of the model.
 
''Disadvantages'': The main disadvantages of linear least squares are limitations in the shapes that linear models can assume over long ranges, possibly poor extrapolation properties, and sensitivity to outliers. Linear models with nonlinear terms in the predictor variables curve relatively slowly, so for inherently nonlinear processes it becomes increasingly difficult to find a linear model that fits the data well as the range of the data increases. As the explanatory variables become extreme, the output of the linear model will also always more extreme. This means that linear models may not be effective for extrapolating the results of a process for which data cannot be collected in the region of interest. Of course extrapolation is potentially dangerous regardless of the model type. Finally, while the method of least squares often gives optimal estimates of the unknown parameters, it is very sensitive to the presence of unusual data points in the data used to fit a model. One or two outliers can sometimes seriously skew the results of a least squares analysis. This makes model validation, especially with respect to outliers, critical to obtaining sound answers to the questions motivating the construction of the model.
 +
 +
=====Inverse-Computation Trick for Matrices that are Nearly-Singular=====
 +
 +
The calculation of <math>\, \underline{\beta}</math> in linear regression and in logistic regression (described in the next lecture) requires the calculation of a matrix inverse.  For linear regression, <math>\, (\mathbf{X}\mathbf{X}^T)^{-1} </math> must be calculated.  Likewise, <math>\, (XWX^T)^{-1}</math> must be produced during the iterative method used for logistic regression. When the matrix <math>\, \mathbf{X}\mathbf{X}^T </math> or <math>\, XWX^T</math> is nearly singular, error resulting from numerical roundoff can be very large.  In the case of logistic regression, it may not be possible to determine a solution because the iterative method relies on convergence; with such large error in calculation of the inverse, the solution for entries of <math>\, \underline{\beta}</math> may grow without bound.  To improve the condition of the nearly-singular matrix prior to calculating its inverse, one trick is to add to it a very small identity matrix like <math>\, (10^{-10})I</math>.  This modification has very little effect on the exact result for the inverse matrix, but it improves the numerical calculation considerably.  Now, the inverses to be calculated are <math>\, (\mathbf{X}\mathbf{X}^T + (10^{-10})I)^{-1} </math> and <math>\, (XWX^T + (10^{-10})I)^{-1}</math>
 +
 +
====Multiple Linear Regression Analysis====
 +
Multiple linear regression is a statistical analysis which is similar to Linear Regression with the exception that there can be more than one predictor variable. The assumptions of outliers, linearity and constant variance need to be met. One additional assumption that needs to be examined is multicollinearity. Multicollinearity is the extent to which the predictor variables are related to each other. Multicollinearity can be assessed by asking SPSS for the Variance Inflation Factor (VIF). While different researchers have different criteria for what constitutes too high a VIF number, VIF of 10 or greater is certainly reason for pause. If the VIF is 10 or greater, consider collapsing the variables.
  
 
===Logistic Regression===
 
===Logistic Regression===
Line 1,662: Line 1,906:
 
1. <math>y = \frac{1}{1+e^{-x}}</math>
 
1. <math>y = \frac{1}{1+e^{-x}}</math>
  
2. <math>\frac{dy}{dx} = y(1-y)=\frac{e^{x}}{(1+e^{x})^{2}}</math>
+
2. <math>\frac{dy}{dx} = y(1-y)=\frac{-e^{-x}}{(1+e^{-x})^{2}}</math>
  
 
3. <math>y(0) = \frac{1}{2}</math>
 
3. <math>y(0) = \frac{1}{2}</math>
Line 1,706: Line 1,950:
  
 
====Fitting a Logistic Regression====
 
====Fitting a Logistic Regression====
Logistic regression tries to fit a distribution.  The common practice in statistics is to fit density function, posterior density of each class (Pr(Y|X), to data using [http://en.wikipedia.org/wiki/Maximum_likelihood maximum likelihood]. The maximum likelihood estimate of <math>\underline\beta</math>, denoted <math>\hat \beta_{ML}</math>, maximizes the probability of observing the training data <math>\{y_i,\, x_{i1}, \ldots, x_{id}\}, i=1, \ldots, n</math> from the known distribution.  Combining <math>\displaystyle P(Y=1 | X=x)</math> and <math>\displaystyle P(Y=0 | X=x)</math> as follows, we can consider the two classes at the same time:
+
Logistic regression tries to fit a distribution to the data.  The common practice in statistics is to fit a density function to data using [http://en.wikipedia.org/wiki/Maximum_likelihood maximum likelihood]. The maximum likelihood estimate of <math>\underline\beta</math>, denoted <math>\hat \beta_{ML}</math>, maximizes the probability of observing the training data <math>\{y_i,\, x_{i1}, \ldots, x_{id}\}, i=1, \ldots, n</math> from the maximum likelihood distribution.  Combining <math>\displaystyle P(Y=1 | X=x)</math> and <math>\displaystyle P(Y=0 | X=x)</math> as follows, we can consider the two classes at the same time (this is a useful trick, since <math> y_i \in \{0, 1\}</math>):
  
 
:<math>p(\underline{x_{i}};\underline{\beta}) = \left(\frac{\exp(\underline{\beta}^T \underline{x_i})}{1+\exp(\underline{\beta}^T \underline{x_i})}\right)^{y_i} \left(\frac{1}{1+\exp(\underline{\beta}^T \underline{x_i})}\right)^{1-y_i}</math>
 
:<math>p(\underline{x_{i}};\underline{\beta}) = \left(\frac{\exp(\underline{\beta}^T \underline{x_i})}{1+\exp(\underline{\beta}^T \underline{x_i})}\right)^{y_i} \left(\frac{1}{1+\exp(\underline{\beta}^T \underline{x_i})}\right)^{1-y_i}</math>
Line 1,715: Line 1,959:
 
\begin{align}
 
\begin{align}
 
\mathcal{L}(\theta)&=p({x_{1},...,x_{n}};\theta)\\
 
\mathcal{L}(\theta)&=p({x_{1},...,x_{n}};\theta)\\
&=\displaystyle p(x_{1};\theta) p(x_{2};\theta)... p(x_{n};\theta)  \quad   \mbox{(by independence and identical distribution)}\\
+
&=\displaystyle p(x_{1};\theta) p(x_{2};\theta)... p(x_{n};\theta)  \quad
&= \prod_{i=1}^n p(x_{i};\theta)
+
\end{align}
 +
</math>  (by independence and identical distribution)
 +
:::<math>
 +
\begin{align}
 +
    = \prod_{i=1}^n p(x_{i};\theta)
 
\end{align}
 
\end{align}
 
</math>
 
</math>
Line 1,747: Line 1,995:
 
To solve this equation, the [http://numericalmethods.eng.usf.edu/topics/newton_raphson.html Newton-Raphson algorithm] is used which requires the second derivative of the log-likelihood <math>\,l(\beta)</math> with respect to <math>\,\beta</math> in addition to the first derivative of <math>\,l(\beta)</math> with respect to <math>\,\beta</math>. This is demonstrated in the next section.
 
To solve this equation, the [http://numericalmethods.eng.usf.edu/topics/newton_raphson.html Newton-Raphson algorithm] is used which requires the second derivative of the log-likelihood <math>\,l(\beta)</math> with respect to <math>\,\beta</math> in addition to the first derivative of <math>\,l(\beta)</math> with respect to <math>\,\beta</math>. This is demonstrated in the next section.
  
====Extension====
+
=== Example: logistic Regression in MATLAB ===
  
* When we are dealing with a problem with more than two classes, we need to generalize our logistic regression to a [http://en.wikipedia.org/wiki/Multinomial_logit Multinomial Logit model].
+
% function x = logistic(a, y, w, ridge, param)
*An extension of the logistic model to sets of interdependent variables is the [http://en.wikipedia.org/wiki/Conditional_random_field Conditional random field].
+
%
 +
% Logistic regression.  Design matrix A, targets Y, optional instance
 +
% weights W, optional ridge term RIDGE, optional parameters object PARAM.
 +
%
 +
% W is a vector with length equal to the number of training examples; RIDGE
 +
% can be either a vector with length equal to the number of regressors, or
 +
% a scalar (the latter being synonymous to a vector with all entries the
 +
% same).
 +
%
 +
% PARAM has fields PARAM.MAXITER (an iteration limit), PARAM.VERBOSE
 +
% (whether to print diagnostic information), PARAM.EPSILON (used to test
 +
% convergence), and PARAM.MAXPRINT (how many regression coefficients to
 +
% print if VERBOSE==1).
 +
%
 +
% Model is  
 +
%
 +
%  E(Y) = 1 ./ (1+exp(-A*X))
 +
%
 +
% Outputs are regression coefficients X.
  
* Limitations of Logistic Regression:
+
function x = logistic(a, y, w, ridge, param)
:1. We know that there is no assumptions made about the distributions of the features of the data (i.e. the explanatory variables). However, the features should not be highly correlated with one another because this could cause problems with estimation.
 
:2. Large number of data points (i.e.the sample sizes) are required for logistic regression to provide sufficient estimates of the paramters in both classes. The more number of features/dimensions of the data, the larger the sample size required.
 
:3. According to [http://www.google.ca/url?sa=t&source=web&cd=3&ved=0CC0QFjAC&url=http%3A%2F%2Fwww.csun.edu%2F~ata20315%2Fpsy524%2Fdocs%2FPsy524%2520lecture%252018%2520logistic.ppt&rct=j&q=logistic%20regression%20limitations&ei=mN7RTOC5HcWOnwfP0eho&usg=AFQjCNFBQ8BNxnc7xVArBgVgVWJOnDLMlw&sig2=_6j0mR3r92_xVGtzEJl7oA&cad=rja this source] however, the only real limitation of logistic regression as compared to other types of regression such as linear regression is that the response variable <math>\,y</math> can only take discrete values.
 
  
==Lecture summary==
+
% process parameters
 +
 
 +
[n, m] = size(a);
 +
if ((nargin < 3) || (isempty(w)))
 +
  w = ones(n, 1);
 +
end
 +
if ((nargin < 4) || (isempty(ridge)))
 +
  ridge = 1e-5;
 +
end
 +
if (nargin < 5)
 +
  param = [];
 +
end
 +
if (length(ridge) == 1)
 +
    ridgemat = speye(m) * ridge;
 +
elseif (length(ridge(:)) == m)
 +
    ridgemat = spdiags(ridge(:), 0, m, m);
 +
else
 +
    error('ridge weight vector should be length 1 or %d', m);
 +
end
 +
if (~isfield(param, 'maxiter'))
 +
  param.maxiter = 200;
 +
end
 +
if (~isfield(param, 'verbose'))
 +
  param.verbose = 0;
 +
end
 +
if (~isfield(param, 'epsilon'))
 +
  param.epsilon = 1e-10;
 +
end
 +
if (~isfield(param, 'maxprint'))
 +
  param.maxprint = 5;
 +
end
 +
 
 +
% do the regression
 +
x = zeros(m,1);
 +
oldexpy = -ones(size(y));
 +
for iter = 1:param.maxiter
 +
  adjy = a * x;
 +
  expy = 1 ./ (1 + exp(-adjy));
 +
  deriv = expy .* (1-expy);
 +
  wadjy = w .* (deriv .* adjy + (y-expy));
 +
  weights = spdiags(deriv .* w, 0, n, n);
 +
  x = inv(a' * weights * a + ridgemat) * a' * wadjy;
 +
  if (param.verbose)
 +
    len = min(param.maxprint, length(x));
 +
    fprintf('%3d: [',iter);
 +
    fprintf(' %g', x(1:len));
 +
    if (len < length(x))
 +
      fprintf(' ... ');
 +
    end
 +
    fprintf(' ]\n');
 +
  end
 +
  if (sum(abs(expy-oldexpy)) < n*param.epsilon)
 +
    if (param.verbose)
 +
      fprintf('Converged.\n');
 +
    end
 +
    return;
 +
  end
 +
    oldexpy = expy;
 +
end
 +
warning('logistic:notconverged', 'Failed to converge');
 +
 
 +
 
 +
====Extension====
 +
 
 +
* When we are dealing with a problem with more than two classes, we need to generalize our logistic regression to a [http://en.wikipedia.org/wiki/Multinomial_logit Multinomial Logit model].
 +
*An extension of the logistic model to sets of interdependent variables is the [http://en.wikipedia.org/wiki/Conditional_random_field Conditional random field].
 +
 
 +
* Advantages and Limitations of Linear Regression Model:
 +
:1. Linear regression implements a statistical model that, when relationships between the independent variables and the dependent variable are almost linear, shows optimal results.
 +
:2. Linear regression is often inappropriately used to model non-linear relationships.
 +
:3. Linear regression is limited to predicting numeric output.
 +
:4. A lack of explanation about what has been learned can be a problem.
 +
 
 +
* Limitations of Logistic Regression:
 +
:1. We know that there is no assumptions made about the distributions of the features of the data (i.e. the explanatory variables). However, the features should not be highly correlated with one another because this could cause problems with estimation.
 +
:2. Large number of data points (i.e.the sample sizes) are required for logistic regression to provide sufficient estimates of the paramters in both classes. The more number of features/dimensions of the data, the larger the sample size required.
 +
:3. According to [http://www.google.ca/url?sa=t&source=web&cd=3&ved=0CC0QFjAC&url=http%3A%2F%2Fwww.csun.edu%2F~ata20315%2Fpsy524%2Fdocs%2FPsy524%2520lecture%252018%2520logistic.ppt&rct=j&q=logistic%20regression%20limitations&ei=mN7RTOC5HcWOnwfP0eho&usg=AFQjCNFBQ8BNxnc7xVArBgVgVWJOnDLMlw&sig2=_6j0mR3r92_xVGtzEJl7oA&cad=rja this source] however, the only real limitation of logistic regression as compared to other types of regression such as linear regression is that the response variable <math>\,y</math> can only take discrete values.
 +
 
 +
====Further reading ====
 +
Some supplemental readings on linear and logistic regression:
 +
 
 +
1- A simple method of sample size calculation for linear and logistic regression [http://onlinelibrary.wiley.com/doi/10.1002/%28SICI%291097-0258%2819980730%2917:14%3C1623::AID-SIM871%3E3.0.CO;2-S/pdf here]
 +
 
 +
2- Choosing Between Logistic Regression and Discriminant Analysis [http://www.jstor.org/stable/pdfplus/2286261.pdf?acceptTC=true here]
 +
 
 +
3- On the existence of maximum likelihood estimates in logistic regression models [http://biomet.oxfordjournals.org/content/71/1/1.full.pdf+html here]
 +
 
 +
==Lecture summary==
  
 
This lecture introduced logistic regression as a classification technique by using linear regression as a stepping-stone.  Classification using models found by linear regression is discouraged, but linear regression provides insight into other forms of regression.  However, one important difference between linear and logistic regression is that the former uses the Least-Squares technique to estimate parameters while the latter uses Maximum Likelihood Estimation for this task.  Maximum Likelihood Estimation works by fitting a density function (in this case, a logistic function) that maximizes the probability of observing the training data.  The lecture finishes by noting some caveats of using logistic regression.
 
This lecture introduced logistic regression as a classification technique by using linear regression as a stepping-stone.  Classification using models found by linear regression is discouraged, but linear regression provides insight into other forms of regression.  However, one important difference between linear and logistic regression is that the former uses the Least-Squares technique to estimate parameters while the latter uses Maximum Likelihood Estimation for this task.  Maximum Likelihood Estimation works by fitting a density function (in this case, a logistic function) that maximizes the probability of observing the training data.  The lecture finishes by noting some caveats of using logistic regression.
Line 1,765: Line 2,115:
 
===Logistic Regression Model===
 
===Logistic Regression Model===
  
Recall that in the last lecture, we learned the logistic regression model.
+
In statistics, '''logistic regression''' (sometimes called the '''logistic model''' or '''logit model''') is used for prediction of the probability of occurrence of an event by fitting data to a logit function logistic curve. It is a generalized linear model used for binomial regression. Like many forms of regression analysis, it makes use of several predictor variables that may be either numerical or categorical. For example, the probability that a person has a heart attack within a specified time period might be predicted from knowledge of the person's age, sex and body mass index. Logistic regression is used extensively in the medical and social sciences fields, as well as marketing applications such as prediction of a customer's propensity to purchase a product or cease a subscription.
 +
 
 +
 
 +
Recall that in the last lecture, we learned the logistic regression model:
  
 
* <math>P(Y=1 | X=x)=P(\underline{x};\underline{\beta})=\frac{exp(\underline{\beta}^T \underline{x})}{1+exp(\underline{\beta}^T \underline{x})}</math>
 
* <math>P(Y=1 | X=x)=P(\underline{x};\underline{\beta})=\frac{exp(\underline{\beta}^T \underline{x})}{1+exp(\underline{\beta}^T \underline{x})}</math>
Line 1,803: Line 2,156:
  
 
<math>
 
<math>
X^{new} \leftarrow X^{old} - H^{-1}\nabla
+
X^{new} \leftarrow X^{old} - H^{-1}(f)(X^{old})\nabla f(X^{old})
 
</math>
 
</math>
 
<br />
 
<br />
H is the [http://en.wikipedia.org/wiki/Hessian_matrix Hessian matrix] or second derivative matrix and <math>\,\nabla</math> is the [http://en.wikipedia.org/wiki/Gradient gradient] or first derivative vector, and both H and <math>\,\nabla</math> are evaluated at <math>\,X^{old}</math>
+
where <math>\,H</math> is the [http://en.wikipedia.org/wiki/Hessian_matrix Hessian matrix] or second derivative matrix and <math>\,\nabla</math> is the [http://en.wikipedia.org/wiki/Gradient gradient] or first derivative vector.
 
<br />
 
<br />
  
Line 1,833: Line 2,186:
  
  
In each of the iterative steps, starting with the existing <math>\,\underline{\beta}^{old}</math> which is initialized with an arbitrarily chosen value, the Newton-Raphson updating rule for obtaining <math>\,\underline{\beta}^{new}</math> is
+
In each of the iterative steps, starting with the existing <math>\,\underline{\beta}^{old}</math> which is initialized with an arbitrarily chosen value, the [http://en.wikipedia.org/wiki/Newton-Raphson Newton-Raphson] updating rule for obtaining <math>\,\underline{\beta}^{new}</math> is
  
 
<math>\,\underline{\beta}^{new}\leftarrow \,\underline{\beta}^{old}- (\frac{\partial ^2 l}{\partial \underline{\beta}\partial \underline{\beta}^T})^{-1}(\frac{\partial l}{\partial \underline{\beta}})</math> where the derivatives are evaluated at <math>\,\underline{\beta}^{old}</math>
 
<math>\,\underline{\beta}^{new}\leftarrow \,\underline{\beta}^{old}- (\frac{\partial ^2 l}{\partial \underline{\beta}\partial \underline{\beta}^T})^{-1}(\frac{\partial l}{\partial \underline{\beta}})</math> where the derivatives are evaluated at <math>\,\underline{\beta}^{old}</math>
Line 1,852: Line 2,205:
 
<math>\frac{\partial ^2 l}{\partial \underline{\beta}\partial \underline{\beta}^T} = -XWX^T</math>
 
<math>\frac{\partial ^2 l}{\partial \underline{\beta}\partial \underline{\beta}^T} = -XWX^T</math>
  
The Newton-Raphson step is
+
The [http://en.wikipedia.org/wiki/Newton-Raphson Newton-Raphson] step is
  
 
<math>\underline{\beta}^{new} \leftarrow \underline{\beta}^{old}+(XWX^T)^{-1}X(\underline{Y}-\underline{P})</math>
 
<math>\underline{\beta}^{new} \leftarrow \underline{\beta}^{old}+(XWX^T)^{-1}X(\underline{Y}-\underline{P})</math>
Line 1,888: Line 2,241:
 
#<math>\underline{\beta}^{new} \leftarrow (XWX^T)^{-1}XWZ</math>.
 
#<math>\underline{\beta}^{new} \leftarrow (XWX^T)^{-1}XWZ</math>.
 
#If <math>\underline{\beta}^{new}</math> is sufficiently close to <math>\underline{\beta}^{old}</math> according to an arbitrarily defined criterion, then stop; otherwise, set <math>\,\underline{\beta}^{old} \leftarrow \underline{\beta}^{new}</math> and another iterative step is made towards convergence between <math>\underline{\beta}^{new}</math> and <math>\underline{\beta}^{old}</math>.
 
#If <math>\underline{\beta}^{new}</math> is sufficiently close to <math>\underline{\beta}^{old}</math> according to an arbitrarily defined criterion, then stop; otherwise, set <math>\,\underline{\beta}^{old} \leftarrow \underline{\beta}^{new}</math> and another iterative step is made towards convergence between <math>\underline{\beta}^{new}</math> and <math>\underline{\beta}^{old}</math>.
 +
 +
The following Matlab code implements the method above:
 +
 +
  Error = 0.01;
 +
 
 +
  %Initialize logistic variables
 +
  B_old=0.1*ones(m,1); %beta
 +
  W=0.5*ones(n,n); %weights
 +
  P=zeros(n,1);
 +
  Norm=1;
 +
 
 +
  while Norm>Error %while the change in Beta (represented by the norm between B_new and B_old) is higher than the threshold, iterate
 +
      for i=1:n
 +
          P(i,1)=exp(B_old'*Xnew(:,i))/(1+exp(B_old'*Xnew(:,i)));
 +
          W(i,i)=P(i,1)*(1-P(i,1));
 +
      end
 +
      z = Xnew'*B_old + pinv(W)*(ytrain-P);
 +
      B_new = pinv(Xnew*W*Xnew')*Xnew*W*z;
 +
      Norm=sqrt((B_new-B_old)'*(B_new-B_old));
 +
      B_old = B_new;
 +
  end
  
 
====Classification====
 
====Classification====
Line 1,971: Line 2,345:
 
[[File:Boundary-lda.png‎|frame|center| From this figure, we can see that the results of Logistic Regression and LDA are very similar.]]
 
[[File:Boundary-lda.png‎|frame|center| From this figure, we can see that the results of Logistic Regression and LDA are very similar.]]
  
===Lecture Summary===
+
=== Extra Matlab Examples ===
  
Traditionally, regression parameters are estimated using maximum likelihood. However, other optimization techniques may be used as well.
+
==== Example 1 ====
<br />
+
 
In the case of logistic regression, since there is no closed-form solution for finding zero of the first derivative of the log-likelihood function, the Newton-Raphson algorithm is typically used to estimate parameters. This problem is convex, so the Newton-Raphson algorithm is guaranteed to converge to a global optimum.
+
% This Matlab code provides a function that uses the Newton-Raphson algorithm
<br />
+
% to calculate ML estimates of a simple logistic regression. Most of the
Logistic regression requires less parameters than LDA or QDA, which makes it favorable for high-dimensional data.
+
% code comes from Anders Swensen, "Non-linear regression." There are two
 +
% elements in the beta vector, which we wish to estimate.
 +
 +
function [beta,J_bar] = NR_logistic(data,beta_start)
 +
x=data(:,1); % x is first column of data
 +
y=data(:,2); % y is second column of data
 +
n=length(x)
 +
diff = 1; beta = beta_start; % initial values
 +
while diff>0.0001 % convergence criterion
 +
  beta_old = beta;
 +
  p = exp(beta(1)+beta(2)*x)./(1+exp(beta(1)+beta(2)*x));
 +
  l = sum(y.*log(p)+(1-y).*log(1-p))
 +
  s = [sum(y-p); % scoring function
 +
  sum((y-p).*x)];
 +
  J_bar = [sum(p.*(1-p)) sum(p.*(1-p).*x); % information matrix
 +
  sum(p.*(1-p).*x) sum(p.*(1-p).*x.*x)]
 +
  beta = beta_old + J_bar\s % new value of beta
 +
  diff = sum(abs(beta-beta_old)); % sum of absolute differences
 +
end
 +
 
 +
==== Example 2 ====
 +
 
 +
% This Matlab program illustrates the use of the Newton-Raphson algorithm
 +
% to obtain maximum likelihood estimates of a logistic regression. The data
 +
% and much of the code are taken from Anders Swensen, "Non-linear regression,"
 +
% www.math.uio_no/avdc/kurs/ST110/materiale/opti_30.ps.
 +
% First, load and transform data:
 +
load 'beetle.dat'; % load data
 +
m=length(beetle(:,1)) % count the rows in the data matrix
 +
x=[]; % create empty vectors
 +
y=[];
 +
for j=1:m % expand group data into individual data
 +
  x=[x,beetle(j,1)*ones(1,beetle(j,2))];
 +
  y=[y,ones(1,beetle(j,3)),zeros(1,beetle(j,2)-beetle(j,3))];
 +
end
 +
beetle2=[x;y]';
 +
 +
% Next, specify starting points for iteration on parameter values:
 +
beta0 = [0; 0]
 +
 +
% Finally, call the function NR_logistic and use its output
 +
[betaml,Jbar] = NR_logistic(beetle2,beta0)
 +
covmat = inv(Jbar)
 +
stderr = sqrt(diag(covmat))
 +
 
 +
==== Example 3 ====
 +
 
 +
% function x = logistic(a, y, w)
 +
% Logistic regression.  Design matrix A, targets Y, optional
 +
% instance weights W.  Model is E(Y) = 1 ./ (1+exp(-A*X)).
 +
% Outputs are regression coefficients X.
 +
function x = logistic(a, y, w)
 +
epsilon = 1e-10;
 +
ridge = 1e-5;
 +
maxiter = 200;
 +
[n, m] = size(a);
 +
if nargin < 3
 +
  w = ones(n, 1);
 +
end
 +
x = zeros(m,1);
 +
oldexpy = -ones(size(y));
 +
for iter = 1:maxiter
 +
  adjy = a * x;
 +
  expy = 1 ./ (1 + exp(-adjy));
 +
  deriv = max(epsilon*0.001, expy .* (1-expy));
 +
  adjy = adjy + (y-expy) ./ deriv;
 +
  weights = spdiags(deriv .* w, 0, n, n);
 +
  x = inv(a' * weights * a + ridge*speye(m)) * a' * weights * adjy;
 +
  fprintf('%3d: [',iter);
 +
  fprintf(' %g', x);
 +
  fprintf(' ]\n');
 +
  if (sum(abs(expy-oldexpy)) < n*epsilon)
 +
  fprintf('Converged.\n');
 +
  break;
 +
  end
 +
  oldexpy = expy;
 +
end
 +
 
 +
===Lecture Summary===
 +
 
 +
Traditionally, regression parameters are estimated using maximum likelihood. However, other optimization techniques may be used as well.
 +
<br />
 +
In the case of logistic regression, since there is no closed-form solution for finding zero of the first derivative of the log-likelihood function, the Newton-Raphson algorithm is typically used to estimate parameters. This problem is convex, so the Newton-Raphson algorithm is guaranteed to converge to a global optimum.
 +
<br />
 +
Logistic regression requires less parameters than LDA or QDA, which makes it favorable for high-dimensional data.
  
 
===Supplements===
 
===Supplements===
  
 
A detailed proof that logistic regression is convex is available [http://people.csail.mit.edu/jrennie/writing/convexLR.pdf here]. See '1 Binary LR' for the case we discussed in lecture.
 
A detailed proof that logistic regression is convex is available [http://people.csail.mit.edu/jrennie/writing/convexLR.pdf here]. See '1 Binary LR' for the case we discussed in lecture.
 +
 +
 +
===[http://komarix.org/ac/lr Applications]===
 +
 +
1. Collaborative filtering.
 +
 +
2. Link Analysis.
 +
 +
3. Times Series with Logistic Regression.
 +
 +
4. Alias Detection.
  
 
===References===
 
===References===
Line 1,990: Line 2,459:
 
2. External validity of predictive models: a comparison of logistic regression, classification trees, and neural networks
 
2. External validity of predictive models: a comparison of logistic regression, classification trees, and neural networks
 
[http://www.jclinepi.com/article/S0895-4356%2803%2900120-3/abstract]
 
[http://www.jclinepi.com/article/S0895-4356%2803%2900120-3/abstract]
 +
 +
3. Logistic Regression: A Self-Learning Text by David G. Kleinbaum, Mitchel Klein [http://books.google.ca/books?id=J7E0JQweHkoC&printsec=frontcover&dq=logistic+regression&hl=en&ei=7WECTcvqMp-KnAeaq6HlDQ&sa=X&oi=book_result&ct=result&resnum=3&ved=0CD8Q6AEwAg#v=onepage&q&f=false]
 +
 +
4. Two useful ppt files introducing concepts of logistic regression
 +
[http://www.csun.edu/~ata20315/psy524/docs/Psy524%20lecture%2018%20logistic.pdf] [http://www.daniel-wiechmann.eu/downloads/logreg1.pdf]
  
 
== '''Multi-Class Logistic Regression & Perceptron - October 19, 2010''' ==
 
== '''Multi-Class Logistic Regression & Perceptron - October 19, 2010''' ==
Line 2,033: Line 2,507:
 
These class-conditional probabilities clearly sum to one. <br /><br />
 
These class-conditional probabilities clearly sum to one. <br /><br />
  
In the case of the two-classes problem, it is pretty simple to find the <math>\,\underline{\beta}</math> parameter (the <math>\,\underline{\beta}</math> in two-class logistic regression problems has dimension <math>\,(d+1)\times1</math>), as mentioned in previous lectures. In the multi-class case the iterative Newton method can be used, but here <math>\,\underline{\beta}</math> is of dimension <math>(d+1)\times(k-1)</math> and the weight matrix <math>W</math> is a dense and non-diagonal matrix. This results in a computationally inefficient yet feasible-to-be-solved algorithm. A trick would be to re-parametrize the logistic regression problem. This is done by suitably expanding the following: the input vector <math>\,x</math>, the vector of parameters <math>\,\beta</math>, the vector of responses <math>\,y</math>, as well as the <math>\,\underline{P}</math> vector and the <math>\,W</math> matrix used in the Newton-Raphson updating rule. For interested readers, details regarding this re-parametrization can be found in [http://www.stat.psu.edu/~jiali/course/stat597e/notes2/logit.pdf Jia Li's "Logistic Regression" slides]. Another major difference between the two-classes logistic regression and the general multi-classes logistic regression is that, unlike the former which uses the logistic sigmoid function, the latter uses the softmax function instead. Details regarding the softmax function can be found in [http://www.cedar.buffalo.edu/~srihari/CSE574/Chap4/Chap4-Part3.pdf Sargur N. Srihari's "Logistic Regression" slides].   
+
In the case of the two-classes problem, it is pretty simple to find the <math>\,\underline{\beta}</math> parameter (the <math>\,\underline{\beta}</math> in two-class logistic regression problems has dimension <math>\,(d+1)\times1</math>), as mentioned in previous lectures. In the multi-class case the iterative Newton method can be used, but here <math>\,\underline{\beta}</math> is of dimension <math>\ (d+1)\times(k-1)</math> and the weight matrix <math>W</math> is a dense and non-diagonal matrix. This results in a computationally inefficient yet feasible-to-be-solved algorithm. A trick would be to re-parametrize the logistic regression problem. This is done by suitably expanding the following: the input vector <math>\,x</math>, the vector of parameters <math>\,\beta</math>, the vector of responses <math>\,y</math>, as well as the <math>\,\underline{P}</math> vector and the <math>\,W</math> matrix used in the Newton-Raphson updating rule. For interested readers, details regarding this re-parametrization can be found in [http://www.stat.psu.edu/~jiali/course/stat597e/notes2/logit.pdf Jia Li's "Logistic Regression" slides]. Another major difference between the two-classes logistic regression and the general multi-classes logistic regression is that, unlike the former which uses the logistic sigmoid function, the latter uses the softmax function instead. Details regarding the softmax function can be found in [http://www.cedar.buffalo.edu/~srihari/CSE574/Chap4/Chap4-Part3.pdf Sargur N. Srihari's "Logistic Regression" slides].   
 
The Newton-Raphson updating rule however, remains the same as it is in the two-classes case, i.e. it is still <math>\underline{\beta}^{new} \leftarrow \underline{\beta}^{old}+(XWX^T)^{-1}X(\underline{Y}-\underline{P})</math>. This key point is also addressed in [http://www.stat.psu.edu/~jiali/course/stat597e/notes2/logit.pdf Jia Li's slides] given above.  
 
The Newton-Raphson updating rule however, remains the same as it is in the two-classes case, i.e. it is still <math>\underline{\beta}^{new} \leftarrow \underline{\beta}^{old}+(XWX^T)^{-1}X(\underline{Y}-\underline{P})</math>. This key point is also addressed in [http://www.stat.psu.edu/~jiali/course/stat597e/notes2/logit.pdf Jia Li's slides] given above.  
 
<br /><br />
 
<br /><br />
  
Note that logistic regression does not assume a distribution for the prior. whereas LDA assumes the prior to be Bernoulli. <br /><br />
+
Note that logistic regression does not assume a distribution for the prior, whereas LDA assumes the prior to be Bernoulli. <br /><br />
  
[http://en.wikipedia.org/wiki/Random_multinomial_logit Ramdon multinomail logit] models combine a random ensemble of multinomial logit models for use as a classifier.
+
[http://en.wikipedia.org/wiki/Random_multinomial_logit Random multinomial logit] models combine a random ensemble of multinomial logit models for use as a classifier.
 +
 
 +
=== Multiple Logistic Regression in Matlab ===
  
===Neural Network Concept[http://en.wikipedia.org/wiki/Neural_network]===
 
The concept of constructing an artificial neural network came from scientists who were interested in simulating the human neural network in their computers. They were trying to create computer programs that could learn like people. A neural network is a method in artificial intelligence, and it is a simplified model of neural processing in the brain, even though the relation between this model and brain biological architecture is not yet clear.
 
  
=== Perceptron ===
+
% Examples: Multiple linear regression im Matlab
  
[http://en.wikipedia.org/wiki/Perceptron Perceptron] was invented in 1957 by [http://en.wikipedia.org/wiki/Frank_Rosenblatt Frank Rosenblatt]. It is the basic building block of Feed-Forward neural networks. The perceptron quickly became very popular after it was introduced, because it was shown to be able to solve many classes of useful problems. However, in 1969, [http://en.wikipedia.org/wiki/Marvin_Minsky Marvin Minsky] and [http://en.wikipedia.org/wiki/Seymour_Papert Seymour Papert] published their book [http://en.wikipedia.org/wiki/Perceptrons_%28book%29 ''Perceptrons'' (1969)] in which the authors strongly criticized the perceptron regarding its inability of solving simple [http://en.wikipedia.org/wiki/XOR exclusive-or (XOR)] problems, which are not linearly separable. Indeed, the simple perceptron and the single hidden-layer perceptron neural network are not able to solve any problem that is not linearly-separable. However, it was known to the authors of this book that the multi-layer perceptron neural network can in fact solve any type of problem, including ones that are not linearly separable such as exclusive-or problems, although no efficient learning algorithm was available at that time for this type of neural network. Because of the book ''Perceptrons'', interest regarding perceptrons and neural networks in general greatly declined to a much lower point as compared to before this book was published and things stayed that way until 1986 when the [http://en.wikipedia.org/wiki/Back-propagation back-propagation] learning algorithm (which is discussed in detail below) for neural networks was popularized. <br /><br />
+
% Load data on cars identify weight and horsepower as predictors, mileage as the response:
 +
load carsmall
 +
x1 = Weight;
 +
x2 = Horsepower; % Contains NaN data
 +
y = MPG;
  
We know that the least-squares obtained by regression of -1/1 response variable <math>\displaystyle Y</math> on observation <math>\displaystyle x</math> leads to the same coefficients as LDA (recall that LDA minimizes the distance between discriminant function (decision boundary) and the data points). Least squares returns the sign of the linear combination of features as the class labels (Figure 2). This concept was called the Perceptron in Engineering literature during the 1950's. <br /><br />
+
% Compute regression coefficients for a linear model with an interaction term:
  
[[File:Perceptron.jpg|371px|thumb|right| Fig.2 Diagram of a linear perceptron ]]
+
X = [ones(size(x1)) x1 x2 x1.*x2];
 +
b = regress(y,X); % Removes NaN data
  
There is a cost function <math>\,\displaystyle D</math> that the Perceptron tries to minimize:<br />
+
[[File:mra1.jpg]]
  
<math>D(\underline{\beta},\beta_0)=-\sum_{i \in M}y_{i}(\underline{\beta}^T \underline{x_{i}}+\beta_0)</math><br />
+
% Plot the data and the model:
  
where <math>\,\displaystyle M</math> is the set of misclassified points. <br><br />
+
scatter3(x1,x2,y,'filled','r')
 +
hold on
 +
x1fit = min(x1):100:max(x1);
 +
x2fit = min(x2):10:max(x2);
 +
[X1FIT,X2FIT] = meshgrid(x1fit,x2fit);
 +
YFIT = b(1) + b(2)*X1FIT + b(3)*X2FIT + b(4)*X1FIT.*X2FIT;
 +
mesh(X1FIT,X2FIT,YFIT);
 +
xlabel('Weight');
 +
ylabel('Horsepower');
 +
zlabel('MPG');
 +
view(50,10);
  
By minimizing D, we minimize the sum of the distances between the misclassified points and the decision boundary.<br /><br />
+
=== Matlab Code for Multiple Logistic Regression ===
 +
% Calculation of gradient and objective for Logistic
 +
% Multi-Class Classifcation.
 +
%
 +
% function [obj,grad] = mcclogistic(v,Y,V,lambda,l,varargin)
 +
% v - vector of parameters [n*p*l,1]
 +
% Y - rating matrix (labels) [n,m]
 +
% V - the feature matrix [m,p]
 +
% lambda - regularization parameter [scalar]
 +
% l - # of labels (1..l)
 +
% obj - value of objective at v [scalar]
 +
% grad - gradient at v [n*p*l,1]
 +
%
 +
% Written by Jason Rennie, April 2005
 +
% Last modified: Tue Jul 25 15:08:38 2006
 +
function [obj,grad] = mcclogistic(v,Y,V,lambda,l,varargin)
 +
  fn = mfilename;
 +
  if nargin < 5
 +
    error('insufficient parameters')
 +
  end
 +
  % Parameters that can be set via varargin
 +
  verbose = 1;
 +
  % Process varargin
 +
  paramgt;
 +
 
 +
  t0 = clock;
 +
  [n,m] = size(Y);
 +
  p = length(v)./n./l;
 +
  if p ~= floor(p) | p < 1
 +
    error('dimensions of v and Y don''t match l');
 +
  end
 +
  U = reshape(v,n,p,l);
 +
  Z = zeros(n,m,l);
 +
  for i=1:l
 +
    Z(:,:,i) = U(:,:,i)*V';
 +
  end
 +
  obj = lambda.*sum(sum(sum(U.^2)))./2;
 +
  dU = zeros(n,p,l);
 +
  YY = full(Y==0) + Y;
 +
  YI = sub2ind(size(Z),(1:n)'*ones(1,m),ones(n,1)*(1:m),YY);
 +
  ZY = Z(YI);
 +
  for i=1:l
 +
    obj = obj + sum(sum(h(ZY-Z(:,:,i)).*(Y~=i).*(Y>0)));
 +
  end
 +
  ZHP = zeros(n,m);
 +
  for i=1:l
 +
    ZHP = ZHP + hprime(ZY-Z(:,:,i)).*(Y~=i).*(Y>0);
 +
  end
 +
  for i=1:l
 +
    dU(:,:,i) = ((Y==i).*ZHP - (Y~=i).*(Y>0).*hprime(ZY-Z(:,:,i)))*V + lambda.*U(:,:,i);
 +
  end
 +
  grad = dU(:);
 +
  if verbose
 +
    fprintf(1,'lambda=%.2e obj=%.4e grad''*grad=%.4e time=%.1f\n',lambda,obj,grad'*grad,etime(clock,t0));
 +
  end
  
'''Derivation''':'' The distances between the misclassified points and the decision boundary''.<br /><br />
+
function [ret] = h(z)
 +
  ret = log(1+exp(-z));
  
Consider points <math>\underline{x_1}</math>, <math>\underline{x_2}</math> and a decision boundary defined as <math>\underline{\beta}^T\underline{x}+\beta_0</math> as shown in Figure 3.<br><br />
+
function [ret] = hprime(z)
 +
  ret = -(exp(-z)./(1+exp(-z)));
  
[[File:DB.jpg|248px|thumb|right| Fig.3 Distance from the decision boundary ]]
+
% ChangeLog
 +
% 7/25/06 - Added varargin, verbose
 +
% 3/23/05 - made calcultions take better advantage of sparseness
 +
% 3/18/05 - fixed bug in objective (wasn't squaring fro norms)
 +
% 3/1/05 - added objective calculation
 +
% 2/23/05 - fixed bug in hprime()
  
Both <math>\underline{x_1}</math> and <math>\underline{x_2}</math> lie on the decision boundary, thus:<br />
+
The code is from [http://people.csail.mit.edu/jrennie/matlab/mcclogistic.m here].
 +
Click [http://people.csail.mit.edu/jrennie/matlab/ here] for more information.
 +
 
 +
===Neural Network Concept[http://en.wikipedia.org/wiki/Neural_network]===
 +
The concept of constructing an artificial neural network came from scientists who were interested in simulating the human neural network in their computers. They were trying to create computer programs that could learn like people. A neural network is a method in artificial intelligence, and it was thought to be a simplified model of neural processing in the brain. Later studies showed that the human neural network is much more complicated, and the structure described here is not a good model for the biological architecture of the brain. Although neural network was developed in an attempt to synthesize the human brain, in actuality it has nothing to do with the human neural system.
 +
 
 +
=== Perceptron ===
 +
 
 +
==== Content ====
 +
 
 +
[http://en.wikipedia.org/wiki/Perceptron Perceptron] was invented in 1957 by [http://en.wikipedia.org/wiki/Frank_Rosenblatt Frank Rosenblatt]. It is the basic building block of Feed-Forward neural networks. The perceptron quickly became very popular after it was introduced, because it was shown to be able to solve many classes of useful problems. However, in 1969, [http://en.wikipedia.org/wiki/Marvin_Minsky Marvin Minsky] and [http://en.wikipedia.org/wiki/Seymour_Papert Seymour Papert] published their book [http://en.wikipedia.org/wiki/Perceptrons_%28book%29 ''Perceptrons'' (1969)] in which the authors strongly criticized the perceptron regarding its inability of solving simple [http://en.wikipedia.org/wiki/XOR exclusive-or (XOR)] problems, which are not linearly separable. Indeed, the simple perceptron and the single hidden-layer perceptron neural network [http://homepages.gold.ac.uk/nikolaev/311perc.htm] are not able to solve any problem that is not linearly-separable. However, it was known to the authors of this book that the multi-layer perceptron neural network can in fact solve any type of problem, including ones that are not linearly separable such as exclusive-or problems, although no efficient learning algorithm was available at that time for this type of neural network. Because of the book ''Perceptrons'', interest regarding perceptrons and neural networks in general greatly declined to a much lower point as compared to before this book was published and things stayed that way until 1986 when the [http://en.wikipedia.org/wiki/Back-propagation back-propagation] learning algorithm (which is discussed in detail below) for neural networks was popularized. <br /><br />
 +
 
 +
We know that the least-squares obtained by regression of -1/1 response variable <math>\displaystyle Y</math> on observation <math>\displaystyle x</math> leads to the same coefficients as LDA (recall that LDA minimizes the distance between discriminant function (decision boundary) and the data points). Least squares returns the sign of the linear combination of features as the class labels (Figure 2). This concept was called the Perceptron in Engineering literature during the 1950's. <br /><br />
 +
 
 +
[[File:Perceptron.jpg|371px|thumb|right| Fig.2 Diagram of a linear perceptron ]]
 +
 
 +
There is a cost function <math>\,\displaystyle D</math> that the Perceptron tries to minimize:<br />
 +
 
 +
<math>D(\underline{\beta},\beta_0)=-\sum_{i \in M}y_{i}(\underline{\beta}^T \underline{x_{i}}+\beta_0)</math><br />
 +
 
 +
where <math>\,\displaystyle M</math> is the set of misclassified points. <br><br />
 +
 
 +
By minimizing D, we minimize the sum of the distances between the misclassified points and the decision boundary.<br /><br />
 +
 
 +
'''Derivation''':'' The distances between the misclassified points and the decision boundary''.<br /><br />
 +
 
 +
Consider points <math>\underline{x_1}</math>, <math>\underline{x_2}</math> and a decision boundary defined as <math>\underline{\beta}^T\underline{x}+\beta_0</math> as shown in Figure 3.<br><br />
 +
 
 +
[[File:DB.jpg|248px|thumb|right| Fig.3 Distance from the decision boundary ]]
 +
 
 +
Both <math>\underline{x_1}</math> and <math>\underline{x_2}</math> lie on the decision boundary, thus:<br />
 
<math>\underline{\beta}^T\underline{x_1}+\beta_0=0 \rightarrow (1)</math><br />
 
<math>\underline{\beta}^T\underline{x_1}+\beta_0=0 \rightarrow (1)</math><br />
 
<math>\underline{\beta}^T\underline{x_2}+\beta_0=0 \rightarrow (2)</math><br><br />
 
<math>\underline{\beta}^T\underline{x_2}+\beta_0=0 \rightarrow (2)</math><br><br />
Line 2,087: Line 2,667:
  
 
However, this quantity is not always positive. Consider <math>\,y_{i}(\underline{\beta}^T \underline{x_{i}}+\beta_0)</math>. If <math>\underline{x_{i}}</math> is classified ''correctly'' then this product is positive, since both (<math>\underline{\beta}^T\underline{x_{i}}+\beta_0)</math> and <math>\displaystyle y_{i}</math> are positive or both are negative. However, if <math>\underline{x_{i}}</math> is classified ''incorrectly'', then one of <math>(\underline{\beta}^T\underline{x_{i}}+\beta_0)</math> and <math>\displaystyle y_{i}</math> is positive and the other one is negative; hence, the product <math>y_{i}(\underline{\beta}^T \underline{x_{i}}+\beta_0)</math> will be negative for a misclassified point. The "-" sign in <math>D(\underline{\beta},\beta_0)</math> makes this cost function always positive (since only misclassified points are passed to D). <br /><br />
 
However, this quantity is not always positive. Consider <math>\,y_{i}(\underline{\beta}^T \underline{x_{i}}+\beta_0)</math>. If <math>\underline{x_{i}}</math> is classified ''correctly'' then this product is positive, since both (<math>\underline{\beta}^T\underline{x_{i}}+\beta_0)</math> and <math>\displaystyle y_{i}</math> are positive or both are negative. However, if <math>\underline{x_{i}}</math> is classified ''incorrectly'', then one of <math>(\underline{\beta}^T\underline{x_{i}}+\beta_0)</math> and <math>\displaystyle y_{i}</math> is positive and the other one is negative; hence, the product <math>y_{i}(\underline{\beta}^T \underline{x_{i}}+\beta_0)</math> will be negative for a misclassified point. The "-" sign in <math>D(\underline{\beta},\beta_0)</math> makes this cost function always positive (since only misclassified points are passed to D). <br /><br />
 +
 +
=== Perceptron in Action ===
 +
Here is a Java applet [http://lcn.epfl.ch/tutorial/english/perceptron/html/index.html] which may help with the procedure of Perceptron perception. This applet has been developed in the Laboratory of Computational Neuroscience, University of EPFL, Lausanne, Switzerland.
 +
 +
This second applet [http://www.eee.metu.edu.tr/~alatan/Courses/Demo/AppletPerceptron.html] is developed in the Department of Electrical and Electronics Engineering, Middle East Technical University, Ankara, Turkey.
 +
 +
This third Java applet [http://neuron.eng.wayne.edu/java/Perceptron/New38.html] has been provided by the Computation and Neural Networks Laboratory, College of Engineering, Wayne State University, Detroit, Michigan.
 +
 +
This fourth applet [http://husky.if.uidaho.edu/nn/jdemos/05/Fred%20Corbett/www.etimage.com/java/appletNN/NeuronTyper/MultiLayerPerceptron/perceptron.html] is provided on the official website of the University of Idaho at Idaho Falls.
 +
 +
=== Further Reading for Perceptron ===
 +
 +
1. Neural Network Classifiers Estimate Bayesian a posteriori Probabilities
 +
[http://www.mitpressjournals.org/doi/abs/10.1162/neco.1991.3.4.461]
 +
 +
2. A perceptron network for functional identification and control of nonlinear systems
 +
[http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=286893]
 +
 +
3. Neural network classifiers estimate Bayesian a posteriori probabilities
 +
[http://www.mitpressjournals.org/doi/abs/10.1162/neco.1991.3.4.461]
  
 
==Perceptron Learning Algorithm and Feed Forward Neural Networks - October 21, 2010 ==
 
==Perceptron Learning Algorithm and Feed Forward Neural Networks - October 21, 2010 ==
Line 2,093: Line 2,693:
  
 
To open the Neural Network discussion, we present a formulation of the [http://en.wikipedia.org/wiki/Universal_approximation_theorem universal function approximator]. The mathematical model for Neural Networks is then built upon this formulation. We also discuss the trade-off between training error and testing error -- known as the generalization problem -- under the universal function approximator section.
 
To open the Neural Network discussion, we present a formulation of the [http://en.wikipedia.org/wiki/Universal_approximation_theorem universal function approximator]. The mathematical model for Neural Networks is then built upon this formulation. We also discuss the trade-off between training error and testing error -- known as the generalization problem -- under the universal function approximator section.
 +
 +
There is useful information in [http://page.mi.fu-berlin.de/rojas/neural/chapter/K4.pdf] by R. Rojas about Perceptron learning.
  
 
===Perceptron===
 
===Perceptron===
Line 2,174: Line 2,776:
 
====Some notes on the Perceptron Learning Algorithm====
 
====Some notes on the Perceptron Learning Algorithm====
  
* If there is access to the training data points in a batch form, we should better take advantage of a closed optimization technique like least-squares or maximum-likelihood estimation for linear classifiers. (These closed solutions has been around many years before invention of the Perceptron).
+
* If there is access to the training data points in a batch form, it is better to take advantage of a closed optimization technique like least-squares or maximum-likelihood estimation for linear classifiers. (These closed form solutions have been around many years before the invention of Perceptron).
  
* Just like the linear classifier, a Perceptron can discriminate between only two classes at a time, and one can generalize its performance for multi-class problems by using one of the <math>k-1</math>, <math>k</math>, or <math>k(k-1)/2</math>-hyperplane methods.
+
* Just like a linear classifier, a Perceptron can discriminate between only two classes at a time, and one can generalize its performance for multi-class problems by using one of the <math>k-1</math>, <math>k</math>, or <math>k(k-1)/2</math>-hyperplane methods.
  
 
* If the two classes are linearly separable, the algorithm will converge in a finite number of iterations to a hyperplane, which makes the error of training data zero. The convergence is guaranteed if the learning rate is set adequately.
 
* If the two classes are linearly separable, the algorithm will converge in a finite number of iterations to a hyperplane, which makes the error of training data zero. The convergence is guaranteed if the learning rate is set adequately.
  
* If the two classes are not linearly separable, the algorithm will never converge. So, one may think of a termination criterion in these cases. (e.g. a maximum number of iterations in which convergence is expected, or the rate of changes in both a cost function and its derivative).
+
* If the two classes are not linearly separable, the algorithm will never converge. So, one may think of a termination criterion in these cases (e.g. a maximum number of iterations in which convergence is expected, or the rate of changes in both a cost function and its derivative).
  
* In the case of linearly separable classes, the final solution and number of iterations will be dependent on the initial values (which are arbitrarily chosen), the learning rate (for example, fixed or adaptive), and the gap between the two classes. In general, a smaller gap between classes requires a greater number of iterations for the algorithm to converge.
+
* In the case of linearly separable classes, the final solution and the number of iterations will be dependent on the initial values (which are arbitrarily chosen), the learning rate (for example, fixed or adaptive), and the gap between the two classes. In general, a smaller gap between classes requires a greater number of iterations for the algorithm to converge.
  
* Learning rate --or updating step-- has a direct impact on both the number of iterations and the accuracy of the solution for the optimization problem. Smaller quantities of this factor make convergence slower, even though we will end up with a more accurate solution. In the opposite way, larger values of the learning rate make the process faster, even though we may lose some precision. So, one may make a balance for this trade-off in order to get fast enough to an accurate enough solution (exploration vs. exploitation). In addition, an adaptive learning rate that starts off with a large value and then gradually decreases to a small value over the steps toward convergence can be used in place of a fixed learning rate.  
+
* Learning rate --or updating step-- has a direct impact on both the number of iterations and the accuracy of the solution for the optimization problem. Smaller quantities of this factor make convergence slower, even though we will end up with a more accurate solution. In the opposite way, larger values of the learning rate make the process faster, even though we may lose some precision. So, one may make a balance for this trade-off in order to get to an accurate enough solution fast enough (exploration vs. exploitation). In addition, an adaptive learning rate that starts off with a large value and then gradually decreases to a small value over the steps toward convergence can be used in place of a fixed learning rate.  
  
In the upcoming lectures, we introduce the Support Vector Machines (SVM), which uses a method similar to the iteration optimization scheme to what the Perceptron suggests, but have a different definition for the cost function.
+
In the upcoming lectures, we introduce the Support Vector Machines (SVM), which use a method similar to the iteration optimization scheme to what the Perceptron suggests, but have a different definition for the cost function.
  
 
===An example of the determination on learning rate===
 
===An example of the determination on learning rate===
Line 2,192: Line 2,794:
 
of Satellite Images (IRS-1D) Scientia Iranica, Vol. 15, No. 6, pp. 558-567 )
 
of Satellite Images (IRS-1D) Scientia Iranica, Vol. 15, No. 6, pp. 558-567 )
  
Learning rate play an important role in the application of Neural Network (NN). Choosing an optimum learning rate helps us to obtain a best regression model with the possible fastest speed. In the application of NN by different algorithms, the optimum learning rate tends to be determined differently. In the paper, Optimum Learning Rate in Back-Propagation Neural Network for Classification of Satellite Images (IRS-1D), the author applied one hidden layer and two hidden layers as networks to satellite images by Variable Learning Rate (VLR) algorithms and compared their optimum learning rates based on the various networks. In practice, the number of neurons should not be very small or very large. If the network with too few neurons does not have enough degree of freedom to train the data, but the network with too many neurons is easier to lead to over fitting, so the range of the number of neurons in the experiment is from 3 to 40. Finally, the optimum learning rate under various cases keeps 0.001-0.006. In practice, we could use the similar way to estimate optimum learning rate to improve our models. For more details, please see the article mentioned above.
+
Learning rate plays an important role in the application of Neural Network (NN). Choosing an optimum learning rate helps us to obtain the best regression model with the fastest possible speed. In the application of NN by different algorithms, the optimum learning rate tends to be determined differently. In the paper, Optimum Learning Rate in Back-Propagation Neural Network for Classification of Satellite Images (IRS-1D), the author applied one hidden layer and two hidden layers as networks to satellite images by Variable Learning Rate (VLR) algorithms and compared their optimum learning rates based on the various networks. In practice, the number of neurons should not be very small or very large. Since the network with too few neurons does not have enough degrees of freedom to train the data, but the network with too many neurons is more likely to lead to over fitting, the range of the number of neurons in the experiment is from 3 to 40. Finally, the optimum learning rate under various cases keeps 0.001-0.006. In practice, we could use a similar way to estimate the optimum learning rate to improve our models. For more details, please see the article mentioned above.
 
 
  
 
===Universal Function Approximator===
 
===Universal Function Approximator===
Line 2,219: Line 2,820:
  
 
===Feed-Forward Neural Network===
 
===Feed-Forward Neural Network===
Neural Network (NN) is one instance for the universal function approximator. It can be thought of as a system of Perceptrons linked together as units of a network.  One particular NN useful for classification is the Feed-Forward Neural Network ([http://www.learnartificialneuralnetworks.com/robotcontrol.html#aproach1 FFNN]), which consists of multiple "hidden layers" of Perceptron units (also known as neurons). Our discussion here is based around the FFNN, which has a toplogy shown in Figure 1. The neurons in the first hidden layer take their inputs, the original features (the <math>\,x_i</math>'s), and pass their inputs unchanged as their outputs to the first hidden layer. From the first layer (the input layer) to the last hidden layer, connections from each neuron are always directed to the neurons in the next adjacent layer.  In the output layer, which receives input only from the last hidden layer, each neuron produces a target measurement for a distinct class. <math>\,K</math> classes typically require <math>\,K</math> output neurons in the output layer. In the case where the target variable has two values, it suffices to have one output node in the output layer, although it is generally necessary for the single output node to have a sigmoid activation function so as to restrict the output of the neural network to be a value between 0 and 1. As shown in Figure 1, the neurons in a single layer are typically distributed vertically, and the inputs and outputs of the network are shown as the far left layer and the far right layer, respectively. Furthermore, as shown in Figure 1, it is often useful to add an extra hidden node to each hidden layer that represents the bias term (or the intercept term) of that hidden layer's hyperplane. Each bias node usually outputs a constant value of -1. The purpose of adding a bias node to each hidden layer is to ensure that the hyperplane of that hidden layer does not necessarily have to pass through the origin. In Figure 1, the bias node in the single hidden layer is the topmost hidden node in that layer.   
+
Neural Network (NN) is one instance of the universal function approximator. It can be thought of as a system of Perceptrons linked together as units of a network.  One particular NN useful for classification is the Feed-Forward Neural Network ([http://www.learnartificialneuralnetworks.com/robotcontrol.html#aproach1 FFNN]), which consists of multiple "hidden layers" of Perceptron units (also known as neurons). Our discussion here is based around the FFNN, which has a topology shown in Figure 1. The neurons in the first hidden layer take their inputs, the original features (the <math>\,x_i</math>'s), and pass their inputs unchanged as their outputs to the first hidden layer. From the first layer (the input layer) to the last hidden layer, connections from each neuron are always directed to the neurons in the next adjacent layer.  In the output layer, which receives input only from the last hidden layer, each neuron produces a target measurement for a distinct class. <math>\,K</math> classes typically require <math>\,K</math> output neurons in the output layer. In the case where the target variable has two values, it suffices to have one output node in the output layer, although it is generally necessary for the single output node to have a sigmoid activation function so as to restrict the output of the neural network to be a value between 0 and 1. As shown in Figure 1, the neurons in a single layer are typically distributed vertically, and the inputs and outputs of the network are shown as the far left layer and the far right layer, respectively. Furthermore, as shown in Figure 1, it is often useful to add an extra hidden node to each hidden layer that represents the bias term (or the intercept term) of that hidden layer's hyperplane. Each bias node usually outputs a constant value of -1. The purpose of adding a bias node to each hidden layer is to ensure that the hyperplane of that hidden layer does not necessarily have to pass through the origin. In Figure 1, the bias node in the single hidden layer is the topmost hidden node in that layer.   
  
 
[[File:FFNN.png|300px|thumb|right|Fig.1 A common architecture for the FFNN]]
 
[[File:FFNN.png|300px|thumb|right|Fig.1 A common architecture for the FFNN]]
Line 2,244: Line 2,845:
 
<math>\hat{y}_k = \sum_{j=1}^{p}\underline{w}_{kj}^T\underline{z}_j, k={1,...,K}</math>.
 
<math>\hat{y}_k = \sum_{j=1}^{p}\underline{w}_{kj}^T\underline{z}_j, k={1,...,K}</math>.
  
<math>\,\hat y_k</math> is thus the target measurement for the <math>\,k</math>th class. It is not necessary to use an activation function <math>\,\sigma</math> for each of the hidden and output neurons in the case of regression since the outputs are continuous, though it is necessary to use an activation function <math>\,\sigma</math> for each of the hidden and output neurons in the case of classification so as to ensure that the outputs are discrete.  
+
<math>\,\hat y_k</math> is thus the target measurement for the <math>\,k</math>th class. It is not necessary to use an activation function <math>\,\sigma</math> for each of the hidden and output neurons in the case of regression since the outputs are continuous, though it is necessary to use an activation function <math>\,\sigma</math> for each of the hidden and output neurons in the case of classification so as to ensure that the outputs are in the <math> [0, 1]</math> interval.
 +
{{Cleanup|date=December 2010|reason=The sentence above is misleading, I think. The outputs will not be discrete, we need the activation function in order to keep them in the {0,1} interval. Please correct me if I'm wrong.}}
  
 
Notice that in each neuron, two operations take place one after the other:
 
Notice that in each neuron, two operations take place one after the other:
Line 2,259: Line 2,861:
 
[[File:actfcn.png|300px|thumb|right|Fig.3 <math>tanh</math> as activation function]]
 
[[File:actfcn.png|300px|thumb|right|Fig.3 <math>tanh</math> as activation function]]
  
The NN can be applied as a regression method or as a classifier, and the output layer differs depending on the application. The major difference between regression and classification is in the output space of the model, which is continuous in the case of regression and discrete in the case of classification. For a regression task, no consideration is needed beyond what has already been mentioned earlier, since the outputs of the network would already be continuous. However, to use the neural network as a classifier, as mentioned above, it is necessary to have a threshold stage for each of the hidden and output neurons using an activation function.  
+
The NN can be applied as a regression method or as a classifier, and the output layer differs depending on the application. The major difference between regression and classification is in the output space of the model, which is continuous in the case of regression and discrete in the case of classification. For a regression task, no consideration is needed beyond what has already been mentioned earlier, since the outputs of the network would already be continuous. However, to use the neural network as a classifier, as mentioned above, it is necessary to have a threshold stage for each of the hidden and output neurons using an activation function.
  
 
====Mathematical Model of the FFNN with Multiple Hidden Layers====
 
====Mathematical Model of the FFNN with Multiple Hidden Layers====
Line 2,324: Line 2,926:
  
  
* Propagate <math>\underline{x}</math> forward through the hidden layers one layer at a time, and calculate the outputs of all hidden neurons.
+
* Propagate the outputs of each hidden layer forward, one hidden layer at a time, and calculate the outputs of all hidden neurons.
  
  
* Once <math>\underline{x}</math> reaches the output layer, calculate the output(s) of all output neuron(s).
+
* Once <math>\underline{x}</math> reaches the output layer, calculate the output(s) of all output neuron(s) given the outputs of the previous hidden layer.
  
  
Line 2,337: Line 2,939:
  
 
Usually, a fairly large number of epochs is necessary for training the FFNN so that the network weights would be close to being their optimal values. The learning rate <math> \,\rho </math> should be chosen carefully. Usually, <math> \,\rho </math> should satisfy <math> \,\rho \rightarrow 0 </math> as the iteration times <math> i \rightarrow \infty </math>. [http://www.youtube.com/watch?v=fJ7eH0Y7xEM This] is an interesting video depicting the training procedure of the weights of an FFNN using the back-propagation algorithm.
 
Usually, a fairly large number of epochs is necessary for training the FFNN so that the network weights would be close to being their optimal values. The learning rate <math> \,\rho </math> should be chosen carefully. Usually, <math> \,\rho </math> should satisfy <math> \,\rho \rightarrow 0 </math> as the iteration times <math> i \rightarrow \infty </math>. [http://www.youtube.com/watch?v=fJ7eH0Y7xEM This] is an interesting video depicting the training procedure of the weights of an FFNN using the back-propagation algorithm.
 +
 +
A Matlab implementation of the pseudocode above is given as an example in the Weight Decay subsection under the [[Regularization for Neural Network - November 4, 2010|Regularization]] title.
  
 
====Alternative Description of the Back-Propagation Algorithm====
 
====Alternative Description of the Back-Propagation Algorithm====
Line 2,402: Line 3,006:
 
  end
 
  end
  
====Some notes on the neural network and its learning algorithm====
+
=== The Neural Network Toolbox in Matlab ===
  
* The activation functions are usually linear around the origin. If this is the case, choosing random weights between the <math>\,-0.5</math> and <math>\,0.5</math>, and normalizing the data may boost up the algorithm in the very first steps of the procedure, as the linear combination of the inputs and weights falls within the linear area of the activation function.
+
% Here is a problem consisting of inputs P and targets T that we would like to solve with a network.
 +
P = [0 1 2 3 4 5 6 7 8 9 10];
 +
T = [0 1 2 3 4 3 2 1 2 3 4];
  
* Learning of the neural network using backpropagation algorithm takes place in epochs. An Epoch is a single pass through the entire training set.
+
% Here a network is created with one hidden layer of 5 neurons.
 +
net = newff(P,T,5);
  
* It is a common practice to randomly change the permutation of the training data in each one of the epochs, to make the learning independent of the data permutation.
+
% Here the network is simulated and its output plotted against the targets.
 +
Y = sim(net,P);
 +
plot(P,T,P,Y,’o’)
  
* Given a set of data for training a neural network, one should keep aside a ratio of it as the validation dataset, to obtain a sufficient number of layers and number of neurons in each of the layers. The best construction may be the one which leads to the least error for the validation dataset. Validation data may not be used as the training data of the network (refer to cross-validation and k-fold validation explained in the next lecture).
+
[[File:nn1.jpg]]
 +
 
 +
% Here the network is trained for 50 epochs. Again the network’s output is plotted.
 +
net.trainParam.epochs = 50;
 +
net = train(net,P,T);
 +
Y = sim(net,P);
 +
plot(P,T,P,Y,’o’)
 +
 +
[[File:nn2.jpg]]
 +
 
 +
====Some notes on the neural network and its learning algorithm====
 +
 
 +
* The activation functions are usually linear around the origin. If this is the case, choosing random weights between the <math>\,-0.5</math> and <math>\,0.5</math>, and normalizing the data may boost up the algorithm in the very first steps of the procedure, as the linear combination of the inputs and weights falls within the linear area of the activation function.
 +
 
 +
* Learning of the neural network using backpropagation algorithm takes place in epochs. An Epoch is a single pass through the entire training set.
 +
 
 +
* It is a common practice to randomly change the permutation of the training data in each one of the epochs, to make the learning independent of the data permutation.
 +
 
 +
* Given a set of data for training a neural network, one should keep aside a ratio of it as the validation dataset, to obtain a sufficient number of layers and number of neurons in each of the layers. The best construction may be the one which leads to the least error for the validation dataset. Validation data may not be used as the training data of the network (refer to cross-validation and k-fold validation explained in the next lecture).
  
 
* We can also use the validation-training scheme to estimate how many epochs is enough for training the network.
 
* We can also use the validation-training scheme to estimate how many epochs is enough for training the network.
Line 2,420: Line 3,047:
 
Neural Network with Back-propagation faces some subtleties.
 
Neural Network with Back-propagation faces some subtleties.
  
Deep Neural Networks became popular two or three years ago, when introduced by Dr. Geoffrey E. Hinton. Deep Neural Network training algorithm deals with the training of a Neural Network with a large number of layers.
+
Deep Neural Networks became popular two or three years ago, when introduced by Dr. Geoffrey E. Hinton, a Professor in computer science at the University of Toronto. Deep Neural Network training algorithm [http://www.cs.toronto.edu/~hinton/absps/ncfast.pdf] deals with the training of a Neural Network with a large number of layers.
  
 
The approach of training the deep network is to assume the network has only two layers first and train these two layers. After that we train the next two layers, so on and so forth.
 
The approach of training the deep network is to assume the network has only two layers first and train these two layers. After that we train the next two layers, so on and so forth.
Line 2,495: Line 3,122:
 
* Money Laundering Detection with a Neural-Network  
 
* Money Laundering Detection with a Neural-Network  
 
* Utilizing Fuzzy Logic and Neurofuzzy for Business Advantage
 
* Utilizing Fuzzy Logic and Neurofuzzy for Business Advantage
 +
 +
=== Further readings ===
 +
Bishop,C. "Neural Networks for Pattern Recognition"
 +
 +
Haykin, Simon. "Neural Networks. A Comprehensive Foundation" Available [http://www.esnips.com/doc/83becbe7-0fa6-4f90-a7c4-34697b63a8cb/Neural-Networks---A-Comprehensive-Foundation---Simon-Haykin here]
 +
 +
Nilsson,N. "Introduction to Machine Learning", Chapter 4: Neural Networks. Available [http://robotics.stanford.edu/people/nilsson/mlbook.html here]
 +
 +
Brian D. Ripley "Pattern Recognition and Neural Networks" Available [http://books.google.com/books?id=m12UR8QmLqoC&printsec=frontcover&dq=Neural+Networks+for+Pattern+Recognition&hl=en&ei=r3YCTbOlDMiYnAfh_JXmDQ&sa=X&oi=book_result&ct=result&resnum=3&ved=0CDYQ6AEwAg#v=onepage&q&f=false here]
 +
 +
G. Dreyfus "Neural networks: methodology and applications" Available [http://books.google.com/books?id=g2J4J2bLgRQC&printsec=frontcover&dq=Neural+Networks&hl=en&ei=WncCTaimM86lngeg-OzlDQ&sa=X&oi=book_result&ct=result&resnum=3&ved=0CD4Q6AEwAg#v=onepage&q&f=false here]
  
 
===References===
 
===References===
<references/>
 
  
2. On fuzzy modeling using fuzzy neural networks with the back-propagation algorithm
+
1. On fuzzy modeling using fuzzy neural networks with the back-propagation algorithm
 
[http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=159069]
 
[http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=159069]
  
3. Thirty years of adaptive neural networks: perceptron, madaline and backpropagation
+
2. Thirty years of adaptive neural networks: perceptron, madaline and backpropagation
 
[http://onlinelibrary.wiley.com/doi/10.1002/9780470231616.app7/pdf]
 
[http://onlinelibrary.wiley.com/doi/10.1002/9780470231616.app7/pdf]
  
Line 2,508: Line 3,145:
  
 
=== Lecture Summary ===  
 
=== Lecture Summary ===  
Selecting the model structure with an appropriate complexity is a standard problem in pattern recognition and machine learning. Systems with the optimal complexity have a good [http://www.csc.kth.se/~orre/snns-manual/UserManual/node16.html generalization] to unseen data.
+
Selecting the model structure with an appropriate complexity is a standard problem in pattern recognition and machine learning. Systems with the optimal complexity have a good [http://www.csc.kth.se/~orre/snns-manual/UserManual/node16.html generalization] to yet unobserved data.
  
A wide range of techniques may be used which alter the system complexity. In this lecture, we present the concepts of over-fitting & under-fitting with an example to illustrate how we choose a good classifier and how to avoid over-fitting.
+
A wide range of techniques may be used which alter the system complexity. In this lecture, we present the concepts of over-fitting & under-fitting, and an example to illustrate how we choose a good classifier and how to avoid over-fitting.
  
Moreover, [http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29 cross-validation] has been introduced during the lecture which is a method for estimating generalization error based on "resampling" (Weiss and Kulikowski 1991; Plutowski, Sakata, and White 1994; Shao and Tu 1995)[1],[2],[3]. The resulting estimates of generalization error are often used for choosing among various models. Also, it can be used for model selection by choosing one of several models that has the smallest estimated generalization error. Finally, the common types of cross-validation have been addressed.
+
Moreover, [http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29 cross-validation] has been introduced during the lecture which is a method for estimating generalization error based on "re-sampling" (Weiss and Kulikowski 1991; Plutowski, Sakata, and White 1994; Shao and Tu 1995)[1],[2],[3]. The resulting estimates of generalization error are often used for choosing among various models. A model which is associated with the smallest estimated generalization error would be selected. Finally, the common types of cross-validation have been addressed.
  
Before starting of next section a short description of model complexity is necessary. As the words show model complexity somehow describes complication of our model. Suppose we have a feed forward neural network if we increase the number of the hidden layers or the number of nodes in a specific layer it makes sense that our model is becoming more complex. Or suppose we want to fit a polynomial function on some data points if we add degree of this polynomial it seems that we are choosing a more complex model. It seems that when we choose a complex model it would be better since we have more degree of freedom and we can get more exact answer. In next section will explain why the case is not like this and why there is a trade-off between model complexity and optimal result. This make it necessary to find methods for controlling complexity in model selection. We will see this procedure in an example.
+
Before starting the next section, a short description of model complexity is necessary. As the name suggests, model complexity somehow describes how complicated our model is. Suppose we have a feed forward neural network -- if we increase the number of hidden layers or the number of nodes in a specific layer, it makes sense that our model is becoming more complex. Or, suppose we want to fit a polynomial function on some data points -- if we add to the degree of this polynomial it seems that we are choosing a more complex model. Intuitively, it seems that fitting a more complex model would be better, since we have more degrees of freedom and can get a more exact answer. The next section will explain why this is not the case, and why there is a trade-off between model complexity and optimal result. This makes it necessary to find methods for controlling complexity in model selection. We will see this procedure in an example.
  
 
=== Over-fitting and Under-fitting ===
 
=== Over-fitting and Under-fitting ===
Line 2,524: Line 3,161:
 
#Underfitting
 
#Underfitting
  
Suppose there is no noise in the training data, then we would face no problem with over-fitting, because in this case every training data point lies on the underlying function and the only goal would be to build a model that is as complex as needed to pass through every training data point.  
+
Suppose there is no noise in the training data, then we would face no problem with over-fitting, because in this case every training data point lies on the underlying function, and the only goal is to build a model that is as complex as needed to pass through every training data point.  
  
However, in the real-world, the training data are [http://en.wikipedia.org/wiki/Statistical_noise noisy], i.e. they tend to not lie on the underlying function and instead they tend to be shifted away from the underlying function to unpredictable locations by random noise. If the model is more complex than what it needs to be in order to accurately fit the underlying function, then it would end up fitting through most or all of the training data. Consequently, it would be a poor approximation of the underlying function and it would have poor prediction ability on new unseen data.  
+
However, in the real-world, the training data are [http://en.wikipedia.org/wiki/Statistical_noise noisy], i.e. they tend to not lie exactly on the underlying function, instead they may be shifted to unpredictable locations by random noise. If the model is more complex than what it needs to be in order to accurately fit the underlying function, then it would end up fitting most or all of the training data. Consequently, it would be a poor approximation of the underlying function and have poor prediction ability on new, unseen data.  
  
The dangers of overfitting is that it can easily lead the predictions to the range that is far beyond that of the training data, and produce wild predictions in multilayer perceptrons even with noise-free data. The best way to avoid overfitting is to use lots of training data. But unfortunately it is not always useful. Increasing the training data alone does not guarantee that it will avoid over-fitting. In fact, it is the combination of a good number of training examples and the complexity of the model. The training set should have a sufficient number of data points, so that it is representative of the whole data space which needs to be sampled appropriately.  
+
The danger of overfitting is that the model becomes susceptible to predicting values outside of the range of training data. It can cause wild predictions in multilayer perceptrons, even with noise-free data. The best way to avoid overfitting is to use lots of training data. Unfortunately, that is not always useful. Increasing the training data alone does not guarantee that over-fitting will be avoided. The best strategy is to use a large-enough size training set, and control the complexity of the model. The training set should have a sufficient number of data points which are sampled appropriately, so that it is representative of the whole data space.  
  
In a Neural Network if the depth is too much, the network will have many degrees of freedom and will learn every characteristic of the training data set. That means it will show a very precise outcome of the training set but will not be able to generalize the commonality of the training set to predict the outcome of new cases.
+
In a Neural Network, if the number of hidden layers or nodes is too high, the network will have many degrees of freedom and will learn every characteristic of the training data set. That means it will fit the training set very precisely, but will not be able to generalize the commonality of the training set to predict the outcome of new cases.
  
Underfitting occurs when the model we picked to describe the data is not complex enough, and has high error rate on the training set.
+
Underfitting occurs when the model we picked to describe the data is not complex enough, and has a high error rate on the training set.
 
There is always a trade-off. If our model is too simple, underfitting could occur and if it is too complex, overfitting can occur.
 
There is always a trade-off. If our model is too simple, underfitting could occur and if it is too complex, overfitting can occur.
  
 
'''Example'''
 
'''Example'''
#Consider the example showed in the figure. We have a training set and we want to find a model which fits it the best. We can find a polynomial of high degree which almost passes through all points in the training set. But, in fact the training set is coming from a line model. Although the complex model has less error on the training set it diverges from the line in other ranges which we have no training point. As a result the high degree polynomial has very poor prediction on the test cases. This is an example of overfitting model.
+
#Consider the example shown in the figure. We have a training set and want to find a model which fits it best. We can find a polynomial of high degree which passes through almost all points in the training set. But in reality, the training set comes from a linear model. Although the complex model has little error on the training set, it diverges from the line in other ranges in which we have no training points. As a result, the high degree polynomial has very poor prediction power on the test cases. This is an example of overfitted model.
 
#Now consider a training set which comes from a polynomial of degree two model. If we model this training set with a polynomial of degree one, our model will have high error rate on the training set, and is not complex enough to describe the problem.
 
#Now consider a training set which comes from a polynomial of degree two model. If we model this training set with a polynomial of degree one, our model will have high error rate on the training set, and is not complex enough to describe the problem.
#Consider a simple classification example.  If our classification rule takes as input only the colour of a fruit and concludes that it is a banana, then it is not a good classifier.   The reason is that just because a fruit is a yellow, does not mean that it is a banana.   We can add complexity to our model to make it a better classifier by considering more features typical of bananas, such as size and shape.   If we continue to make our model more and more complex in order to improve our classifier, we will eventually reach a point where the quality of our classifier no longer improves, ie., we have overfit the data.  This occurs when we have considered so many features that we have perfectly described the existing bananas, but if presented with a new banana of slightly different shape than the existing bananas, for example, it cannot be detected.  This is the tradeoff; what is the right level of complexity?
+
#Consider a simple classification example: if our classification rule takes as input only the colour of a fruit and concludes that it is a banana, then it is not a good classifier. The reason is that just because a fruit is yellow, does not mean that it is a banana. We can add complexity to our model to make it a better classifier by considering more features, such as size and shape. If we continue to make our model more and more complex in order to improve our classifier, we will eventually reach a point where the quality of our classifier no longer improves, ie. we have overfit the data.  This occurs when we have considered so many features that we have perfectly described our existing banana that we training on, but if presented with a new banana of a slightly different shape for example, it may not be detected.  This is the tradeoff: what is the right level of complexity?
  
Overfitting occurs when the model becomes too complex and underfitting occurs when it is not complex enough, both of which are not desirable.  To control complexity, it is necessary to make assumptions for the model before fitting the data. Assumptions that we can make for a model are with polynomials or a neural network. There are other ways as well.
+
Overfitting occurs when the model is too complex and underfitting occurs when it is not complex enough, both of which are not desirable.  To control complexity, it is necessary to make assumptions for the model before fitting the data. Some of the assumptions that we can make for a model are with polynomials or a neural network. There are other ways as well.
  
 
[[File:Family_of_polynomials.jpg|200px|thumb|right|Figure 2: An example of a model with a family of polynomials]]
 
[[File:Family_of_polynomials.jpg|200px|thumb|right|Figure 2: An example of a model with a family of polynomials]]
Line 2,548: Line 3,185:
  
 
[[File:extrem_model.jpg|400px|thumb|right|Figure 3]]
 
[[File:extrem_model.jpg|400px|thumb|right|Figure 3]]
After the construction of model is determined, the next problem we meet is do the model selection, that is, how to estimate the parameters effectively, especially when we use iteration method to do the estimation. In the iteration method, the key point is to determine the best time to stop update parameters.
+
After the structure of the model is determined, the next step is do the model selection. The problem encountered is how to estimate the parameters effectively, especially when we use iteration methods to do the estimation. In the iteration method, the key point is to determine the best time to stop updating parameters.
Let us see a very simple example;   assume the dotted line on the graph can be expressed as a function, and the data points, the circles, are generated by the function with added noise.  
+
Let us see a very simple example; assume the dotted line on the graph can be expressed as a function <math>\,h(x)</math>, and the data points (the circles) are generated by the function with added noise.  
  
  
 
'''Model 1'''(as shown on the left of Figure 3)
 
'''Model 1'''(as shown on the left of Figure 3)
A line can be used to describe the data points, where two parameter are needed to construct the estimate of the function.  However, it is clear that it performs badly. This model is a typical example of underfitted model. In this case, the model will perform well in prediction, but a large bias could be generated.
+
A line <math>\,g(x)</math> can be used to describe the data points, where two parameters are needed to construct the estimated function.  However, it is clear that it performs badly. This model is a typical example of an underfitted model. In this case, the model will perform well in prediction, but a large bias could be generated.
  
 
'''Model 2''' (as shown on the right of Figure 3)
 
'''Model 2''' (as shown on the right of Figure 3)
in this model, lots of parameter are used to fit the data points. Although it looks pretty good on fitting, the performance on prediction could be very bad, which means this model will generate a large variance when we use it on the data points which are not in the training data.
+
In this model, lots of parameters are used to fit the data. Although it looks like a fairly good fit, the prediction performance could be very bad. This means that this model will generate a large variance when we use it on points not part of the training data.
The models above are the extreme case in the model selection, we do not want to choose any of them in our classification work. So the key thing is to stop our training work at the optimal time such that the balance of bias and variance would be obtained, that is, the time t in the following graph.  
+
The models above are the extreme cases in the model selection, we do not want to choose any of them in our classification task. The key is to stop our training process at the optimal time, such that a balance of bias and variance is obtained, that is, the time t in the following graph.  
  
 
[[File:optimal_time.jpg|300px|thumb|right|Figure 4]]
 
[[File:optimal_time.jpg|300px|thumb|right|Figure 4]]
  
To achieve the aim, one approach we can use is to divide our data points into two groups and make them independently; one (training set) is used in the training test to obtain parameters, the other one (validation set) is used for determining the optimal time. After every updated parameter, the test in the validation set is implement  and plot the curve of error in the two test in order to find the optimal point t. Here, the validation test is a good measure of generalization. Remember that do not update the parameters in the validation test. If one more independent test we need to follow, three independent groups should be determined at the beginning. In addition, this approach is suitable for the case of more data points, especially a finite data set, since the effect on noise could be decreased to the lowest level.  
+
To achieve this goal, one approach we can use is to divide our data points into two groups: one (training set) is used in the training process to obtain parameters, the other one (validation set) is used for determining the optimal model. After every update of parameters, the test in the validation set is implemented and the error curve is plotted to find the optimal point <math>\,t</math>. Here, the validation test is a good measure of generalization. Remember to not update the parameters in the validation test. If another, independent test is needed to follow validation, three independent groups should be determined at the beginning. In addition, this approach is suitable for the case of more data points, especially a finite data set, since the effect of noise could be decreased to the lowest level.  
  
So far, we have learn two most popular ways to estimate the expected level of fit of a model to a data set that is independent of the data that were used to train the model:
+
So far, we have learned two of the most popular ways to estimate the expected level of fit of a model to a test data set that is independent of the data used to train the model:
 
:1. Cross validation
 
:1. Cross validation
:2. Regularization: refers to a series of techniques we can use to suppress overfitting,that is, making our function not so curved such that it performance badly in the prediction. The specific way is to add a new penalty term into the error function, and it tends to limit the over-increasing the weight when the weight update by iteration.
+
:2. Regularization: refers to a series of techniques we can use to suppress overfitting, that is, making our function not so curved that it performs badly in prediction. The specific way is to add a new penalty term into the error function, this prevents increasing the weights too much when they are updated at each iteration.
  
 
Indeed, there are many techniques could be used, such as:
 
Indeed, there are many techniques could be used, such as:
Line 2,573: Line 3,210:
  
 
===='''Note'''====
 
===='''Note'''====
When the model is linear, the true error form AIC approach is identical to that from Cp approach; When the model is nonlinear, they are different.
+
When the model is linear, the true error form AIC approach is identical to that from Cp approach; when the model is nonlinear, they are different.
  
 
=== '''How do we choose a good classifier?''' ===
 
=== '''How do we choose a good classifier?''' ===
Line 2,586: Line 3,223:
  
 
<span id="prediction-error">[[File:Prediction_Error.jpg|200px|thumb|right|Figure 3]]</span>
 
<span id="prediction-error">[[File:Prediction_Error.jpg|200px|thumb|right|Figure 3]]</span>
There is a downward bias to training error estimate that it is always less than the true error rate.  
+
There is a downward bias to the training error estimate, it is always less than the true error rate.  
  
If there is a change in our complexity from low to high, our training (empirical) error rate is always decreasing. When we apply our model to the test data, our error rate will start to decrease to a point, but then it will increase since the model hasn't seen the datapoints in the test set before.  This results in a convex test error curve as a function of learning model complexity. This can be explained since training error will decrease when we fit the model better by increasing its complexity, but as we have seen, this complex model will not generalize well, resulting in a larger test error.  
+
If there is a change in our complexity from low to high, our training (empirical) error rate is always decreased. When we apply our model to the test data, our error rate will decrease to a point, but then it will increase because the model has not seen the test data points before.  This results in a convex test error curve as a function of learning model complexity. The training error will decrease when we keep fitting increasingly complex models, but as we have seen, a model too complex will not generalize well, resulting in a large test error.  
  
 
We use our test data (from the test sample line shown on Figure 2) to get our true error rate.
 
We use our test data (from the test sample line shown on Figure 2) to get our true error rate.
Right complexity is defined as where true error rate ( the error rate associated with the test data) is minimum; and this is one idea behind complexity control.
+
Right complexity is defined as the point where the true error rate ( the error rate associated with the test data) is minimum; this is one idea behind complexity control.
  
 
[[File:Bias.jpg|200px|thumb|left|Figure 4]]
 
[[File:Bias.jpg|200px|thumb|left|Figure 4]]
Line 2,607: Line 3,244:
  
 
One desired property of the estimator is that it is correct on average, that is, it is unbiased. <math>Bias (\hat{f}) = E(\hat{f}) - f=0</math>.
 
One desired property of the estimator is that it is correct on average, that is, it is unbiased. <math>Bias (\hat{f}) = E(\hat{f}) - f=0</math>.
However, there is a more important property for an estimator than just being unbiased: the mean squared error. In statistics, there are problems for which it may be good to use an estimator with a small bias. In some cases, an estimator with a small bias may have lesser mean squared error or be median-unbiased (rather than mean-unbiased, the standard unbiasedness property). The property of median-unbiasedness is invariant under transformations while the property of mean-unbiasedness may be lost under nonlinear transformations. For example, while using an unbiased estimator with large mean square error to estimate the parameter, we highly risk a big error. In contrast, a biased estimator with small mean square error will improve the precision of our prediction.
+
However, there is a more important property for an estimator than just being unbiased: low mean squared error. In statistics, there are problems for which it may be good to use an estimator with a small bias. In some cases, an estimator with a small bias may have lesser mean squared error or be median-unbiased (rather than mean-unbiased, the standard unbiasedness property). The property of median-unbiasedness is invariant under transformations while the property of mean-unbiasedness may be lost under nonlinear transformations. For example, while using an unbiased estimator with large mean square error to estimate the parameter, we risk a big error. In contrast, a biased estimator with small mean square error will improve the precision of our predictions.
  
 
Hence, our goal is to minimize <math>MSE (\hat{f})</math>.
 
Hence, our goal is to minimize <math>MSE (\hat{f})</math>.
Line 2,637: Line 3,274:
  
  
A test error is a good estimation on MSE. We want to have a somewhat balanced bias and variance (not high on bias or variance), although it will have some bias.
+
A test error is a good estimation of MSE. We want to have a somewhat balanced bias and variance (not high on bias or variance), although it will have some bias.
  
Referring to Figure 3, overfitting happens after the point where training error starts to decrease and test error starts to increase.
+
=== References ===
 +
 
 +
1. A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms
 +
[http://www.springerlink.com/content/u751321011502645.pdf]
 +
 
 +
2. Model complexity control and statistical learning theory
 +
[http://www.springerlink.com/content/wh40jlnrbr6cnh9x/]
 +
 
 +
3. On Dimensionality, Sample Size, Classification Error, and Complexity of Classification Algorithm in Pattern Recognition
 +
[http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4767011]
 +
 
 +
4. Overfitting, Underfitting and Model Complexity
 +
[http://www.chemometrie.com/phd/2_8_1.html]
  
 
=== Avoid Overfitting ===
 
=== Avoid Overfitting ===
Line 2,666: Line 3,315:
  
 
=== Cross-Validation ===
 
=== Cross-Validation ===
 +
 +
'''Cross-validation''', sometimes called '''rotation estimation''', is a technique for assessing how the results of a statistical analysis will generalize to an independent data set.  It is mainly used in settings where the goal is prediction, and one wants to estimate how accurately a predictive model will perform in practice.  One round of cross-validation involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the ''training set''), and validating the analysis on the other subset (called the ''validation set'' or ''testing set'').  To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.
  
 
[[File:Cv.jpg|200px|thumb|right|Figure 1: Illustration of Cross-Validation]]
 
[[File:Cv.jpg|200px|thumb|right|Figure 1: Illustration of Cross-Validation]]
Line 2,692: Line 3,343:
 
[[File:k-fold.png|350px|thumb|right|Figure 2: K-fold cross-validation]]
 
[[File:k-fold.png|350px|thumb|right|Figure 2: K-fold cross-validation]]
 
The results from the method above may differ significantly based on the initial choice of T and V. Therefore, we improve simple cross-validation by introducing K-fold cross-validation.  
 
The results from the method above may differ significantly based on the initial choice of T and V. Therefore, we improve simple cross-validation by introducing K-fold cross-validation.  
The advantage of K-fold cross validation is that all the values in the dataset are eventually used for both training and testing.  
+
The advantage of K-fold cross validation is that all the values in the dataset are eventually used for both training and testing. When using K-fold cross validation the number of folds must be considered. If the user has a large data set then more folds can be used because a smaller portion of the total data is needed to train the classifier. This leaves more test data and therefore a better estimate on the test error. Unfortunately, the more folds one uses the longer the cross-validation will run. If the user has a small data set then fewer, larger folds must be taken to properly train the classifier.
  
 
In this case, the algorithm is:
 
In this case, the algorithm is:
Line 2,699: Line 3,350:
  
 
: 1) Randomly divide the data into K parts with approximately equal size
 
: 1) Randomly divide the data into K parts with approximately equal size
 
=== References ===
 
 
1. A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms
 
[http://www.springerlink.com/content/u751321011502645.pdf]
 
 
2. Model complexity control and statistical learning theory
 
[http://www.springerlink.com/content/wh40jlnrbr6cnh9x/]
 
 
3. On Dimensionality, Sample Size, Classification Error, and Complexity of Classification Algorithm in Pattern Recognition
 
[http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4767011]
 
 
==[[statf10841Scribe|Editor sign up]] ==
 
{{Cleanup|date=November 14 2010|reason= I think there is not enough description in this part, especially about how to specify the number of folds.}}
 
 
{{Cleanup|date=November 14 2010|reason= We also should consider that as the number of folds increases, the time that is required to perform cross-validation increases accordingly, because a model must be generated and tested for each fold. We might experience performance problems if the number of folds is too high.}}
 
  
 
: 2) For k = 1,...,K
 
: 2) For k = 1,...,K
Line 2,751: Line 3,386:
  
  
Leave-one-out cross-validation is similar to k-fold validation by selecting sets of equal size for error estimation. Leave-one-out cross-validation instead removes a single data point, with n-partitions. Each partition is used systematically for testing exactly once whereas the remaining partitions are used for training. For example, we estimate the <math>\,n-1</math> data points with <math>\,m</math> linear models over the <math>\,n</math> sets, and compare the average error rates.The leave-one-out error is the average error over all partitions.<br />  
+
Leave-one-out cross-validation is similar to k-fold validation by selecting sets of equal size for error estimation. Leave-one-out cross-validation instead removes a single data point, with n-partitions. Each partition is used systematically for testing exactly once whereas the remaining partitions are used for training. For example, we estimate the <math>\,n-1</math> data points with <math>\,m</math> linear models over the <math>\,n</math> sets, and compare the average error rates of the m linear model.The leave-one-out error is the average error over all partitions.<br />  
{{Cleanup|date=November 12 2010|reason= " we estimate the <math>\,n-1</math> data points with <math>\,m</math> linear models over the <math>\,n</math> sets" I am not sure if this is right!<br />
 
In Leave-one-out cross-validation, in each iteration, we leave a point out (the kth point) and fit a model to the remaining <math>\,n-1</math> datapoints, and then test the removed datapoint and compute the error. The overall error rate will be the average error computed over all individual errors. <br /> The above procedure should be repeated for all the model options and the model with the least error rate will be selected.}}
 
  
  
Line 2,772: Line 3,405:
  
 
Leave-one-out cross-validation can perform poorly in comparison to k-fold validation. A paper by Breiman compares k-fold (leave-many-out) cross-validation to leave-one-out cross-validation, noting that average prediction loss and downward bias increase from k-fold to leave-one-out <ref>Breiman, L. ''Heuristics of instability and stabilization in model selection,'' Annals of Statistics, 24, 2350-2383 (1996).</ref>. This can be explained by the lower bias of leave-one-out validation, causing an increase in variance. The bias is relative to the size of the sample set compared to the training set [http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29#Leave-one-out_cross-validation]. As such, as k becomes larger, it becomes more biased and has less variance. Similarly, larger data sets will direct the bias toward zero.<br /><br />
 
Leave-one-out cross-validation can perform poorly in comparison to k-fold validation. A paper by Breiman compares k-fold (leave-many-out) cross-validation to leave-one-out cross-validation, noting that average prediction loss and downward bias increase from k-fold to leave-one-out <ref>Breiman, L. ''Heuristics of instability and stabilization in model selection,'' Annals of Statistics, 24, 2350-2383 (1996).</ref>. This can be explained by the lower bias of leave-one-out validation, causing an increase in variance. The bias is relative to the size of the sample set compared to the training set [http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29#Leave-one-out_cross-validation]. As such, as k becomes larger, it becomes more biased and has less variance. Similarly, larger data sets will direct the bias toward zero.<br /><br />
 
==== Further Reading ====
 
  
 
====k × 2 cross-validation====
 
====k × 2 cross-validation====
Line 2,781: Line 3,412:
 
* One-item-out: [http://biomet.oxfordjournals.org/content/64/1/29.abstract Asymptotics for and against cross-validation]
 
* One-item-out: [http://biomet.oxfordjournals.org/content/64/1/29.abstract Asymptotics for and against cross-validation]
 
* [http://www.springerlink.com/content/tfvyva1cqvtqacvy/fulltext.pdf Leave-one-out style crossvalidation bound for Kernel methods applied to some classification and regression problems]
 
* [http://www.springerlink.com/content/tfvyva1cqvtqacvy/fulltext.pdf Leave-one-out style crossvalidation bound for Kernel methods applied to some classification and regression problems]
 +
 +
=== Matlab Code for Cross Validation ===
 +
1. Generate cross validation index using matlab build-in function 'crossvalind.m'. Click [http://www.mathworks.com/help/toolbox/bioinfo/ref/crossvalind.html here] for details.
 +
 +
2. Use 'cvpartition.m' to partition data. Click [http://www.mathworks.com/help/toolbox/stats/cvpartition.html here].
 +
 +
=== Further Reading ===
 +
1. Two useful pdf's introducing concepts of cross validation. [http://www.autonlab.org/tutorials/overfit10.pdf] [http://www.autonlab.org/tutorials/overfit10.pdf]
  
 
=== References ===
 
=== References ===
Line 2,793: Line 3,432:
  
 
== Radial Basis Function (RBF) Network  - October 28, 2010==
 
== Radial Basis Function (RBF) Network  - October 28, 2010==
{{Cleanup|date=October 2010|reason= It might be interesting to explain the reason for over fitting in MLP. Based on what we learnt on the class we might think that over-fitting occurs due to complexity in structure but the most common way that occurs in MLP has to do nothing with the structure. In MLP over-fitting occurs to do iterations more than what is needed. Since if you this then weights of neural network get very adapted to training data and the training error rate will decrease, this will cause over-fitting. To avoid this problem training of neural network is stopped heuristically. Making this automatic for MLP's can be a good topic for research! }}
+
 
{{Cleanup|date=November 2010|reason= Could you please clarify what you mean? }}
 
{{Cleanup|date=November 2010|reason= I meant that generally our sense is that model complexity is related to structure. It means that we think that if we choose a more complex model it will result in overfitting but sometimes the structure is not bad and appropriate for our problem like a MLP. But we encounter overfitting due to over-training. Which means that we do more than what is needed iterations and this will lead to fitting the training data more than what is needed.    }}
 
{{Cleanup|date=November 2010|reason= In that case, this issue is addressed when we talk about weight decay which penalizes iteration-driven overfitting in NN's}}
 
 
[[File:Rbf_net.png|350px|thumb|right|Figure 1: Radial Basis Function Network]]
 
[[File:Rbf_net.png|350px|thumb|right|Figure 1: Radial Basis Function Network]]
  
Line 2,808: Line 3,444:
 
* and no weights from the first layer to the hidden layer.
 
* and no weights from the first layer to the hidden layer.
  
An RBF network can be trained without back propagation since it has a closed-form solution. The neurons in the hidden layer contain basis functions. A common basis function is RBF function which is a kind of Gaussian function without scaling factor.
+
An RBF network can be trained without back propagation since it has a closed-form solution. The neurons in the hidden layer contain basis functions. A common basis function for  RBF network is a kind of Gaussian function without the scaling factor.
  
 
* Note: [http://ibiblio.org/e-notes/Splines/Intro.htm Spline], RBF, [http://www.aaai.org/Papers/Workshops/1999/WS-99-04/WS99-04-008.pdf Fourier], and similar methods differ only in the basis function.<br />  
 
* Note: [http://ibiblio.org/e-notes/Splines/Intro.htm Spline], RBF, [http://www.aaai.org/Papers/Workshops/1999/WS-99-04/WS99-04-008.pdf Fourier], and similar methods differ only in the basis function.<br />  
  
RBF networks were first used in solving multivariate interpolation problems and numerical analysis. Their prospect is similar in neural network applications, where the training and query targets are rather continuous. RBF networks are artificial neural networks and they can be applied to Regression, Classification and Time series prediction.
+
RBF networks were first used in solving multivariate interpolation problems and in numerical analysis. Their prospect is similar in neural network applications, where the training and query targets are continuous. RBF networks are artificial neural networks and they can be applied to Regression, Classification and Time series prediction.
  
 
For example, if we consider <math>\,n</math> data points along a one dimensional line and <math>\,m</math> clusters. An RBF network with radial basis (Gaussian) functions will cluster points around the <math>\,m</math> means, <math>\displaystyle\mu_{j}</math> for <math>j= 1, ..., m</math>. The other data points will be distributed normally around these centers.
 
For example, if we consider <math>\,n</math> data points along a one dimensional line and <math>\,m</math> clusters. An RBF network with radial basis (Gaussian) functions will cluster points around the <math>\,m</math> means, <math>\displaystyle\mu_{j}</math> for <math>j= 1, ..., m</math>. The other data points will be distributed normally around these centers.
  
* Note: The hidden layer can have a variable number of basis functions (the optimal number of basis function can be determined using the complexity control techniques discussed in the previous section). As usual the more basis functions in the hidden layer, the higher the model complexity will be.<br />
+
* Note: The hidden layer can have a variable number of basis functions (the optimal number of basis function can be determined using the complexity control techniques discussed in the previous section). As usual, the more basis functions are in the hidden layer, the higher the model complexity will be.<br />
 +
 
 +
RBF networks, K-Means clustering, Probabilistic Neural Networks(PNN) and General Regression Neural Networks(GRNN) are almost the same. The main difference is that PNN/GRNN networks have one neuron for each point in the training file, whereas the number of RBF networks  neurons (basis functions) is not set, and it is usually much less than the number of training points. When the size of the training set is not very large, PNN and GRNN perform well. But for large size data sets RBF will be more useful, since PNN/GRNN are impractical.
  
RBF networks, K-Means clustering ,Probabilistic Neural Networks(PNN) and General Regression Neural Networks (GRNN) are almost the same. The main difference is that PNN/GRNN networks have one neuron for each point in the training file, whereas the number of RBF networks  neurons (basis functions) is not distinct and it is usually much less than the number of training points. When the training size sets is not very large PNN/GRNN perform well. But for large size data sets RBF will be more useful since PNN/GRNN are impractical.
+
====A brief introduction to the K-means algorithm====
 +
K-means is a commonly applied technique in clustering, which aims to divide <math>\,n</math> observations into <math>\,k</math> groups by computing the distance from each of individual observations to the <math>\,k</math> cluster centers. A typical K-means algorithm can be described as follows:
  
====A brief introduction on the algorithm of K-means====
+
Step1: Select <math>\,k</math> as the number of clusters
K-means is a well-applied technique in clustering, which aims to divide n observations into k groups by computing the distance from each of individual observations to k cluster centers. A typical algorithm of K-means can be described as follows:
 
  
Step1: Select randomly k observations from n observations as k initial centers.  
+
Step2: Randomly select <math>\,k</math> observations from the <math>\,n</math> observations, to be used as <math>\,k</math> initial centers.  
  
Step2: For each of the rest observations, compute the distance to each of k initial centers and classify it into the cluster with the minimum distance.
+
Step3: For each data point from the rest of observations, compute the distance to each of the <math>\,k</math> initial centers and classify it into the cluster with the minimum distance.
  
Step3: Obtain updated k cluster centers by computing the mean of all the observations in the corresponding clusters.
+
Step4: Obtain updated <math>\,k</math> cluster centers by computing the mean of all the observations in the corresponding clusters.
  
Step4: Repeat step2 and step3 until all the difference between the old cluster centers and new cluster centers is acceptable.
+
Step5: Repeat Step 3 and Step 4 until all of the differences between the old cluster centers and new cluster centers are acceptable.
 +
====Typical Radial Function====
 +
 
 +
Gaussian :
 +
 
 +
<math>\ \phi(r) = e^{- \frac{r^{2}}{2 \sigma^2}} </math>
 +
 
 +
Hardy Multi-quadratic :
 +
 
 +
<math>\ \phi(r) = \frac{\sqrt{r^2+c^2}}{c} ,  c>0 </math>
 +
 
 +
Hardy Multi-quadratic :
 +
 
 +
<math>\ \phi(r) = \frac{c}{\sqrt{r^2+c^2}} ,  c>0 </math>
  
 
==== Reference for the above paragraph ====
 
==== Reference for the above paragraph ====
Line 2,835: Line 3,486:
 
[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97.548&rep=rep1&type=pdf]
 
[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97.548&rep=rep1&type=pdf]
  
2. GA-RBF: A selfoptimising RBF network
+
2. GA-RBF: A self-optimising RBF network
 
[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.60.7406&rep=rep1&type=pdf]
 
[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.60.7406&rep=rep1&type=pdf]
  
Line 2,856: Line 3,507:
 
==== RBF Network's Hidden Layer ====
 
==== RBF Network's Hidden Layer ====
  
The hidden layer has <math>\, m</math> neurons, where the optimal number can be determined using the cross validation techniques discussed in the previous section.
+
The hidden layer has <math>\, m</math> neurons, where the optimal number for <math>\, m</math> can be determined using cross validation techniques discussed in the previous section.
For example, if the data is generated from mixture of Gaussian distribution, you can cluster the data and estimate each Gaussian distribution mean and variance by EM algorithm. Their mean and variance can be used for constructing the basis functions. Each neuron consists of a basis function of an input layer point <math>\underline x_{i}</math> referred to as <math>\,\Phi_{j}(\underline x_{i}) </math> where <math>\, j \in \{1 ... m\}</math> and <math>\, i \in \{1 ... n\}</math>. <br>
+
For example, if the data is generated from mixture of Gaussian distribution, you can cluster the data and estimate each Gaussian distribution mean and variance by [http://en.wikipedia.org/wiki/Expectation-maximization_algorithm EM algorithm]. Their mean and variance can be used for constructing the basis functions. Each neuron consists of a basis function of an input layer point <math>\underline x_{i}</math> referred to as <math>\,\Phi_{j}(\underline x_{i}) </math> where <math>\, j \in \{1 ... m\}</math> and <math>\, i \in \{1 ... n\}</math>. <br>
  
 
* Note: In the following section, <math>k</math> is the number of outputs, <math>n</math> is the number of data points, and <math>m</math> is the number of hidden units.  If <math>\,k = 1</math>, <math>\,\hat Y</math> and <math>\,W</math> are column vectors. <br>
 
* Note: In the following section, <math>k</math> is the number of outputs, <math>n</math> is the number of data points, and <math>m</math> is the number of hidden units.  If <math>\,k = 1</math>, <math>\,\hat Y</math> and <math>\,W</math> are column vectors. <br>
Line 2,866: Line 3,517:
 
* Note: An RBF function <math>\Phi</math> is a real-valued function whose value depends only on the distance from a centre <math>\underline c</math>, such that <math>\Phi(\underline x,\underline c) = \Phi(\|\underline x - \underline c \|)</math>. Other commonly used radial basis functions are Multiquadric, Polyharmonic spline, and Thin plate spline.
 
* Note: An RBF function <math>\Phi</math> is a real-valued function whose value depends only on the distance from a centre <math>\underline c</math>, such that <math>\Phi(\underline x,\underline c) = \Phi(\|\underline x - \underline c \|)</math>. Other commonly used radial basis functions are Multiquadric, Polyharmonic spline, and Thin plate spline.
  
{{Cleanup|date=November 17 2010|reason= above, it is said that the value of Gaussian radial basis function depends only on the distance from a centre. I am not sure if this is correct, since the Guassian RBF value depends on the distance to the center as well as the variance of the Gaussian.}}
 
The radii of the RBF functions may be different. The centers and radii can be determined by clustering the data into <math>\, m</math> groups using different clustering algorithms available. <br>
 
  
 
:<math>\Phi_{n,m} = \left[ \begin{matrix}
 
:<math>\Phi_{n,m} = \left[ \begin{matrix}
Line 2,924: Line 3,573:
 
<math> \,= E[(f(x) + \epsilon - \hat f(x))^2]</math><br>
 
<math> \,= E[(f(x) + \epsilon - \hat f(x))^2]</math><br>
 
<math> \,= E[(f(x) - \hat f(x))^2 + \epsilon^2 - 2\epsilon(f(x) - \hat f(x))]</math><br>
 
<math> \,= E[(f(x) - \hat f(x))^2 + \epsilon^2 - 2\epsilon(f(x) - \hat f(x))]</math><br>
The part of the error term we want to approximate is <math>\, E[(f(x) - \hat f(x))^2] </math>. We will try to estimate this by finding the other terms of the above expression.
+
The part of the error term we want to approximate is <math>\, E[(f(x) - \hat f(x))^2] </math>. We will try to estimate this by finding the other terms of the above expression. See lecture titled "Model Selection for an RBF network", November 2, 2010, below.
  
 
==== Conceptualizing RBF Networks ====
 
==== Conceptualizing RBF Networks ====
Line 2,941: Line 3,590:
  
 
An Example of RBF Networks [http://reference.wolfram.com/applications/neuralnetworks/ApplicationExamples/12.1.2.html]
 
An Example of RBF Networks [http://reference.wolfram.com/applications/neuralnetworks/ApplicationExamples/12.1.2.html]
 +
 +
This paper suggests an objective approach in determining proper samples to find good RBF networks with respect to accuracy[http://www.wseas.us/e-library/conferences/2009/hangzhou/MUSP/MUSP41.pdf].
  
 
=====Improvement for RBF Neural Networks Based on Cloud Theory=====
 
=====Improvement for RBF Neural Networks Based on Cloud Theory=====
 
Base on cloud theory, an improved algorithm for RBF neural networks was introduced to transfer the problem of determining the center and its corresponding bandwidth of cluster of RBF to determine the parameters of normal cloud model in order to make the output of each of hidden layers having vague and random properties and the randomness of each of data are kept and passed to the output layer. The conclusion shows that the improved algorithm is superior to the classical RBF in prediction and the actual result is well. Simultaneously, the improved algorithm can be transplanted to the improvement of RBF neural networks algorithms. For more information, see Lingfang Sun, Shouguo Wang, Ce Xu, Dong Ren, Jian Zhang, Research on the improvement for RBF neural networks based on cloud theory, Proceedings of the World Congress on Intelligent Control and Automation (WCICA), pp. 3110-3113, 2008.
 
Base on cloud theory, an improved algorithm for RBF neural networks was introduced to transfer the problem of determining the center and its corresponding bandwidth of cluster of RBF to determine the parameters of normal cloud model in order to make the output of each of hidden layers having vague and random properties and the randomness of each of data are kept and passed to the output layer. The conclusion shows that the improved algorithm is superior to the classical RBF in prediction and the actual result is well. Simultaneously, the improved algorithm can be transplanted to the improvement of RBF neural networks algorithms. For more information, see Lingfang Sun, Shouguo Wang, Ce Xu, Dong Ren, Jian Zhang, Research on the improvement for RBF neural networks based on cloud theory, Proceedings of the World Congress on Intelligent Control and Automation (WCICA), pp. 3110-3113, 2008.
 +
 +
=== Radial Basis Approximation Implementation ===
 +
 +
% This code uses the built-in NEWRB MATLAB function to create a radial basis network that
 +
% approximates a function defined by a set of data points.
 +
 +
% Define 21 inputs P and associated targets T.
 +
 +
P = -1:.1:1;
 +
T = [-.9602 -.5770 -.0729  .3771  .6405  .6600  .4609 ...
 +
      .1336 -.2013 -.4344 -.5000 -.3930 -.1647  .0988 ...
 +
      .3072  .3960  .3449  .1816 -.0312 -.2189 -.3201];
 +
plot(P,T,'+');
 +
title('Training Vectors');
 +
xlabel('Input Vector P');
 +
ylabel('Target Vector T');
 +
 +
% We would like to find a function which fits the 21 data points.  One way to do this is with a radial % basis network. A radial basis network is a network with two layers.  A hidden layer of radial basis  % neurons and an output layer of linear neurons.  Here is the radial basis transfer function used by  % the hidden layer.
 +
 +
p = -3:.1:3;
 +
a = radbas(p);
 +
plot(p,a)
 +
title('Radial Basis Transfer Function');
 +
xlabel('Input p');
 +
ylabel('Output a');
 +
 +
% The weights and biases of each neuron in the hidden layer define the position
 +
% and width of a radial basis function.  Each linear output neuron forms a
 +
% weighted sum of these radial basis functions.  With the correct weight and
 +
% bias values for each layer, and enough hidden neurons, a radial basis network
 +
% can fit any function with any desired accuracy.  This is an example of three
 +
% radial basis functions are scaled and summed to produce a function
 +
 +
a2 = radbas(p-1.5);
 +
a3 = radbas(p+2);
 +
a4 = a + a2*1 + a3*0.5;
 +
plot(p,a,'b-',p,a2,'b--',p,a3,'b--',p,a4,'m-')
 +
title('Weighted Sum of Radial Basis Transfer Functions');
 +
xlabel('Input p');
 +
ylabel('Output a');
 +
 +
[[File:rbf1.jpg]]
 +
 +
% The function NEWRB quickly creates a radial basis network which approximates
 +
% the function defined by P and T.  In addition to the training set and targets,
 +
% NEWRB takes two arguments, the sum-squared error goal and the spread constant.
 +
 +
eg = 0.02; % sum-squared error goal
 +
sc = 1;    % spread constant
 +
net = newrb(P,T,eg,sc);
 +
 +
% To see how the network performs, replot the training set.  Then simulate the
 +
% network response for inputs over the same range.  Finally, plot the results on
 +
% the same graph.
 +
 +
plot(P,T,'+');
 +
xlabel('Input');
 +
X = -1:.01:1;
 +
Y = sim(net,X);
 +
hold on;
 +
plot(X,Y);
 +
hold off;
 +
legend({'Target','Output'})
 +
 +
=== Linear Basis Network ===
 +
A piece-wise linear trajectory maybe modeled more easily and meaningfully using linear bases, rather than Gaussians. To replace Gaussian with linear, one may need to replace the clustering algorithm and modify the basis function. C-varieties algorithm developed by H.H.Bock and J.C.Bezdek is a good way for finding linear clusters (for details on this algorithm you may want to refer to the Fuzzy Cluster Analysis, by Frank Hoeppner et al). And then the resulting linear functions should be accompanied with some Gaussian functions, or any other localizing functions to localize the working area of each one of the linear clusters. The combination is a general function approximator, just like the RBF, NN, etc.
 +
 +
<math>\begin{align}\hat{y}=\sum_{i=1}^{P}f_i(x)\Phi_i(x) \end{align}</math>
 +
 +
Where in this equation, <math>\ f_i(x)</math> is a linear function, coming the clustering stage and corresponding to the ith cluster and <math>\ \Phi_i</math> is the validity or localizing function, corresponding to the ith cluster.
  
 
== '''Model Selection for RBF Network (Stein's Unbiased Risk Estimator) - November 2nd, 2010''' ==
 
== '''Model Selection for RBF Network (Stein's Unbiased Risk Estimator) - November 2nd, 2010''' ==
Line 2,952: Line 3,673:
  
  
However, training error and testing error do not demonstrate a linear relationship. In particular, a smaller training error does not necessarily result in a smaller testing error. In practice, one often observes that up to a certain point the model error on testing data tends to decrease as the training error decreases. However, if one attempts to decrease the training error too much by increasing the model complexity, the testing error often can take a dramatic turn and begin to increase. This was explained and a graphic provided in the lecture on October 26th concerning complexity control.  
+
However, training error and testing error do not demonstrate a linear relationship. In particular, a smaller training error does not necessarily result in a smaller testing error. In practice, one often observes that up to a certain point the model error on testing data tends to decrease as the training error decreases. However, if one attempts to decrease the training error too much by increasing the model complexity, the testing error often can take a dramatic turn and begin to increase. This behavior was explained and related figures illustrating this concept were provided in the lecture on complexity control on October 26th.  
  
[[File:data_noise.jpg|500px|thumb|right|Figure 1. Data sampled from a smooth function (in black) cannot be over-fit. Data sampled from a smooth function with noise (in red) can be over-fit when the noise is modelled along with the smooth function.]]
+
[[File:data_noise.jpg|500px|thumb|right|Figure 1. Data sampled from a smooth function (in black) cannot be over-fit. Data sampled from a smooth function with noise (in red) can be over-fit when the noise is modeled along with the smooth function.]]
  
The basic reason behind this phenomenon of the training and testing errors is that in the process of minimizing training error, after a certain point, the model begins to over-fit the training set. Over-fitting in this context means fitting the model to the training data at the expense of losing generality. As seen in Figure 1, the red data points have been over-fit as the general form of the underlying smooth function has been lost in the red-curve model. In the extreme case, a set of <math>\displaystyle N</math> training data points can be modeled exactly with <math>\displaystyle N</math> radial basis functions. Such a model will fit the training data set perfectly. However, the perfectly-fit model fails to be as accurate or perform as well on the training data set because it has modelled not only the true function <math>\displaystyle f(X)</math> but the random noise as well, and thus has over-fit the data (as the red curve in Figure 1 has done). It is interesting to note that in the case of no noise, over-fitting will not occur and hence the complexity of the model can be increased without bound. However, this is not realistic in practice as random noise is almost always present in the data.  
+
The basic reason behind this phenomenon of the training and testing errors is that in the process of minimizing training error, after a certain point, the model begins to over-fit the training set. Over-fitting in this context means fitting the model to the training data at the expense of losing generality. As seen in Figure 1, the red data points have been over-fit as the general form of the underlying smooth function has been lost in the red-curve model. In the extreme case, a set of <math>\displaystyle N</math> training data points can be modeled exactly with <math>\displaystyle N</math> radial basis functions. Such a model will fit the training data set perfectly. However, the perfectly-fit model fails to be as accurate or perform as well on the training data set because it has modeled not only the true function <math>\displaystyle f(X)</math> but the random noise as well, and thus has over-fit the data (as the red curve in Figure 1 has done). It is interesting to note that in the case of no noise, over-fitting will not occur and hence the complexity of the model can be increased without bound. However, this is not realistic in practice as random noise is almost always present in the data.  
  
In general, the training error rate will be less than the testing error on the new data. A model typically adapts to the training data, and hence the training error will be an overly optimistic estimate of the testing error. An obvious way to estimate testing error is to add a penalty term to the training error to compensate for the difference. SURE is developed based on this idea.
+
All in all, we can expect the training error will be an overly optimistic estimate of the testing error. An obvious way to estimate testing error is to add a penalty term to the training error to compensate for the difference. SURE, a technique developed by Charles Stein, a professor of statistics at Stanford University, is based on this idea.
  
 
===Stein's unbiased risk estimate (SURE)===
 
===Stein's unbiased risk estimate (SURE)===
  
Stein's unbiased risk estimate (SURE) is an unbiased estimator of the mean-squared error of a given estimator in a deterministic estimation scenario. In other words, it provides an indication of the accuracy of a given estimator. This is important since, in deterministic estimation, the true mean-squared error of an estimator generally depends on the value of the unknown parameter, and thus cannot be determined completely. A standard application of SURE is to choose a parametric form for an estimator, and then optimize the values of the parameters to minimize the risk estimate. This technique has been applied in several settings. For example, a variant of the James-Stein estimator can be derived by finding the optimal shrinkage estimator. The technique has also been used by Donoho and Johnstone to determine the optimal shrinkage factor in a wavelet denoising setting [http://en.wikipedia.org/wiki/Stein's_unbiased_risk_estimate].
+
Stein's unbiased risk estimate (SURE) is an unbiased estimator of the mean-squared error of a given estimator in a deterministic estimation scenario. In other words, it provides an indication of the accuracy of a given estimator. This is important since, in deterministic estimation, the true mean-squared error of an estimator generally depends on the value of the unknown parameter, and thus cannot be determined completely. A standard application of SURE is to choose a parametric form for an estimator, and then optimize the values of the parameters to minimize the risk estimate.
  
For more information about the relation between Stein's unbiased risk estimator and Stein's lemma refer to[ http://www.cc.gatech.edu/~lebanon/notes/sure.pdf]. The following is the description of Stein's lemma and its use to derive Stein's unbiased risk estimator (SURE).
+
Stein’s unbiased risk estimation (SURE) theory gives a rigorous definition of the degrees of freedom for any fitting procedure. [http://www.ams.org/mathscinet-getitem?mr=0630098]. For more information about the relation between Stein's unbiased risk estimator and Stein's lemma refer to[http://www.cc.gatech.edu/~lebanon/notes/sure.pdf]. The following is the description of Stein's lemma and its use to derive Stein's unbiased risk estimator (SURE).
  
Note that the material presented here is applicable to model selection in general, and is not specific to RBF networks.  
+
Note that the material presented here is applicable to model selection in general, and is not specific to RBF networks.
 +
 
 +
===Applications of Stein's unbiased risk estimate===
 +
A standard application of SURE is to choose a parametric form for an estimator, and then optimize the values of the parameters to minimize the risk estimate. This technique has been applied in several settings. For example, a variant of the James–Stein estimator[http://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator] can be derived by finding the optimal shrinkage estimator. The technique has also been used by Donoho and Johnstone to determine the optimal shrinkage factor in a wavelet denoising setting [http://www.jstor.org/sici?sici=0162-1459%28199512%2990%3A432%3C1200%3AATUSVW%3E2.0.CO%3B2-K].
 +
SURE has also been used for optical flow estimation by Mingren Shi [http://www.sci.usq.edu.au/research/seminars/files//seminar1/OpSureTalk.pdf].
  
 
====Important Notation [http://en.wikipedia.org/wiki/Stein's_unbiased_risk_estimate]====
 
====Important Notation [http://en.wikipedia.org/wiki/Stein's_unbiased_risk_estimate]====
Line 3,059: Line 3,784:
  
 
This means that equation <math>\displaystyle (1)</math> now becomes, for one data point:
 
This means that equation <math>\displaystyle (1)</math> now becomes, for one data point:
<math>\displaystyle E[(\hat y_i-y_i)^2 ]=E[(\hat f_i-f_i)^2]+\sigma^2-2\sigma^2E\left[\frac {\partial \hat f}{\partial y_i}\right]</math>.
+
<math>\displaystyle E[(\hat y-y)^2 ]=E[(\hat f-f)^2]+\sigma^2-2\sigma^2E\left[\frac {\partial \hat f}{\partial y}\right]</math>.
  
  
Line 3,090: Line 3,815:
 
<math>\,Trace(H)= Trace(\Phi(\Phi^{T}\Phi)^{-1}\Phi^{T})= Trace(\Phi^{T}\Phi(\Phi^{T}\Phi)^{-1})=m</math>, by the trace cyclical permutation property, where <math>\displaystyle m</math> is the number of basis functions in the RBF network (and hence <math>\displaystyle \Phi</math> has dimension <math>\displaystyle n \times m</math>).<br>
 
<math>\,Trace(H)= Trace(\Phi(\Phi^{T}\Phi)^{-1}\Phi^{T})= Trace(\Phi^{T}\Phi(\Phi^{T}\Phi)^{-1})=m</math>, by the trace cyclical permutation property, where <math>\displaystyle m</math> is the number of basis functions in the RBF network (and hence <math>\displaystyle \Phi</math> has dimension <math>\displaystyle n \times m</math>).<br>
  
====Sketch of trace cyclical property proof:====
+
====Sketch of Trace Cyclical Property Proof:====
 
For <math>\, A_{mn}, B_{nm}, Tr(AB) = \sum_{i=1}^{n}\sum_{j=1}^{m}A_{ij}B_{ji} = \sum_{j=1}^{m}\sum_{i=1}^{n}B_{ji}A_{ij} = Tr(BA)</math>.<br>
 
For <math>\, A_{mn}, B_{nm}, Tr(AB) = \sum_{i=1}^{n}\sum_{j=1}^{m}A_{ij}B_{ji} = \sum_{j=1}^{m}\sum_{i=1}^{n}B_{ji}A_{ij} = Tr(BA)</math>.<br>
 
With that in mind, for <math>\, A_{nn}, B_{nn} = CD, Tr(AB) = Tr(ACD) = Tr(BA)</math> (from above) <math>\, = Tr(CDA)</math>.<br><br>
 
With that in mind, for <math>\, A_{nn}, B_{nn} = CD, Tr(AB) = Tr(ACD) = Tr(BA)</math> (from above) <math>\, = Tr(CDA)</math>.<br><br>
Line 3,097: Line 3,822:
  
  
Substituing <math>\sum_{i=1}^n \,H_{ii} = m+1</math> into equation <math>\displaystyle (3)</math> gives the following:
+
Substituting <math>\sum_{i=1}^n \,H_{ii} = m+1</math> into equation <math>\displaystyle (3)</math> gives the following:
 
<math>\displaystyle Err=err-n\sigma^2+2\sigma^2(m+1)</math>.
 
<math>\displaystyle Err=err-n\sigma^2+2\sigma^2(m+1)</math>.
  
Line 3,113: Line 3,838:
 
====Reference:====
 
====Reference:====
  
Automatic basis selection for RBF networks using Stein’s unbiased risk estimator
+
* Automatic basis selection for RBF networks using Stein’s unbiased risk estimator [http://www.google.ca/url?sa=t&source=web&cd=2&sqi=2&ved=0CB4QFjAB&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.5.5344%26rep%3Drep1%26type%3Dpdf&rct=j&q=Stein%27s%20Unbiased%20Risk%20Estimator%29%20RBF&ei=YsHSTKzgDYausAO-4IWrCw&usg=AFQjCNHO9oFBQ6tljsEqdLOjFgtiQz9gxQ&sig2=Cx9Sh0Uk-h8pDgihKkU_HA&cad=rja.pdf]
[http://www.google.ca/url?sa=t&source=web&cd=2&sqi=2&ved=0CB4QFjAB&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.5.5344%26rep%3Drep1%26type%3Dpdf&rct=j&q=Stein%27s%20Unbiased%20Risk%20Estimator%29%20RBF&ei=YsHSTKzgDYausAO-4IWrCw&usg=AFQjCNHO9oFBQ6tljsEqdLOjFgtiQz9gxQ&sig2=Cx9Sh0Uk-h8pDgihKkU_HA&cad=rja.pdf]
+
* J. Moody and C. J. Darken, "Fast learning in networks of locally tuned processing units," Neural Computation, 1, 281-294 (1989). Also see [http://www.ki.inf.tu-dresden.de/~fritzke/FuzzyPaper/node5.html Radial basis function networks according to Moody and Darken]
 +
* T. Poggio and F. Girosi, "Networks for approximation and learning," Proc. IEEE 78(9), 1484-1487 (1990).
 +
* Roger D. Jones, Y. C. Lee, C. W. Barnes, G. W. Flake, K. Lee, P. S. Lewis, and S. Qian, [http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=137644 Function approximation and time series prediction with neural networks], Proceedings of the International Joint Conference on Neural Networks, June 17-21, p. I-649 (1990).
 +
* John R. Davies, Stephen V. Coggeshall, Roger D. Jones, and Daniel Schutzer, "Intelligent Security Systems,"
 +
* S. Chen, C. F. N. Cowan, and P. M. Grant, "Orthogonal Least Squares Learning Algorithm for Radial Basis Function Networks", IEEE Transactions on Neural Networks, Vol 2, No 2 (Mar) 1991.
  
 
====Further Reading:====
 
====Further Reading:====
Line 3,134: Line 3,863:
  
 
== '''Regularization for Neural Network - November 4, 2010'''==
 
== '''Regularization for Neural Network - November 4, 2010'''==
 +
 +
Large weights in a neural network can hurt generalization in two different ways. First, excessively large weights leading to hidden units can cause the output function to be too rough, possibly with near discontinuities. Second, excessively large weights leading to output units can cause wild outputs far beyond the range of the data if the output activation function is not bounded to the same range as the data. To put it another way, large weights can cause excessive variance of the output. The effort to reduce the size of these weights is called regularization.
 +
 +
Training of the weights in a neural network is usually accomplished by iteratively developing them from some regularized set of small initial values.  The weights tend to increase in absolute values as training proceeds.  When neural networks were first developed, weights were prevented from getting too large by simply stopping the training session early; to determine when to stop training the neural network, a set of test data was used to detect overfitting. Using this method, the stopping point would be determined by finding the length of training time that results in minimal classification error for the test set.  However, in this section, a somewhat different for regularization method is presented that does not require the training session to be terminated early; rather, this method directly penalizes overfitting in the optimization calculation.
 +
 
=== ''' Weight decay'''===
 
=== ''' Weight decay'''===
Weight decay is a subset of regularization methods. The penalty term in
+
Weight decay is a subset of regularization methods, which aim to prevent overfitting in model selection. The penalty term in
 
weight decay, by definition, penalizes large weights. Other regularization
 
weight decay, by definition, penalizes large weights. Other regularization
 
methods may involve not only the weights but various derivatives of the
 
methods may involve not only the weights but various derivatives of the
 
output function [http://research.microsoft.com/en-us/um/people/cmbishop/nnpr.htm].  
 
output function [http://research.microsoft.com/en-us/um/people/cmbishop/nnpr.htm].  
 +
The weight decay penalty term causes the weights to converge to smaller
 +
absolute values than they otherwise would.
  
 
[[File:figure 2.png|350px|thumb|right|Figure 3: activation function]]
 
[[File:figure 2.png|350px|thumb|right|Figure 3: activation function]]
 
Weight decay training is suggested as a method useful in achieving a robust [http://en.wikipedia.org/wiki/Neural_network neural network] which is insensitive to noise. Since the number of hidden layers in a NN is usually decided by certain domain knowledge, it may easily get into the problem of overfitting.
 
Weight decay training is suggested as a method useful in achieving a robust [http://en.wikipedia.org/wiki/Neural_network neural network] which is insensitive to noise. Since the number of hidden layers in a NN is usually decided by certain domain knowledge, it may easily get into the problem of overfitting.
  
The weight–decay method is an effective way to improve the generalization ability of neural networks. In general, the trained weights are constrained to be small when the weight-decay method is applied. Large weights leading to output units can cause outputs that are far beyond the range of the data (when test data is used); in other words, large weights can result in high output variance.  
+
The weight–decay method is an effective way to improve the generalization ability of neural networks. In general, the trained weights are constrained to be small when the weight-decay method is applied. Large weights in the output layer can cause outputs that are far beyond the range of the data (when test data is used); in other words, large weights can result in high output variance.  
  
 
It can be seen from Figure 3 that when the weight is in the vicinity of zero, the operative part of the activation function shows linear behavior. That is, the operative part of a sigmoid function is almost linear for small weights. The NN then collapses to an approximately linear model. Note that a linear model is the simplest model, and we can avoid overfitting by constraining the weights to be small. This gives us a hint on why we initialize the random weights to be close to zero. If the weights are large, the model is more complex and the activation function tends to be nonlinear.
 
It can be seen from Figure 3 that when the weight is in the vicinity of zero, the operative part of the activation function shows linear behavior. That is, the operative part of a sigmoid function is almost linear for small weights. The NN then collapses to an approximately linear model. Note that a linear model is the simplest model, and we can avoid overfitting by constraining the weights to be small. This gives us a hint on why we initialize the random weights to be close to zero. If the weights are large, the model is more complex and the activation function tends to be nonlinear.
  
Note that it is not necessarily bad to go to the nonlinear section of the activation function. In fact we use nonlinear activation functions to increase the ability of neural networks and make it possible to estimate nonlinear functions. What we must avoid is using the nonlinear section more than required, which would result in overfitting the training data. For this reason we add a penalty term to the error function.
+
Note that it is not necessarily bad to go to the nonlinear section of the activation function. In fact, we use nonlinear activation functions to increase the ability of neural networks and make it possible to estimate nonlinear functions. What we must avoid is using the nonlinear section more than required, which would result in overfitting of the training data. To achieve this we add a penalty term to the error function.
  
Our goal is keeping the weights small. Formally, we penalize nonlinear weights by adding a penalty term to the error function; the penalty term in weight decay, by definition, penalizes large weights. The usual penalty is the sum of squared weights times a decay constant. In a linear model, this form of weight decay is equivalent to ridge regression. Now the regularized error function becomes:
+
The usual penalty is the sum of squared weights times a decay constant. In a linear model, this form of weight decay is equivalent to ridge regression [http://komarix.org/ac/papers/thesis/thesis_html/node15.html]. Now the regularized error function becomes:
  
  
Line 3,167: Line 3,903:
 
<math>u^{new} \leftarrow u^{old} - \rho\left(\frac{\partial err}{\partial u} + 2\lambda u^{old}\right)</math>
 
<math>u^{new} \leftarrow u^{old} - \rho\left(\frac{\partial err}{\partial u} + 2\lambda u^{old}\right)</math>
  
{{Cleanup|date=November 2010|reason=It is a good to have a discussion on value of weight decay parameter. If we choose it to be very small then we will not see the effect of weight decay as we increase it we can see the effect of weight decay but if we set it to be very large then back-propagation will not work properly. It means that weights will not change in iteration steps so there is a optimum value for this parameters and some methods try to find it.}}
 
  
 
To conclude, the weight decay penalty term lead the weights to converge to smaller
 
To conclude, the weight decay penalty term lead the weights to converge to smaller
Line 3,174: Line 3,909:
 
near discontinuities. Excessively large weights leading to output units can
 
near discontinuities. Excessively large weights leading to output units can
 
cause wild outputs far beyond the range of the data if the output activation
 
cause wild outputs far beyond the range of the data if the output activation
function is not bounded to the same range as the data. In another words, large weights can cause large variance of the output [http://portal.acm.org/citation.cfm?id=148062]. According to [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.57.2302], the size (L_1
+
function is not bounded to the same range as the data. In another words, large weights can cause large variance of the output [http://portal.acm.org/citation.cfm?id=148062]. According to [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.57.2302], the size (L1-
 
norm) of the weights is more important than the number of weights in
 
norm) of the weights is more important than the number of weights in
 
determining generalization.  
 
determining generalization.  
Line 3,184: Line 3,919:
 
<math>\,\lambda</math>is different for different types of weights in the NN. We can have different <math>\,\lambda</math> for input-to-hidden, hidden-to-hidden, and hidden-to-output weights.
 
<math>\,\lambda</math>is different for different types of weights in the NN. We can have different <math>\,\lambda</math> for input-to-hidden, hidden-to-hidden, and hidden-to-output weights.
  
For more reading about the effect of weight decay training for backpropagation on noisy data sets please refer to [http://www.sciencedirect.com/science?_ob=MImg&_imagekey=B6T08-3TYVWK9-F-P&_cdi=4856&_user=1067412&_pii=S089360809800046X&_origin=search&_coverDate=08%2F31%2F1998&_sk=999889993&view=c&wchp=dGLbVzW-zSkzS&md5=52846ec8e0ba54b28000ef1de34c7bc5&ie=/sdarticle.pdf] and how weight decay can improve generalization in feed forward network refer to [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.47.4221]
+
The following Matlab code implements a neural network with weight decay using the backpropagation method:
  
 +
    %initialize
 +
    m;         %features
 +
    n;         %datapoints
 +
    nodes; %number of hidden nodes
 +
    u1=rand(m,nodes)-0.5; %input-to-hidden weights
 +
    u2=rand(nodes,1)-0.5; %hidden-to-output weights
 +
    ro; %learning rate
 +
    weightPenalty;
 +
   
 +
    Z_output=zeros(n,1);
 +
    Z=zeros(nodes,n);
 +
   
 +
    %% TRAIN DATA
 +
    for epoch=1:100
 +
   
 +
        for i=1:n
 +
            %% Forward Pass
 +
            %determine inputs to hidden layer
 +
            A=u1'*training_X(:,i);
 +
           
 +
        %apply activation function to hidden layer weighted inputs
 +
            for j=1:nodes
 +
                Z(j,i)=1/(1+exp(-A(j)));
 +
            end
 +
               
 +
        %apply weights to get fitted outputs;
 +
            Z_output(i,1) = u2'*Z(:,i);
 +
           
 +
            %% Backward Pass
 +
            %output delta
 +
            delta_O = -2*(training_Y(i)-Z_output(i,1));
 +
               
 +
        %tweak the hidden-output weights
 +
            for j=1:nodes
 +
                u2(j)=u2(j)-ro*(delta_O*Z(j)+2*weightPenalty*u2(j));
 +
            end
 +
            for j=1:nodes
 +
                sigmaPrime=exp(-A(j))/(1+exp(-A(j)))^2;
 +
                delta_H = sigmaPrime*delta_O*u2(j);
 +
                u1(:,j)=u1(:,j)-ro*(delta_H*training_X(:,i)+2*weightPenalty*u1(:,j));
 +
            end
 +
        end
 +
        yhat(:,1) = Z_output(:,1) > 0.5;
 +
    end
  
====Methods to estimate the weight decay parameter====
 
One of the biggest problems in weight decay for neural networks is how to estimate its parameter. There are many ways proposed in the literature to estimate the weight decay parameter. The following papers are good start for some one who is looking for further reading.
 
  
1- On the selection of weight decay parameter for faulty networks [http://portal.acm.org/citation.cfm?id=1862025 here]
+
For more reading about the effect of weight decay training for backpropagation on noisy data sets please refer to [http://www.sciencedirect.com/science?_ob=MImg&_imagekey=B6T08-3TYVWK9-F-P&_cdi=4856&_user=1067412&_pii=S089360809800046X&_origin=search&_coverDate=08%2F31%2F1998&_sk=999889993&view=c&wchp=dGLbVzW-zSkzS&md5=52846ec8e0ba54b28000ef1de34c7bc5&ie=/sdarticle.pdf] and how weight decay can improve generalization in feed forward network refer to [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.47.4221]
  
2- A Simple Trick for Estimating the Weight Decay Parameter [http://www.springerlink.com/content/0889d07ufuwgql03/ here]
 
  
===Regularization invariant under transformation===
+
A fundamental problem with weight decay is that different types of weights
 +
in the network will usually require different decay constants for good
 +
generalization. At the very least, you need three different decay constants
 +
for input-to-hidden, hidden-to-hidden, and hidden-to-output weights.
 +
Adjusting all these decay constants to produce the best estimated
 +
generalization error often requires vast amounts of computation.
  
A major drawback of the simple weight decay is that it is inconsistent with regard to transformations done to the input variables and / or the target variables of the training data. This fact is remarkably easy to show. For the interested reader, a simple derivation of it is available on page 8 of Sargur Srihari's [http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.5-Regularization.pdf ''Regularization in Neural Networks''] slides.  
+
Fortunately, there is a superior alternative to weight decay: hierarchical
 +
Bayesian learning. Bayesian learning makes it possible to estimate
 +
efficiently numerous decay constants.
  
It is desirable for regularization to have the property of being invariant under transformation of the inputs and / or the outputs of the training data. This is so that, if one is given a set of training data and uses regularization during training, and trains one network after transforming the training data using one transformation and trains another network after transforming the training data using another transformation, then the two solutions represented by the two trained networks should only differ from each other with regard to the weights as given so that neither solution would be arbitrarily favored over the other.   
+
Weight decay is proposed to reduce overfitting as it often
 +
appears in the learning tasks of artificial neural networks. For example, in [http://www.springerlink.com/content/f21781218007l750/fulltext.pdf] weight decay is applied to a well
 +
defined model system based on a single
 +
layer perceptron, which exhibits strong overfitting. Since the optimal
 +
non-overfitting solution is known for this system, they have compared the effect
 +
of the weight decay with this solution. A strategy to find the optimal
 +
weight decay strength is proposed, which leads to the optimal solution
 +
for any number of examples.
 +
 
 +
 
 +
====Methods to estimate the weight decay parameter====
 +
One of the biggest problems in weight decay regularization of neural networks is how to estimate its parameter. There are many ways proposed in the literature to estimate the weight decay parameter.
 +
 
 +
Typically,the weight decay parameter is set between 0.001 and 0.1  that is based on that is based on network training .An inappropriate estimate of the decay parameter may cause over-fitting or over smoothing . Determining the correct value of the parameter is a very tedious process which needs a lot of trial and error . Typically, the optimal value of the weight decay is determined by training the network many times .That is, performing network training based on the same set of initial weights ,same network configuration with fixed  number of neutral layers , and fit the network with various weight decay parameters . Then determine the optimal value of weight decay values by the smallest generalization error.
 +
 
 +
The following papers are good start for some one who is looking for further reading.
 +
 
 +
1- On the selection of weight decay parameter for faulty networks [http://portal.acm.org/citation.cfm?id=1862025 here]
 +
 
 +
2- A Simple Trick for Estimating the Weight Decay Parameter [http://www.springerlink.com/content/0889d07ufuwgql03/ here]
 +
 
 +
===Regularization invariant under transformation===
 +
 
 +
A major drawback of the simple weight decay is that it is inconsistent with regard to transformations done to the input variables and / or the target variables of the training data. This fact is remarkably easy to show. For the interested reader, a simple derivation of it is available on page 8 of Sargur Srihari's [http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.5-Regularization.pdf ''Regularization in Neural Networks''] slides.
 +
 
 +
It is desirable for regularization to have the property of being invariant under transformation of the inputs and / or the outputs of the training data. This is so that, if one is given a set of training data and uses regularization during training, and trains one network after transforming the training data using one transformation and trains another network after transforming the training data using another transformation, then the two solutions represented by the two trained networks should only differ from each other with regard to the weights as given so that neither solution would be arbitrarily favored over the other.   
  
 
Many approaches have been devised so that, when regularization is used during the training process of a network, the resulting predictions would be invariant under any transformation(s) made to the input variable(s). One such approach is to add a regularization term to the error function that serves to penalize any possible changes to the outputs resulting from any transformation(s) applied to the inputs. A common example of this approach is [http://arts.uwaterloo.ca/~cnrglab/?q=system/files/tangent_prop.pdf tangent propagation], which is described in Sargur Srihari's [http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.5-Regularization.pdf slides] and which is discussed in detail in Simard ''et al.'''s 1998 [http://yann.lecun.com/exdb/publis/pdf/simard-98.pdf paper] regarding transformation invariance. Several other approaches are also described in Sargur Srihari's slides.   
 
Many approaches have been devised so that, when regularization is used during the training process of a network, the resulting predictions would be invariant under any transformation(s) made to the input variable(s). One such approach is to add a regularization term to the error function that serves to penalize any possible changes to the outputs resulting from any transformation(s) applied to the inputs. A common example of this approach is [http://arts.uwaterloo.ca/~cnrglab/?q=system/files/tangent_prop.pdf tangent propagation], which is described in Sargur Srihari's [http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.5-Regularization.pdf slides] and which is discussed in detail in Simard ''et al.'''s 1998 [http://yann.lecun.com/exdb/publis/pdf/simard-98.pdf paper] regarding transformation invariance. Several other approaches are also described in Sargur Srihari's slides.   
  
 +
=== Other Alternatives for NN Regularization ===
 +
As enumerated before, there are some drawbacks for the weight decay, which makes it necessary to think of some other regularization methods. Consistent Gaussian priors, early Stopping, invariances, tangent propagation, training with transformed data, convolutional networks, and soft weight sharing are some other alternatives for neural network regularization. A zealous reader may find a great deal of information about these topics in the Pattern Recognition and Machine Learning by Christopher M. Bishop (chapter 5, section 5).
  
===='''Further reading'''====
+
==='''Further reading'''===
 
The generalization ability of the network can depend crucially on the decay constant, especially with small training sets. One approach to choosing the decay constant is to train several networks with different amounts of decay and estimate the generalization error for each; then choose the decay constant that minimizes the estimated generalization error.  
 
The generalization ability of the network can depend crucially on the decay constant, especially with small training sets. One approach to choosing the decay constant is to train several networks with different amounts of decay and estimate the generalization error for each; then choose the decay constant that minimizes the estimated generalization error.  
  
Line 3,211: Line 4,021:
  
 
Fortunately, there is a superior alternative to weight decay: hierarchical Bayesian learning. Bayesian learning makes it possible to estimate efficiently numerous decay constants.For information about bayesian learning, please refer to [http://en.wikipedia.org/wiki/Bayesian_inference Bayesian inference]
 
Fortunately, there is a superior alternative to weight decay: hierarchical Bayesian learning. Bayesian learning makes it possible to estimate efficiently numerous decay constants.For information about bayesian learning, please refer to [http://en.wikipedia.org/wiki/Bayesian_inference Bayesian inference]
 +
 +
[http://books.google.ca/books?id=jFAbzhrDqRcC&pg=PA1125&lpg=PA1125&dq=regularization+in+neural+networks+weight+decay&source=bl&ots=6YX8KIhxyO&sig=Dcwk5Y1_LPvtLhukEx3gDcVNEik&hl=en&ei=b0HzTLbfBYmgnwfv-5mXCg&sa=X&oi=book_result&ct=result&resnum=2&ved=0CCIQ6AEwATgK#v=onepage&q&f=false]
  
 
===='''References'''====
 
===='''References'''====
Line 3,224: Line 4,036:
  
 
4. Sargur Srihari. ''Regularization in Neural Networks'' slides. [http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.5-Regularization.pdf]
 
4. Sargur Srihari. ''Regularization in Neural Networks'' slides. [http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.5-Regularization.pdf]
 +
 +
5. Neural Network Modeling using SAS Enterprise Miner [http://www.sasenterpriseminer.com/neural_networks.htm]
 +
 +
6. Valentina Corradi, Halbert White ''Regularized neural networks: some convergence rate results'' [http://portal.acm.org/citation.cfm?id=211706]
 +
 +
7. Bayesian Regularization in a Neural Network Model to Estimate Lines of Code Using Function Points [http://www.scipub.org/fulltext/jcs/jcs14505-509.pdf]
 +
 +
7. An useful pdf introducing regularization of neural network: [http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.5-Regularization.pdf]
  
 
=='''Support Vector Machine - November 09, 2010'''==
 
=='''Support Vector Machine - November 09, 2010'''==
  
 
===Introduction===
 
===Introduction===
Through the course we have seen different methods for solving linearly separable problems, e.g.: Linear regression, LDA, Neural Networks. In most cases we can find a lot of linear boundaries for a problem that separates classes (see figure 1) and all having the same training error. A question arises, which of these boundaries is optimal and have minimum true error? Answer to this question lead to a new type of classifiers called [http://en.wikipedia.org/wiki/Support_vector_machine Support Vector Machines (SVM)].
+
 
 +
 
 +
Through the course we have seen different methods for solving linearly separable problems, e.g.: Linear regression, LDA, Neural Networks. In most cases, we can find many linear boundaries for a problem which separate classes (see figure 1) and all have the same training error. A question arises: which of these boundaries is optimal and has minimum true error? The answer to this question leads to a new type of classifiers called [http://en.wikipedia.org/wiki/Support_vector_machine Support Vector Machines (SVM)].
 
SVMs are a set of supervised learning methods.  
 
SVMs are a set of supervised learning methods.  
  
  
The original algorithm was proposed by Vladmier [http://en.wikipedia.org/wiki/Vapnik Vapnik] and later formulated to what is in current literature by Corinna Cortes and Vapnik. The modern history of SVM can be traced to 1974 when the field of [http://www.econ.upf.edu/~lugosi/mlss_slt.pdf statistical learning theory] was pioneered by [http://en.wikipedia.org/wiki/Vladimir_Vapnik Vladimir Vapnik] and [http://en.wikipedia.org/wiki/Alexey_Chervonenkis Alexey Chervonenkis]. In 1979, SVM was established when Vapnik further developed statistical learning theory and wrote a book in 1979 documenting his works. Since Vapnik's 1979 book was written in Russian, SVM did not become popular until Vapnik immigrated to the US and, in 1982, translated his 1979 book into English. More of SVM's history can be found in this [http://www.svms.org/history.html link].
+
The original algorithm was proposed by Vladimir [http://en.wikipedia.org/wiki/Vapnik Vapnik] and later formulated to what is in current literature by Corinna Cortes and Vapnik. The modern history of SVM can be traced to 1974 when the field of [http://www.econ.upf.edu/~lugosi/mlss_slt.pdf statistical learning theory] was pioneered by [http://en.wikipedia.org/wiki/Vladimir_Vapnik Vladimir Vapnik] and [http://en.wikipedia.org/wiki/Alexey_Chervonenkis Alexey Chervonenkis]. In 1979, SVM was established when Vapnik further developed statistical learning theory and wrote a book in 1979 documenting his works. Since Vapnik's 1979 book was written in Russian, SVM did not become popular until Vapnik immigrated to the US and, in 1982, translated his 1979 book into English. More of SVM's history can be found in this [http://www.svms.org/history.html link].
  
  
SVM was introduced after neural networks but got popular due to the outperformance compared to the neural networks in many applications e.g. bioinformatics, text, image recognition, until recently when notion of deep network was introduced by Hinton which outperforms SVM in some applications. A support vector machine constructs a hyperplane which can be used as classification boundary. These linear decision boundaries explicitly try to separate the data into different classes while maximizing the margin of separation. Intuitively, -if we are dealing with separable data clusters- a good separation is achieved by the hyperplane that has the largest distance to the nearest training data point(s) from each of the classes since in general the larger the margin the lower is the generalization error of the classifier, i.e. the lower is the probability that a new data point would be misclassified into its wrong class. The techniques that make the extensions to the non-linearly-separable case, where the classes overlap no matter what linear boundary is created, are generalized to what is known as the kernel support vector machine. Kernel SVM produces a nonlinear boundary by constructing a linear boundary in a higher-dimensional and transformed version of the feature space. This non-linear boundary is a linear boundary in the original feature space, making kernel SVM a linear classifier just as the original form of SVM is. No matter whether the training data are linearly-separable, the linear boundary produced by any of the versions of SVM is calculated using only a small fraction of the training data rather than using all of the training data points. This is much like the difference between the median and the mean. SVM can also be considered a special case of [http://en.wikipedia.org/wiki/Tikhonov_regularization Tikhonov regularization]. A special property is that they simultaneously minimize the empirical classification error and maximize the geometric margin; hence they are also known as maximum margin classifiers. The key features of SVM are the use of kernels, the absence of local minima, the sparseness of the solution (i.e. few training data points are needed to construct the linear decision boundary) and the capacity control obtained by optimising the margin."(Shawe-Taylor and Cristianini (2004)). Another key feature of SVM, as discussed below, is the use of [http://en.wikipedia.org/wiki/Slack_variable slack variables] to control the amount of tolerable misclassification on the training data. This key feature can serve to improve the generalization of SVM to new data. SVM has been used successfully in many real-world problems
+
The current standard incarnation of SVM is known as "soft margin" and was proposed by Corinna Cortes and Vladimir Vapnik [http://en.wikipedia.org/wiki/Vladimir_Vapnik]. In practice the data is not usually linearly separable. Although theoretically we can make the data linearly separable by mapping it into higher dimensions, the issues of how to obtain the mapping and how to avoid overfitting are still of concern. A more practical approach to classifying non-linearly separable data is to add some error tolerance to the separating hyperplane between the two classes, meaning that a data point in class A can cross the separating hyperplane into class B by a certain specified distance. This more generalized version of SVM is the so-called "soft margin" support vector machine and is generally accepted as the standard form of SVM over the hard margin case in practice today. [http://en.wikipedia.org/wiki/Support_vector_machine#Soft_margin]
 +
 
 +
 
 +
SVM was introduced after neural networks and gathered attention by outperforming neural networks in many applications e.g. bioinformatics, text, image recognition, etc. It retained popularity until recently, when the notion of deep network (introduced by Hinton) outperformed SVM in some applications. A support vector machine constructs a hyperplane which can be used as a classification boundary. These linear decision boundaries explicitly try to separate the data into different classes while maximizing the margin of separation. Intuitively, if we are dealing with separable data clusters, a good separation is achieved by the hyperplane that has the largest distance to the nearest training data point(s) from each of the classes since in general the larger the margin the lower is the generalization error of the classifier, i.e. the lower is the probability that a new data point would be misclassified into the wrong class.  
 +
 
 +
 
 +
The techniques that make the extensions to the non-linearly separable case, where the classes overlap no matter what linear boundary is created, are generalized to what is known as the "kernel support vector machine". Kernel SVM produces a nonlinear boundary by constructing a linear boundary in a higher-dimensional space and transformed feature space. This non-linear boundary is a linear boundary in the transformed feature space obtained by application of kernel, making kernel SVM a linear classifier just as the original form of SVM.  
 +
 
 +
 
 +
No matter whether the training data are linearly-separable or not, the linear boundary produced by any of the versions of SVM is calculated using only a small fraction of the training data rather than using all of the training data points. This is much like the difference between the median and the mean. SVM can also be considered a special case of [http://en.wikipedia.org/wiki/Tikhonov_regularization Tikhonov regularization]. A special property is that they simultaneously minimize the empirical classification error and maximize the geometric margin; hence they are also known as maximum margin classifiers. The key features of SVM are the use of kernels, the absence of local minima, the sparseness of the solution (i.e. few training data points are needed to construct the linear decision boundary) and the capacity control obtained by optimizing the margin.(Shawe-Taylor and Cristianini (2004)). Another key feature of SVM, as discussed below, is the use of [http://en.wikipedia.org/wiki/Slack_variable slack variables] to control the amount of tolerable misclassification on the training data, which form the soft margin SVM. This key feature can serve to improve the generalization of SVM to new data. SVM has been used successfully in many real-world problems:
  
 
- Pattern Recognition (Face Detection [17], Face Verification [18], Object Recognition [19], Handwritten Character/Digit Recognition  [20], Speaker/Speech Recognition [21], Image Retrieval [22], Prediction [23])
 
- Pattern Recognition (Face Detection [17], Face Verification [18], Object Recognition [19], Handwritten Character/Digit Recognition  [20], Speaker/Speech Recognition [21], Image Retrieval [22], Prediction [23])
Line 3,250: Line 4,081:
 
===Optimal Separating Hyperplane===
 
===Optimal Separating Hyperplane===
  
As can be seen in figure 1, there exists an infinite number of linear hyperplanes between the classes.A Support Vector Machine (SVM) performs classification by constructing an N-dimensional hyperplane that optimally separates the data into two categories
+
As can be seen in Figure 1, there exists an infinite number of linear hyperplanes between the two classes. A Support Vector Machine (SVM) performs classification by constructing an N-dimensional hyperplane that optimally separates the data into two categories.
The data points which are indicated in figure 2 are the datapoints that the Margin is pushed up against and are called Support Vectors.
+
The data points which are indicated in the green circles in Figure 2 are the data points that are on the boundary of the margin and are called Support Vectors.
 
[[File:Yyy.png|250px|thumb|right|Fig. 1 Linear Classifiers]]  
 
[[File:Yyy.png|250px|thumb|right|Fig. 1 Linear Classifiers]]  
 
[[File:Xxx.png|250px|thumb|right|Fig. 2 Maximum Margin]]
 
[[File:Xxx.png|250px|thumb|right|Fig. 2 Maximum Margin]]
Line 3,260: Line 4,091:
 
Figure 3 shows the linear algebra of the hyperplane, where <math>\,d_i</math> is the distance between the origin and a point <math>\,x_i</math> on the hyperplane.
 
Figure 3 shows the linear algebra of the hyperplane, where <math>\,d_i</math> is the distance between the origin and a point <math>\,x_i</math> on the hyperplane.
  
Suppose hyperplane is defined as <math>\displaystyle \beta^{T}x+\beta_0=0</math> as shown in figure 3 and suppose that data is linearly separable and <math>\displaystyle y_i \in \{-1,1 \} </math>.
+
Suppose a hyperplane is defined as <math>\displaystyle \beta^{T}x+\beta_0=0</math>, as shown in figure 3, and suppose that the data is linearly separable and <math>\displaystyle y_i \in \{-1,1 \} </math>. Where <math>\displaystyle \beta_0</math> is the distance of the hyperplane to the origin.
  
<math>\displaystyle \beta_0</math> is distance of the hyperplane to the origin
 
  
 +
Property 1:  <math>\displaystyle \beta </math> is orthogonal to the hyperplane
  
1)
+
Suppose that <math>\displaystyle x_1,x_2</math> are <br />lying on the hyperplane. Then we have
  
<math>\displaystyle \beta </math> is orthogonal to the hyperplane <br />
+
: <math>\displaystyle \beta^{T}x_1+\beta_0=0</math> , and
  
Suppose that <math>\displaystyle x_1,x_2</math> are lying on the hyperplane.
+
: <math>\displaystyle \beta^{T}x_2+\beta_0=0</math> .
  
<math>\displaystyle \beta^{T}x_1+\beta_0=0</math>
+
Therefore,
  
<math>\displaystyle \beta^{T}x_2+\beta_0=0</math>
+
: <math>\displaystyle \beta^{T}x_1+\beta_0 - (\beta^{T}x_2+\beta_0)=0</math> , and
  
therefore : <math>\displaystyle \beta^{T}x_1+\beta_0 - (\beta^{T}x_2+\beta_0)=0</math>
+
: <math>\displaystyle \beta^{T}(x_1-x_2)=0</math> .
  
<math>\displaystyle \beta^{T}(x_1-x_2)=0</math>
+
Hence,
  
Hence,<math>\displaystyle \beta \bot \displaystyle (x_1 - x_2)</math>. But <math>\displaystyle x_1-x_2</math> is a vector lying in the hyperplane and since the two points were arbitrary so <math>\displaystyle \beta </math> is orthogonal to every vector lying in the hyperplane and by definition orthogonal to hyperplane
+
: <math>\displaystyle \beta \bot \displaystyle (x_1 - x_2)</math> .
  
 +
But <math>\displaystyle x_1-x_2</math> is a vector lying in the hyperplane, since the two points were arbitrary. So, <math>\displaystyle \beta </math> is orthogonal to every vector lying in the hyperplane and by definition orthogonal to hyperplane.
  
2)
+
 
 +
Property 2:
  
 
For any point <math>\displaystyle  x_0 </math> on the hyperplane, we can say that
 
For any point <math>\displaystyle  x_0 </math> on the hyperplane, we can say that
  
<math>\displaystyle \beta^{T}x_0+\beta_0=0</math>
+
: <math>\displaystyle \beta^{T}x_0+\beta_0=0</math>&nbsp; and
  
<math>\displaystyle \beta^{T}x_0=-\beta_0</math>
+
: <math>\displaystyle \beta^{T}x_0=-\beta_0</math> .
For any point on the hyperplane, multiplying by <math>\displaystyle \beta^{T}</math> gives negative value of the intercept of the hyperplane. <br/>
 
3)
 
  
For any point <math>\displaystyle x_i</math> the distance of the point to the hyperplane is denoted by <math>\displaystyle d_i</math>, which is the projection of (<math>\displaystyle x_i - x_0</math>) on <math>\displaystyle\beta</math>.
+
For any point on the hyperplane, multiplying by <math>\displaystyle \beta^{T}</math> gives negative value of the intercept of the hyperplane.
The signed distance for any point <math>\displaystyle x_i </math> to the hyperplane is <math> \displaystyle d_i = \beta^{T}(x_i - x_0)</math>. <br/>Since the length of <math>\displaystyle \beta </math> changes the value of distance , normalize it by diving it into its length.
+
<br />
  
<math>\displaystyle d_i=\frac{\beta^{T}(x_i-x_0)}{\|\beta\|}  </math> <math>\displaystyle i=1,2,....,N </math>
 
  
<math>\displaystyle d_i=\frac{\beta^{T}x_i-\beta^{T}x_0}{\|\beta\|} </math>
+
Property 3:
  
by property 2
+
For any point <math>\displaystyle x_i</math>, let the distance of the point to the hyperplane be denoted by <math>\displaystyle d_i</math>, which is the projection of (<math>\displaystyle x_i - x_0</math>) onto <math>\displaystyle\beta</math>. The signed distance for any point <math>\displaystyle x_i </math> to the hyperplane is <math> \displaystyle d_i = \beta^{T}(x_i - x_0)</math>. Since the length of <math>\displaystyle \beta </math> changes the value of the distance, we can normalize it by dividing <math>\displaystyle \beta </math> into its length. Thus, we get
  
<math>\displaystyle d_i=\frac{\beta^{T}x_i+\beta_0}{\|\beta\|} </math>
+
: <math>\displaystyle d_i=\frac{\beta^{T}(x_i-x_0)}{\|\beta\|} </math> <math>\displaystyle i=1,2,....,N </math> ,
  
Therefore, for any point if we want to find its distance to the hyperplane we simply put it in the above equation.
+
: <math>\displaystyle d_i=\frac{\beta^{T}x_i-\beta^{T}x_0}{\|\beta\|} </math>&nbsp; by property 2, and
  
4)
+
: <math>\displaystyle d_i=\frac{\beta^{T}x_i+\beta_0}{\|\beta\|} </math> .
  
We use lables to make the distance positive, therefore: <math>\displaystyle  Margin=(y_id_i)</math>
+
Therefore, for any point if we want to find it's distance to the hyperplane we simply put it in the above equation.
and since we would like to maximize the Margin:
 
  
<math>\displaystyle  Margin=max(y_id_i)</math>      <math>\displaystyle i=1,2,....,N </math>, and since we now know how to compute <math>\displaystyle  d_i \Rightarrow</math>
 
  
<math>\displaystyle  Margin=max\{y_i\frac{\beta^{T}x_i+\beta_0}{\|\beta\|}\} </math>
+
Property 4:
  
<math>\displaystyle y_i(\beta^{T}x_i+\beta_0)\ge 0</math>
+
We use labels to make the distance positive. Therefore, let <math>\displaystyle Margin=(y_id_i)</math>. Since we would like to maximize the Margin, we have
  
 +
: <math>\displaystyle  Margin=max(y_id_i)</math>      <math>\displaystyle i=1,2,....,N </math> .
  
Since it is a distance it is always non-negative. If the point is on the hyperplane, it is zero. Otherwise, it is greater than zero.
+
Since we now know how to compute <math>\displaystyle  d_i </math> , by property 3,
 +
 
 +
: <math>\displaystyle  Margin=max\{y_i\frac{\beta^{T}x_i+\beta_0}{\|\beta\|}\}    \quad    (1)</math> , and
 +
 
 +
: <math>\displaystyle y_i(\beta^{T}x_i+\beta_0)\ge 0</math> .
 +
 
 +
Since the margin is a distance it is always non-negative. If the point is on the hyperplane, it is zero. Otherwise, it is greater than zero.
 
For all training data points <math>\,i</math> that are not on the hyperplane,  
 
For all training data points <math>\,i</math> that are not on the hyperplane,  
  
<math>\displaystyle y_i(\beta^{T}x_i+\beta_0)>0 </math>
+
: <math>\displaystyle y_i(\beta^{T}x_i+\beta_0)>0 </math> .
  
Let <math> \displaystyle c >0 </math> be the least of the distances between the training data points that are not on hyperplane and the hyperplane. We have <math>\, y_i(\beta^{T}x_i+\beta_0)\ge c </math> for all training data points <math> \displaystyle i </math> that are not on the hyperplane.
+
Let <math> \displaystyle c>0 </math> be the minimum distance between the hyperplane and the training data points not on the hyperplane. We have
  
<math>\displaystyle y_i(\frac{\beta^{T}x_i}{c}+\frac{\beta_0}{c})\geq1</math>
+
: <math>\, y_i(\beta^{T}x_i+\beta_0)\ge c </math>
  
This is known as the canonical representation of the decision hyperplane.
+
for all training data points <math> \displaystyle i </math> that are not on the hyperplane. Thus,
  
For <math>\displaystyle \beta^{T} </math> only the direction is important, so <math>\displaystyle \frac{\beta^{T}}{c} </math> does not change its direction, the hyperplane will be the same.
+
: <math>\displaystyle y_i(\frac{\beta^{T}x_i}{c}+\frac{\beta_0}{c})\ge 1</math> .
  
<math>\displaystyle y_i(\beta^{T}x_i+\beta_0)\geq1 </math>
+
This is known as the canonical representation of the decision hyperplane. For <math>\displaystyle \beta^{T} </math> only the direction is important, so <math>\displaystyle \frac{\beta^{T}}{c} </math> does not change its direction and the hyperplane will be the same. Thus,
  
<math>\displaystyle y_i\frac{\beta^{T}x_i+\beta_0}{\|\beta\|}\geq1 \frac{1}{\|\beta\|} </math>
+
: <math>\displaystyle y_i(\beta^{T}x_i+\beta_0)\ge 1   \quad  (2)</math> ,<br />
  
 +
equivalently, as we care only about the direction of the <math>\displaystyle\beta</math>, we can write:<br />
  
<math>\displaystyle  y_i \beta^{T}x_i+\beta_0 \geq 1 </math>
+
: <math>\displaystyle  y_i\frac{\beta^{T}x_i+\beta_0}{\|\beta\|}\geq1  </math> <br /><br />
  
Therefore, in order to maximize the Margin we have to minimize the norm of <math>\,\beta</math>.
+
Considering (2) and (1), for the the closest datapoints to the margin (those datapoints, which are placed at the distance 1 to the margin as shown above), (1) becomes:<br />
  
 +
: <math>\displaystyle  Margin=max\{\frac{1}{\|\beta\|}\}    </math>
  
minimize <math>\displaystyle\|\beta\|^2</math>
 
  
minimize <math>\displaystyle\frac{1}{2}\|\beta\|^2</math>   s.t <math> \displaystyle y_i(\beta^T x_i + \beta_0) \geq 1  \forall</math> i
+
Therefore, in order to maximize the margin we have to minimize the norm of <math>\,\beta</math>. So, we get
  
for the <math>\displaystyle\beta</math> s which have distances greater or equal to one. We have used <math>\displaystyle\frac{1}{2}</math> factor only for convenience in derivation of derivative.
+
: minimize <math>\displaystyle\|\beta\|^2</math>&nbsp; and
====Writing Lagrangian Form of Support Vector Machine====
+
 
 +
: minimize <math>\displaystyle\frac{1}{2}\|\beta\|^2</math>    s.t <math> \displaystyle y_i(\beta^T x_i + \beta_0) \geq 1  \forall</math> i
 +
 
 +
for the <math>\displaystyle\beta</math> s which have distance greater than or equal to one.  
 +
<br />
 +
we choose to minimize norm 2 of <math>\displaystyle\beta</math> mainly for the sake of simplified optimization.
 +
We have used <math>\displaystyle\frac{1}{2}</math> factor only for convenience in derivation of the derivative.
 +
 
 +
===Writing Lagrangian Form of Support Vector Machine===
  
 
The Lagrangian form using [http://en.wikipedia.org/wiki/Lagrange_multipliers Lagrange multipliers] and constraints that are discussed below is introduced to ensure that the optimization conditions are satisfied, as well as finding an optimal solution (the optimal saddle point of the Lagrangian for the [http://en.wikipedia.org/wiki/Quadratic_programming classic quadratic optimization]). The problem will be solved in dual space by introducing  <math>\,\alpha_i</math> as dual constraints, this is in contrast to solving the problem in primal space as function of the betas.  A [http://www.cs.wisc.edu/dmi/lsvm/ simple algorithm] for iteratively solving the Lagrangian has been found to run well on very large data sets, making SVM more usable.  Note that this algorithm is intended to solve Support Vector Machines with some tolerance for errors - not all points are necessarily classified correctly.  Several papers by Mangasarian explore different algorithms for solving SVM.
 
The Lagrangian form using [http://en.wikipedia.org/wiki/Lagrange_multipliers Lagrange multipliers] and constraints that are discussed below is introduced to ensure that the optimization conditions are satisfied, as well as finding an optimal solution (the optimal saddle point of the Lagrangian for the [http://en.wikipedia.org/wiki/Quadratic_programming classic quadratic optimization]). The problem will be solved in dual space by introducing  <math>\,\alpha_i</math> as dual constraints, this is in contrast to solving the problem in primal space as function of the betas.  A [http://www.cs.wisc.edu/dmi/lsvm/ simple algorithm] for iteratively solving the Lagrangian has been found to run well on very large data sets, making SVM more usable.  Note that this algorithm is intended to solve Support Vector Machines with some tolerance for errors - not all points are necessarily classified correctly.  Several papers by Mangasarian explore different algorithms for solving SVM.
Line 3,351: Line 4,195:
 
Dual form of the optimization problem:
 
Dual form of the optimization problem:
  
<math>\,L(\beta,\beta_0,\alpha) = \frac{1}{2}\|\beta\|^2 - \sum_{i=1}^n{\alpha_i\left(y_i(\beta^Tx_i+\beta_0)-1\right)}</math>.   
+
: <math>\,L(\beta,\beta_0,\alpha) = \frac{1}{2}\|\beta\|^2 - \sum_{i=1}^n{\alpha_i\left(y_i(\beta^Tx_i+\beta_0)-1\right)}</math> .   
 +
 
 +
To find the optimal value, we set the derivative equal to zero:
 +
 
 +
: <math>\,\frac{\partial L}{\partial \beta} = 0</math> and <math>\,\frac{\partial L}{\partial \beta_0} = 0</math> .
  
To find the optimal value, set the derivative equal to zero.
+
Note that <math>\,\frac{\partial L}{\partial \alpha_i}</math> is equivalent to the constraints <math>\left(y_i(\beta^Tx_i+\beta_0)-1\right) \geq 0, \,\forall\, i</math>.
  
<math>\,\frac{\partial L}{\partial \beta} = 0</math>, <math>\,\frac{\partial L}{\partial \beta_0} = 0</math>.
+
First, setting <math>\,\frac{\partial L}{\partial \beta} = 0</math>:
Note that <math>\,\frac{\partial L}{\partial \alpha_i}</math> is equivalent to the constraints <math>\left(y_i(\beta^Tx_i+\beta_0)-1\right) \geq 0, \,\forall\, i</math>
 
  
First, <math>\,\frac{\partial L}{\partial \beta} = 0</math>
+
: <math>\,\frac{\partial L}{\partial \beta} = \frac{\partial}{\partial \beta}\frac{1}{2}\|\beta\|^2 - \sum_{i=1}^n{\left\{\frac{\partial}{\partial \beta}(\alpha_iy_i\beta^Tx_i)+\frac{\partial}{\partial \beta}\alpha_iy_i\beta_0-\frac{\partial}{\partial \beta}\alpha_iy_i\right\}}</math> ,
  
<math>\,\frac{\partial L}{\partial \beta} = \frac{\partial}{\partial \beta}\frac{1}{2}\|\beta\|^2 - \sum_{i=1}^n{\left\{\frac{\partial}{\partial \beta}(\alpha_iy_i\beta^Tx_i)+\frac{\partial}{\partial \beta}\alpha_iy_i\beta_0-\frac{\partial}{\partial \beta}\alpha_iy_i\right\}}</math>
+
: <math>\frac{\partial}{\partial \beta}\frac{1}{2}\|\beta\|^2 = \beta</math> ,
  
: <math>\frac{\partial}{\partial \beta}\frac{1}{2}\|\beta\|^2 = \beta</math>.
+
: <math>\,\frac{\partial}{\partial \beta}(\alpha_iy_i\beta^Tx_i) = \alpha_iy_ix_i</math> ,
  
: <math>\,\frac{\partial}{\partial \beta}(\alpha_iy_i\beta^Tx_i) = \alpha_iy_ix_i</math>
+
: <math>\,\frac{\partial}{\partial \beta}\alpha_iy_i\beta_0 = 0</math> , and
  
: <math>\,\frac{\partial}{\partial \beta}\alpha_iy_i\beta_0 = 0</math>.  
+
: <math>\,\frac{\partial}{\partial \beta}\alpha_iy_i = 0</math> .
  
: <math>\,\frac{\partial}{\partial \beta}\alpha_iy_i = 0</math>.
+
So this simplifies to <math>\,\frac{\partial L}{\partial \beta} = \beta - \sum_{i=1}^n{\alpha_iy_ix_i} = 0</math> . In other words,
  
So this simplifies to <math>\,\frac{\partial L}{\partial \beta} = \beta - \sum_{i=1}^n{\alpha_iy_ix_i} = 0</math>.   In other words,
+
: <math>\,\beta = \sum_{i=1}^n{\alpha_iy_ix_i}</math> and <math>\,\beta^T = \sum_{i=1}^n{\alpha_iy_ix_i^T}</math> .
  
<math>\,\beta = \sum_{i=1}^n{\alpha_iy_ix_i}</math>, <math>\,\beta^T = \sum_{i=1}^n{\alpha_iy_ix_i^T}</math>
+
Similarly, <math>\,\frac{\partial L}{\partial \beta_0} = \sum_{i=1}^n{\alpha_iy_i} = 0</math> .
  
Similarly, <math>\,\frac{\partial L}{\partial \beta_0} = \sum_{i=1}^n{\alpha_iy_i} = 0</math>.
+
Thus, our objective function becomes <math>\,\frac{1}{2}\sum_{i=1}^n{\sum_{j=1}^n{\alpha_i\alpha_jy_iy_jx_i^Tx_j}} - \sum_{i=1}^n{\sum_{j=1}^n{\alpha_i\alpha_jy_iy_jx_i^Tx_j}} + \sum_{i=1}^n{\alpha_i}</math>, which is a dual representation of the maximum margin. Since <math>\,\alpha_i</math> is the Lagrange multiplier, <math>\,\alpha_i \geq 0 \forall i</math>. Therefore, we have a new optimization problem:
  
 +
: <math>\underset{\alpha}{\max} \sum_{i=1}^n{\alpha_i}- \,\frac{1}{2}\sum_{i=1}^n{\sum_{j=1}^n{\alpha_i\alpha_jy_iy_jx_i^Tx_j}} </math> , where
  
Thus, our objective function becomes <math>\,\frac{1}{2}\sum_{i=1}^n{\sum_{j=1}^n{\alpha_i\alpha_jy_iy_jx_i^Tx_j}} - \sum_{i=1}^n{\sum_{j=1}^n{\alpha_i\alpha_jy_iy_jx_i^Tx_j}} + \sum_{i=1}^n{\alpha_i}</math>, which is a dual representation of the maximum margin.
+
: <math>\,\alpha_i \ge 0 \forall i</math>&nbsp; and
  
 +
: <math>\,\Sigma_i{\alpha_i y_i} = 0</math> .
  
Because <math>\,\alpha_i</math> is the Lagrange multiplier, <math>\,\alpha_i \geq 0 \forall i</math>.
+
This is a much simpler optimization problem and we can solve it by [http://en.wikipedia.org/wiki/Quadratic_programming]. Quadratic programming (QP) is a special type of mathematical optimization problem. It is the problem of optimizing (minimizing or maximizing) a quadratic function of several variables subject to linear constraints on these variables.
 +
The general form of such a problem is minimize with respect to <math>\,x</math>
 +
: <math>f(x) = \frac{1}{2}x^TQx + c^Tx</math>
 +
subject to one or more constraints of the form <math>\,Ax\le b</math>, <math>\,Ex=d</math>.
  
 +
See this [http://www.me.utexas.edu/~jensen/ORMM/supplements/methods/nlpmethod/S2_quadratic.pdf link] for a good description of general QP problem formulation and solution.
  
Therefore, we have a new optimization problem:
+
====Remark====
 +
Using Lagrangian Primal/Lagrangian Dual will give us two very different results when formulating this problem:
 +
-------------
 +
If use Lagrangian Primal:
  
<math>\underset{\alpha}{\max} \sum_{i=1}^n{\alpha_i}- \,\frac{1}{2}\sum_{i=1}^n{\sum_{j=1}^n{\alpha_i\alpha_jy_iy_jx_i^Tx_j}} </math>
+
:<math>\,min{L(\beta,\beta_0,\alpha)}</math>
 +
where
 +
:<math>\,L(\beta,\beta_0,\alpha) = \frac{1}{2}\|\beta\|^2 - \sum_{i=1}^n{\alpha_i\left(y_i(\beta^Tx_i+\beta_0)-1\right)}</math>
 +
such that
 +
:<math>\,\frac{\partial L}{\partial \beta} = 0</math> , <math>\,\frac{\partial L}{\partial \beta_0} = 0</math> and <math>\alpha_i\left(y_i(\beta^Tx_i+\beta_0)-1\right) = 0, \,\forall\, i</math>
  
1)  <math>\,\alpha_i \ge 0 \forall i</math>
+
-------------
 +
If use Lagrangian Dual:
  
2) <math>\,\Sigma_i{\alpha_i y_i} = 0</math>  
+
:<math>\,max{L(\beta,\beta_0,\alpha)}</math>
 +
where
 +
:<math>\,L(\beta,\beta_0,\alpha) = \frac{1}{2}\|\beta\|^2 - \sum_{i=1}^n{\alpha_i\left(y_i(\beta^Tx_i+\beta_0)-1\right)}</math>
 +
such that
 +
:<math>\,\frac{\partial L}{\partial \beta} = 0</math> and <math>\,\frac{\partial L}{\partial \beta_0} = 0</math> (Note: no <math>\alpha_i\left(y_i(\beta^Tx_i+\beta_0)-1\right) = 0, \,\forall\, i</math> constraint anymore)
  
This is a much simpler optimization problem and we can solve it by [http://en.wikipedia.org/wiki/Quadratic_programming]. Quadratic programming (QP) is a special type of mathematical optimization problem. It is the problem of optimizing (minimizing or maximizing) a quadratic function of several variables subject to linear constraints on these variables.
+
-------------
 +
We do not use Lagrangian Primal here because this formulation will generate quadratic constraints, which cannot be solved by 'quadprog.m'.
 +
Click [http://www.icom.nctu.edu.tw/CLSCM/download/Handouts/or706_week13.pdf here] for more details about Lagrangian Duality.
  
====Quadratic Programming Problem of SVMs and Dual Problem====
+
===Quadratic Programming Problem of SVMs and Dual Problem===
  
 
We have to find <math>\,\beta</math> and <math>\,\beta_0</math> such that <math>\,\frac{1}{2}\|\beta\|^2 </math> is minimized subject to <math> \,y_i (\beta^T x_i + \beta_0) \geq 1 \forall i </math>.
 
We have to find <math>\,\beta</math> and <math>\,\beta_0</math> such that <math>\,\frac{1}{2}\|\beta\|^2 </math> is minimized subject to <math> \,y_i (\beta^T x_i + \beta_0) \geq 1 \forall i </math>.
 
 
Therefore, we need to optimize a quadratic function subject to linear constraints.
 
Therefore, we need to optimize a quadratic function subject to linear constraints.
 
Quadratic optimization problems are a well-known class of mathematical programming problems, and many (rather intricate) algorithms exist for solving them.  
 
Quadratic optimization problems are a well-known class of mathematical programming problems, and many (rather intricate) algorithms exist for solving them.  
Line 3,407: Line 4,273:
 
Lagrange multipliers that maximize the objective function
 
Lagrange multipliers that maximize the objective function
  
<math>\,Q(\alpha)= \underset{\alpha}{\max}  \sum_{i=1}^n{\alpha_i}- \,\frac{1}{2}\sum_{i=1}^n{\sum_{j=1}^n{\alpha_i\alpha_jy_iy_jx_i^Tx_j}} </math>
+
: <math>\,Q(\alpha)= \underset{\alpha}{\max}  \sum_{i=1}^n{\alpha_i}- \,\frac{1}{2}\sum_{i=1}^n{\sum_{j=1}^n{\alpha_i\alpha_jy_iy_jx_i^Tx_j}} </math>
  
 
subject to the constraints  
 
subject to the constraints  
  
1)  <math>\,\alpha_i \ge 0 \forall i</math>  
+
: <math>\,\alpha_i \ge 0 \forall i</math>&nbsp; and
  
2)  <math>\,\Sigma_i{\alpha_i y_i} = 0</math>
+
: <math>\,\Sigma_i{\alpha_i y_i} = 0</math> .
  
===Implementation===
+
=====Feasibility of the Primal and Dual Programming Problems=====
 +
 
 +
For Hard-Margin SVM, the primal and dual programming problems are feasible with any values of <math>\, y_i </math> and <math>\, x_i</math>.  Feasibility depends on satisfaction of the problem's constraints.  In the primal problem, the constraint <math> \,y_i (\beta^T x_i + \beta_0) \geq 1 \forall i </math> can be satisfied by setting <math>\, \beta = \underline{0} </math> and by setting <math>\, \beta_0 = \frac{1}{y_i} </math> for any values of <math>\, y_i </math> and <math>\, x_i</math>.  In the dual problem, the constrants <math>\,\alpha_i \ge 0 \forall i</math> and <math>\,\Sigma_i{\alpha_i y_i} = 0</math> can be satisfied by setting <math>\,\alpha_i = 0 \forall i</math> for any values of <math>\, y_i </math> and <math>\, x_i</math>.  The constaints can always be satisfied, so the programming problems are always feasible.
 +
 
 +
===Implementation===
  
 
The parameters of the maximum-margin hyperplane are derived by solving the optimization. There exist several specialized algorithms for quickly solving the QP problem that arises from SVMs, mostly reliant on heuristics for breaking the problem down into smaller, more-manageable chunks. A common method for solving the QP problem is the Sequential Minimal Optimization (SMO) algorithm by John Platt in 1998. A link to the original paper of which is available [http://research.microsoft.com/en-us/um/people/jplatt/smoTR.pdf here]. SMO which breaks the problem down into 2-dimensional sub-problems that may be solved analytically, eliminating the need for a numerical optimization algorithm.
 
The parameters of the maximum-margin hyperplane are derived by solving the optimization. There exist several specialized algorithms for quickly solving the QP problem that arises from SVMs, mostly reliant on heuristics for breaking the problem down into smaller, more-manageable chunks. A common method for solving the QP problem is the Sequential Minimal Optimization (SMO) algorithm by John Platt in 1998. A link to the original paper of which is available [http://research.microsoft.com/en-us/um/people/jplatt/smoTR.pdf here]. SMO which breaks the problem down into 2-dimensional sub-problems that may be solved analytically, eliminating the need for a numerical optimization algorithm.
Line 3,429: Line 4,299:
  
 
SVM is only directly applicable for two-class case. We want to generalize this algorithm to multi-class tasks. Multiclass SVM aims to assign labels to instances by using support vector machines, where the labels are drawn from a finite set of several elements. The dominating approach for doing so is to reduce the single multiclass problem into multiple binary problems. Each of the problems yields a binary classifier, which is assumed to produce an output function that gives relatively large values for examples from the positive class and relatively small values for examples belonging to the negative classes. Two common methods to build such binary classifiers are where each classifier distinguishes between (i) one of the labels to the rest (one-versus-all) or (ii) between every pair of classes (one-versus-one). Classification of new instances for one-versus-all case is done by a winner-takes-all strategy, in which the classifier with the highest output function assigns the class (it is important that the output functions be calibrated to produce comparable scores). For the one-versus-one approach, classification is done by a max-wins voting strategy, in which every classifier assigns the instance to one of the two classes, then the vote for the assigned class is increased by one vote, and finally the class with most votes determines the instance classification.
 
SVM is only directly applicable for two-class case. We want to generalize this algorithm to multi-class tasks. Multiclass SVM aims to assign labels to instances by using support vector machines, where the labels are drawn from a finite set of several elements. The dominating approach for doing so is to reduce the single multiclass problem into multiple binary problems. Each of the problems yields a binary classifier, which is assumed to produce an output function that gives relatively large values for examples from the positive class and relatively small values for examples belonging to the negative classes. Two common methods to build such binary classifiers are where each classifier distinguishes between (i) one of the labels to the rest (one-versus-all) or (ii) between every pair of classes (one-versus-one). Classification of new instances for one-versus-all case is done by a winner-takes-all strategy, in which the classifier with the highest output function assigns the class (it is important that the output functions be calibrated to produce comparable scores). For the one-versus-one approach, classification is done by a max-wins voting strategy, in which every classifier assigns the instance to one of the two classes, then the vote for the assigned class is increased by one vote, and finally the class with most votes determines the instance classification.
 +
 +
LIBSVM is an integrated software for support vector classification, regression and distribution estimation. It supports multi-class classification.
 +
[http://www.csie.ntu.edu.tw/~cjlin/libsvm/]
 +
 +
Koby Cramer and  Yoram Singer have proposed a direct way to learn multiclass SVMs. It is described here in their paper:
 +
[http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.69.8716 On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines]
 +
 +
  
 
==== Implements SVM multi-class ====
 
==== Implements SVM multi-class ====
Line 3,438: Line 4,316:
 
The development of ANNs followed a heuristic path, with applications and extensive experimentation preceding theory. In contrast, the development of SVMs involved sound theory first, then implementation and experiments. A significant advantage of SVMs is that whilst ANNs can suffer from multiple local minima, the solution to an SVM is global and unique. Two more advantages of SVMs are that they have a simple geometric interpretation and also a sparse solution. Unlike ANNs, the computational complexity of SVMs does not depend on the dimensionality of the input space. ANNs use empirical risk minimization, whilst SVMs use structural risk minimization. The reason that SVMs often outperform ANNs in practice is that they deal with the biggest problem with ANNs, SVMs are less prone to over-fitting since their solution is sparse. In contrast to neural networks SVMs automatically select their model size (by selecting the Support vectors)(Rychetsky (2001)).While the weight decay term is an important aspect for obtaining good generalization in the context of neural networks for regression, the gamma parameter (in soft-margin SVM) that is discussed below plays a somewhat similar role in classification problems.
 
The development of ANNs followed a heuristic path, with applications and extensive experimentation preceding theory. In contrast, the development of SVMs involved sound theory first, then implementation and experiments. A significant advantage of SVMs is that whilst ANNs can suffer from multiple local minima, the solution to an SVM is global and unique. Two more advantages of SVMs are that they have a simple geometric interpretation and also a sparse solution. Unlike ANNs, the computational complexity of SVMs does not depend on the dimensionality of the input space. ANNs use empirical risk minimization, whilst SVMs use structural risk minimization. The reason that SVMs often outperform ANNs in practice is that they deal with the biggest problem with ANNs, SVMs are less prone to over-fitting since their solution is sparse. In contrast to neural networks SVMs automatically select their model size (by selecting the Support vectors)(Rychetsky (2001)).While the weight decay term is an important aspect for obtaining good generalization in the context of neural networks for regression, the gamma parameter (in soft-margin SVM) that is discussed below plays a somewhat similar role in classification problems.
  
===Advantages of Support Vector Machines===
+
===SVM packages===
 +
One of the popular Matlab toolboxes for SVM is [http://www.csie.ntu.edu.tw/~cjlin/libsvm/ LIBSVM], which has been developed in the department of Computer Science and Information Engineering, National Taiwan University, under supervision of Chih-Chung Chang and Chih-Jen Lin. In this page they have provided the society with many different interfaces for LIBSVM like Matlab, C++, Python, Perl, and many other languages, each one of those has been developed in different institutes and by variety of engineers and mathematicians. In this page you can also find a thorough introduction to the package and its various parameters.
  
* SVMs provide a good out-of-sample generalization. This means that, by choosing an appropriate generalization grade,
+
A very helpful tool which you can find on the [http://www.csie.ntu.edu.tw/~cjlin/libsvm/ LIBSVM] page is a graphical interface for SVM; it is an applet by which we can draw points corresponding to each of the two classes of the classification problem and by adjusting the SVM parameters, observe the resulting solution.
SVMs can be robust, even when the training sample has some bias. This is mainly due to selection of optimal hyperplane.
 
* SVMs deliver a unique solution, since the optimality problem is convex. This is an advantage compared
 
to Neural Networks, which have multiple solutions associated with local minima and for this reason may
 
not be robust over different samples.  
 
  
===Disadvantages of Support Vector Machines [http://www.cse.unr.edu/~bebis/MathMethods/SVM/lecture.pdf]===
+
If you found LIBSVM helpful and wanted to use it for your research, [http://www.csie.ntu.edu.tw/~cjlin/libsvm/faq.html#f203 please cite the toolbox].
  
*Perhaps the biggest limitation of the support vector approach lies in choice of the kernel (Which we will study about in future).
+
A pretty long list of other SVM packages and comparison between all of them in terms of language, execution platform, multiclass and regression capabilities, is available [http://www.cs.ubc.ca/~murphyk/Software/svm.htm here].
  
*A second limitation is speed and size, both in training and testing (mostly in training - for large training sets, it typically selects a small number of support vectors, thereby minimizing the computational requirements during testing).
+
The top 3 SVM software are:
  
*Discrete data presents another problem, although with suitable rescaling excellent results have nevertheless been obtained.
+
1. LIBSVM
  
*The optimal design for multiclass SVM classifiers is a further area for research.
+
2. SVMlight
 
*Although SVMs have good generalization performance, they can be abysmally slow in test phase.
 
 
*Besides the advantages of SVMs - from a practical point of view - they have some drawbacks. An important practical question that is not entirely solved, is the selection of the kernel function parameters - for Gaussian kernels the width parameter [sigma] - and the value of [epsilon] in the [epsilon]-insensitive loss function.
 
  
*However, from a practical point of view perhaps the most serious problem with SVMs is the high algorithmic complexity and extensive memory requirements of the required quadratic programming in large-scale tasks.
+
3. SVMTorch
  
===Applications of Support Vector Machines===
+
Also, there are other two web pages introducing SVM software and their comparison: [http://www.svms.org/software.html] and [http://www.support-vector-machines.org/SVM_soft.html].
  
The following papers describe some of the possible applications of support vector machines:
+
===References===
  
1- Training support vector machines: an application to face detection [http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=609310 here]
+
1. V. Vapnik and A. Chervonenkis, Theory of Pattern Recognition, Nauka, Moscow, 1974.
  
2- Application of support vector machines in financial time series forecasting [http://svms.org/regression/TaCa01.pdf here]
+
2. V. Vapnik, Estimation of Dependencies Based on Empirical Data, Nauka, Moscow, 1979.
  
3- Support vector machine active learning with applications to text classification [http://portal.acm.org/citation.cfm?id=944793&dl=GUIDE, here]
+
3. V. Vapnik, The Nature of Statistical Learning Theory, Springer, 1995.
  
4- Support Vector Machines: Training and Applications [http://dspace.mit.edu/bitstream/handle/1721.1/7290/AIM-1602.pdf?sequence=2 here]
+
4. V. Vapnik, Statistical Learning Theory, Wiley-Interscience, New York, 1998.  
  
5- Application of support vector machines for T-cell epitopes prediction [http://bioinformatics.oxfordjournals.org/content/19/15/1978.full.pdf here]
+
5. P. H. Chen, C. J. Lin, and B. Schölkopf, A tutorial on ν-support vector machines, Appl. Stoch. Models. Bus. Ind. 21, 111-136, 2005.
  
===SVM packages===
+
6. S.-W. Lee and A. Verri (Eds.),Applications of Support Vector Machines for Pattern Recognition: SVM 2002, LNCS 2388, pp. 213-236, 2002.
One of the popular Matlab toolboxes for SVM is [http://www.csie.ntu.edu.tw/~cjlin/libsvm/ LIBSVM], which has been developed in the department of Computer Science and Information Engineering, National Taiwan University, under supervision of Chih-Chung Chang and Chih-Jen Lin. In this page they have provided the society with many different interfaces for LIBSVM like Matlab, C++, Python, Perl, and many other languages, each one of those has been developed in different institutes and by variety of engineers and mathematicians. In this page you can also find a thorough introduction to the package and its various parameters.
 
  
A very helpful tool which you can find on the [http://www.csie.ntu.edu.tw/~cjlin/libsvm/ LIBSVM] page is a graphical interface for SVM; it is an applet by which we can draw points corresponding to each of the two classes of the classification problem and by adjusting the SVM parameters, observe the resulting solution.
+
7. V. D. Sanchez, Advanced support vector machines and kernel methods, Neurocomputing 55, 5-20, 2003.
 
 
If you found LIBSVM helpful and wanted to use it for your research, [http://www.csie.ntu.edu.tw/~cjlin/libsvm/faq.html#f203 please cite the toolbox].
 
 
 
A pretty long list of other SVM packages and comparison between all of them in terms of language, execution platform, multiclass and regression capabilities, is available [http://www.cs.ubc.ca/~murphyk/Software/svm.htm here].
 
 
 
===References===
 
 
 
1. V. Vapnik and A. Chervonenkis, Theory of Pattern Recognition, Nauka, Moscow, 1974.
 
 
 
2. V. Vapnik, Estimation of Dependencies Based on Empirical Data, Nauka, Moscow, 1979.
 
 
 
3. V. Vapnik, The Nature of Statistical Learning Theory, Springer, 1995.
 
 
 
4. V. Vapnik, Statistical Learning Theory, Wiley-Interscience, New York, 1998.
 
 
 
5. P. H. Chen, C. J. Lin, and B. Schölkopf, A tutorial on ν-support vector machines, Appl. Stoch. Models. Bus. Ind. 21, 111-136, 2005.
 
 
 
6. S.-W. Lee and A. Verri (Eds.),Applications of Support Vector Machines for Pattern Recognition: SVM 2002, LNCS 2388, pp. 213-236, 2002.
 
 
 
7. V. D. Sanchez, Advanced support vector machines and kernel methods, Neurocomputing 55, 5-20, 2003.
 
  
 
8. C. Campbell, Kernel methods: a survey of current techniques, Neurocomputing, 48, 63-84, 2002.
 
8. C. Campbell, Kernel methods: a survey of current techniques, Neurocomputing, 48, 63-84, 2002.
Line 3,616: Line 4,466:
 
     ineqlin: [3x1 double]
 
     ineqlin: [3x1 double]
  
 +
=====Upper bound for Hard Margin SVM in MATLAB's Quadprog=====
 +
 +
The theoretical optimization problem for Hard Margin SVM provided at the beginning of this lecture shows that <math>\, \underline{\alpha}</math> is given no upper bound.  However, when this problem is implemented using MATLAB's <code>quadprog</code>, there are some cases where the correct solution is not produced. It is the experience of students who have taken this course that, for some Hard-Margin SVM problems, <code>quadprog</code> unexpectedly outputs a solution that has all elements of <math>\, \underline{\alpha}</math> set to zero.  Of course, some elements of <math>\, \underline{\alpha}</math> need to be non-zero in order for the support vectors to be determined, so this all-zero result is obviously incorrect.  One numerical trick that seems to correct this trouble with <code>quadprog</code> is to set the upper bound on the elements of <math>\, \underline{\alpha}</math> to a large number like <math>\, 10^6</math>. This can be done by setting the eighth parameter of <code>quadprog</code> to a vector with the same dimension of <math>\, \underline{\alpha}</math> and with all elements having a value of <math>\, 10^6</math>.
  
 
===Examining K.K.T. conditions===
 
===Examining K.K.T. conditions===
Line 3,665: Line 4,518:
 
# Find <math>\,\beta_0</math> by choosing a support vector (a point with <math>\,\alpha_i > 0</math>) and solving <math>\,y_i(\beta^Tx_i+\beta_0) = 1</math>
 
# Find <math>\,\beta_0</math> by choosing a support vector (a point with <math>\,\alpha_i > 0</math>) and solving <math>\,y_i(\beta^Tx_i+\beta_0) = 1</math>
  
==== Advantages of SVM ====
+
===Advantages of Support Vector Machines===
  
The support vector machine algorithm is insensitive to outliers. If <math>\,\alpha = 0</math>, then the cost function is also 0, and won't contribute to the solution of the SVM problem; only points on the margin — support vectors — contribute. Hence the model given by SVM is entirely defined by the support vectors, which is a very small subset of the entire training set. In this case we have a data-driven or 'nonparametric' model in which is the training set and algorithm will determine the support vectors, instead of fitting a set of parameters.
+
* SVMs provide a good out-of-sample generalization. This means that, by choosing an appropriate generalization grade,
 +
SVMs can be robust, even when the training sample has some bias. This is mainly due to selection of optimal hyperplane.
 +
* SVMs deliver a unique solution, since the optimality problem is convex. This is an advantage compared
 +
to Neural Networks, which have multiple solutions associated with local minima and for this reason may
 +
not be robust over different samples.
 +
*State-of-the-art accuracy on many problems.
 +
*SVM can handle any data types by changing the kernel.
 +
*The support vector machine algorithm is insensitive to outliers. If <math>\,\alpha = 0</math>, then the cost function is also 0, and won't contribute to the solution of the SVM problem; only points on the margin — support vectors — contribute. Hence the model given by SVM is entirely defined by the support vectors, which is a very small subset of the entire training set. In this case we have a data-driven or 'nonparametric' model in which is the training set and algorithm will determine the support vectors, instead of fitting a set of parameters.
  
 
References:
 
References:
 
Wang, L, 2005. Support Vector Machines: Theory and Applications, Springer, 3
 
Wang, L, 2005. Support Vector Machines: Theory and Applications, Springer, 3
  
Note that SVMs start from the goal of separating the data with a hyperplane, and could be extended to non-linear decision boundaries using the kernel trick.
 
  
===Kernel Trick===
+
===Disadvantages of Support Vector Machines [http://www.cse.unr.edu/~bebis/MathMethods/SVM/lecture.pdf]===
{{Cleanup|date=November 2010|reason=It would be better to provide a link to exact proof of the fact that of we project data into high dimensional space then data will become linearly separable.}}
 
We talked about the [http://www.armyconference.org/ACAS00-02/ACAS02ShortCourse/ACASCourse10.pdf curse of dimensionality] at the beginning of this course. However, we now turn to the power of high dimensions in order to find a hyperplane between two classes of data points that can linearly separate the transformed (mapped) data in a space that has a higher dimension than the space in which the training data points reside. To understand this, imagine a two dimensional prison where a two dimensional person is constrained. Suppose magically we give the person a third dimension, then he can escape from the prison. In other words, the prison and the person are linearly separable now with respect to the third dimension. The intuition behind the [http://www.cs.berkeley.edu/~jordan/courses/281B-spring04/lectures/lec3.pdf kernel trick] is basically to map data to a higher dimension in which the mapped data are linearly separable by a hyperplane, even if the original data are not linearly separable.
 
  
[[File:Point_2d.png|200px|thumb|right|Imagine the point is a person.  They're stuck.]]
+
*Perhaps the biggest limitation of the support vector approach lies in choice of the kernel (Which we will study about in future).
[[File:Point_3d.png|200px|thumb|right|Escape through the third dimension!]]
 
[[File:Unsep.png|200px|thumb|right|It's not possible to put a hyperplane through these points.]]
 
[[File:Sep2.png|200px|thumb|right|After a simple transformation, a perfect classification plane can be found.]]
 
  
The original optimal hyperplane algorithm proposed by [http://en.wikipedia.org/wiki/Vladimir_Vapnik Vladimir Vapnik] in 1963 was a linear classifier. However, in 1992, Bernhard Boser, Isabelle Guyon and Vapnik suggested a way to create non-linear classifiers by applying the kernel trick to maximum-margin hyperplanes. The algorithm is very similar, except that every dot product is replaced by a non-linear kernel function as below. This allows the algorithm to fit the maximum-margin hyperplane in a transformed feature space. We have seen SVM as a linear classification problem that finds the maximum margin hyperplane in the given input space. However, for many real world problems a more complex decision boundary is required. The following simple method was devised in order to solve the same linear classification problem but in a higher dimensional space, a [http://en.wikipedia.org/wiki/Feature_space feature space], under which the maximum margin hyperplane is better suited.
+
*A second limitation is speed and size, both in training and testing (mostly in training - for large training sets, it typically selects a small number of support vectors, thereby minimizing the computational requirements during testing).
  
Let <math>\,\phi</math> be a mapping,
+
*Discrete data presents another problem, although with suitable rescaling excellent results have nevertheless been obtained.
  
<math>\phi:\mathbb{R}^d \rightarrow \mathbb{R}^D </math>, where <math>\,D > d</math>.<br /><br />
+
*The optimal design for multiclass SVM classifiers is a further area for research.
 +
 +
*Although SVMs have good generalization performance, they can be abysmally slow in test phase.
 +
 +
*Besides the advantages of SVMs - from a practical point of view - they have some drawbacks. An important practical question that is not entirely solved, is the selection of the kernel function parameters - for Gaussian kernels the width parameter [sigma] - and the value of [epsilon] in the [epsilon]-insensitive loss function.
 +
 
 +
*However, from a practical point of view perhaps the most serious problem with SVMs is the high algorithmic complexity and extensive memory requirements of the required quadratic programming in large-scale tasks.
 +
 
 +
===Applications of Support Vector Machines===
 +
 
 +
The following papers describe some of the possible applications of support vector machines:
 +
 
 +
1- Training support vector machines: an application to face detection [http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=609310 here]
 +
 
 +
2- Application of support vector machines in financial time series forecasting [http://svms.org/regression/TaCa01.pdf here]
 +
 
 +
3- Support vector machine active learning with applications to text classification [http://portal.acm.org/citation.cfm?id=944793&dl=GUIDE, here]
 +
 
 +
 
 +
 
 +
Note that SVMs start from the goal of separating the data with a hyperplane, and could be extended to non-linear decision boundaries using the kernel trick.
 +
 
 +
===Kernel Trick===
 +
{{Cleanup|date=November 2010|reason=It would be better to provide a link to exact proof of the fact that of we project data into high dimensional space then data will become linearly separable.}}
 +
 
 +
{{Cleanup|date=November 2010|reason=I dont know if such a proof exists, would this not depend on the data whether or not a high dimensional projection would make the data linearly seperable}}
 +
 
 +
Fig.1 shows how transforming the data into the higher dimension makes it linearly separable.
 +
 
 +
[[File:dimensoin.png|350px|thumb|right|Fig1.Transforming the data can make it linearly separable.]]
 +
 
 +
We talked about the [http://www.armyconference.org/ACAS00-02/ACAS02ShortCourse/ACASCourse10.pdf curse of dimensionality] at the beginning of this course. However, we now turn to the power of high dimensions in order to find a hyperplane between two classes of data points that can linearly separate the transformed (mapped) data in a space that has a higher dimension than the space in which the training data points reside. To understand this, imagine a two dimensional prison where a two dimensional person is constrained. Suppose magically we give the person a third dimension, then he can escape from the prison. In other words, the prison and the person are linearly separable now with respect to the third dimension. The intuition behind the [http://www.cs.berkeley.edu/~jordan/courses/281B-spring04/lectures/lec3.pdf kernel trick] is basically to map data to a higher dimension in which the mapped data are linearly separable by a hyperplane, even if the original data are not linearly separable.
 +
 
 +
[[File:Point_2d.png|200px|thumb|right|Imagine the point is a person.  They're stuck.]]
 +
[[File:Point_3d.png|200px|thumb|right|Escape through the third dimension!]]
 +
[[File:Unsep.png|200px|thumb|right|It's not possible to put a hyperplane through these points.]]
 +
[[File:Sep2.png|200px|thumb|right|After a simple transformation, a perfect classification plane can be found.]]
 +
 
 +
The original optimal hyperplane algorithm proposed by [http://en.wikipedia.org/wiki/Vladimir_Vapnik Vladimir Vapnik] in 1963 was a linear classifier. However, in 1992, Bernhard Boser, Isabelle Guyon and Vapnik suggested a way to create non-linear classifiers by applying the kernel trick to maximum-margin hyperplanes. The algorithm is very similar, except that every dot product is replaced by a non-linear kernel function as below. This allows the algorithm to fit the maximum-margin hyperplane in a transformed feature space. We have seen SVM as a linear classification problem that finds the maximum margin hyperplane in the given input space. However, for many real world problems a more complex decision boundary is required. The following simple method was devised in order to solve the same linear classification problem but in a higher dimensional space, a [http://en.wikipedia.org/wiki/Feature_space feature space], under which the maximum margin hyperplane is better suited.
 +
 
 +
Let <math>\,\phi</math> be a mapping,
 +
 
 +
<math>\phi:\mathbb{R}^d \rightarrow \mathbb{R}^D </math>, where <math>\,D > d</math>.<br /><br />
  
 
We wish to find a <math>\,\phi</math> such that our data will be suited for separation by a hyperplane. Given this function, we are led to solve the previous constrained quadratic optimization on the transformed dataset,<br /><br />
 
We wish to find a <math>\,\phi</math> such that our data will be suited for separation by a hyperplane. Given this function, we are led to solve the previous constrained quadratic optimization on the transformed dataset,<br /><br />
Line 3,723: Line 4,619:
  
  
{{Cleanup|date=November 2010|reason= Could someobody please format this properly. I could not figure out how to do so.I believe that there are two other rules for constructing kernels from other kernels.}}
+
Kernels can also be constructed from other kernels using the following rules:
  
Kernels can also be constructed from other kernels using the following rules
 
 
<br>
 
<br>
 
Let a(x,x') , b(x,x') both be kernel functions<br>
 
Let a(x,x') , b(x,x') both be kernel functions<br>
Line 3,734: Line 4,629:
 
<math> k(x,x') =  p(a(x,x')) \forall </math> polynomial functions p with non negative coefficients  
 
<math> k(x,x') =  p(a(x,x')) \forall </math> polynomial functions p with non negative coefficients  
 
<br>
 
<br>
<math> k(x,x') = e^{a(x,x')} </math>
+
<math>\, k(x,x') = e^{a(x,x')} </math>
 
<br>  
 
<br>  
<math> k(x,x') = a(x,x') + b(x,x') </math>
+
<math>\, k(x,x') = a(x,x') + b(x,x') </math>
 
<br>
 
<br>
<math> k(x,x') = a(x,x')b(x,x') </math>
+
<math>\, k(x,x') = a(x,x')b(x,x') </math>
 
<br>
 
<br>
 
<math> k(x,x') = k3(\phi(x),\phi(x')) \forall </math> valid kernels k3 over the dimension of <math>\phi(x)</math>
 
<math> k(x,x') = k3(\phi(x),\phi(x')) \forall </math> valid kernels k3 over the dimension of <math>\phi(x)</math>
  
 
<math> k(x,x') = x^{T}Ax' \forall A \succeq 0 </math>
 
<math> k(x,x') = x^{T}Ax' \forall A \succeq 0 </math>
 +
<br>
 +
<math>\, k(x,x') = k_c(x_c,x_d') + k_d(x_d,x_d') </math> where <math>\, x_c, x_d </math> are variables with <math>\, x = (x_c,x_d) </math> and where <math>\, k_c, k_d </math> are valid kernel functions
 +
<br>
 +
<math>\, k(x,x') = k_c(x_c,x_a')k_d(x_d,x_d') </math> where <math>\, x_c, x_d </math> are variables with <math>\, x = (x_c,x_d) </math> and where <math>\, k_c, k_d </math> are valid kernel functions
 
<br>
 
<br>
  
Once we have chosen the Kernel function, we don't need to figure out what <math>\,\phi</math> is, just use <math>\,\phi(\underline{x}_i)^T\phi(\underline{x}_j) = K(\underline{x}_i,\underline{x}_j) </math> to replace <math>\,\underline{x}_i^T\underline{x}_j</math>
+
Once we have chosen the Kernel function, we don't need to figure out what <math>\,\phi</math> is, just use <math>\,\phi(\underline{x}_i)^T\phi(\underline{x}_j) = K(\underline{x}_i,\underline{x}_j) </math> to replace <math>\,\underline{x}_i^T\underline{x}_j</math>.
 +
 
 +
Reference for rules: [[http://www.cedar.buffalo.edu/~srihari/CSE574/Chap6/Chap6.1-KernelMethods.pdf Rules]]
  
 
Since the transformation chosen is dependent on the shape of the data, the only automated way to choose an appropriate kernel is by trial and error. Otherwise it is chosen manually.
 
Since the transformation chosen is dependent on the shape of the data, the only automated way to choose an appropriate kernel is by trial and error. Otherwise it is chosen manually.
  
 
====Kernel Functions for Machine Learning Applications====
 
====Kernel Functions for Machine Learning Applications====
Except the kernel functions, Linear Kernel , Polynomial Kernel and Gaussian Kernel, we discussed in class, more kernel functions can be used in the application of kernel methods for machine learning, such that Exponential Kernel , Laplacian Kernel , ANOVA Kernel , Hyperbolic Tangent (Sigmoid) Kernel , Rational Quadratic Kernel, Multiquadric Kernel , Inverse Multiquadric Kernel, Circular Kernel , Spherical Kernel , Wave Kernel , Power Kernel , Log Kernel , Spline Kernel , B-Spline Kernel , Bessel Kernel , Cauchy Kernel , Chi-Square Kernel , Histogram Intersection Kernel , Generalized Histogram Intersection Kernel , Generalized T-Student Kernel , Bayesian Kernel , Wavelet Kernel , etc. For more details, see http://crsouza.blogspot.com/2010/03/kernel-functions-for-machine-learning.html#kernel_functions.  
+
Beyond the kernel functions we discussed in class (Linear Kernel, Polynomial Kernel and Gaussian Kernel), many more kernel functions can be used in the application of kernel methods for machine learning. Some examples of other kernels are: Exponential Kernel, Laplacian Kernel, ANOVA Kernel, Hyperbolic Tangent (Sigmoid) Kernel, Rational Quadratic Kernel, Multiquadric Kernel, Inverse Multiquadric Kernel, Circular Kernel, Spherical Kernel, Wave Kernel, Power Kernel, Log Kernel, Spline Kernel, B-Spline Kernel, Bessel Kernel, Cauchy Kernel, Chi-Square Kernel, Histogram Intersection Kernel, Generalized Histogram Intersection Kernel, Generalized T-Student Kernel, Bayesian Kernel, Wavelet Kernel, etc. For more details, see http://crsouza.blogspot.com/2010/03/kernel-functions-for-machine-learning.html#kernel_functions.
  
 
===Example in Matlab===
 
===Example in Matlab===
  
 
The following code, taken verbatim from the lecture, shows how to use Matlab built-in SVM routines (found in the Bioinformatics toolkit) to do classification through support vector machines.
 
The following code, taken verbatim from the lecture, shows how to use Matlab built-in SVM routines (found in the Bioinformatics toolkit) to do classification through support vector machines.
{{Cleanup|date=November 2010|reason= It is good to add that old matlab built in SVM function has some bugs. for example it marks some points as support vectors while they are not}}
+
 
 +
Note: in Matlab R2009, the svmtrain function returns support vectors which should not be marked as such. This could be due to svmtrain centering and rescaling the data automatically. The issue has been fixed in later versions.
 +
 
 
  load 2_3;
 
  load 2_3;
 
  [U,Y] = princomp(X');
 
  [U,Y] = princomp(X');
Line 3,794: Line 4,697:
 
Source: John Shawe-Taylor and Nello Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, illustrated edition edition, June 2004.
 
Source: John Shawe-Taylor and Nello Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, illustrated edition edition, June 2004.
  
===Extention:Support Vector Machines===
+
===Extension:Support Vector Machines===
  
 
==== Pattern Recognition ====
 
==== Pattern Recognition ====
Line 3,833: Line 4,736:
  
 
==='''Case II: Non-separable data (Soft Margin)'''===
 
==='''Case II: Non-separable data (Soft Margin)'''===
{{Cleanup|date=November 2010|reason=It is good to answer this question: when must we use soft margin? since when we want to start the problem we don't know if the data is separable. One answer is that we can always start with soft margin and after we see the boundary we may be able to guess if the classes are separable.}}
 
  
  
{{Cleanup|date=November 2010|reason= What is the benefit of using hard margin. If the classes are truly linearly separable would not the softmargin classifier find this boundary as well. Also if we allow points to be misclassified would not this be better in terms of over fitting?}}
+
We have seen how SVMs are able to find an optimally separating hyperplane of two separable classes of data, in which case the margin contains no data points. However, in the real world, data of different classes are usually mixed together at the boundary and it's hard to find a perfect boundary to totally separate them. In this, one may want to separate the training data set with the minimal number of errors . To address this problem, we slacken the classification rule to allow data cross the margin. Now each data point can have some error <math>\,\xi_i</math>. However, we only want data to cross the boundary when they have to and make the minimum sacrifice; thus, a penalty term is added correspondingly in the objective function to constrain the number of points that cross the margin. The optimization problem now becomes:
 
 
 
 
We have seen how SVMs are able to find an optimally separating hyperplane of two separable classes of data, in which case the margin contains no data points. However, in the real world, data of different classes are usually mixed together at the boundary and it's hard to find a perfect boundary to totally separate them.  
 
To address this problem, we slacken the classification rule to allow data cross the margin. Now each data point can have some error <math>\,\xi_i</math>. However, we only want data to cross the boundary when they have to and make the minimum sacrifice; thus, a penalty term is added correspondingly in the objective function to constrain the number of points that cross the margin. The optimization problem now becomes:
 
 
[[File:non-separable.JPG|350px|thumb|right|Figure non-separable case]]
 
[[File:non-separable.JPG|350px|thumb|right|Figure non-separable case]]
  
Line 3,849: Line 4,747:
 
<br\>Note that <math>\,\xi_i</math> is not necessarily smaller than one, which means data can not only enter the margin but can also cross the separating hyperplane.
 
<br\>Note that <math>\,\xi_i</math> is not necessarily smaller than one, which means data can not only enter the margin but can also cross the separating hyperplane.
  
<br\>Note that <math>\,\gamma \Rightarrow \infty </math> is feasible in the separable case, as all <math>\,\xi_i = 0</math>. In general, for higher <math>\,\gamma</math>, the sets are more separable.
+
<br\>Minimizing the objective, one finds some minimal subset of errors. If these error data are excluded from the training data set, then one can separate the remaining part of training data without errors.
 +
 
 +
<br\>Soft-margin SVM is a generalization of hard-margin SVM. The hard-margin SVM was presented here before the soft-margin SVM because hard-margin SVM is more intuitive and it was historically conceived first. Note that as <math>\,\gamma \Rightarrow \infty </math>, all <math>\,\xi_i \Rightarrow 0</math>. Intuitively, we would set <math>\,\xi_i = 0</math> if it was known that the classes were separable, that is, if we knew a hard-margin SVM boundary could be used to separate the classes. As shown later in this lecture, the softmargin classifier with <math>\,\gamma = \infty </math> is, in fact, the same optimization problem as the hardmargin classifier mentioned in the previous lecture. By similar logic, it is wise to set higher <math>\,\gamma</math>, for sets that are more separable.
  
  
With the formulation of Primal form for non-seperable case above, we can form the lagrangian.
+
With the formulation of the Primal form for non-separable case above, we can form the Lagrangian.
  
 
===Forming the Lagrangian===
 
===Forming the Lagrangian===
In this case we have have two constraints in the [http://en.wikipedia.org/wiki/Lagrangian Lagrangian] and therefore we optimize with respect to two dual variables <math>\,\alpha</math> and <math>\,\lambda</math>,<br>
+
In this case we have have two constraints in the [http://en.wikipedia.org/wiki/Lagrangian Lagrangian] primal form and therefore we optimize with respect to two dual variables <math>\,\alpha</math> and <math>\,\lambda</math>,<br>
 
:<math>L: \frac{1}{2} |\beta|^2 + \gamma \sum_{i} \xi_i - \sum_{i} \alpha_i[y_i(\beta^T x_i+\beta_0)-1+\xi_i]-\sum_{i} \lambda_i \xi_i</math>
 
:<math>L: \frac{1}{2} |\beta|^2 + \gamma \sum_{i} \xi_i - \sum_{i} \alpha_i[y_i(\beta^T x_i+\beta_0)-1+\xi_i]-\sum_{i} \lambda_i \xi_i</math>
 
:<math>\alpha_i \geq 0, \lambda_i \geq 0</math>
 
:<math>\alpha_i \geq 0, \lambda_i \geq 0</math>
  
Now apply KKT conditions, and come up with a new function to optimize. As we will see, the equation that we will attempt to optimize in the SVM algorithm for non-separable data sets is the same as the optimization for the separable case, with slightly different conditions.
+
Now we apply KKT conditions, and come up with a new function to optimize. As we will see, the equation that we will attempt to optimize in the SVM algorithm for non-separable data sets is the same as the optimization for the separable case, with slightly different conditions.
 
 
  
 
===Applying KKT conditions[http://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions]===
 
===Applying KKT conditions[http://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions]===
 
# <math>\frac{\partial L}{\partial p} = 0</math> at an optimal solution <math>\, \hat p</math>, for each primal variable <math>\,p = \{\beta, \beta_0, \xi\}</math><br><math>\frac{\partial L}{\partial \beta}=\beta - \sum_{i} \alpha_i y_i x_i = 0 \Rightarrow \beta=\sum_{i}\alpha_i y_i x_i</math> <br\><math>\frac{\partial L}{\partial \beta_0}=-\sum_{i} \alpha_i y_i =0 \Rightarrow \sum_{i} \alpha_i y_i =0</math> since the sign does not make a difference<br><math>\frac{\partial L}{\partial \xi_i}=\gamma - \alpha_i - \lambda_i \Rightarrow \gamma = \alpha_i+\lambda_i</math>.  This is the only new condition added here
 
# <math>\frac{\partial L}{\partial p} = 0</math> at an optimal solution <math>\, \hat p</math>, for each primal variable <math>\,p = \{\beta, \beta_0, \xi\}</math><br><math>\frac{\partial L}{\partial \beta}=\beta - \sum_{i} \alpha_i y_i x_i = 0 \Rightarrow \beta=\sum_{i}\alpha_i y_i x_i</math> <br\><math>\frac{\partial L}{\partial \beta_0}=-\sum_{i} \alpha_i y_i =0 \Rightarrow \sum_{i} \alpha_i y_i =0</math> since the sign does not make a difference<br><math>\frac{\partial L}{\partial \xi_i}=\gamma - \alpha_i - \lambda_i \Rightarrow \gamma = \alpha_i+\lambda_i</math>.  This is the only new condition added here
 
#<math>\,\alpha_i \geq 0, \lambda_i \geq 0</math>, dual feasibility
 
#<math>\,\alpha_i \geq 0, \lambda_i \geq 0</math>, dual feasibility
#<math>\,\alpha_i[y_i(\beta^T x_i+\beta_0)-1+\xi_i]=0</math> and <math>\,\alpha_i \xi_i=0</math>
+
#<math>\,\alpha_i[y_i(\beta^T x_i+\beta_0)-1+\xi_i]=0</math> and <math>\,\lambda_i\xi_i=0</math>
 
#<math>\,y_i( \beta^T x_i+ \beta_0)-1+ \xi_i \geq 0</math>
 
#<math>\,y_i( \beta^T x_i+ \beta_0)-1+ \xi_i \geq 0</math>
 
  
 
=== Objective Function ===
 
=== Objective Function ===
 
With our KKT conditions and the Lagrangian equation, <math>\,\alpha</math> could be estimated by Quadratic programming.  
 
With our KKT conditions and the Lagrangian equation, <math>\,\alpha</math> could be estimated by Quadratic programming.  
  
<br\> Similar to what we did for the separable case after apply KKT conditions, replace the primal variables in terms of dual variables into the Lagrangian equations and simplify as follows:
+
<br\> Similar to what we did for the separable case after applying KKT conditions, replace the primal variables in terms of dual variables into the Lagrangian equations and simplify as follows:
  
 
:<math>L =  \frac{1}{2} |\beta|^2 + \gamma \sum_{i} \xi_i - \beta^T \sum_{i} \alpha_i y_i x_i - \beta_0 \sum_{i} \alpha_i  
 
:<math>L =  \frac{1}{2} |\beta|^2 + \gamma \sum_{i} \xi_i - \beta^T \sum_{i} \alpha_i y_i x_i - \beta_0 \sum_{i} \alpha_i  
Line 3,884: Line 4,782:
 
:<math> \displaystyle \sum_{i} \alpha_i y_i = 0</math>
 
:<math> \displaystyle \sum_{i} \alpha_i y_i = 0</math>
  
Rewritting the above equation we have:
+
Rewriting the above equation we have:
  
 
:<math>L =  \frac{1}{2} |\beta|^2 - |\beta|^2 + \gamma \sum_{i} \xi_i + \sum_{i} \alpha_i - \sum_{i} \alpha_i \xi_i -  
 
:<math>L =  \frac{1}{2} |\beta|^2 - |\beta|^2 + \gamma \sum_{i} \xi_i + \sum_{i} \alpha_i - \sum_{i} \alpha_i \xi_i -  
Line 3,895: Line 4,793:
  
  
:<math>\Rightarrow L = - \frac{1}{2} \sum_{i} \sum_{j} \alpha_i\alpha_j  y_i y_j x_i^T x_i + \sum_{i} \alpha_i + \sum_{i}  
+
:<math>\Rightarrow L = - \frac{1}{2} \sum_{i} \sum_{j} \alpha_i\alpha_j  y_i y_j x_i^T x_j + \sum_{i} \alpha_i + \sum_{i}  
  
 
\gamma \xi_i - \sum_{i} \alpha_i \xi_i - \sum_{i} \lambda_i \xi_i</math>
 
\gamma \xi_i - \sum_{i} \alpha_i \xi_i - \sum_{i} \lambda_i \xi_i</math>
  
:<math>\Rightarrow L = - \frac{1}{2} \sum_{i} \sum_{j} \alpha_i\alpha_j  y_i y_j x_i^T x_i + \sum_{i} \alpha_i + \sum_{i}  
+
:<math>\Rightarrow L = - \frac{1}{2} \sum_{i} \sum_{j} \alpha_i\alpha_j  y_i y_j x_i^T x_j + \sum_{i} \alpha_i + \sum_{i}  
  
 
(\gamma - \alpha_i - \lambda_i) \xi_i</math>
 
(\gamma - \alpha_i - \lambda_i) \xi_i</math>
Line 3,905: Line 4,803:
 
We know that by KKT condition <math>\displaystyle \gamma - \alpha_i - \lambda_i = 0 </math>
 
We know that by KKT condition <math>\displaystyle \gamma - \alpha_i - \lambda_i = 0 </math>
  
Finally we have the simplest form of lagrangian for non-seperable case:
+
Finally we have the simplest form of Lagrangian for non-separable case:
  
:<math>L = \sum_{i} \alpha_i - \frac{1}{2} \sum_{i} \sum_{j} \alpha_i\alpha_j  y_i y_j x_i^T x_i </math>
+
:<math>L = \sum_{i} \alpha_i - \frac{1}{2} \sum_{i} \sum_{j} \alpha_i\alpha_j  y_i y_j x_i^T x_j </math>
  
  
You can see that there is no difference in objective funtion of Hard & Soft Margin Objective function. Now let's see the constraints for above objective funtion.
+
You can see that there is no difference in objective function of Hard & Soft Margin. Now let's see the constraints for above objective function.
  
 
=== Constraints ===
 
=== Constraints ===
Following will be the constraints of above objective funtion:
+
Following will be the constraints of above objective function:
  
:<math>\displaystyle \alpha_i > 0 \forall i</math>
+
:<math>\,\alpha_i \geq 0 \forall i</math>
:<math>\displaystyle \lambda_i > 0 \forall i</math>
+
:<math>\lambda_i \geq 0 \forall i</math>
:<math>\displaystyle \sum_{i} \alpha_i y_i = 0</math>
+
:<math>\displaystyle \sum_{i} \alpha_i y_i = 0</math><br />
  
If <math>\displaystyle \lambda_i \ge 0 \,\Rightarrow \, \alpha_i \le \gamma</math>
+
From the KKT conditions above, we have:<br />
 +
<math>\frac{\partial L}{\partial \xi_i}=\gamma - \alpha_i - \lambda_i \Rightarrow \gamma = \alpha_i+\lambda_i</math><br />
 +
 
 +
Therefore, If <math>\displaystyle \lambda_i \ge 0 \,\Rightarrow \, \alpha_i \le \gamma</math>, hence, <math>\,\lambda_i \geq 0 </math> constraint can be replaced by <math>\displaystyle \alpha_i \le \gamma</math>.
  
 
===Dual Problem or Quadratic Programming Problem===
 
===Dual Problem or Quadratic Programming Problem===
  
We have formallized the Dual Problem which is as follows:
+
We have formalized the Dual Problem which is as follows:
  
 
:<math>\displaystyle \max_{\alpha_i} \sum_{i}{\alpha_i} - \frac{1}{2}\sum_{i}{\sum_{j}{\alpha_i \alpha_j y_i y_j x_i^T x_j}}</math>  
 
:<math>\displaystyle \max_{\alpha_i} \sum_{i}{\alpha_i} - \frac{1}{2}\sum_{i}{\sum_{j}{\alpha_i \alpha_j y_i y_j x_i^T x_j}}</math>  
Line 3,933: Line 4,834:
 
You can see that the only difference in the Hard and Soft Margin is the upper bound of <math>\displaystyle \alpha</math> i.e. <math>\displaystyle \alpha \le \gamma</math>.
 
You can see that the only difference in the Hard and Soft Margin is the upper bound of <math>\displaystyle \alpha</math> i.e. <math>\displaystyle \alpha \le \gamma</math>.
  
As <math>\displaystyle \gamma \rightarrow \propto</math>  soft margin  <math>\displaystyle \rightarrow</math> Hard margin.
+
As <math>\displaystyle \gamma \rightarrow \infty </math>  soft margin  <math>\displaystyle \rightarrow</math> Hard margin.
  
 
=== Recovery of Hyperplane ===
 
=== Recovery of Hyperplane ===
Line 3,960: Line 4,861:
 
applied to classification or regression.<ref name="cccc"> Ovidiu Ivanciuc, Review: Applications of Support Vector Machines in Chemistry, Rev. Comput. Chem. 2007, 23, 291-400</ref>Support vectors are the training points that determine the optimal separating hyperplane that we seek. Also, they are the most difficult points to classify and at the same time the most informative for classification.
 
applied to classification or regression.<ref name="cccc"> Ovidiu Ivanciuc, Review: Applications of Support Vector Machines in Chemistry, Rev. Comput. Chem. 2007, 23, 291-400</ref>Support vectors are the training points that determine the optimal separating hyperplane that we seek. Also, they are the most difficult points to classify and at the same time the most informative for classification.
  
For non-seperable case from third KKT condition, if <math>\displaystyle \alpha_i > 0 \Rightarrow y_i(\underline \beta^T \underline x_i+\beta_0)-1+\xi_i=0</math>, these points are called support vectors.
+
For the non-separable case, the third KKT condition yields: if <math>\displaystyle \alpha_i > 0 \Rightarrow y_i(\underline \beta^T \underline x_i+\beta_0)-1+\xi_i=0</math>. These above points are called support vectors.
  
 
* Case 1: Support Vectors are on the Margin
 
* Case 1: Support Vectors are on the Margin
::If <math>\displaystyle \lambda_i > 0 \Rightarrow \xi_i = 0 </math>, then support vectors are on the margin.
+
::If <math>\displaystyle \lambda_i > 0 \Rightarrow \xi_i = 0 </math>, then this support vector is on the margin.
  
 
* Case 2: Support Vectors are inside the Margin  
 
* Case 2: Support Vectors are inside the Margin  
::If <math>\displaystyle \alpha = \gamma</math>, then support vectors are inside the margin.
+
::If <math>\displaystyle \alpha = \gamma</math>, then this support vectors is inside the margin.
  
 
===  Support Vectors Machine Demo Tool ===
 
===  Support Vectors Machine Demo Tool ===
Line 3,972: Line 4,873:
 
[[image:SVM_Demo.png]]
 
[[image:SVM_Demo.png]]
  
This demo tool shows the linear SVM and illustrates its behaviour on some 2D data. This should be great for getting to grips with maximizing geometric margins, support vectors, and the optimisation involved in computing an optimal separating hyperplane.[http://www.mathworks.com/matlabcentral/fileexchange/28302-svm-demo File]
+
This demo tool shows the linear boundary found by SVM and illustrates its behaviour on some 2D data. This is an interactive demonstration to provide insight into how SVM finds the classification boundary.[http://www.mathworks.com/matlabcentral/fileexchange/28302-svm-demo File]
  
 +
 +
=== Relevance Vector Machines ===
 +
Support vector machines have been used in a variety of classification and regression applications. Nevertheless, they suffer from a number of limitations, several of which have been highlighted already in earlier sessions. In particular, the outputs of an SVM represent decisions rather than posterior probabilities. Also, the SVM was originally formulated for two classes, and the extension toK > 2 classes is problematic.
 +
There is a complexity parameter C, or ν (as well as a parameter epsilon in the case of regression), that must be found using a hold-out method such as cross-validation. Finally, predictions are expressed as linear combinations of kernel functions that are centred on training data points and that are required to be positive definite.
 +
 +
The relevance vector machine or RVM (Tipping, 2001) is a Bayesian sparse kernel technique for regression and classification that shares many of the characteristics of the SVM whilst avoiding its principal limitations. Additionally, it typically leads to much sparser models resulting in correspondingly faster performance on test data whilst maintaining comparable generalization error.
 +
 +
Borrowed from Bishop's great book, Pattern Recognition and Machine Learning [http://research.microsoft.com/en-us/um/people/cmbishop/prml/]. For more details on RVM, you may refer to this book.
  
 
===  Further reading on the Kernel Trick ===
 
===  Further reading on the Kernel Trick ===
Line 3,985: Line 4,894:
  
 
5- SVM application list[http://www.clopinet.com/isabelle/Projects/SVM/applist.html]
 
5- SVM application list[http://www.clopinet.com/isabelle/Projects/SVM/applist.html]
 +
 +
6- Some readings about SVM  and the kernel trick [http://www.cs.cmu.edu/~guestrin/Class/10701-S07/Slides/kernels.pdf] and [http://www.cs.cmu.edu/~tom/10601_sp08/slides/svm3-26.ppt]
 +
 +
7- General overview of SVM and Kernel Methods. Easy to understand presentation. [http://www.support-vector.net/icml-tutorial.pdf]
  
 
== ''' Naive Bayes,  K Nearest Neighbours, Boosting, Bagging and Decision Trees,  - November 18, 2010 ''' ==
 
== ''' Naive Bayes,  K Nearest Neighbours, Boosting, Bagging and Decision Trees,  - November 18, 2010 ''' ==
Line 3,993: Line 4,906:
 
=== [http://en.wikipedia.org/wiki/Naive_Bayes_classifier Naive Bayes Classifiers] ===
 
=== [http://en.wikipedia.org/wiki/Naive_Bayes_classifier Naive Bayes Classifiers] ===
  
Recall that one of the major drawbacks of the Bayes classifier was the difficulty in estimating a joint density in a multidimensional space.  Naive Bayes classifiers are one possible solution to the problem.  They are especially popular for problems with high-dimension feature problems.  
+
Recall that one of the major drawbacks of the Bayes classifier was the difficulty in estimating a joint density in a multidimensional space.  Naive Bayes classifiers are one possible solution to the problem.  They are especially popular for problems with high-dimensional features.  
 +
 
 +
 
 +
A naive Bayes classifier applies a strong independence assumption to the conditional probability <math>\ P(X|Y) = P(x_1,x_2,...,x_d |Y)</math>. It assumes that inputs within each class are conditionally independent.  In other words, it assumes the dimensions of the inputs in each class are independent. The Naive Bayes classifier does this by reducing the number of parameters to be estimated dramatically when modeling <math>\ P(X|Y)</math>.
 +
 
 +
Under the conditional independence assumption:
  
 +
<math>\ P(X|Y) = P(x_1,x_2,...,x_d |Y) =\prod_{i=1}^{d}P(X = x_i | Y)</math>.
  
A naive Bayes classifier applies a strong independence assumption to the conditional probability <math>\ P(X|Y) = P(X_1,X_2,...,X_d |Y)</math>. It assumes that inputs within each class are conditionally independent.  In other words, it assumes the dimensions of the inputs in each class are independent. The Naive Bayes classifier does this by reducing the number of parameters to be estimated dramatically when modeling <math>\ P(X|Y)</math>.
+
====Naive Bayes is Equivalent to a Generalized Additive Model====
  
Under the conditionally independence assumption:
+
Continuing with the notation used above, consider a binary classification problem <math>\, y \in \{0,1\}</math>. Beginning with the corresponding Bayes classifier, we have
  
<math>\ P(X|Y) = P(X_1,X_2,...,X_d |Y) =\prod_{i=1}^{d}P(X_i | Y)</math>.  
+
:<math>\, h^*(x)= \left\{\begin{matrix}
 +
1 &\text{if } P(Y=1|X=x)>P(Y=0|X=x)  \\
 +
0 &\mathrm{otherwise}  \end{matrix}\right.</math>
 +
 
 +
:<math>\, = \left\{\begin{matrix}
 +
1 &\text{if } \log\left(\frac{P(Y=1|X=x)}{P(Y=0|X=x)}\right)>0  \\
 +
0 &\mathrm{otherwise}  \end{matrix}\right.</math>
 +
 
 +
:<math>\, = \frac{1}{2}\text{sign}\left[\log\left(\frac{P(Y=1|X=x)}{P(Y=0|X=x)}\right)\right] + \frac{1}{2}</math>
 +
 
 +
:<math>\, = \frac{1}{2}\text{sign}\left[\log\left(\frac{P(Y=1)P(X=x|Y=1)}{P(Y=0)P(X=x|Y=0)}\right)\right] + \frac{1}{2}</math>
 +
 
 +
then, by the conditional independence assumption provided by Naive Bayes,
 +
 
 +
:<math>\, = \frac{1}{2}\text{sign}\left[\log\left(\frac{P(Y=1)}{P(Y=0)} \prod_{i=1}^d \frac{P(X_i=x_i|Y=1)}{P(X_i=x_i|Y=0)}\right)\right] + \frac{1}{2}</math>
 +
 +
:<math>\, = \frac{1}{2}\text{sign}\left[\log\left(\frac{P(Y=1)}{P(Y=0)}\right) + \sum_{i=1}^d \log\left(\frac{P(X_i=x_i|Y=1)}{P(X_i=x_i|Y=0)}\right)\right] + \frac{1}{2}</math>
 +
 
 +
which is a generalized additive model.
  
 
==== Naive Bayes for Continuous Input ====
 
==== Naive Bayes for Continuous Input ====
Line 4,008: Line 4,945:
 
Recall that the Bayes rule is :
 
Recall that the Bayes rule is :
  
<math>\ h(x) = argmax_{ \{k\}}  \pi_{k}f_{k}(x). </math>
+
<math>\ h(x) = \arg\max_{ k}  \pi_{k}f_{k}(x). </math>
  
 +
Although the Bayes classifier is the best classifier, in practice, it is difficult to give an estimate for the multi-variable prior probabilities which are required to determine the classification. Therefore, by assuming independence between the features, we can transform an n-variable distribution into n independent one-variable distributions which are easier to handle, and then apply the Bayes classification.
  
 
The density function of inputs can be written as below under the independence assumption :
 
The density function of inputs can be written as below under the independence assumption :
Line 4,023: Line 4,961:
 
An interesting example by Jose M. Vidal that shows how the naive Bayes classifier can be used to solve a real-world classification task is available [http://jmvidal.cse.sc.edu/talks/bayesianlearning/nbex.xml here].
 
An interesting example by Jose M. Vidal that shows how the naive Bayes classifier can be used to solve a real-world classification task is available [http://jmvidal.cse.sc.edu/talks/bayesianlearning/nbex.xml here].
  
{{Cleanup|date=November 19 2010|reason=If someone can add some cotents aboutthe Naive bayes for discrete input, it will be very nice.}}
+
==== Naive Bayes for Discrete Inputs ====
  
==== Further reading Naive Bayes ====
+
Naive Bayes with discrete inputs is very similar to that of continuous inputs. From examples researched, the major difference is that instead of using a probability distribution to characterize the likelihood, we use feature frequencies, or (in English) the proportion of time cases in which variables X fall under class C vs. total number of cases that fall under class C. The following example shows how this would work:
  
{{Cleanup|date=November 21 2010|reason=I felt that Naive Bayes is not very clear to me. That's why I posted few papers speaking on this point}}
+
You are running a scientific study meant to find the optimal features under which a girl you encounter will wear her glasses. The data you collect represent the setting of your encounter (library, park, bar), whether she is a student or not (yes, no), and what her eye colour is (blue, green, brown).
  
The following are some papers to show how Naive Bayes is used in different aspects of classifications.
+
{|
 
+
|-
1- An empirical study of the naive Bayes classifier [http://www.cc.gatech.edu/home/isbell/classes/reading/papers/Rish.pdf here]
+
! scope="col" | Case
 +
! scope="col" | Setting
 +
! scope="col" | Student
 +
! scope="col" | Eye colour
 +
! scope="col" | Wears glasses?
 +
|-
 +
! scope="row" | 1
 +
| Bar || yes || Blue || no
 +
|-
 +
! scope="row" | 2
 +
| Park || yes || Brown || yes
 +
|-
 +
! scope="row" | 3
 +
| Library || no || Green || yes
 +
|-
 +
! scope="row" | 4
 +
| Library || no || Blue || no
 +
|-
 +
! scope="row" | 5
 +
| Bar || no || Brown || yes
 +
|-
 +
! scope="row" | 6
 +
| Park || yes || Green || yes
 +
|-
 +
! scope="row" | 7
 +
| Bar || no || Brown || yes
 +
|-
 +
! scope="row" | 8
 +
| Library || yes || Brown || yes
 +
|-
 +
! scope="row" | 9
 +
| Bar || yes || Green || no
 +
|-
 +
! scope="row" | 10
 +
| Park || yes || Blue || no
 +
|}
  
2- Naive (Bayes) at forty: The independence assumption in information retrieval [http://www.springerlink.com/content/wu3g458834583125/ here]
 
  
3- Emotion Recognition Using a Cauchy Naive Bayes Classifier [http://www.computer.org/portal/web/csdl/doi/10.1109/ICPR.2002.1044578 here]
+
From this, we extract the following feature frequencies:
  
===  Reference ===
+
{|
 +
|-
 +
! scope="col" | Eye Colour
 +
! scope="col" | Wearing glasses
 +
! scope="col" | Not wearing glasses
 +
|-
 +
! scope="row" | Blue
 +
| 0 || 3
 +
|-
 +
! scope="row" | Brown
 +
| 4 || 0
 +
|-
 +
! scope="row" | Green
 +
| 2 || 1
 +
|}
 +
 
 +
{|
 +
|-
 +
! scope="col" | Student?
 +
! scope="col" | Wearing glasses
 +
! scope="col" | Not wearing glasses
 +
|-
 +
! scope="row" | Not a student
 +
| 3 || 1
 +
|-
 +
! scope="row" | Student
 +
| 3 || 3
 +
|}
 +
 
 +
{|
 +
|-
 +
! scope="col" | Setting
 +
! scope="col" | Wearing glasses
 +
! scope="col" | Not wearing glasses
 +
|-
 +
! scope="row" | Bar
 +
| 2 || 2
 +
|-
 +
! scope="row" | Library
 +
| 2 || 1
 +
|-
 +
! scope="row" | Park
 +
| 2 || 1
 +
|}
 +
 
 +
You also note that of the 10 girls you saw, 6 were wearing their glasses and 4 weren't. Therefore, given the new case of a green eyed student in a bar, we calculate the probabilities of her wearing vs. not wearing her glasses as such:
 +
 
 +
P(Wearing glasses | green eyed student in a bar) = P(Wearing glasses | student)*P(Wearing glasses | green eyed)*P(Wearing glasses | in a bar) = 3/6 * 2/6 * 2/6 = 0.0556<br>
 +
P(Not wearing glasses | green eyed student in a bar) = P(Not wearing glasses | student)*P(Not wearing glasses | green eyed)*P(Not wearing glasses | in a bar) = 3/4 * 1/4 * 2/4 = 0.09375<br>
 +
 
 +
Since P(Wearing glasses | green eyed student in a bar) < P(Not wearing glasses | green eyed student in a bar), it is not likely that a green eyed student will be wearing her glasses in a bar.
 +
 
 +
==== Further reading Naive Bayes  ====
 +
 
 +
The following are some papers to show how Naive Bayes is used in different aspects of classifications.
 +
 
 +
1- An empirical study of the naive Bayes classifier [http://www.cc.gatech.edu/home/isbell/classes/reading/papers/Rish.pdf here]
 +
 
 +
2- Naive (Bayes) at forty: The independence assumption in information retrieval [http://www.springerlink.com/content/wu3g458834583125/ here]
 +
 
 +
3- Emotion Recognition Using a Cauchy Naive Bayes Classifier [http://www.computer.org/portal/web/csdl/doi/10.1109/ICPR.2002.1044578 here]
 +
 
 +
===  References ===
  
 
1. Scaling up the accuracy of naive-Bayes classifiers: A decision-tree hybrid
 
1. Scaling up the accuracy of naive-Bayes classifiers: A decision-tree hybrid
Line 4,048: Line 5,082:
 
[http://www.springerlink.com/content/m4p7863g61502515/]
 
[http://www.springerlink.com/content/m4p7863g61502515/]
  
=== [http://en.wikipedia.org/wiki/K-nearest_neighbor_algorithm K-Nearest Neighbours Classification] ===
+
=== [http://en.wikipedia.org/wiki/K-nearest_neighbor_algorithm K-Nearest Neighbors Classification] ===
  
  
<math>\,K</math>-nearest neighbours is a very simple algorithm that classifies points based on a majority vote of the <math>\ k</math> nearest points in the feature space, with the object being assigned to the class most common among its <math>\ k</math> nearest neighbors. <math>\ k</math> is a positive integer, typically small which is chosen by cross validation. If <math>\ k=1</math>, then the object is simply assigned to the class of its nearest neighbor.
+
<math>\,K</math>-Nearest Neighbors is a very simple algorithm that classifies points based on a majority vote of the <math>\ k</math> nearest points in the feature space, with the object being assigned to the class most common among its <math>\ k</math> nearest neighbors. <math>\ k</math> is a positive integer, typically small which can be chosen using cross validation. If <math>\ k=1</math>, then the object is simply assigned to the class of its nearest neighbor.
  
 
1. Ties are broken at random.
 
1. Ties are broken at random.
Line 4,062: Line 5,096:
  
 
====  Advantage:====
 
====  Advantage:====
 
 
Robust to noisy training data (especially if we use inverse square of weighted distance as the “distance”)[http://people.revoledu.com/kardi/tutorial/KNN/What-is-K-Nearest-Neighbor-Algorithm.html]
 
Robust to noisy training data (especially if we use inverse square of weighted distance as the “distance”)[http://people.revoledu.com/kardi/tutorial/KNN/What-is-K-Nearest-Neighbor-Algorithm.html]
  
Line 4,076: Line 5,109:
  
 
A major drawback is that if  the frequency of one class is greater than the other ones significantly , the samples in this class with the largest frequency tend to dominate the prediction of a new point . An approach to overcome it is  attaching weights to the samples ,for instance ,add  larger weights to the neighbors which are closer to the new points than those that are further away.
 
A major drawback is that if  the frequency of one class is greater than the other ones significantly , the samples in this class with the largest frequency tend to dominate the prediction of a new point . An approach to overcome it is  attaching weights to the samples ,for instance ,add  larger weights to the neighbors which are closer to the new points than those that are further away.
 +
 +
====Note of Interest:====
 +
In K-nearest neighbours over fitting may occur when a small k is used. In contrast to other methods, in K-nearest neighbours k=1 is the most complex case.
  
 
====Property[http://en.wikipedia.org/wiki/K-nearest_neighbor_algorithm#Properties]====
 
====Property[http://en.wikipedia.org/wiki/K-nearest_neighbor_algorithm#Properties]====
  
 
K-nearest neighbor algorithm has some good and strong results. As the number of data points goes to infinity, the algorithm is guaranteed to yield an error rate no worse than twice the Bayes error rate (the minimum achievable error rate given the distribution of the data). K-nearest neighbor is guaranteed to approach the Bayes error rate, for some value of k (where k increases as a function of the number of data points).
 
K-nearest neighbor algorithm has some good and strong results. As the number of data points goes to infinity, the algorithm is guaranteed to yield an error rate no worse than twice the Bayes error rate (the minimum achievable error rate given the distribution of the data). K-nearest neighbor is guaranteed to approach the Bayes error rate, for some value of k (where k increases as a function of the number of data points).
 +
See ''Nearest Neighbour Pattern Classification'', T.M. Cover and P.E. Hart, for interesting theoretical results about the algorithm, including proof of the above properties.
 +
 +
Effect of K on Bias Variance Trade-off for KNN
 +
 +
Large K high bias , low variance
 +
Samll K low bias , high variance
  
 
==== Algorithm ====
 
==== Algorithm ====
Line 4,221: Line 5,263:
 
[http://en.wikipedia.org/wiki/Boosting Boosting] algorithms are a class of machine learning meta-algorithms that can improve weak classifiers.The idea is to incorporate unequal weights in learning process given higher weights to misclassified points . If we have different weak classifiers which slightly do better than random classification, then by assigning larger weights to points which are misclassified and  minimizing the new cost function by choosing an optimal weak classifier,we can update the weights in a way related to  the minimum value of the new cost function. This procedure can be repeated for a finite number of times and then a new classifier which is a weighed aggregation of the generated classifiers will be used as the boosted classifier. The better each generated classifier is the more its weight is in the final classifier.
 
[http://en.wikipedia.org/wiki/Boosting Boosting] algorithms are a class of machine learning meta-algorithms that can improve weak classifiers.The idea is to incorporate unequal weights in learning process given higher weights to misclassified points . If we have different weak classifiers which slightly do better than random classification, then by assigning larger weights to points which are misclassified and  minimizing the new cost function by choosing an optimal weak classifier,we can update the weights in a way related to  the minimum value of the new cost function. This procedure can be repeated for a finite number of times and then a new classifier which is a weighed aggregation of the generated classifiers will be used as the boosted classifier. The better each generated classifier is the more its weight is in the final classifier.
  
[http://www.site.uottawa.ca/~stan/csi5387/boost-tut-ppr.pdf Paper about Booting]:
+
[http://www.site.uottawa.ca/~stan/csi5387/boost-tut-ppr.pdf Paper about Boosting]:
 
Boosting is a general method for improving the accuracy of any given learning algorithm.
 
Boosting is a general method for improving the accuracy of any given learning algorithm.
 
This paper introduces the boosting algorithm AdaBoost, and explains the underlying theory of boosting, including an explanation of why boosting often does not suffer
 
This paper introduces the boosting algorithm AdaBoost, and explains the underlying theory of boosting, including an explanation of why boosting often does not suffer
from overfitting as well as boosting’s relationship to support-vector machines. Finally, this paper gives some examples of applications of boosting recently.
+
from overfitting as well as boosting’s relationship to support-vector machines. Finally, this paper gives some examples of recent applications of boosting.
  
 
Boosting is a general method of producing a very accurate prediction rule by combining rough and moderately inaccurate "rules of thumb." Much recent work has been on the "AdaBoost" boosting algorithm and its extensions.  
 
Boosting is a general method of producing a very accurate prediction rule by combining rough and moderately inaccurate "rules of thumb." Much recent work has been on the "AdaBoost" boosting algorithm and its extensions.  
 
[http://www.cs.princeton.edu/~schapire/boost.html]
 
[http://www.cs.princeton.edu/~schapire/boost.html]
 +
 +
==== [http://en.wikipedia.org/wiki/AdaBoost AdaBoost] ====
 +
AdaBoost (which is short for adaptive boosting) is a linear classifier with all its desirable properties. Its output converges to the logarithm of likelihood ratio.
 +
It has good generalization properties and is a feature selector with a principled strategy (minimization of upper
 +
bound on empirical error).
 +
AdaBoost produces a sequence of gradually more complex classifiers).
 +
 +
Advantages
 +
 +
*Very simple to implement
 +
*Feature selection on very large sets of features
 +
*Fairly good generalization
 +
 +
Disadvantages
 +
 +
*Suboptimal solution for <math>\,\Rightarrow\alpha</math>
 +
*Can overfit in presence of noise
 +
 +
[[File:1111.JPG|200px|thumb|right|Fig1.j=1]]
 +
[[File:2222.JPG|200px|thumb|right|Fig2.j=2]]
 +
[[File:3333.JPG|200px|thumb|right|Fig3.j=3]]
 +
[[File:4444.JPG|200px|thumb|right|Fig4.j=4]]
 +
[[File:5555.JPG|200px|thumb|right|Fig5.j=5]]
 +
[[File:6666.JPG|200px|thumb|right|Fig6.j=6]]
 +
[[File:7777.JPG|200px|thumb|right|Fig7.j=7]]
 +
[[File:8888.JPG|200px|thumb|right|Fig8.j=J]]
  
 
==== AdaBoost Algorithm ====
 
==== AdaBoost Algorithm ====
 +
 +
 +
 
Let's first look at the adaptive boosting algorithm:
 
Let's first look at the adaptive boosting algorithm:
 
#Set all the weights of all points equal <math>w_i\leftarrow \frac{1}{n}</math> where we have <math>\,n</math> points.
 
#Set all the weights of all points equal <math>w_i\leftarrow \frac{1}{n}</math> where we have <math>\,n</math> points.
 
#For <math>j=1,\dots, J</math>
 
#For <math>j=1,\dots, J</math>
## Find <math>h_j:X\rightarrow \{-1,+1\}</math> that minimizes the weighted error <math>\,L_j</math><br><math>h_j=\mbox{argmin}_{h_j \in H} L_j </math> where <math>L_j=\frac{\sum_{i=1}^n w_i I[y_i\neq h_j(x_i)]}{\sum_{i=1}^n w_i} </math>.<math>\ H </math> is a set of classifiers which need to be improved.
+
## Find <math>h_j:X\rightarrow \{-1,+1\}</math> that minimizes the weighted error <math>\,L_j</math><br><math>h_j=\mbox{argmin}_{h_j \in H} L_j </math> where <math>L_j=\frac{\sum_{i=1}^n w_i I[y_i\neq h_j(x_i)]}{\sum_{i=1}^n w_i} </math>.<br /><math>\ H </math> is a set of classifiers which need to be improved and <math>I</math> is:<br />
 +
:<math>\, I= \left\{\begin{matrix}
 +
1 &  for \quad  y_i\neq h_j(x_i) \\
 +
0 &  for \quad y_i = h_j(x_i)  \end{matrix}\right.</math><br />
 
## Let <math>\alpha_j\leftarrow\log(\frac{1-L_j}{L_j})</math>  
 
## Let <math>\alpha_j\leftarrow\log(\frac{1-L_j}{L_j})</math>  
 
## Update the weights: <math>w_i\leftarrow w_i e^{a_j I[y_j\neq h_j(x_i)]}</math>
 
## Update the weights: <math>w_i\leftarrow w_i e^{a_j I[y_j\neq h_j(x_i)]}</math>
#The final hypothesis is <math>h(x)=\mbox{sign}\left(\sum_{j=1}^J \alpha_j h_j(x)\right)</math>
+
#The final hypothesis is <math>h(x)=\mbox{sign}\left(\sum_{j=1}^J \alpha_j h_j(x)\right)</math><br />
 +
 
 +
The final hypothesis <math>h(x)</math> can be completely nonlinear. <br />
 +
 
  
* If we have a classifier that is random <math> {L_j} = 0 \Rightarrow \alpha_j = 0</math>, where else if the classifier is a little bit better than chance <math> \alpha_j\ >= 0 </math>
+
* If we have a classifier that is random <math> {L_j} = 0 \Rightarrow \alpha_j = 0</math>, where else if the classifier is a little bit better than chance <math> \alpha_j\ \geq 0 </math>
 
* If we have a good classifier and incorrectly misclassified <math>{x_i}</math>, then <math>{w_i}</math> is increased heavily
 
* If we have a good classifier and incorrectly misclassified <math>{x_i}</math>, then <math>{w_i}</math> is increased heavily
  
Line 4,270: Line 5,347:
 
==== AnyBoost ====
 
==== AnyBoost ====
  
Many boosting algorithms belong to a class called AnyBoost which are gradient descent algorithms for choosing linear combinations of elements of an inner product space in order to minimize some cost functional.
+
Many boosting algorithms belong to a class called AnyBoost which are gradient descent algorithms for choosing linear combinations of elements of an inner product space in order to minimize some cost function.
  
We are primarily interested in voted combinations of classifiers <math>H(x) = sgn(\sum_{j=1}^J \alpha_j h_j(x))</math>
+
We are primarily interested in weighted combinations of classifiers <math>H(x) = sgn(\sum_{j=1}^J \alpha_j h_j(x))</math>
  
 
We want to find H such that the cost functional <math>C(F) = \frac{1}{m}\sum_{i=1}^m c(y_i F(x_i))</math> is minimized for a suitable cost function <math>c</math>
 
We want to find H such that the cost functional <math>C(F) = \frac{1}{m}\sum_{i=1}^m c(y_i F(x_i))</math> is minimized for a suitable cost function <math>c</math>
Line 4,293: Line 5,370:
 
## <math>\ H_{j+1} = H_j + \alpha_{j+1} h_{j+1}</math>
 
## <math>\ H_{j+1} = H_j + \alpha_{j+1} h_{j+1}</math>
 
#The final classifier is <math>\ H_{J+1}</math>
 
#The final classifier is <math>\ H_{J+1}</math>
 +
 +
Figures 1 to 8 show how Anyboost algorithm works to classify the data.
  
  
Line 4,326: Line 5,405:
  
 
Bagging, or [http://en.wikipedia.org/wiki/Bootstrap_aggregating bootstrap aggregating], is another technique used to reduce the variance of classifiers with high variability.  It exploits the fact that a bootstrap mean is approximately equal to the posterior average.  It is most effective for highly nonlinear classifiers such as decision trees. In particular because of the highly unstable nature of these classifiers, they stand most likely to benefit from bagging.
 
Bagging, or [http://en.wikipedia.org/wiki/Bootstrap_aggregating bootstrap aggregating], is another technique used to reduce the variance of classifiers with high variability.  It exploits the fact that a bootstrap mean is approximately equal to the posterior average.  It is most effective for highly nonlinear classifiers such as decision trees. In particular because of the highly unstable nature of these classifiers, they stand most likely to benefit from bagging.
 +
 +
Bagging is one of the most effective computationally intensive procedures to improve on unstable estimators or classifiers, useful especially for high dimensional data set problems. Hard decisions create instability, and bagging is shown to smooth such hard decisions, yielding smaller variance and mean squared error.
  
 
==== Bagging Classifier ====
 
==== Bagging Classifier ====
Line 4,376: Line 5,457:
 
<math> \hat p_{s}(j) = \frac{\sum_{i = 1}^{n} I(Y_{i} = j,X_{i} \in R_{s})}{\sum_{i = 1}^{n} I(X_{i} \in R_{s})}</math>  
 
<math> \hat p_{s}(j) = \frac{\sum_{i = 1}^{n} I(Y_{i} = j,X_{i} \in R_{s})}{\sum_{i = 1}^{n} I(X_{i} \in R_{s})}</math>  
  
 +
==== CART ====
 +
Classification and regression trees (CART) is a non-parametric Decision tree learning technique that produces either classification or regression trees, depending on whether the dependent variable is categorical or numeric, respectively. (Wikipedia)
 +
 +
Classification and Regression Trees is a classification method which uses historical data to construct so-called decision trees. Decision trees are then used to classify new data. In order to use CART we need to know number of classes a priori. ([http://edoc.hu-berlin.de/master/timofeev-roman-2004-12-20/PDF/timofeev.pdf])
 +
 +
CART methodology was developed in 80s by Breiman, Freidman, Olshen, Stone in their paper ”Classification and Regression Trees” (1984). For building decision trees, CART uses so-called learning sample - a set of historical data with pre-assigned classes for all observations. For example, learning sample for credit scoring system would be fundamental information about previous borrows (variables) matched with actual
 +
payoff results (classes). ([http://edoc.hu-berlin.de/master/timofeev-roman-2004-12-20/PDF/timofeev.pdf])
 +
 +
Official Statistics Toolbox of Matlab provides CART. Here is a simple code for training and evaluation of a CART.
 +
 +
% Tree Construction - Learning Phase - Statistics Toolbox Built-in Function
 +
tree = classregtree(data_train,labels_train,'method','classification');
 +
% Tree in Action - Recalling Phase - Statistics Toolbox Built-in Function
 +
labels_test_hat = tree.eval(data_test));
 +
% Confusion Matrix Estimation - Statistics Toolbox Built-in Function
 +
C = confusionmat(labels_test,labels_test_hat);
 +
CCR = sum(diag(C))/sum(sum(C));
 +
 +
These are some pros and cons of CART (from here: [http://edoc.hu-berlin.de/master/timofeev-roman-2004-12-20/PDF/timofeev.pdf])
 +
 +
1. CART is nonparametric. Therefore this method does not require specification of any functional form.
 +
 +
2. CART does not require variables to be selected in advance. CART algorithm will itself identify the most significant variables and eliminate non-significant ones.
 +
 +
3. CART results are invariant to monotone transformations of its independent variables. Changing one or several variables to its logarithm or square root will not change the structure of the tree. Only the splitting values (but not variables) in the questions will be different.
 +
 +
4. CART can easily handle outliers. Outliers can negatively affect the results of some statistical models, like Principal Component Analysis (PCA) and linear regression. But the splitting algorithm of CART will easily handle noisy data: CART will isolate the outliers in a separate node. This property is very important, because financial data very often have outliers due to financial crisises or defaults.
  
 
==== Examples====
 
==== Examples====
Line 4,577: Line 5,685:
  
 
4. Papadimitriou, Christos H. and Mihalis Yannakakis. "On Limited Nondeterminism and the Complexity of the V-C Dimension." Journal of Computer and System Sciences, 53. pp. 161-170 (1996).
 
4. Papadimitriou, Christos H. and Mihalis Yannakakis. "On Limited Nondeterminism and the Complexity of the V-C Dimension." Journal of Computer and System Sciences, 53. pp. 161-170 (1996).
 +
 +
 +
== Supervised PCA - December 3, 2010==
 +
As we had in our very last (unofficial) meeting, we can briefly describe a possible approach for having PCA as a supervised dimensionality reduction methodology. This approach is based on the Hilbert-Schmidt Independence Criterion, or briefly HSIC.
 +
 +
Let's assume that we want to departure from a <math>\ D</math> dimensional space to a <math>\ d</math> dimensional one, using the following mapping:
 +
<math>\begin{align}Z=u.X \end{align}</math>
 +
 +
Where <math>\ X</math> is a <math>D\times n</math> matrix of the data points in the primary space, <math>\ Z</math> is a <math>d\times n</math> matrix of the the same data points in a reduced dimension space, and <math>\ u</math> is the <math>d\times D</math> mapping matrix. <math>\ n</math> is the total number of available data samples.
 +
 +
Here is the dual optimization problem we would like to solve: (you may find details on the primary problem in this paper: Zhang, Y., Zhi-Hua, Z., "Multi-label dimensionality reduction via dependence maximization",
 +
Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (2008))
 +
 +
<math>\begin{align}
 +
\max~&tr(u^T.X.H.B.H.X^T.u)\\
 +
s.t.~&u^Tu=I
 +
\end{align}</math>
 +
 +
Where <math>\ H</math> is a centering matrix defined like this: <math>H=I-\frac{1}{n}e.e^T</math> and <math>\ e</math> is a <math>n\times 1</math> vector of all ones. And <math>\ B</math> is the transformed target labels (class labels) using a arbitrarily chosen kernel.
 +
 +
If one consider the matrix <math>\ S</math> as a <math>1\times D</math> vector of <math>\ X.H.B.H.X^T</math> eigenvalues in a descending order, so that <math>s_1>s_2>\ldots>s_D</math>, where <math>\ s_i</math> is the ith element of the Matrix <math>\ S</math>, then the optimal solution for this optimization problem would be a matrix whose columns are <math>\ d</math> eigenvectors corresponding to the first <math>\ d</math> eigenvalues.
 +
 +
[[File:012DR-PCA.jpg|300px|thumb|right|Dimensionality Reduction of the 0-1-2 Data, Using PCA]]
 +
[[File:012DR-SPCA.jpg|300px|thumb|right|Dimensionality Reduction of the 0-1-2 Data, Using Supervised PCA]]
 +
 +
And here is a Matlab function for supervised PCA, based on HSIC.
 +
function [Z,u] = HSICPCA(X,Y,k)
 +
%---------- Supervised Principal Component Analysis
 +
%- X: samples, q*p
 +
%- Y: class labels, q*1 and \in{1,2,...,C}
 +
[q,p] = size(X);
 +
C = max(Y);
 +
X = sortrows([X,Y],p+1);
 +
Y = X(:,p+1);
 +
X = X(:,1:p);
 +
B = zeros(q,q);
 +
Q = zeros(1,C);
 +
for i = 1:C
 +
    Q(i) = sum(Y==i);
 +
    B(sum(Q(1:i-1))+1:sum(Q(1:i)),sum(Q(1:i-1))+1:sum(Q(1:i))) = ones(Q(i),Q(i));
 +
end
 +
H = eye(q) - ones(q,q)/q;
 +
gamma = X'*H*B*H*X;
 +
[V,D] = eig(gamma);
 +
D = diag(abs(D));
 +
D = [D,(1:p)'];
 +
D = sortrows(D,-1);
 +
ind = zeros(1,p);
 +
ind(D(1:k,2)) = 1;
 +
ind = logical(ind);
 +
u = V(:,ind);
 +
Z = X*u;
 +
 +
and PCA
 +
 +
function [Y,X_h,w] = PCA(X,d)
 +
%---------- Principal Component Analysis
 +
%- X:  p*q, Matrix of Samples (p: dimension of the space, q: no. of samples)
 +
%- d:  1*1, Dimension of the New Space
 +
%- Y:  d*q, Mapped Data into the New Space
 +
%- w:  p*d, Matrix of Mapping
 +
%- X_h: p*q, Reconstructed Data, Using the d Largest Eigen Values
 +
q = length(X(1,:));
 +
mu = mean(X,2);
 +
X_ao = X - mu*ones(1,q);
 +
[U,S,V] = svd(X_ao);
 +
X_h = U(:,1:d)*S(1:d,1:d)*V(:,1:d)'+mu*ones(1,q);
 +
w = U(:,1:d);
 +
Y = w'*X_ao;

Latest revision as of 08:45, 30 August 2017

Contents

Schedule of Project Presentations

Proposal Fall 2010

Mark your contribution here

Editor sign up

{{

 Template:namespace detect

| type = style | image = | imageright = | style = | textstyle = | text = This article may require cleanup to meet Wikicoursenote's quality standards. The specific problem is: Provide a summary for each topic here.. Please improve this article if you can. (October 8 2010) | small = | smallimage = | smallimageright = | smalltext = }}

Digest

Reference Textbook

The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition, February 2009 Trevor Hastie, Robert Tibshirani, Jerome Friedman (3rd Edition is available)

Classification - September 21, 2010

Classification

Statistical classification, or simply known as classification, is an area of supervised learning that addresses the problem of how to systematically assign unlabeled (classes unknown) novel data to their labels (classes or groups or types) by using knowledge of their features (characteristics or attributes) that are obtained from observation and/or measurement. A classifier is a specific technique or method for performing classification. To classify new data, a classifier first uses labeled (classes are known) training data to train a model, and then it uses a function known as its classification rule to assign a label to each new data input after feeding the input's known feature values into the model to determine how much the input belongs to each class.

Classification has been an important task for people and society since the beginnings of history. According to this link, the earliest application of classification in human society was probably done by prehistory peoples for recognizing which wild animals were beneficial to people and which ones were harmful, and the earliest systematic use of classification was done by the famous Greek philosopher Aristotle (384 BC - 322 BC) when he, for example, grouped all living things into the two groups of plants and animals. Classification is generally regarded as one of four major areas of statistics, with the other three major areas being regression, clustering, and dimensionality reduction (feature extraction or manifold learning). Please be noted that some people consider classification to be a broad area that consists of both supervised and unsupervised methods of classifying data. In this view, as can be seen in this link, clustering is simply a special case of classification and it may be called unsupervised classification.

In classical statistics, classification techniques were developed to learn useful information using small data sets where there is usually not enough of data. When machine learning was developed after the application of computers to statistics, classification techniques were developed to work with very large data sets where there is usually too many data. A major challenge facing data mining using machine learning is how to efficiently find useful patterns in very large amounts of data. An interesting quote that describes this problem quite well is the following one made by the retired Yale University Librarian Rutherford D. Rogers, a link to a source of which can be found here.

       "We are drowning in information and starving for knowledge."  
                                                         - Rutherford D. Rogers                          

In the Information Age, machine learning when it is combined with efficient classification techniques can be very useful for data mining using very large data sets. This is most useful when the structure of the data is not well understood but the data nevertheless exhibit strong statistical regularity. Areas in which machine learning and classification have been successfully used together include search and recommendation (e.g. Google, Amazon), automatic speech recognition and speaker verification, medical diagnosis, analysis of gene expression, drug discovery etc.

The formal mathematical definition of classification is as follows:

Definition: Classification is the prediction of a discrete random variable [math] \mathcal{Y} [/math] from another random variable [math] \mathcal{X} [/math], where [math] \mathcal{Y} [/math] represents the label assigned to a new data input and [math] \mathcal{X} [/math] represents the known feature values of the input.

A set of training data used by a classifier to train its model consists of [math]\,n[/math] independently and identically distributed (i.i.d) ordered pairs [math]\,\{(X_{1},Y_{1}), (X_{2},Y_{2}), \dots , (X_{n},Y_{n})\}[/math], where the values of the [math]\,ith[/math] training input's feature values [math]\,X_{i} = (\,X_{i1}, \dots , X_{id}) \in \mathcal{X} \subset \mathbb{R}^{d}[/math] is a d-dimensional vector and the label of the [math]\, ith[/math] training input is [math]\,Y_{i} \in \mathcal{Y} [/math] that can take a finite number of values. The classification rule used by a classifier has the form [math]\,h: \mathcal{X} \mapsto \mathcal{Y} [/math]. After the model is trained, each new data input whose feature values is [math]\,x[/math] is given the label [math]\,\hat{Y}=h(x)[/math].

As an example, if we would like to classify some vegetables and fruits, then our training data might look something like the one shown in the following picture from Professor Ali Ghodsi's Fall 2010 STAT 841 slides.

Data1.jpg

After we have selected a classifier and then built our model using our training data, we could use the classifier's classification rule [math]\ h [/math] to classify any newly-given vegetable or fruit such as the one shown in the following picture from Professor Ali Ghodsi's Fall 2010 STAT 841 slides after first obtaining its feature values.

Data3.jpg

As another example, suppose we wish to classify newly-given fruits into apples and oranges by considering three features of a fruit that comprise its color, its diameter, and its weight. After selecting a classifier and constructing a model using training data [math]\,\{(X_{color, 1}, X_{diameter, 1}, X_{weight, 1}, Y_{1}), \dots , (X_{color, n}, X_{diameter, n}, X_{weight, n}, Y_{n})\}[/math], we could then use the classifier's classification rule [math]\,h[/math] to assign any newly-given fruit having known feature values [math]\,x = (\,x_{color}, x_{diameter} , x_{weight})[/math] the label [math]\, \hat{Y}=h(x) \in \mathcal{Y}= \{apple,orange\}[/math].

Examples of Classification

• Email spam filtering (spam vs not spam).

• Detecting credit card fraud (fraudulent or legitimate).

• Face detection in images (face or background).

• Web page classification (sports vs politics vs entertainment etc).

• Steering an autonomous car across the US (turn left, right, or go straight).

• Medical diagnosis (classification of disease based on observed symptoms).

Independent and Identically Distributed (iid) Data Assumption

Suppose that we have training data X which contains N data points. The Independent and Identically Distributed (IID) assumption declares that the datapoints are drawn independently from identical distributions. This assumption implies that ordering of the data points does not matter, and the assumption is used in many classification problems. For an example of data that is not IID, consider daily temperature: today's temperature is not independent of the yesterday's temperature -- rather, there is a strong correlation between the temperatures of the two days.

Error rate

The empirical error rate (or training error rate) of a classifier having classification rule [math]\,h[/math] is defined as the frequency at which [math]\,h[/math] does not correctly classify the data inputs in the training set, i.e., it is defined as [math]\,\hat{L}_{n} = \frac{1}{n} \sum_{i=1}^{n} I(h(X_{i}) \neq Y_{i})[/math], where [math]\,I[/math] is an indicator variable and [math]\,I = \left\{\begin{matrix} 1 &\text{if } h(X_i) \neq Y_i \\ 0 &\text{if } h(X_i) = Y_i \end{matrix}\right.[/math]. Here, [math]\,X_{i} \in \mathcal{X}[/math] and [math]\,Y_{i} \in \mathcal{Y}[/math] are the known feature values and the true class of the [math]\,ith[/math] training input, respectively.


The true error rate [math]\,L(h)[/math] of a classifier having classification rule [math]\,h[/math] is defined as the probability that [math]\,h[/math] does not correctly classify any new data input, i.e., it is defined as [math]\,L(h)=P(h(X) \neq Y)[/math]. Here, [math]\,X \in \mathcal{X}[/math] and [math]\,Y \in \mathcal{Y}[/math] are the known feature values and the true class of that input, respectively.


In practice, the empirical error rate is obtained to estimate the true error rate, whose value is impossible to be known because the parameter values of the underlying process cannot be known but can only be estimated using available data. The empirical error rate, in practice, estimates the true error rate quite well in that, as mentioned here, it is an unbiased estimator of the true error rate.

An Error Rate Comparison of Classification Methods [1]

Decision Theory

we can identify three distinct approaches to solving decision problems, all of which have been used in practical applications. These are given, in decreasing order of complexity, by:

a. First solve the inference problem of determining the class-conditional densities [math]\ p(x|C_k)[/math] for each class [math]\ C_k[/math] individually. Also separately infer the prior class probabilities [math]\ p(C_k)[/math]. Then use Bayes’ theorem in the form

[math]\begin{align}p(C_k|x)=\frac{p(x|C_k)p(C_k)}{p(x)} \end{align}[/math]

to find the posterior class probabilities [math]\ p(C_k|x)[/math]. As usual, the denominator in Bayes’ theorem can be found in terms of the quantities appearing in the numerator, because

[math]\begin{align}p(x)=\sum_{k} p(x|C_k)p(C_k) \end{align}[/math]

Equivalently, we can model the joint distribution [math]\ p(x, C_k)[/math] directly and then normalize to obtain the posterior probabilities. Having found the posterior probabilities, we use decision theory to determine class membership for each new input [math]\ x[/math]. Approaches that explicitly or implicitly model the distribution of inputs as well as outputs are known as "generative models", because by sampling from them it is possible to generate synthetic data points in the input space.

b. First solve the inference problem of determining the posterior class probabilities [math]\ p(C_k|x)[/math], and then subsequently use decision theory to assign each new [math]\ x[/math] to one of the classes. Approaches that model the posterior probabilities directly are called "discriminative models".

c. Find a function [math]\ f(x)[/math], called a discriminant function, which maps each input [math]\ x[/math] directly onto a class label. For instance, in the case of two-class problems, [math]\ f(.)[/math] might be binary valued and such that [math]\ f = 0[/math] represents class [math]\ C_1[/math] and [math]\ f = 1[/math] represents class [math]\ C_2[/math]. In this case, probabilities play no role.

This topic has brought to you from Pattern Recognition and Machine Learning by Christopher M. Bishop (Chapter 1)

Bayes Classifier

A Bayes classifier is a simple probabilistic classifier based on applying Bayes' Theorem (from Bayesian statistics) with strong (naive) independence assumptions. A more descriptive term for the underlying probability model would be "independent feature model".

In simple terms, a Bayes classifier assumes that the presence (or absence) of a particular feature of a class is unrelated to the presence (or absence) of any other feature. For example, a fruit may be considered to be an apple if it is red, round, and about 4" in diameter. Even if these features depend on each other or upon the existence of the other features, a Bayes classifier considers all of these properties to independently contribute to the probability that this fruit is an apple.

Depending on the precise nature of the probability model, naive Bayes classifiers can be trained very efficiently in a supervised learning setting. In many practical applications, parameter estimation for Bayes models uses the method of maximum likelihood; in other words, one can work with the naive Bayes model without believing in Bayesian probability or using any Bayesian methods.

In spite of their design and apparently over-simplified assumptions, naive Bayes classifiers have worked quite well in many complex real-world situations. In 2004, analysis of the Bayesian classification problem has shown that there are some theoretical reasons for the apparently unreasonable efficacy of Bayes classifiers [1]. Still, a comprehensive comparison with other classification methods in 2006 showed that Bayes classification is outperformed by more current approaches, such as boosted trees or random forests[2].

An advantage of the naive Bayes classifier is that it requires a small amount of training data to estimate the parameters (means and variances of the variables) necessary for classification. Because independent variables are assumed, only the variances of the variables for each class need to be determined and not the entire covariance matrix.

After training its model using training data, the Bayes classifier classifies any new data input in two steps. First, it uses the input's known feature values and the Bayes formula to calculate the input's posterior probability of belonging to each class. Then, it uses its classification rule to place the input into the most-probable class, which is the one associated with the input's largest posterior probability.

In mathematical terms, for a new data input having feature values [math]\,(X = x)\in \mathcal{X}[/math], the Bayes classifier labels the input as [math](Y = y) \in \mathcal{Y}[/math], such that the input's posterior probability [math]\,P(Y = y|X = x)[/math] is maximum over all of the members of [math]\mathcal{Y}[/math].

Suppose there are [math]\,k[/math] classes and we are given a new data input having feature values [math]\,x[/math]. The following derivation shows how the Bayes classifier finds the input's posterior probability [math]\,P(Y = y|X = x)[/math] of belonging to each class [math] y \in \mathcal{Y} [/math].

[math] \begin{align} P(Y=y|X=x) &= \frac{P(X=x|Y=y)P(Y=y)}{P(X=x)} \\ &=\frac{P(X=x|Y=y)P(Y=y)}{\Sigma_{\forall i \in \mathcal{Y}}P(X=x|Y=i)P(Y=i)} \end{align} [/math]

Here, [math]\,P(Y=y|X=x)[/math] is known as the posterior probability as mentioned above, [math]\,P(Y=y)[/math] is known as the prior probability, [math]\,P(X=x|Y=y)[/math] is known as the likelihood, and [math]\,P(X=x)[/math] is known as the evidence.

In the special case where there are two classes, i.e., [math]\, \mathcal{Y}=\{0, 1\}[/math], the Bayes classifier makes use of the function [math]\,r(x)=P\{Y=1|X=x\}[/math] which is the posterior probability of a new data input having feature values [math]\,x[/math] belonging to the class [math]\,Y = 1[/math]. Following the above derivation for the posterior probabilities of a new data input, the Bayes classifier calculates [math]\,r(x)[/math] as follows:

[math] \begin{align} r(x)&=P(Y=1|X=x) \\ &=\frac{P(X=x|Y=1)P(Y=1)}{P(X=x)}\\ &=\frac{P(X=x|Y=1)P(Y=1)}{P(X=x|Y=1)P(Y=1)+P(X=x|Y=0)P(Y=0)} \end{align} [/math]

The Bayes classifier's classification rule [math]\,h^*: \mathcal{X} \mapsto \mathcal{Y}[/math], then, is

[math]\, h^*(x)= \left\{\begin{matrix} 1 &\text{if } \hat r(x)\gt \frac{1}{2} \\ 0 &\mathrm{otherwise} \end{matrix}\right.[/math].

Here, [math]\,x[/math] is the feature values of a new data input and [math]\hat r(x)[/math] is the estimated value of the function [math]\,r(x)[/math] given by the Bayes classifier's model after feeding [math]\,x[/math] into the model. Still in this special case of two classes, the Bayes classifier's decision boundary is defined as the set [math]\,D(h)=\{x: P(Y=1|X=x)=P(Y=0|X=x)\}[/math]. The decision boundary [math]\,D(h)[/math] essentially combines together the trained model and the decision function [math]\,h^*[/math], and it is used by the Bayes classifier to assign any new data input to a label of either [math]\,Y = 0[/math] or [math]\,Y = 1[/math] depending on which side of the decision boundary the input lies in. From this decision boundary, it is easy to see that, in the case where there are two classes, the Bayes classifier's classification rule can be re-expressed as

[math]\, h^*(x)= \left\{\begin{matrix} 1 &\text{if } P(Y=1|X=x)\gt P(Y=0|X=x) \\ 0 &\mathrm{otherwise} \end{matrix}\right.[/math].

Bayes Classification Rule Optimality Theorem The Bayes classifier is the optimal classifier in that it results in the least possible true probability of misclassification for any given new data input, i.e., for any generic classifier having classification rule [math]\,h[/math], it is always true that [math]\,L(h^*(x)) \le L(h(x))[/math]. Here, [math]\,L[/math] represents the true error rate, [math]\,h^*[/math] is the Bayes classifier's classification rule, and [math]\,x[/math] is any given data input's feature values.

Although the Bayes classifier is optimal in the theoretical sense, other classifiers may nevertheless outperform it in practice. The reason for this is that various components which make up the Bayes classifier's model, such as the likelihood and prior probabilities, must either be estimated using training data or be guessed with a certain degree of belief. As a result, the estimated values of the components in the trained model may deviate quite a bit from their true population values, and this can ultimately cause the calculated posterior probabilities of inputs to deviate quite a bit from their true values. Estimation of all these probability functions, as likelihood, prior probability, and evidence function is a very expensive task, computationally, which also makes some other classifiers more favorable than Bayes classifier.

A detailed proof of this theorem is available here.

Defining the classification rule:

In the special case of two classes, the Bayes classifier can use three main approaches to define its classification rule [math]\,h^*[/math]:

1) Empirical Risk Minimization: Choose a set of classifiers [math]\mathcal{H}[/math] and find [math]\,h^*\in \mathcal{H}[/math] that minimizes some estimate of the true error rate [math]\,L(h^*)[/math].
2) Regression: Find an estimate [math] \hat r [/math] of the function [math] x [/math] and define
[math]\, h^*(x)= \left\{\begin{matrix} 1 &\text{if } \hat r(x)\gt \frac{1}{2} \\ 0 &\mathrm{otherwise} \end{matrix}\right.[/math].
3) Density Estimation: Estimate [math]\,P(X=x|Y=0)[/math] from the [math]\,X_{i}[/math]'s for which [math]\,Y_{i} = 0[/math], estimate [math]\,P(X=x|Y=1)[/math] from the [math]\,X_{i}[/math]'s for which [math]\,Y_{i} = 1[/math], and estimate [math]\,P(Y = 1)[/math] as [math]\,\frac{1}{n} \sum_{i=1}^{n} Y_{i}[/math]. Then, calculate [math]\,\hat r(x) = \hat P(Y=1|X=x)[/math] and define
[math]\, h^*(x)= \left\{\begin{matrix} 1 &\text{if } \hat r(x)\gt \frac{1}{2} \\ 0 &\mathrm{otherwise} \end{matrix}\right.[/math].

Typically, the Bayes classifier uses approach 3 to define its classification rule. These three approaches can easily be generalized to the case where the number of classes exceeds two.

Multi-class classification:

Suppose there are [math]\,k[/math] classes, where [math]\,k \ge 2[/math].

In the above discussion, we introduced the Bayes formula for this general case:

[math] \begin{align} P(Y=y|X=x) &=\frac{P(X=x|Y=y)P(Y=y)}{\Sigma_{\forall i \in \mathcal{Y}}P(X=x|Y=i)P(Y=i)} \end{align} [/math]

which can re-worded as:

[math] \begin{align} P(Y=y|X=x) &=\frac{f_y(x)\pi_y}{\Sigma_{\forall i \in \mathcal{Y}} f_i(x)\pi_i} \end{align} [/math]

Here, [math]\,f_y(x) = P(X=x|Y=y)[/math] is known as the likelihood function and [math]\,\pi_y = P(Y=y)[/math] is known as the prior probability.

In the general case where there are at least two classes, the Bayes classifier uses the following theorem to assign any new data input having feature values [math]\,x[/math] into one of the [math]\,k[/math] classes.

Theorem

Suppose that [math] \mathcal{Y}= \{1, \dots, k\}[/math], where [math]\,k \ge 2[/math]. Then, the optimal classification rule is [math]\,h^*(x) = arg max_{i} P(Y=i|X=x)[/math], where [math]\,i \in \{1, \dots, k\}[/math].

Example: We are going to predict if a particular student will pass STAT 441/841. There are two classes represented by [math]\, \mathcal{Y}\in \{ 0,1 \} [/math], where 1 refers to pass and 0 refers to fail. Suppose that the prior probabilities are estimated or guessed to be [math]\,\hat P(Y = 1) = \hat P(Y = 0) = 0.5[/math]. We have data on past student performances, which we shall use to train the model. For each student, we know the following:

Whether or not the student’s GPA was greater than 3.0 (G).
Whether or not the student had a strong math background (M).
Whether or not the student was a hard worker (H).
Whether or not the student passed or failed the course. Note: these are the known y values in the training data.

These known data are summarized in the following tables:

裁剪.jpg

For each student, his/her feature values is [math]\, x = \{G, M, H\} [/math] and his or her class is [math]\, y \in \{0, 1\} [/math].

Suppose there is a new student having feature values [math]\, x = \{0, 1, 0\}[/math], and we would like to predict whether he/she would pass the course. [math]\,\hat r(x)[/math] is found as follows:


[math]\, \hat r(x) = P(Y=1|X =(0,1,0))=\frac{P(X=(0,1,0)|Y=1)P(Y=1)}{P(X=(0,1,0)|Y=0)P(Y=0)+P(X=(0,1,0)|Y=1)P(Y=1)}=\frac{0.05*0.5}{0.05*0.5+0.2*0.5}=\frac{0.025}{0.125}=\frac{1}{5}\lt \frac{1}{2}.[/math]

The Bayes classifier assigns the new student into the class [math]\, h^*(x)=0 [/math]. Therefore, we predict that the new student would fail the course.

Naive Bayes Classifier:

The naive Bayes classifier is a special (simpler) case of the Bayes classifier. It uses an extra assumption: that the presence (or absence) of a particular feature of a class is unrelated to the presence (or absence) of any other feature. This assumption allows for an easier likelihood function [math]\,f_y(x)[/math] in the equation:

[math] \begin{align} P(Y=y|X=x) &=\frac{f_y(x)\pi_y}{\Sigma_{\forall i \in \mathcal{Y}} f_i(x)\pi_i} \end{align} [/math]

The simper form of the likelihood function seen in the naive Bayes is:

[math] \begin{align} f_y(x) = P(X=x|Y=y) = {\prod_{i=1}^{n} P(X_{i}=x_{i}|Y=y)} \end{align} [/math]

The Bayes classifier taught in class was not the naive Bayes classifier.

Bayesian vs. Frequentist

The Bayesian view of probability and the frequentist view of probability are the two major schools of thought in the field of statistics regarding how to interpret the probability of an event.


The Bayesian view of probability states that, for any event E, event E has a prior probability that represents how believable event E would occur prior to knowing anything about any other event whose occurrence could have an impact on event E's occurrence. Theoretically, this prior probability is a belief that represents the baseline probability for event E's occurrence. In practice, however, event E's prior probability is unknown, and therefore it must either be guessed at or be estimated using a sample of available data. After obtaining a guessed or estimated value of event E's prior probability, the Bayesian view holds that the probability, that is, the believability of event E's occurrence, can always be made more accurate should any new information regarding events that are relevant to event E become available. The Bayesian view also holds that the accuracy for the estimate of the probability of event E's occurrence is higher as long as there are more useful information available regarding events that are relevant to event E. The Bayesian view therefore holds that there is no intrinsic probability of occurrence associated with any event. If one adherers to the Bayesian view, one can then, for instance, predict tomorrow's weather as having a probability of, say, [math]\,50%[/math] for rain. The Bayes classifier as described above is a good example of a classifier developed from the Bayesian view of probability. The earliest works that lay the framework for the Bayesian view of probability is accredited to Thomas Bayes (1702–1761).


In contrast to the Bayesian view of probability, the frequentist view of probability holds that there is an intrinsic probability of occurrence associated with every event to which one can carry out many, if not an infinite number, of well-defined independent random trials. In each trial for an event, the event either occurs or it does not occur. Suppose [math]n_x[/math] denotes the number of times that an event occurs during its trials and [math]n_t[/math] denotes the total number of trials carried out for the event. The frequentist view of probability holds that, in the long run, where the number of trials for an event approaches infinity, one could theoretically approach the intrinsic value of the event's probability of occurrence to any arbitrary degree of accuracy, i.e., :[math]P(x) = \lim_{n_t\rightarrow \infty}\frac{n_x}{n_t}[/math]. In practice, however, one can only carry out a finite number of trials for an event and, as a result, the probability of the event's occurrence can only be approximated as [math]P(x) \approx \frac{n_x}{n_t}[/math]. If one adherers to the frequentist view, one cannot, for instance, predict the probability that there would be rain tomorrow. This is because one cannot possibly carry out trials for any event that is set in the future. The founder of the frequentist school of thought is arguably the famous Greek philosopher Aristotle. In his work Rhetoric, Aristotle gave the famous line "the probable is that which for the most part happens".


More information regarding the Bayesian and the frequentist schools of thought are available here. Furthermore, an interesting and informative youtube video that explains the Bayesian and frequentist views of probability is available here.

There is useful information about Machine Learning, Neural and Statistical Classification in this link [2] Machine Learning, Neural and Statistical Classification; there is some description of Classification in chapter 2 Classical Statistical Methods in chapter 3 and Modern Statistical Techniques in chapter 4.

Extension: Statistical Classification Framework

In statistical classification, each object is represented by 'd' (a set of features) a measurement vector, and the goal of classifier becomes finding compact and disjoint regions for classes in a d-dimensional feature space. Such decision regions are defined by decision rules that are known or can be trained. The simplest configuration of a classification consists of a decision rule and multiple membership functions; each membership function represents a class. The following figures illustrate this general framework.

File:cs1.png

Simple Conceptual Classifier.

File:cs2.png

Statistical Classification Framework


The classification process can be described as follows:

A measurement vector is input to each membership function. Membership functions feed the membership scores to the decision rule. A decision rule compares the membership scores and returns a class label.

Linear and Quadratic Discriminant Analysis

Introduction

Linear discriminant analysis (LDA) and the related Fisher's linear discriminant are methods used in statistics, pattern recognition and machine learning to find a linear combination of features which characterize or separate two or more classes of objects or events. The resulting combination may be used as a linear classifier, or, more commonly, for dimensionality reduction before later classification.

LDA is also closely related to principal component analysis (PCA) and factor analysis in that both look for linear combinations of variables which best explain the data. LDA explicitly attempts to model the difference between the classes of data. PCA on the other hand does not take into account any difference in class, and factor analysis builds the feature combinations based on differences rather than similarities. Discriminant analysis is also different from factor analysis in that it is not an interdependence technique: a distinction between independent variables and dependent variables (also called criterion variables) must be made.

LDA works when the measurements made on independent variables for each observation are continuous quantities. When dealing with categorical independent variables, the equivalent technique is discriminant correspondence analysis.

Content

First, we shall limit ourselves to the case where there are two classes, i.e. [math]\, \mathcal{Y}=\{0, 1\}[/math]. In the above discussion, we introduced the Bayes classifier's decision boundary [math]\,D(h^*)=\{x: P(Y=1|X=x)=P(Y=0|X=x)\}[/math], which represents a hyperplane that determines the class of any new data input depending on which side of the hyperplane the input lies in. Now, we shall look at how to derive the Bayes classifier's decision boundary under certain assumptions of the data. Linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA) are two of the most well-known ways for deriving the Bayes classifier's decision boundary, and we shall look at each of them in turn.

Let us denote the likelihood [math]\ P(X=x|Y=y) [/math] as [math]\ f_y(x) [/math] and the prior probability [math]\ P(Y=y) [/math] as [math]\ \pi_y [/math].

First, we shall examine LDA. As explained above, the Bayes classifier is optimal. However, in practice, the prior and conditional densities are not known. Under LDA, one gets around this problem by making the assumptions that both of the two classes have multivariate normal (Gaussian) distributions and the two classes have the same covariance matrix [math]\, \Sigma[/math]. Under the assumptions of LDA, we have: [math]\ P(X=x|Y=y) = f_y(x) = \frac{1}{ (2\pi)^{d/2}|\Sigma|^{1/2} }\exp\left( -\frac{1}{2} (x - \mu_k)^\top \Sigma^{-1} (x - \mu_k) \right)[/math]. Now, to derive the Bayes classifier's decision boundary using LDA, we equate [math]\, P(Y=1|X=x) [/math] to [math]\, P(Y=0|X=x) [/math] and proceed from there. The derivation of [math]\,D(h^*)[/math] is as follows:

[math]\,Pr(Y=1|X=x)=Pr(Y=0|X=x)[/math]
[math]\,\Rightarrow \frac{Pr(X=x|Y=1)Pr(Y=1)}{Pr(X=x)}=\frac{Pr(X=x|Y=0)Pr(Y=0)}{Pr(X=x)}[/math] (using Bayes' Theorem)
[math]\,\Rightarrow Pr(X=x|Y=1)Pr(Y=1)=Pr(X=x|Y=0)Pr(Y=0)[/math] (canceling the denominators)
[math]\,\Rightarrow f_1(x)\pi_1=f_0(x)\pi_0[/math]
[math]\,\Rightarrow \frac{1}{ (2\pi)^{d/2}|\Sigma|^{1/2} }\exp\left( -\frac{1}{2} (x - \mu_1)^\top \Sigma^{-1} (x - \mu_1) \right)\pi_1=\frac{1}{ (2\pi)^{d/2}|\Sigma|^{1/2} }\exp\left( -\frac{1}{2} (x - \mu_0)^\top \Sigma^{-1} (x - \mu_0) \right)\pi_0[/math]
[math]\,\Rightarrow \exp\left( -\frac{1}{2} (x - \mu_1)^\top \Sigma^{-1} (x - \mu_1) \right)\pi_1=\exp\left( -\frac{1}{2} (x - \mu_0)^\top \Sigma^{-1} (x - \mu_0) \right)\pi_0[/math]
[math]\,\Rightarrow -\frac{1}{2} (x - \mu_1)^\top \Sigma^{-1} (x - \mu_1) + \log(\pi_1)=-\frac{1}{2} (x - \mu_0)^\top \Sigma^{-1} (x - \mu_0) +\log(\pi_0)[/math] (taking the log of both sides).
[math]\,\Rightarrow \log(\frac{\pi_1}{\pi_0})-\frac{1}{2}\left( x^\top\Sigma^{-1}x + \mu_1^\top\Sigma^{-1}\mu_1 - 2x^\top\Sigma^{-1}\mu_1 - x^\top\Sigma^{-1}x - \mu_0^\top\Sigma^{-1}\mu_0 + 2x^\top\Sigma^{-1}\mu_0 \right)=0[/math] (expanding out)
[math]\,\Rightarrow \log(\frac{\pi_1}{\pi_0})-\frac{1}{2}\left( \mu_1^\top\Sigma^{-1} \mu_1-\mu_0^\top\Sigma^{-1}\mu_0 - 2x^\top\Sigma^{-1}(\mu_1-\mu_0) \right)=0[/math] (canceling out alike terms and factoring).

It is easy to see that, under LDA, the Bayes's classifier's decision boundary [math]\,D(h^*)[/math] has the form [math]\,ax+b=0[/math] and it is linear in [math]\,x[/math]. This is where the word linear in linear discriminant analysis comes from.


LDA under the two-classes case can easily be generalized to the general case where there are [math]\,k \ge 2[/math] classes. In the general case, suppose we wish to find the Bayes classifier's decision boundary between the two classes [math]\,m [/math] and [math]\,n[/math], then all we need to do is follow a derivation very similar to the one shown above, except with the classes [math]\,1 [/math] and [math]\,0[/math] being replaced by the classes [math]\,m [/math] and [math]\,n[/math]. Following through with a similar derivation as the one shown above, one obtains the Bayes classifier's decision boundary [math]\,D(h^*)[/math] between classes [math]\,m [/math] and [math]\,n[/math] to be [math]\,\log(\frac{\pi_m}{\pi_n})-\frac{1}{2}\left( \mu_m^\top\Sigma^{-1} \mu_m-\mu_n^\top\Sigma^{-1}\mu_n - 2x^\top\Sigma^{-1}(\mu_m-\mu_n) \right)=0[/math] . In addition, for any two classes [math]\,m [/math] and [math]\,n[/math] for whom we would like to find the Bayes classifier's decision boundary using LDA, if [math]\,m [/math] and [math]\,n[/math] both have the same number of data, then, in this special case, the resulting decision boundary would lie exactly halfway between the centers (means) of [math]\,m [/math] and [math]\,n[/math].


The Bayes classifier's decision boundary for any two classes as derived using LDA looks something like the one that can be found in this link:


Although the assumption under LDA may not hold true for most real-world data, it nevertheless usually performs quite well in practice, where it often provides near-optimal classifications. For instance, the Z-Score credit risk model that was designed by Edward Altman in 1968 and revisited in 2000, is essentially a weighted LDA. This model has demonstrated a 85-90% success rate in predicting bankruptcy, and for this reason it is still in use today.


According to this link, some of the limitations of LDA include:

  • LDA implicitly assumes that the data in each class has a Gaussian distribution.
  • LDA implicitly assumes that the mean rather than the variance is the discriminating factor.
  • LDA may over-fit the training data.

The following link provides a comparison of discriminant analysis and artificial neural networks [3]

Different Approaches to LDA

Data sets can be transformed and test vectors can be classified in the transformed space by two different approaches.

  • Class-dependent transformation: This type of approach involves maximizing the ratio of between

class variance to within class variance. The main objective is to maximize this ratio so that adequate class separability is obtained. The class-specific type approach involves using two optimizing criteria for transforming the data sets independently.

  • Class-independent transformation: This approach involves maximizing the ratio of overall variance

to within class variance. This approach uses only one optimizing criterion to transform the data sets and hence all data points irrespective of their class identity are transformed using this transform. In this type of LDA, each class is considered as a separate class against all other classes.

Further reading

The following are some applications that use LDA and QDA:

1- Linear discriminant analysis for improved large vocabulary continuous speech recognition here

2- 2D-LDA: A statistical linear discriminant analysis for image matrix here

3- Regularization studies of linear discriminant analysis in small sample size scenarios with application to face recognition here

4- The sparse discriminant vectors are useful for supervised dimension reduction for high dimensional data. Naive application of classical Fisher’s LDA to high dimensional, low sample size settings suffers from the data piling problem. In [4] they have use sparse LDA method which selects important variables for discriminant analysis and thereby yield improved classification. Introducing sparsity in the discriminant vectors is very effective in eliminating data piling and the associated overfitting problem.

Linear and Quadratic Discriminant Analysis cont'd - September 23, 2010

LDA x QDA

Linear discriminant analysis[5] is a statistical method used to find the linear combination of features which best separate two or more classes of objects or events. It is widely applied in classifying diseases, positioning, product management, and marketing research. LDA assumes that the different classes have the same covariance matrix [math]\, \Sigma[/math].

Quadratic Discriminant Analysis[6], on the other hand, aims to find the quadratic combination of features. It is more general than linear discriminant analysis. Unlike LDA, QDA does not make the assumption that the different classes have the same covariance matrix [math]\, \Sigma[/math]. Instead, QDA makes the assumption that each class [math]\, k[/math] has its own covariance matrix [math]\, \Sigma_k[/math].

The derivation of the Bayes classifier's decision boundary [math]\,D(h^*)[/math] under QDA is similar to that under LDA. Again, let us first consider the two-classes case where [math]\, \mathcal{Y}=\{0, 1\}[/math]. This derivation is given as follows:

[math]\,Pr(Y=1|X=x)=Pr(Y=0|X=x)[/math]
[math]\,\Rightarrow \frac{Pr(X=x|Y=1)Pr(Y=1)}{Pr(X=x)}=\frac{Pr(X=x|Y=0)Pr(Y=0)}{Pr(X=x)}[/math] (using Bayes' Theorem)
[math]\,\Rightarrow Pr(X=x|Y=1)Pr(Y=1)=Pr(X=x|Y=0)Pr(Y=0)[/math] (canceling the denominators)
[math]\,\Rightarrow f_1(x)\pi_1=f_0(x)\pi_0[/math]
[math]\,\Rightarrow \frac{1}{ (2\pi)^{d/2}|\Sigma_1|^{1/2} }\exp\left( -\frac{1}{2} (x - \mu_1)^\top \Sigma_1^{-1} (x - \mu_1) \right)\pi_1=\frac{1}{ (2\pi)^{d/2}|\Sigma_0|^{1/2} }\exp\left( -\frac{1}{2} (x - \mu_0)^\top \Sigma_0^{-1} (x - \mu_0) \right)\pi_0[/math]
[math]\,\Rightarrow \frac{1}{|\Sigma_1|^{1/2} }\exp\left( -\frac{1}{2} (x - \mu_1)^\top \Sigma_1^{-1} (x - \mu_1) \right)\pi_1=\frac{1}{|\Sigma_0|^{1/2} }\exp\left( -\frac{1}{2} (x - \mu_0)^\top \Sigma_0^{-1} (x - \mu_0) \right)\pi_0[/math] (by cancellation)
[math]\,\Rightarrow -\frac{1}{2}\log(|\Sigma_1|)-\frac{1}{2} (x - \mu_1)^\top \Sigma_1^{-1} (x - \mu_1)+\log(\pi_1)=-\frac{1}{2}\log(|\Sigma_0|)-\frac{1}{2} (x - \mu_0)^\top \Sigma_0^{-1} (x - \mu_0)+\log(\pi_0)[/math] (by taking the log of both sides)
[math]\,\Rightarrow \log(\frac{\pi_1}{\pi_0})-\frac{1}{2}\log(\frac{|\Sigma_1|}{|\Sigma_0|})-\frac{1}{2}\left( x^\top\Sigma_1^{-1}x + \mu_1^\top\Sigma_1^{-1}\mu_1 - 2x^\top\Sigma_1^{-1}\mu_1 - x^\top\Sigma_0^{-1}x - \mu_0^\top\Sigma_0^{-1}\mu_0 + 2x^\top\Sigma_0^{-1}\mu_0 \right)=0[/math] (by expanding out)
[math]\,\Rightarrow \log(\frac{\pi_1}{\pi_0})-\frac{1}{2}\log(\frac{|\Sigma_1|}{|\Sigma_0|})-\frac{1}{2}\left( x^\top(\Sigma_1^{-1}-\Sigma_0^{-1})x + \mu_1^\top\Sigma_1^{-1}\mu_1 - \mu_0^\top\Sigma_0^{-1}\mu_0 - 2x^\top(\Sigma_1^{-1}\mu_1-\Sigma_0^{-1}\mu_0) \right)=0[/math]

It is easy to see that, under QDA, the decision boundary [math]\,D(h^*)[/math] has the form [math]\,ax^2+bx+c=0[/math] and it is quadratic in [math]\,x[/math]. This is where the word quadratic in quadratic discriminant analysis comes from.

As is the case with LDA, QDA under the two-classes case can easily be generalized to the general case where there are [math]\,k \ge 2[/math] classes. In the general case, suppose we wish to find the Bayes classifier's decision boundary between the two classes [math]\,m [/math] and [math]\,n[/math], then all we need to do is follow a derivation very similar to the one shown above, except with the classes [math]\,1 [/math] and [math]\,0[/math] being replaced by the classes [math]\,m [/math] and [math]\,n[/math]. Following through with a similar derivation as the one shown above, one obtains the Bayes classifier's decision boundary [math]\,D(h^*)[/math] between classes [math]\,m [/math] and [math]\,n[/math] to be [math]\,\log(\frac{\pi_m}{\pi_n})-\frac{1}{2}\log(\frac{|\Sigma_m|}{|\Sigma_n|})-\frac{1}{2}\left( x^\top(\Sigma_m^{-1}-\Sigma_n^{-1})x + \mu_m^\top\Sigma_m^{-1}\mu_m - \mu_n^\top\Sigma_n^{-1}\mu_n - 2x^\top(\Sigma_m^{-1}\mu_m-\Sigma_n^{-1}\mu_n) \right)=0[/math].

Summarizing LDA and QDA

We can summarize what we have learned so far into the following theorem.

Theorem:


Suppose that [math]\,Y \in \{1,\dots,K\}[/math], if [math]\,f_k(x) = Pr(X=x|Y=k)[/math] is Gaussian, the Bayes Classifier rule is

[math]\,h^*(x) = \arg\max_{k} \delta_k(x)[/math]

where,

  • In the case of LDA, which assumes that a common covariance matrix is shared by all classes, [math] \,\delta_k(x) = x^\top\Sigma^{-1}\mu_k - \frac{1}{2}\mu_k^\top\Sigma^{-1}\mu_k + log (\pi_k) [/math], and the Bayes classifier's decision boundary [math]\,D(h^*)[/math] is linear in [math]\,x[/math].
  • In the case of QDA, which assumes that each class has its own covariance matrix, [math] \,\delta_k(x) = - \frac{1}{2}log(|\Sigma_k|) - \frac{1}{2}(x-\mu_k)^\top\Sigma_k^{-1}(x-\mu_k) + log (\pi_k) [/math], and the Bayes classifier's decision boundary [math]\,D(h^*)[/math] is quadratic in [math]\,x[/math].


Note [math]\,\arg\max_{k} \delta_k(x)[/math]returns the set of k for which [math]\,\delta_k(x)[/math] attains its largest value.

See Theorem 46.6 Page 133

In practice

We need to estimate the prior, so in order to do this, we use the Maximum Likelihood estimates from the sample for [math]\,\pi,\mu_k,\Sigma_k[/math] in place of their true values, i.e.

File:estimation.png
Estimation of the probability of belonging to either class k or l

[math]\,\hat{\pi_k} = \hat{Pr}(y=k) = \frac{n_k}{n}[/math]

[math]\,\hat{\mu_k} = \frac{1}{n_k}\sum_{i:y_i=k}x_i[/math]

[math]\,\hat{\Sigma_k} = \frac{1}{n_k}\sum_{i:y_i=k}(x_i-\hat{\mu_k})(x_i-\hat{\mu_k})^\top[/math]

Common covariance, denoted [math]\Sigma[/math], is defined as the weighted average of the covariance for each class.

In the case where we need a common covariance matrix, we get the estimate using the following equation:

[math]\,\Sigma=\frac{\sum_{r=1}^{k}(n_r\Sigma_r)}{\sum_{l=1}^{k}(n_l)} [/math]

Where: [math]\,n_r[/math] is the number of data points in class r, [math]\,\Sigma_r[/math] is the covariance of class r and [math]\,n[/math] is the total number of data points, [math]\,k[/math] is the number of classes.

See the details about the estimation of covarience matrices.

Computation For QDA And LDA

First, let us consider QDA, and examine each of the following two cases.

Case 1: (Example) [math]\, \Sigma_k = I [/math]

[math]\, \Sigma_k = I [/math] for every class [math]\,k[/math] implies that our data is spherical. This means that the data of each class [math]\,k[/math] is distributed symmetrically around the center [math]\,\mu_k[/math], i.e. the isocontours are all circles.

We have:

[math] \,\delta_k = - \frac{1}{2}log(|I|) - \frac{1}{2}(x-\mu_k)^\top I(x-\mu_k) + log (\pi_k) [/math]

We see that the first term in the above equation, [math]\,\frac{-1}{2}log(|I|)[/math], is zero since [math]\ |I|=1 [/math]. The second term contains [math]\, (x-\mu_k)^\top I(x-\mu_k) = (x-\mu_k)^\top(x-\mu_k) [/math], which is the squared Euclidean distance between [math]\,x[/math] and [math]\,\mu_k[/math]. Therefore we can find the distance between a point and each center and adjust it with the log of the prior, [math]\,log(\pi_k)[/math]. The class that has the minimum distance will maximize [math]\,\delta_k[/math]. According to the theorem, we can then classify the point to a specific class [math]\,k[/math].


Case 2: (General Case) [math]\, \Sigma_k \ne I [/math]

We can decompose this as:

[math] \, \Sigma_k = U_kS_kV_k^\top = U_kS_kU_k^\top [/math] (In general when [math]\,X=U_kS_kV_k^\top[/math], [math]\,U_k[/math] is the eigenvectors of [math]\,X_kX_k^T[/math] and [math]\,V_k[/math] is the eigenvectors of [math]\,X_k^\top X_k[/math]. So if [math]\, X_k[/math] is symmetric, we will have [math]\, U_k=V_k[/math]. Here [math]\, \Sigma_k [/math] is symmetric, because it is the covariance matrix of [math] X_k [/math]) and the inverse of [math]\,\Sigma_k[/math] is

[math] \, \Sigma_k^{-1} = (U_kS_kU_k^\top)^{-1} = (U_k^\top)^{-1}S_k^{-1}U_k^{-1} = U_kS_k^{-1}U_k^\top [/math] (since [math]\,U_k[/math] is orthonormal)

So from the formula for [math]\,\delta_k[/math], the second term is

[math]\begin{align} (x-\mu_k)^\top\Sigma_k^{-1}(x-\mu_k)&= (x-\mu_k)^\top U_kS_k^{-1}U_k^T(x-\mu_k)\\ & = (U_k^\top x-U_k^\top\mu_k)^\top S_k^{-1}(U_k^\top x-U_k^\top \mu_k)\\ & = (U_k^\top x-U_k^\top\mu_k)^\top S_k^{-\frac{1}{2}}S_k^{-\frac{1}{2}}(U_k^\top x-U_k^\top\mu_k) \\ & = (S_k^{-\frac{1}{2}}U_k^\top x-S_k^{-\frac{1}{2}}U_k^\top\mu_k)^\top I(S_k^{-\frac{1}{2}}U_k^\top x-S_k^{-\frac{1}{2}}U_k^\top \mu_k) \\ & = (S_k^{-\frac{1}{2}}U_k^\top x-S_k^{-\frac{1}{2}}U_k^\top\mu_k)^\top(S_k^{-\frac{1}{2}}U_k^\top x-S_k^{-\frac{1}{2}}U_k^\top \mu_k) \\ \end{align} [/math]

where we have the squared Euclidean distance between [math] \, S_k^{-\frac{1}{2}}U_k^\top x [/math] and [math]\, S_k^{-\frac{1}{2}}U_k^\top\mu_k[/math].

A transformation of all the data points can be done from [math]\,x[/math] to [math]\,x^*[/math] where [math] \, x^* \leftarrow S_k^{-\frac{1}{2}}U_k^\top x [/math].

A similar transformation of all the centers can be done from [math]\,\mu_k[/math] to [math]\,\mu_k^*[/math] where [math] \, \mu_k^* \leftarrow S_k^{-\frac{1}{2}}U_k^\top \mu_k [/math].

It is now possible to do classification with [math]\,x^*[/math] and [math]\,\mu_k^*[/math], treating them as in Case 1 above. This strategy is correct because by transforming [math]\, x[/math] to [math]\,x^*[/math] where [math] \, x^* \leftarrow S_k^{-\frac{1}{2}}U_k^\top x [/math], the new variable variance is [math]I[/math]

Note that when we have multiple classes, we also need to compute [math]\, log{|\Sigma_k|}[/math] respectively. Then we compute [math] \,\delta_k [/math] for QDA .

Note that when we have multiple classes, they must all have the same transformation, in another word, have same covariance [math]\,\Sigma_k[/math],else, ahead of time we would have to assume a data point belongs to one class or the other. All classes therefore need to have the same shape for classification to be applicable using this method. So this method works for LDA.

If the classes have different shapes, in another word, have different covariance [math]\,\Sigma_k[/math], can we use the same method to transform all data points [math]\,x[/math] to [math]\,x^*[/math]?

The answer is Yes. Consider that you have two classes with different shapes. Given a data point, justify which class this point belongs to. You just do the transformations corresponding to the 2 classes respectively, then you get [math]\,\delta_1 ,\delta_2 [/math] ,then you determine which class the data point belongs to by comparing [math] \,\delta_1 [/math] and [math] \,\delta_2 [/math] .

In summary, to apply QDA on a data set [math]\,X[/math], in the general case where [math]\, \Sigma_k \ne I [/math] for each class [math]\,k[/math], one can proceed as follows:

Step 1: For each class [math]\,k[/math], apply singular value decomposition on [math]\,X_k[/math] to obtain [math]\,S_k[/math] and [math]\,U_k[/math].
Step 2: For each class [math]\,k[/math], transform each [math]\,x[/math] belonging to that class to [math]\,x_k^* = S_k^{-\frac{1}{2}}U_k^\top x[/math], and transform its center [math]\,\mu_k[/math] to [math]\,\mu_k^* = S_k^{-\frac{1}{2}}U_k^\top \mu_k[/math].
Step 3: For each data point [math]\,x \in X[/math], find the squared Euclidean distance between the transformed data point [math]\,x_k^*[/math] and the transformed center [math]\,\mu_k^*[/math] of each class [math]\,k[/math], and assign [math]\,x[/math] to class [math]\,k[/math] such that the squared Euclidean distance between [math]\,x_k^*[/math] and [math]\,\mu_k^*[/math] is the least for all possible [math]\,k[/math]'s.


Now, let us consider LDA. Here, one can derive a classification scheme that is quite similar to that shown above. The main difference is the assumption of a common variance across the classes, so we perform the Singular Value Decomposition once, as opposed to k times.

To apply LDA on a data set [math]\,X[/math], one can proceed as follows:

Step 1: Apply singular value decomposition on [math]\,X[/math] to obtain [math]\,S[/math] and [math]\,U[/math].
Step 2: For each [math]\,x \in X[/math], transform [math]\,x[/math] to [math]\,x^* = S^{-\frac{1}{2}}U^\top x[/math], and transform each center [math]\,\mu[/math] to [math]\,\mu^* = S^{-\frac{1}{2}}U^\top \mu[/math].
Step 3: For each data point [math]\,x \in X[/math], find the squared Euclidean distance between the transformed data point [math]\,x^*[/math] and the transformed center [math]\,\mu^*[/math] of each class, and assign [math]\,x[/math] to the class such that the squared Euclidean distance between [math]\,x^*[/math] and [math]\,\mu^*[/math] is the least over all of the classes.


Kernel QDA In actual data scenarios, it is generally true that QDA will provide a better classifier for the data then LDA because QDA does not assume that the covariance matrix for each class is identical, as LDA assumes. However, QDA still assumes that the class conditional distribution is Gaussian, which is not always the case in real-life scenarios. The link provided at the beginning of this paragraph describes a kernel-based QDA method which does not have the Gaussian distribution assumption.

The Number of Parameters in LDA and QDA

Both LDA and QDA require us to estimate parameters. The more estimation we have to do, the less robust our classification algorithm will be.

LDA: Since we just need to compare the differences between one given class and remaining [math]\,K-1[/math] classes, totally, there are [math]\,K-1[/math] differences. For each of them, [math]\,a^{T}x+b[/math] requires [math]\,d+1[/math] parameters. Therefore, there are [math]\,(K-1)\times(d+1)[/math] parameters.

QDA: For each of the differences, [math]\,x^{T}ax + b^{T}x + c[/math] requires [math]\frac{1}{2}(d+1)\times d + d + 1 = \frac{d(d+3)}{2}+1[/math] parameters. Therefore, there are [math](K-1)(\frac{d(d+3)}{2}+1)[/math] parameters.

A plot of the number of parameters that must be estimated, in terms of (K-1). The x-axis represents the number of dimensions in the data. As is easy to see, QDA is far less robust than LDA for high-dimensional data sets.

More information on Regularized Discriminant Analysis (RDA)

Discriminant analysis (DA) is widely used in classification problems. Except LDA and QDA, there is also an intermediate method between LDA and QDA, a regularized version of discriminant analysis (RDA) proposed by Friedman [1989], and it has been shown to be more flexible in dealing with various class distributions. RDA applies the regularization techniques by using two regularization parameters, which are selected to jointly maximize the classification performance. The optimal pair of parameters is commonly estimated via cross-validation from a set of candidate pairs. More detail about this method can be found in the book by Hastie et al. [2001]. On the other hand, the time of computing last long for high dimensional data, especially when the candidate set is large, which limits the applications of RDA to low dimensional data. In 2006, Ye Jieping and Wang Tie develop a novel algorithm for RDA for high dimensional data. It can estimate the optimal regularization parameters from a large set of parameter candidates efficiently. Experiments on a variety of datasets confirm the claimed theoretical estimate of the efficiency, and also show that, for a properly chosen pair of regularization parameters, RDA performs favourably in classification, in comparison with other existing classification methods. For more details, see Ye, Jieping; Wang, Tie Regularized discriminant analysis for high dimensional, low sample size data Conference on Knowledge Discovery in Data: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining; 20-23 Aug. 2006

Further Reading for Regularized Discriminant Analysis (RDA)

1. Regularized Discriminant Analysis and Reduced-Rank LDA [7]

2. Regularized discriminant analysis for the small sample size in face recognition [8]

3. Regularized Discriminant Analysis and Its Application in Microarrays [9]

Trick: Using LDA to do QDA - September 28, 2010

There is a trick that allows us to use the linear discriminant analysis (LDA) algorithm to generate as its output a quadratic function that can be used to classify data. This trick is similar to, but more primitive than, the Kernel trick that will be discussed later in the course.

Essentially, the trick involves adding one or more new features (i.e. new dimensions) that just contain our original data projected to that dimension. We then do LDA on our new higher-dimensional data. The answer provided by LDA can then be collapsed onto a lower dimension, giving us a quadratic answer.

Motivation

Why would we want to use LDA over QDA? In situations where we have fewer data points, LDA turns out to be more robust.

If we look back at the equations for LDA and QDA, we see that in LDA we must estimate [math]\,\mu_1[/math], [math]\,\mu_2[/math] and [math]\,\Sigma[/math]. In QDA we must estimate all of those, plus another [math]\,\Sigma[/math]; the extra [math]\,\frac{d(d-1)}{2}[/math] estimations make QDA less robust with fewer data points.

Theoretically

Suppose we can estimate some vector [math]\underline{w}^T[/math] such that

[math]y = \underline{w}^T\underline{x}[/math]

where [math]\underline{w}[/math] is a d-dimensional column vector, and [math]\underline{x}\ \epsilon\ \mathbb{R}^d[/math] (vector in d dimensions).

We also have a non-linear function [math]g(x) = y = \underline{x}^Tv\underline{x} + \underline{w}^T\underline{x}[/math] that we cannot estimate.

Using our trick, we create two new vectors, [math]\,\underline{w}^*[/math] and [math]\,\underline{x}^*[/math] such that:

[math]\underline{w}^{*T} = [w_1,w_2,...,w_d,v_1,v_2,...,v_d][/math]

and

[math]\underline{x}^{*T} = [x_1,x_2,...,x_d,{x_1}^2,{x_2}^2,...,{x_d}^2][/math]

We can then estimate a new function, [math]g^*(\underline{x},\underline{x}^2) = y^* = \underline{w}^{*T}\underline{x}^*[/math].

Note that we can do this for any [math]\, x[/math] and in any dimension; we could extend a [math]D \times n[/math] matrix to a quadratic dimension by appending another [math]D \times n[/math] matrix with the original matrix squared, to a cubic dimension with the original matrix cubed, or even with a different function altogether, such as a [math]\,sin(x)[/math] dimension. Pay attention, We don't do QDA with LDA. If we try QDA directly on this problem the resulting decision boundary will be different. Here we try to find a nonlinear boundary for a better possible boundary but it is different with general QDA method. We can call it nonlinear LDA.

By Example

Let's use our trick to do a quadratic analysis of the 2_3 data using LDA.

>> load 2_3;
>> [U, sample] = princomp(X');
>> sample = sample(:,1:2);
We start off the same way, by using PCA to reduce the dimensionality of our data to 2.
>> X_star = zeros(400,4);
>> X_star(:,1:2) = sample(:,:);
>> for i=1:400
     for j=1:2
       X_star(i,j+2) = X_star(i,j)^2;
     end
   end
This projects our sample into two more dimensions by squaring our initial two dimensional data set.
>> group = ones(400,1);
>> group(201:400) = 2;
>> [class, error, POSTERIOR, logp, coeff] = classify(X_star, X_star, group, 'linear');
>> sum (class==group)
ans =
   375
We can now display our results.
>> k = coeff(1,2).const;
>> l = coeff(1,2).linear;
>> f = sprintf('0 = %g+%g*x+%g*y+%g*(x)^2+%g*(y)^2', k, l(1), l(2),l(3),l(4));
>> ezplot(f,[min(sample(:,1)), max(sample(:,1)), min(sample(:,2)), max(sample(:,2))]);
The plot shows the quadratic decision boundary obtained using LDA in the four-dimensional space on the 2_3.mat data. Counting the blue and red points that are on the wrong side of the decision boundary, we can confirm that we have correctly classified 375 data points.
Not only does LDA give us a better result than it did previously, it actually beats QDA, which only correctly classified 371 data points for this data set. Continuing this procedure by adding another two dimensions with [math]x^4[/math] (i.e. we set X_star(i,j+2) = X_star(i,j)^4) we can correctly classify 376 points.

Working Example - Diabetes Data Set

Let's take a look at specific data set. This is a diabetes data set from the UC Irvine Machine Learning Repository. It is a fairly small data set by today's standards. The original data had eight variable dimensions. What I did here was to obtain the two prominent principal components from these eight variables. Instead of using the original eight dimensions we will just use these two principal components for this example.

The Diabetes data set has two types of samples in it. One sample type are healthy individuals the other are individuals with a higher risk of diabetes. Here are the prior probabilities estimated for both of the sample types, first for the healthy individuals and second for those individuals at risk:

eq1.png

The first type has a prior probability estimated at 0.651. This means that among the data set, (250 to 300 data points), about 65% of these belong to class one and the other 35% belong to class two. Next, we computed the mean vector for the two classes separately:eq2.png

Then we computed File:eq3.jpg using the formulas discussed earlier.

Once we have done all of this, we compute the linear discriminant function and found the classification rule. Classification rule:File:eq4.jpg

In this example, if you give me an [math]\, x[/math], I then plug this value into the above linear function. If the result is greater than or equal to zero, I claim that it is in class one. Otherwise, it is in class two. Below is a scatter plot of the dominant principle components. The two classes are represented. The first, without diabetes, is shown with red stars (class 1), and the second class, with diabetes, is shown with blue circles (class 2). The solid line represents the classification boundary obtained by LDA. It appears the two classes are not that well separated. The dashed or dotted line is the boundary obtained by linear regression of indicator matrix. In this case, the results of the two different linear boundaries are very close.

File:eq5.jpg

It is always good practice to visualize the scheme to check for any obvious mistakes.

• Within training data classification error rate: 28.26%. • Sensitivity: 45.90%. • Specificity: 85.60%.

Below is the contour plot for the density of the diabetes data (the marginal density for [math]\, x[/math] is a mixture of two Gaussians, 2 classes). It looks like a single Gaussian distribution. The reason for this is that the two classes are so close together that they merge into a single mode.

File:eq6.jpg

LDA and QDA in Matlab

We have examined the theory behind Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA) above; how do we use these algorithms in practice? Matlab offers us a function called classify that allows us to perform LDA and QDA quickly and easily.

In class, we were shown an example of using LDA and QDA on the 2_3 data that is used in the first assignment. The code below applies LDA to the same data set and reproduces that example, slightly modified, and explains each step.

>> load 2_3;
>> [U, sample] = princomp(X');
>> sample = sample(:,1:2);
First, we do principal component analysis (PCA) on the 2_3 data to reduce the dimensionality of the original data from 64 dimensions to 2. Doing this makes it much easier to visualize the results of the LDA and QDA algorithms.


>> plot (sample(1:200,1), sample(1:200,2), '.');
>> hold on;
>> plot (sample(201:400,1), sample(201:400,2), 'r.');
Recall that in the 2_3 data, the first 200 elements are images of the number two handwritten and the last 200 elements are images of the number three handwritten. This code sets up a plot of the data such that the points that represent a 2 are blue, while the points that represent a 3 are red.
See title and legend for information on adding the title and legend.
Before using classify we can set up a vector that contains the actual labels for our data, to train the classification algorithm. If we don't know the labels for the data, then the element in the group vector should be an empty string or NaN. (See grouping data for more information.)
>> group = ones(400,1);
>> group(201:400) = 2;
We can now classify our data.
>> [class, error, POSTERIOR, logp, coeff] = classify(sample, sample, group, 'linear');
The full details of this line can be examined in the Matlab help file linked above. What we care about are class, which contains the labels that the algorithm thinks that each data point belongs to, and coeff, which contains information about the line that the algorithm created to separate the data into the two classes.
We can see the efficacy of the algorithm by comparing class to group.
>> sum (class==group)
ans =
   369
This compares the value in class to the value in group. The answer of 369 tells us that the algorithm correctly determined the classes of the points 369 times, out of a possible 400 data points. This gives us an empirical error rate of 0.0775.
We can see the line produced by LDA using coeff.
>> k = coeff(1,2).const;
>> l = coeff(1,2).linear;
>> f = sprintf('0 = %g+%g*x+%g*y', k, l(1), l(2));
>> ezplot(f, [min(sample(:,1)), max(sample(:,1)), min(sample(:,2)), max(sample(:,2))]);
Those familiar with the programming language C will find the sprintf line refreshingly familiar; those with no exposure to C are directed to Matlab's sprintf page. Essentially, this code sets up the equation of the line in the form 0 = a + bx + cy. We then use the ezplot function to plot the line.
The 2-3 data after LDA is performed. The line shows where the two classes are split.
Let's perform the same steps, except this time using QDA. The main difference with QDA is a slightly different call to classify, and a more complicated procedure to plot the line.
>> [class, error, POSTERIOR, logp, coeff] = classify(sample, sample, group, 'quadratic');
>> sum (class==group)
ans =
   371
>> k = coeff(1,2).const;
>> l = coeff(1,2).linear;
>> q = coeff(1,2).quadratic;
>> f = sprintf('0 = %g+%g*x+%g*y+%g*x^2+%g*x*y+%g*y^2', k, l(1), l(2), q(1,1), q(1,2)+q(2,1), q(2,2));
>> ezplot(f, [min(sample(:,1)), max(sample(:,1)), min(sample(:,2)), max(sample(:,2))]);
The 2-3 data after QDA is performed. The curved line shows where QDA splits the two classes. Note that QDA is only correct in 2 more data points compared to LDA; we can see a blue point and a red point that lie on the correct side of the curve produced by QDA that do not lie on the correct side of the line produced by LDA.

classify can also be used with other discriminant analysis algorithms. The steps laid out above would only need to be modified slightly for those algorithms.

Recall: An analysis of the function of princomp in matlab.
In our assignment 1, we learned how to perform Principal Component Analysis using the SVD (Singular Value Decomposition) method. In fact, matlab offers a built-in function called princomp which performs PCA. From the matlab help file on princomp, you can find the details about this function. Here we will analyze Matlab's princomp() code. We find something different than the SVD method we used on our first assignment. The following is Matlab's code for princomp followed by some explanations to emphasize some key steps.

   function [pc, score, latent, tsquare] = princomp(x);
   %   PRINCOMP Principal Component Analysis (centered and scaled data).
   %   [PC, SCORE, LATENT, TSQUARE] = PRINCOMP(X) takes a data matrix X and
   %   returns the principal components in PC, the so-called Z-scores in SC
   %   ORES, the eigenvalues of the covariance matrix of X in LATENT,
   %   and Hotelling's T-squared statistic for each data point in TSQUARE.
   %   Reference: J. Edward Jackson, A User's Guide to Principal Components
   %   John Wiley & Sons, Inc. 1991 pp. 1-25.
   %   B. Jones 3-17-94
   %   Copyright 1993-2002 The MathWorks, Inc.
   %   $Revision: 2.9 $  $Date: 2002/01/17 21:31:45 $
   [m,n] = size(x);       %  get the lengh of the rows and columns of matrix x. 
   r = min(m-1,n);        %  max possible rank of X                    
   avg = mean(x);         %  the mean of every column of X
   centerx = (x - avg(ones(m,1),:));     
                          %  centers X by subtracting off column means                 
   [U,latent,pc] = svd(centerx./sqrt(m-1),0);                          
                          %  "economy size" decomposition
   score = centerx*pc;      
                          %  the representation of X in the principal component space
   if nargout < 3
      return;
    end
    latent = diag(latent).^2;
    if (r   latent = [latent(1:r); zeros(n-r,1)];
    score(:,r+1:end) = 0;
    end
    if nargout < 4
    return;
    end
    tmp = sqrt(diag(1./latent(1:r)))*score(:,1:r)';
    tsquare = sum(tmp.*tmp)';

We should compare the following aspects of the above code with the SVD method:

First, Rows of [math]\,X[/math] correspond to observations, columns to variables. When using princomp on 2_3 data in assignment 1, note that we take the transpose of [math]\,X[/math].

 >> load 2_3;
 >> [U, score] = princomp(X');

Second, princomp centers X by subtracting off column means.

The third, when [math]\,X=UdV'[/math], princomp uses [math]\,V[/math] as coefficients for principal components, rather than [math]\,U[/math].

The following is an example to perform PCA using princomp and SVD respectively to get the same result.

SVD method
 >> load 2_3
 >> mn=mean(X,2);
 >> X1=X-repmat(mn,1,400);
 >> [s d v]=svd(X1');
 >> y=X1'*v;
princomp
 >>[U score]=princomp(X');

Then we can see that y=score, v=U.

useful resources: LDA and QDA in Matlab[10],[11],[12]

Reference

1. Harry Zhang. The optimality of naive bayes. FLAIRS Conference. AAAI Press, 2004

2. Rich Caruana and Alexandru N. Mizil. An empirical comparison of supervised learning algorithms. In ICML ’06: Proceedings of the 23rd international conference on Machine learning, pages 161–168, New York, NY, USA, 2006, ACM.

Related links to LDA & QDA

LDA:[13]

[14]

Regularized linear discriminant analysis and its application in microarrays

MATHEMATICAL OPERATIONS OF LDA

Application in face recognition and in market

QDA:[15]

Bayes QDA

LDA & QDA

Using discriminant analysis for multi-class classification: an experimental investigation [16]

Reference articles on solving a small sample size problem when LDA is applied

( Based on Li-Fen Chen, Hong-Yuan Mark Liao, Ming-Tat Ko, Ja-Chen Lin, Gwo-Jong Yu A new LDA-based face recognition system which can solve the small sample size problem Pattern Recognition 33 (2000) 1713-1726 )

Small sample size indicates that the number of samples is smaller than the dimension of each sample. In this case, the within-class covariance we stated in class could be a singular matrix and naturally we cannot find its inverse matrix for further analysis.However, many researchers tried to solve it by different techniques:
1.Goudail et al. proposed a technique which calculated 25 local autocorrelation coefficients from each sample image to achieve dimensionality reduction. (Referenced by F. Goudail, E. Lange, T. Iwamoto, K. Kyuma, N. Otsu, Face recognition system using local autocorrelations and multiscale integration, IEEE Trans. Pattern Anal. Mach. Intell. 18 (10) (1996) 1024-1028.)
2.Swets and Weng applied the PCA approach to accomplish reduction of image dimensionality. (Referenced by D. Swets, J. Weng, Using discriminant eigen features for image retrieval, IEEE Trans. Pattern Anal. Mach. Intell.18 (8) (1996) 831-836.)
3.Fukunaga proposed a more efficient algorithm and calculated eigenvalues and eigenvectors from an m*m matrix, where n is the dimensionality of the samples and m is the rank of the within-class scatter matrix Sw. (Referenced by K. Fukunaga, Introduction to Statistical Pattern Recognition, Academic Press, New York, 1990.)
4.Tian et al. used a positive pseudoinverse matrix instead of calculating the inverse matrix Sw. (Referenced by Q. Tian, M. Barbero, Z.H. Gu, S.H. Lee, Image classification by the Foley-Sammon transform, Opt. Eng. 25 (7) (1986) 834-840.)
5.Hong and Yang tried to add the singular value perturbation in Sw and made Sw a nonsingular matrix. (Referenced by Zi-Quan Hong, Jing-Yu Yang, Optimal discriminant plane for a small number of samples and design method of classifier on the plane, Pattern Recognition 24 (4) (1991) 317-324)
6.Cheng et al. proposed another method based on the principle of rank decomposition of matrices. The above three methods are all based on the conventional Fisher's criterion function. (Referenced by Y.Q. Cheng, Y.M. Zhuang, J.Y. Yang, Optimal fisher discriminant analysis using the rank decomposition, Pattern Recognition 25 (1) (1992) 101-111.)
7.Liu et al. modified the conventional Fisher's criterion function and conducted a number of researches based on the new criterion function. They used the total scatter matrix as the divisor of the original Fisher's function instead of merely using the within-class scatter matrix. (Referenced by K. Liu, Y. Cheng, J. Yang, A generalized optimal set of discriminant vectors, Pattern Recognition 25 (7) (1992) 731-739.)

Principal Component Analysis - September 30, 2010

Brief introduction on dimension reduction method

Dimension reduction is a process to reduce the number of variables of the data by some techniques. Principal components analysis (PCA) and factor analysis are two primary classical methods on dimension reduction. PCA is a method to create some new variables by a linear combination of the variables in the data and the number of new variables depends on what proportion of the variance the new ones contribute. On the contrary, factor analysis method tries to express the old variables by the linear combination of new variables. So before creating the expressions, a certain number of factors should be determined firstly by analysis on the features of old variables. In general, the idea of both PCA and factor analysis is to use as less as possible mixed variables to reflect as more as possible information.

Rough definition

Keepings two important aspects of data analysis in mind:

  • Reducing covariance in data
  • Preserving information stored in data(Variance is a source of information)


Principal component analysis (PCA) is a dimensionality-reduction method invented by Karl Pearson in 1901 [17]. Depending on where this methodology is applied, other common names of PCA include the Karhunen–Loève transform (KLT) , the Hotelling transform, and the proper orthogonal decomposition (POD). PCA is the simplist eigenvector-based multivariate analysis. It reduces the dimensionality of the data by revealing the internal structure of the data in a way that best explains the variance in the data. To this end, PCA works by using a user-defined number of the most important directions of variation (dimensions or principal components) of the data to project the data onto these directions so as to produce a lower-dimensional representation of the original data. The resulting lower-dimensional representation of our data is usually much easier to visualize and it also exhibits the most informative aspects (dimensions) of our data whilst capturing as much of the variation exhibited by our data as it possibly could.


Furthermore, if one considers the lower dimensional representation produced by PCA as a least square fit of our original data, then it can also be easily shown that this representation is the one that minimizes the reconstruction error of our data. It should be noted, however, that one usually does not have control over which dimensions PCA deems to be the most informative for a given set of data, and thus one usually does not know which dimensions PCA should be selected to be the most informative dimensions in order to create the lower-dimensional representation.


Suppose [math]\,X[/math] is our data matrix containing [math]\,d[/math]-dimensional data. The idea behind PCA is to apply singular value decomposition to [math]\,X[/math] to replace the rows of [math]\,X[/math] by a subset of it that captures as much of the variance in [math]\,X[/math] as possible. First, through the application of singular value decomposition to [math]\,X[/math], PCA obtains all of our data's directions of variation. These directions would also be ordered from left to right, with the leftmost directions capturing the most amount of variation in our data and the rightmost directions capturing the least amount. Then, PCA uses a subset of these directions to map our data from its original space to a lower-dimensional space.


By applying singular value decomposition to [math]\,X[/math], [math]\,X[/math] is decomposed as [math]\,X = U\Sigma V^T \,[/math]. The [math]\,d[/math] columns of [math]\,U[/math] are the eigenvectors of [math]\,XX^T \,[/math]. The [math]\,d[/math] columns of [math]\,V[/math] are the eigenvectors of [math]\,X^TX \,[/math]. The [math]\,d[/math] diagonal values of [math]\,\Sigma[/math] are the square roots of the eigenvalues of [math]\,XX^T \,[/math] (also of [math]\,X^TX \,[/math]), and they correspond to the columns of [math]\,U[/math] (also of [math]\,V[/math]).


We are interested in [math]\,U[/math], whose [math]\,d[/math] columns are the [math]\,d[/math] directions of variation of our data. Ordered from left to right, the [math]\,ith[/math] column of [math]\,U[/math] is the [math]\,ith[/math] most informative direction of variation of our data. That is, the [math]\,ith[/math] column of [math]\,U[/math] is the [math]\,ith[/math] most effective column in terms of capturing the total variance exhibited by our data. A subset of the columns of [math]\,U[/math] is used by PCA to reduce the dimensionality of [math]\,X[/math] by projecting [math]\,X[/math] onto the columns of this subset. In practice, when we apply PCA to [math]\,X[/math] to reduce the dimensionality of [math]\,X[/math] from [math]\,d[/math] to [math]\,k[/math], where [math]k \lt d\,[/math], we would proceed as follows:

Step 1: Center [math]\,X[/math] so that it would have zero mean.
Step 2: Apply singular value decomposition to [math]\,X[/math] to obtain [math]\,U[/math].
Step 3: Suppose we denote the resulting [math]\,k[/math]-dimensional representation of [math]\,X[/math] by [math]\,Y[/math]. Then, [math]\,Y[/math] is obtained as [math]\,Y = U_k^TX[/math]. Here, [math]\,U_k[/math] consists of the first (leftmost) [math]\,k[/math] columns of [math]\,U[/math] that correspond to the [math]\,k[/math] largest diagonal elements of [math]\,\Sigma[/math].


PCA takes a sample of [math]\, d[/math] - dimensional vectors and produces an orthogonal(zero covariance) set of [math]\, d[/math] 'Principal Components'. The first Principal Component is the direction of greatest variance in the sample. The second principal component is the direction of second greatest variance (orthogonal to the first component), etc.

Then we can preserve most of the variance in the sample in a lower dimension by choosing the first [math]\, k[/math] Principle Components and approximating the data in [math]\, k[/math] - dimensional space, which is easier to analyze and plot.

Principal Components of handwritten digits

Suppose that we have a set of 130 images (28 by 23 pixels) of handwritten threes.


We can represent each image as a vector of length 644 ([math]644 = 23 \times 28[/math]). Then we can represent the entire data set as a 644 by 130 matrix, shown below. Each column represents one image (644 rows = 644 pixels).

File:matrix decomp PCA.png

Using PCA, we can approximate the data as the product of two smaller matrices, which I will call [math]V \in M_{644,2}[/math] and [math]W \in M_{2,103}[/math]. If we expand the matrix product then each image is approximated by a linear combination of the columns of V: [math] \hat{f}(\lambda) = \bar{x} + \lambda_1 v_1 + \lambda_2 v_2 [/math], where [math]\lambda = [\lambda_1, \lambda_2]^T[/math] is a column of W.

File:linear comb PCA.png

To demonstrate this process, we can compare the images of 2s and 3s. We will apply PCA to the data, and compare the images of the labeled data. This is an example in classifying.

Don't worry about the constant term for now. The point is that we can represent an image using just 2 coefficients instead of 644. Also notice that the coefficients correspond to features of the handwritten digits. The picture below shows the first two principal components for the set of handwritten threes.

23plotPCA.jpg

The first coefficient represents the width of the entire digit, and the second coefficient represents the slant of each handwritten digit.

Derivation of the first Principle Component

For finding the direction of maximum variation, Let [math]\begin{align}\textbf{w}\end{align}[/math] be an arbitrary direction, [math]\begin{align}\textbf{x}\end{align}[/math] a data point, and [math]\begin{align}\displaystyle u\end{align}[/math] be the length of the projection of [math]\begin{align}\textbf{x}\end{align}[/math] in the direction [math]\begin{align}\textbf{w}\end{align}[/math].

[math]\begin{align} \textbf{w} &= [w_1, \ldots, w_D]^T \\ \textbf{x} &= [x_1, \ldots, x_D]^T \\ u &= \frac{\textbf{w}^T \textbf{x}}{\sqrt{\textbf{w}^T\textbf{w}}} \end{align} [/math]

The direction [math]\begin{align}\textbf{w}\end{align}[/math] is the same as [math]\begin{align}c\textbf{w}\end{align}[/math], for any scalar [math]c[/math], so without loss of generality we assume that:

[math] \begin{align} |\textbf{w}| &= \sqrt{\textbf{w}^T\textbf{w}} = 1 \\ u &= \textbf{w}^T \textbf{x}. \end{align} [/math]

Let [math]x_1, \ldots, x_D[/math] be random variables, then we set our goal as to maximize the variance of [math]u[/math],

[math] \textrm{var}(u) = \textrm{var}(\textbf{w}^T \textbf{x}) = \textbf{w}^T \Sigma \textbf{w}. [/math]

For a finite data set we replace the covariance matrix [math]\Sigma[/math] by [math]s[/math]. The sample covariance matrix

[math]\textrm{var}(u) = \textbf{w}^T s\textbf{w} .[/math]

The above mentioned variable is the variance of [math]\begin{align}\displaystyle u \end{align}[/math] formed by the weight vector [math]\begin{align}\textbf{w} \end{align}[/math]. The first principal component is the vector [math]\begin{align}\textbf{w} \end{align}[/math] that maximizes the variance,

[math] \textrm{PC} = \underset{\textbf{w}}{\operatorname{arg\,max}} \, \left( \operatorname{var}(u) \right) = \underset{\textbf{w}}{\operatorname{arg\,max}} \, \left( \textbf{w}^T s \textbf{w} \right) [/math]

where arg max denotes the value of [math]\begin{align}\textbf{w} \end{align}[/math] that maximizes the function. Our goal is to find the weight [math]\begin{align}\textbf{w} \end{align}[/math] that maximizes this variability, subject to a constraint. Since our function is convex, it has no maximum value. Therefore we need to add a constraint that restricts the length of [math]\begin{align}\textbf{w} \end{align}[/math]. However, we are only interested in the direction of the variability, so the problem becomes

[math] \underset{\textbf{w}}{\operatorname{max}} \, \left( \textbf{w}^T s \textbf{w} \right) [/math]

s.t. [math]\textbf{w}^T \textbf{w} = 1.[/math]

Notice,

[math] \textbf{w}^T s \textbf{w} \leq \| \textbf{w}^T s \textbf{w} \| \leq \| s \| \| \textbf{w} \| = \| s \|. [/math]

Therefore the variance is bounded, so the maximum exists. We find the this maximum using the method of Lagrange multipliers.

Lagrange Multiplier

Before we can proceed, we must review Lagrange multipliers.

"The red line shows the constraint g(x,y) = c. The blue lines are contours of f(x,y). The point where the red line tangentially touches a blue contour is our solution." [Lagrange Multipliers, Wikipedia]

To find the maximum (or minimum) of a function [math]\displaystyle f(x,y)[/math] subject to constraints [math]\displaystyle g(x,y) = 0 [/math], we define a new variable [math]\displaystyle \lambda[/math] called a Lagrange Multiplier and we form the Lagrangian,

[math]\displaystyle L(x,y,\lambda) = f(x,y) - \lambda g(x,y)[/math]

If [math]\displaystyle (x^*,y^*)[/math] is the max of [math]\displaystyle f(x,y)[/math], there exists [math]\displaystyle \lambda^*[/math] such that [math]\displaystyle (x^*,y^*,\lambda^*) [/math] is a stationary point of [math]\displaystyle L[/math] (partial derivatives are 0).
In addition [math]\displaystyle (x^*,y^*)[/math] is a point in which functions [math]\displaystyle f[/math] and [math]\displaystyle g[/math] touch but do not cross. At this point, the tangents of [math]\displaystyle f[/math] and [math]\displaystyle g[/math] are parallel or gradients of [math]\displaystyle f[/math] and [math]\displaystyle g[/math] are parallel, such that:

[math]\displaystyle \nabla_{x,y } f = \lambda \nabla_{x,y } g[/math]

where,
[math]\displaystyle \nabla_{x,y} f = (\frac{\partial f}{\partial x},\frac{\partial f}{\partial{y}}) \leftarrow[/math] the gradient of [math]\, f[/math]
[math]\displaystyle \nabla_{x,y} g = (\frac{\partial g}{\partial{x}},\frac{\partial{g}}{\partial{y}}) \leftarrow[/math] the gradient of [math]\, g [/math]

Example

Suppose we wish to maximize the function [math]\displaystyle f(x,y)=x-y[/math] subject to the constraint [math]\displaystyle x^{2}+y^{2}=1[/math]. We can apply the Lagrange multiplier method on this example; the lagrangian is:

[math]\displaystyle L(x,y,\lambda) = x-y - \lambda (x^{2}+y^{2}-1)[/math]

We want the partial derivatives equal to zero:


[math]\displaystyle \frac{\partial L}{\partial x}=1+2 \lambda x=0 [/math]

[math]\displaystyle \frac{\partial L}{\partial y}=-1+2\lambda y=0[/math]

[math]\displaystyle \frac{\partial L}{\partial \lambda}=x^2+y^2-1[/math]

Solving the system we obtain 2 stationary points: [math]\displaystyle (\sqrt{2}/2,-\sqrt{2}/2)[/math] and [math]\displaystyle (-\sqrt{2}/2,\sqrt{2}/2)[/math]. In order to understand which one is the maximum, we just need to substitute it in [math]\displaystyle f(x,y)[/math] and see which one as the biggest value. In this case the maximum is [math]\displaystyle (\sqrt{2}/2,-\sqrt{2}/2)[/math].

Determining W

Back to the original problem, from the Lagrangian we obtain,

[math]\displaystyle L(\textbf{w},\lambda) = \textbf{w}^T S \textbf{w} - \lambda (\textbf{w}^T \textbf{w} - 1)[/math]

If [math] \textbf{w}^T \textbf{w} [/math] is a unit vector then the second part of the equation is 0.

If [math] \textbf{w}^T \textbf{w} [/math] is not a unit vector then the second part of the equation increases. Thus decreasing overall [math]\displaystyle L(\textbf{w},\lambda)[/math]. Maximization happens when [math] \textbf{w}^T \textbf{w} =1 [/math]


(Note that to take the derivative with respect to w below, [math] \textbf{w}^T S \textbf{w} [/math] can be thought of as a quadratic function of w, hence the 2sw below. For more matrix derivatives, see section 2 of the Matrix Cookbook)

Taking the derivative with respect to w, we get:

[math]\displaystyle \frac{\partial L}{\partial \textbf{w}} = 2S\textbf{w} - 2\lambda\textbf{w} [/math]

Set [math] \displaystyle \frac{\partial L}{\partial \textbf{w}} = 0 [/math], we get

[math]\displaystyle S\textbf{w}^* = \lambda^*\textbf{w}^* [/math]

From the eigenvalue equation [math]\, \textbf{w}^*[/math] is an eigenvector of S and [math]\, \lambda^*[/math] is the corresponding eigenvalue of S. If we substitute [math]\displaystyle\textbf{w}^*[/math] in [math]\displaystyle \textbf{w}^T S\textbf{w}[/math] we obtain,

[math]\displaystyle\textbf{w}^{*T} S\textbf{w}^* = \textbf{w}^{*T} \lambda^* \textbf{w}^* = \lambda^* \textbf{w}^{*T} \textbf{w}^* = \lambda^* [/math]

In order to maximize the objective function we choose the eigenvector corresponding to the largest eigenvalue. We choose the first PC, u1 to have the maximum variance
(i.e. capturing as much variability in [math]\displaystyle x_1, x_2,...,x_D [/math] as possible.) Subsequent principal components will take up successively smaller parts of the total variability.

D dimensional data will have D eigenvectors

[math]\lambda_1 \geq \lambda_2 \geq ... \geq \lambda_D [/math] where each [math]\, \lambda_i[/math] represents the amount of variation in direction [math]\, i [/math]

so that

[math]Var(u_1) \geq Var(u_2) \geq ... \geq Var(u_D)[/math]

If two eigenvalues happen to be equal, then the data has the same amount of variation in each of the two directions that they correspond to with. If only one of the two equal eigenvalues are to be chosen for dimensionality reduction, then either will do. Note that if ALL of the eigenvalues are the same then this means that the data is on the surface of a d-dimensional sphere (all directions have the same amount of variation).

Note that the Principal Components decompose the total variance in the data:

[math]\displaystyle \sum_{i = 1}^D Var(u_i) = \sum_{i = 1}^D \lambda_i = Tr(S) = Var(\sum_{i = 1}^n x_i)[/math]

i.e. the sum of variations in all directions is the variation in the whole data

Example from class

We apply PCA to the noise data, making the assumption that the intrinsic dimensionality of the data is 10. We now try to compute the reconstructed images using the top 10 eigenvectors and plot the original and reconstructed images

The Matlab code is as follows:

 load noisy
 who
 size(X)
 imagesc(reshape(X(:,1),20,28)')
 colormap gray
 imagesc(reshape(X(:,1),20,28)')
 m_X=mean(X,2);
 mm=repmat(m_X,1,300);
 XX=X-mm;
 [u s v] = svd(XX);
 xHat = u(:,1:10)*s(1:10,1:10)*v(:,1:10)'; % use ten principal components
 xHat=xHat+mm;
 figure
 imagesc(reshape(xHat(:,1000),20,28)') % here '1000' can be changed to different values, e.g. 105, 500, etc.
 colormap gray

Running the above code gives us 2 images - the first one represents the noisy data - we can barely make out the face

The second one is the denoised image



As you can clearly see, more features can be distinguished from the picture of the de-noised face compared to the picture of the noisy face. This is because almost all of the noise in the noisy image is captured by the principal components (directions of variation) that capture the least amount of variation in the image, and these principal components were discarded when we used the few principal components that capture most of the image's variation to generate the image's lower-dimensional representation. If we took more principal components, at first the image would improve since the intrinsic dimensionality is probably more than 10. But if you include all the components you get the noisy image, so not all of the principal components improve the image. In general, it is difficult to choose the optimal number of components.

Application of PCA - Feature Extraction

PCA, depending on the field of application, it is also named the discrete Karhunen–Loève transform (KLT), the Hotelling transform or proper orthogonal decomposition (POD). One of the applications of PCA is to group similar data (e.g. images). There are generally two methods to do this. We can classify the data (i.e. give each data a label and compare different types of data) or cluster (i.e. do not label the data and compare output for classes).

Generally speaking, we can do this with the entire data set (if we have an 8X8 picture, we can use all 64 pixels). However, this is hard, and it is easier to use the reduced data and features of the data.

General PCA Algorithm

The PCA Algorithm is summarized as follows (taken from the Lecture Slides).

Algorithm

Recover basis: Calculate [math] XX^T =\Sigma_{i=1}^{n} x_i x_{i}^{T} [/math] and let [math] U=[/math] eigenvectors of [math] X X^T [/math] corresponding to the top [math] d [/math] eigenvalues.

Encoding training data: Let [math]Y=U^TX [/math] where [math]Y[/math] is a [math]d \times n[/math] matrix of encoding of the original data.

Reconstructing training data: [math]\hat{X}= UY=UU^TX [/math].

Encode set example: [math] y=U^T x [/math] where [math] y [/math] is a [math]d-[/math]dimentional encoding of [math]x[/math].

Reconstruct test example: [math]\hat{x}= Uy=UU^Tx [/math].


Other Notes:

  1. The mean of the data(X) must be 0. This means we may have to preprocess the data by subtracting off the mean(see detailsPCA in Wikipedia.)
  2. Encoding the data means that we are projecting the data onto a lower dimensional subspace by taking the inner product. Encoding: [math]X_{D\times n} \longrightarrow Y_{d\times n}[/math] using mapping [math]\, U^T X_{D \times n} [/math].
  3. When we reconstruct the training set, we are only using the top d dimensions.This will eliminate the dimensions that have lower variance (e.g. noise). Reconstructing: [math] \hat{X}_{D\times n}\longleftarrow Y_{d \times n}[/math] using mapping [math]\, U_dY_{d \times n} [/math], where [math]\,U_d[/math] contains the first (leftmost) [math]\,d[/math] columns of [math]\,U[/math].
  4. We can compare the reconstructed test sample to the reconstructed training sample to classify the new data.


Feature Extraction Uses and Discussion

PCA, as well as other feature extraction methods not within the scope of the course [18] are used as a first step to classification in enhancing generalization capability: one of the classification aspects that will be discussed later in the course is model complexity. As a classification model becomes more complex over its training set, classification error over test data tends to increase. By performing feature extraction prior to attempting classification, we restrict model inputs to only the most important variables, thus decreasing complexity and potentially improving test results.

Feature selection methods, that are used to select subsets of relevant features for building robust learning models, differ from extraction methods, where features are transformed. Feature selection has the added benefit of improving model interpretability.


Independent Component Analysis

As we have already seen, the Principal Component Analysis (PCA) performed by the Karhunen-Lokve transform produces features [math]\ y ( i ) ; i = 0, 1, . . . , N - 1[/math], that are mutually uncorrelated. The obtained by the KL transform solution is optimal when dimensionality reduction is the goal and one wishes to minimize the approximation mean square error. However, for certain applications, the obtained solution falls short of the expectations. In contrast, the more recently developed Independent Component Analysis (ICA) theory, tries to achieve much more than simple decorrelation of the data. The ICA task is casted as follows: Given the set of input samples [math]\ x[/math], determine an [math]\ N \times N[/math] invertible matrix [math]\ W[/math] such that the entries [math]\ y(i), i = 0, 1, . . . , N - 1[/math], of the transformed vector

[math]\ y = W.x[/math]

are mutually independent. The goal of statistical independence is a stronger condition than the uncorrelatedness required by the PCA. The two conditions are equivalent only for Gaussian random variables. Searching for independent rather than uncorrelated features gives us the means of exploiting a lot more of information, hidden in the higher order statistics of the data.

This topic has brought to you from Pattern Recognition by Sergios Theodoridis and Konstantinos Koutroumbas. (Chapter 6) For further details on the ICA and its varieties, refer to this book.

References

1. Probabilistic Principal Component Analysis [19]

2. Nonlinear Component Analysis as a Kernel Eigenvalue Problem [20]

3. Kernel principal component analysis [21]

4. Principal Component Analysis [22] and [23]

Further Readings

1. I. T. Jolliffe "Principal component analysis" Available here.

2. James V. Stone "Independent component analysis: a tutorial introduction" Available here.

3. Aapo Hyvärinen, Juha Karhunen, Erkki Oja "Independent component analysis" Available here.

Fisher's (Linear) Discriminant Analysis (FDA) - Two Class Problem - October 5, 2010

Sir Ronald A. Fisher

Fisher's Discriminant Analysis (FDA), also known as Fisher's Linear Discriminant Analysis (LDA) in some sources, is a classical feature extraction technique. It was originally described in 1936 by Sir Ronald Aylmer Fisher, an English statistician and eugenicist who has been described as one of the founders of modern statistical science. His original paper describing FDA can be found here; a Wikipedia article summarizing the algorithm can be found here. In this paper Fisher used for the first time the term DISCRIMINANT FUNCTION. The term DISCRIMINANT ANALYSIS was introduced later by Fisher himself in a subsequent paper which can be found here.

Introduction

Linear discriminant analysis (LDA) and the related Fisher's linear discriminant are methods used in statistics, pattern recognition and machine learning to find a linear combination of features which characterize or separate two or more classes of objects or events. The resulting combination may be used as a linear classifier, or, more commonly, for dimensionality reduction before later classification.

LDA is also closely related to principal component analysis (PCA) and factor analysis in that both look for linear combinations of variables which best explain the data. LDA explicitly attempts to model the difference between the classes of data. PCA on the other hand does not take into account any difference in class, and factor analysis builds the feature combinations based on differences rather than similarities. Discriminant analysis is also different from factor analysis in that it is not an interdependence technique: a distinction between independent variables and dependent variables (also called criterion variables) must be made.

LDA works when the measurements made on independent variables for each observation are continuous quantities. When dealing with categorical independent variables, the equivalent technique is discriminant correspondence analysis.

Contrasting FDA with PCA

As in PCA, the goal of FDA is to project the data in a lower dimension. You might ask, why was FDA invented when PCA already existed? There is a simple explanation for this that can be found here. PCA is an unsupervised method for classification, so it does not take into account the labels in the data. Suppose we have two clusters that have very different or even opposite labels from each other but are nevertheless positioned in a way such that they are very much parallel to each other and also very near to each other. In this case, most of the total variation of the data is in the direction of these two clusters. If we use PCA in cases like this, then both clusters would be projected onto the direction of greatest variation of the data to become sort of like a single cluster after projection. PCA would therefore mix up these two clusters that, in fact, have very different labels. What we need to do instead, in this cases like this, is to project the data onto a direction that is orthogonal to the direction of greatest variation of the data. This direction is in the least variation of the data. On the 1-dimensional space resulting from such a projection, we would then be able to effectively classify the data, because these two clusters would be perfectly or nearly perfectly separated from each other taking into account of their labels. This is exactly the idea behind FDA.

The main difference between FDA and PCA is that, in FDA, in contrast to PCA, we are not interested in retaining as much of the variance of our original data as possible. Rather, in FDA, our goal is to find a direction that is useful for classifying the data (i.e. in this case, we are looking for a direction that is most representative of a particular characteristic e.g. glasses vs. no-glasses). Suppose we have 2-dimensional data, then FDA would attempt to project the data of each class onto a point in such a way that the resulting two points would be as far apart from each other as possible. Intuitively, this basic idea behind FDA is the optimal way for separating each pair of classes along a certain direction.

Please note dimention reduction in PCA is different from subspace cluster , see the details about the subspace cluser [24] {{

 Template:namespace detect

| type = style | image = | imageright = | style = | textstyle = | text = This article may require cleanup to meet Wikicoursenote's quality standards. The specific problem is: Just a thought: how relevant is "Dimensionality reduction techniques" to the concept of "subspace clustering"? As in subspace clustering, the goal is to find a set of features (relevant features, the concept is referred to as local feature relevance in the literature) in the high dimensional space, where potential subspaces accommodating different classes of data points can be defined. This means; the data points are dense when they are considered in a subset of dimensions (features).. Please improve this article if you can. (October 2010) | small = | smallimage = | smallimageright = | smalltext = }} {{

 Template:namespace detect

| type = style | image = | imageright = | style = | textstyle = | text = This article may require cleanup to meet Wikicoursenote's quality standards. The specific problem is: If I'm not mistaken, classification techniques like FDA use labeled training data whereas clustering techniques use unlabeled training data instead. Any other input regarding this would be much appreciated. Thanks. Please improve this article if you can. (October 2010) | small = | smallimage = | smallimageright = | smalltext = }} {{

 Template:namespace detect

| type = style | image = | imageright = | style = | textstyle = | text = This article may require cleanup to meet Wikicoursenote's quality standards. The specific problem is: An extension of clustering is subspace clustering in which different subspace are searched through to find the relavant and appropriate dimentions. High dimentional data sets are roughly equiedistant from each other, so feature selection methods are used to remove the irrelavant dimentions. These techniques do not keep the relative distance so PCA is not useful for these applications. It should be noted that subspace clustering localize their search unlike feature selection algorithms.for more information click here[25]. Please improve this article if you can. (October 2010) | small = | smallimage = | smallimageright = | smalltext = }}

The number of dimensions that we want to reduce the data to depends on the number of classes:
For a 2-classes problem, we want to reduce the data to one dimension (a line), [math]\displaystyle Z \in \mathbb{R}^{1}[/math]
Generally, for a k-classes problem, we want to reduce the data to k-1 dimensions, [math]\displaystyle Z \in \mathbb{R}^{k-1}[/math]

As we will see from our objective function, we want to maximize the separation of the classes and to minimize the within-variance of each class. That is, our ideal situation is that the individual classes are as far away from each other as possible, and at the same time the data within each class are as close to each other as possible (collapsed to a single point in the most extreme case).

The following diagram summarizes this goal.

FDA.JPG

In fact, the two examples above may represent the same data projected on two different lines.

FDAtwo.PNG

Distance Metric Learning VS FDA

In many fundamental machine learning problems, the Euclidean distances between data points do not represent the desired topology that we are trying to capture. Kernel methods address this problem by mapping the points into new spaces where Euclidean distances may be more useful. An alternative approach is to construct a Mahalanobis distance (quadratic Gaussian metric) over the input space and use it in place of Euclidean distances. This approach can be equivalently interpreted as a linear transformation of the original inputs, followed by Euclidean distance in the projected space. This approach has attracted a lot of recent interest.

Some of the proposed algorithms are iterative and computationally expensive. In the paper,"Distance Metric Learning VS FDA " written by our instructor, they propose a closed-form solution to one algorithm that previously required expensive semidefinite optimization. They provide a new problem setup in which the algorithm performs better or as well as some standard methods, but without the computational complexity. Furthermore, they show a strong relationship between these methods and the Fisher Discriminant Analysis (FDA). They also extend the approach by kernelizing it, allowing for non-linear transformations of the metric.

Example

In the paper "Distance Metric Learning VS FDA ", classification error rate for three of the six UCI datasets, each learned metric is projected onto a lowdimensional subspace, shown along the x axis are shown as below.

Example.png,Example3.png

FDA Goals

An intuitive description of FDA can be given by visualizing two clouds of data, as shown above. Ideally, we would like to collapse all of the data points in each cloud onto one point on some projected line, then make those two points as far apart as possible. In doing so, we make it very easy to tell which class a data point belongs to. In practice, it is not possible to collapse all of the points in a cloud to one point, but we attempt to make all of the points in a cloud close to each other while simultaneously far from the points in the other cloud.

Example in R

PCA and FDA primary dimension for normal multivariate data, using R.
>> X = matrix(nrow=400,ncol=2)
>> X[1:200,] = mvrnorm(n=200,mu=c(1,1),Sigma=matrix(c(1,1.5,1.5,3),2))
>> X[201:400,] = mvrnorm(n=200,mu=c(5,3),Sigma=matrix(c(1,1.5,1.5,3),2))
>> Y = c(rep("red",200),rep("blue",200))
Create 2 multivariate normal random variables with [math]\, \mu_1 = \left( \begin{array}{c}1 \\ 1 \end{array} \right), \mu_2 = \left( \begin{array}{c}5 \\ 3 \end{array} \right). ~\textrm{Cov} = \left( \begin{array}{cc} 1 & 1.5 \\ 1.5 & 3 \end{array} \right)[/math]. Create Y, an index indicating which class they belong to.
>> s <- svd(X,nu=1,nv=1)
Calculate the singular value decomposition of X. The most significant direction is in s$v[,1], and is displayed as a black line.
>> s2 <- lda(X,grouping=Y)
The lda function, given the group for each item, uses Fischer's Linear Discriminant Analysis (FLDA) to find the most discriminant direction. This can be found in s2$scaling.

Now that we've calculated the PCA and FLDA decompositions, we create a plot to demonstrate the differences between the two algorithms. FLDA is clearly better suited to discriminating between two classes whereas PCA is primarily good for reducing the number of dimensions when data is high-dimensional.

>> plot(X,col=Y,main="PCA vs. FDA example")
Plot the set of points, according to colours given in Y.
>> slope = s$v[2]/s$v[1]
>> intercept = mean(X[,2])-slope*mean(X[,1])
>> abline(a=intercept,b=slope)
Plot the main PCA direction, drawn through the mean of the dataset. Only the direction is significant.
>> slope2 = s2$scaling[2]/s2$scaling[1]
>> intercept2 = mean(X[,2])-slope2*mean(X[,1])
>> abline(a=intercept2,b=slope2,col="red")
Plot the FLDA direction, again through the mean.
>> legend(-2,7,legend=c("PCA","FDA"),col=c("black","red"),lty=1)
Labeling the lines directly on the graph makes it easier to interpret.


FDA projects the data into lower dimensional space, where the distances between the projected means are maximum and the within-class variances are minimum. There are two categories of classification problems:

1. Two-class problem

2. Multi-class problem (addressed next lecture)

Two-class problem

In the two-class problem, we have prior knowledge that the data points belong to two classes. Conceptually, points of each class form a cloud around the class mean, and each class has an distinct size. To divide points among the two classes, we must determine the class whose mean is closest to each point, and we must also account for the different size of each class given by the covariance of each class.

Assume [math]\underline{\mu_{1}}=\frac{1}{n_{1}}\displaystyle\sum_{i:y_{i}=1}\underline{x_{i}}[/math] and [math]\displaystyle\Sigma_{1}[/math], represent the mean and covariance of the 1st class, and [math]\underline{\mu_{2}}=\frac{1}{n_{2}}\displaystyle\sum_{i:y_{i}=2}\underline{x_{i}}[/math] and [math]\displaystyle\Sigma_{2}[/math] represent the mean and covariance of the 2nd class. We have to find a transformation which satisfies the following goals:

1.To make the means of these two classes as far apart as possible

In other words, the goal is to maximize the distance after projection between class 1 and class 2. This can be done by maximizing the distance between the means of the classes after projection. When projecting the data points to a one-dimensional space, all points will be projected to a single line; the line we seek is the one with the direction that achieves maximum separation of classes upon projection. If the original points are [math]\underline{x_{i}} \in \mathbb{R}^{d}[/math]and the projected points are [math]\underline{w}^T \underline{x_{i}}[/math] then the mean of the projected points will be [math]\underline{w}^T \underline{\mu_{1}}[/math] and [math]\underline{w}^T \underline{\mu_{2}}[/math] for class 1 and class 2 respectively. The goal now becomes to maximize the Euclidean distance between projected means, [math](\underline{w}^T\underline{\mu_{1}}-\underline{w}^T\underline{\mu_{2}})^T (\underline{w}^T\underline{\mu_{1}}-\underline{w}^T\underline{\mu_{2}})[/math]. The steps of this maximization are given below.

2.We want to collapse all data points of each class to a single point, i.e., minimize the covariance within each class

Notice that the variance of the projected classes 1 and 2 are given by [math]\underline{w}^T\Sigma_{1}\underline{w}[/math] and [math]\underline{w}^T\Sigma_{2}\underline{w}[/math]. The second goal is to minimize the sum of these two covariances (the summation of the two covariances is a valid covariance, satisfying the symmetry and positive semi-definite criteria).

{{

 Template:namespace detect

| type = style | image = | imageright = | style = | textstyle = | text = This article may require cleanup to meet Wikicoursenote's quality standards. The specific problem is: In 2. above, I wonder if the computation would be much more complex if we instead find a weighted sum of the covariances of the two classes where the weights are the sizes of the two classes?. Please improve this article if you can. (October 2010) | small = | smallimage = | smallimageright = | smalltext = }}

{{

 Template:namespace detect

| type = style | image = | imageright = | style = | textstyle = | text = This article may require cleanup to meet Wikicoursenote's quality standards. The specific problem is: If using the weighted sum of two covariances, you will need to use the shared mean of the two classes, and the weighted sum will be the shared covariance. Doing this will result in collapsing the two classes into one point, which contradicts the purpose of using FDA. Please improve this article if you can. (December 2010) | small = | smallimage = | smallimageright = | smalltext = }}

As is demonstrated below, both of these goals can be accomplished simultaneously.

Original points are [math]\underline{x_{i}} \in \mathbb{R}^{d}[/math]
[math]\ \{ \underline x_1 \underline x_2 \cdot \cdot \cdot \underline x_n \} [/math]


Projected points are [math]\underline{z_{i}} \in \mathbb{R}^{1}[/math] with [math]\underline{z_{i}} = \underline{w}^T \cdot\underline{x_{i}}[/math] where [math]\ z_i [/math] is a scalar

1. Minimizing within-class variance

[math]\displaystyle \min_w (\underline{w}^T\sum_1\underline{w}) [/math]

[math]\displaystyle \min_w (\underline{w}^T\sum_2\underline{w}) [/math]

and this problem reduces to [math]\displaystyle \min_w (\underline{w}^T(\sum_1 + \sum_2)\underline{w})[/math]
(where [math]\,\sum_1[/math] and [math]\,\sum_2 [/math] are the covariance matrices of the 1st and 2nd classes of data, respectively)

Let [math]\displaystyle \ s_w=\sum_1 + \sum_2[/math] be the within-classes covariance. Then, this problem can be rewritten as [math]\displaystyle \min_w (\underline{w}^Ts_w\underline{w})[/math].

2. Maximize the distance between the means of the projected data



[math]\displaystyle \max_w ||\underline{w}^T \mu_1 - \underline{w}^T \mu_2||^2, [/math]

[math]\begin{align} ||\underline{w}^T \mu_1 - \underline{w}^T \mu_2||^2 &= (\underline{w}^T \mu_1 - \underline{w}^T \mu_2)^T(\underline{w}^T \mu_1 - \underline{w}^T \mu_2)\\ &= (\mu_1^T\underline{w} - \mu_2^T\underline{w})(\underline{w}^T \mu_1 - \underline{w}^T \mu_2)\\ &= (\mu_1 - \mu_2)^T \underline{w} \underline{w}^T (\mu_1 - \mu_2) \\ &= ((\mu_1 - \mu_2)^T \underline{w})^{T} (\underline{w}^T (\mu_1 - \mu_2))^{T} \\ &= \un