stat841f14
Contents
 1 Editor Sign Up
 2 Principal components analysis (Lecture 1: Sept. 10, 2014)
 3 PCA continued, Lagrange multipliers, singular value decomposition (SVD) (Lecture 2: Sept. 15, 2014)
 4 PCA Algorithm, Dual PCA, and Kernel PCA (Lecture 3: Sept. 17, 2014)
 5 Centering for kernel PCA and multidimensional scaling (Lecture 4: Sept. 22, 2014)
 6 ISOMAP and LLE (Lecture 5: Sept. 29, 2014)
 7 LLE continued, introduction to maximum variance unfolding (MVU) (Lecture 6: Oct. 1, 2014)
 8 Action Respecting Embedding and Spectral Clustering (Lecture 7: Oct. 6, 2014)
 9 Spectral Clustering (Lecture 8: Oct. 8, 2014)
 10 Supervised Principal Component Analysis (Lecture 9: Oct. 15, 2014)
 11 Hilbert Schmidt Independence Criterion (Lecture 10: Oct. 20, 2014)
 12 SPCA continued and Metric Learning(Lecture 12: Oct. 22, 2014)
 12.1 SPCA continued
 12.2 Metric Learning
 13 Nonnegative Matrix Factorization Via Rankone Downdate(NMF) (Lecture 13: Oct 27, 2014)
 14 Text/Image Biclustering, Nyström Approximation, and Clustering (Lecture 14: Oct 29, 2014)
 15 Clustering (Lecture 15: Nov 3, 2014)
 16 EM algorithm and Mixture Model (Cont) (Lecture 16: Nov 5, 2014)
 17 EM algorithm and Mixture Model (Cont) (Lecture 17: Nov 10, 2014)
 18 Feature Selection (Lecture 18: Nov. 12, 2014)
 18.1 Introduction
 18.2 Mixture of Gaussian continued
 18.3 Definitions and Relation to Previous Works
 18.4 Unsupervised Feature Selection
 18.5 Recursive Selection Criterion
 18.6 Greedy Selection Criterion
 18.7 Conclusion
 18.8 Supervised Feature Selection
 18.9 Gap statistics
 18.10 Affinity Propagation
 19 Learning Affine Transformations for Nonlinear Dimensionality Recuction (Lecture 19: Nov. 27, 2014)
Editor Sign Up
Principal components analysis (Lecture 1: Sept. 10, 2014)
Introduction
Principal Component Analysis (PCA), first invented by Karl Pearson in 1901, is a statistical technique for data analysis. Its main purpose is to reduce the dimensionality of the data.
Suppose there is a set of data points in a ddimensional space. The goal of PCA is to find a linear subspace with lower dimensionality
( ), such that maximum variance is retained in this lowerdimensional space. The linear subspace can be specified by p orthogonal vectors, such as , which form a new coordinate system. Ideally (worst case would be to have ). These vectors are called the 'Principal Components'. In other words, PCA aims to reduce the dimensionality of the data, while preserving its information (or minimizing the loss of information). Information comes from variation. In other words, capturing more variation in the lower dimensional subspace means that more information about the original data is preserved.For example, if all data points have the same value along one dimension (as depicted in the figures below), then that dimension does not carry any information. To preserve the original information in a lower dimensional subspace, the subspace needs to contain components (or dimensions) along which data has most of its variability. In this case, we can ignore the dimension where all data points have the same value. In practical problems, however, finding a linear subspace of lower dimensionality, where no information about the data is lost, is not possible. Thus the loss of information is inevitable. Through PCA, we try to reduce this loss and capture most of the features of data.
The figure below demonstrates an example of PCA. Data is transformed from original 3D space to 2D coordinate system where each coordinate is a principal component.
Now, consider the ability of the two of the above example components (PC_{1} and PC_{2}) to retain information from the original data. The data in the original space is projected onto each of these two components separately in the figures below. Notice that PC_{1} is better able to capture the variation in the original data than PC_{2}. If the goal was to reduce the original d=3 dimensional data to p=1 dimension, PC_{1} would be preferable to PC_{2}.
PCA Plot example
For example<ref> https://onlinecourses.science.psu.edu/stat857/sites/onlinecourses.science.psu.edu.stat857/files/lesson05/PCA_plot.gif </ref>, in the top left corner of the image below, the point is shown in a twodimensional space and it's coordinates are and
All the red points in the plot represented by their projected values on the two original coordinators (Feature 1, Feature 2).
In PCA technique, it uses new coordinators (showed in green). As coordinate system changed, the points also are shown by their new pair of coordinate values for example
and . These values are determined by original vector, and the relation between the rotated coordinates and the original ones.In the original coordinates the two vectors
and are not linearly uncorrelated. In other words, after applying PCA, if there are two principal component, and use those as the new coordinates, then it is guaranteed that the data along the two coordinates have correlation = 0By rotating the coordinates we performed an orthonormal transform on the data. After the transform, the correlation is removed  which is the whole point of PCA.
As an example of an extreme case where all of the points lie on a straight line, which in the original coordinates system it is needed two coordinates to describe these points.
In short, after rotating the coordinates, we need only one coordinate and clearly the value on the other one is always zero. This example shows PCA's dimension reduction in an extreme case while in real world points may not fit exactly on a line, but only approximately.
PCA applications
As mentioned, PCA is a method to reduce data dimension if possible to principal components such that those PCs cover as much data variation as possible. This technique is useful in different type of applications which involve data with a huge dimension like data preprocessing, neuroscience, computer graphics, meteorology, oceanography, gene expression, economics, and finance among of all other applications.
Data usually is represented by lots of variables. In data preprocessing, PCA is a technique to select a subset of variables in order to figure our best model for data. In neuroscience, PCA used to identify the specific properties of a stimulus that increase a neuron’s probability of generating an action potential.
Figures below show some example of PCA in real world applications. Click to enlarge.
Tracking the useful features based on PCA:
Since in real world application, we need to know which features are important after dimensionality reduction. Then we can use those features as our new data set to continuous our project. To do this, we need to check the principal components have been chosen. Values of principal components or eigenvectors represent the weight of features within that specific principal component. Based on these weights we can tell the importance of features in the principal component. Then choose several most importance features from the original data instead of using all.
Mathematical details
PCA is a transformation from original (ddimensional) space to a linear subspace with a new (pdimensional) coordinate system. Each coordinate of this subspace is called a Principle Component. First principal component is the coordinate of this system along which the data points has the maximum variation. That is, if we project the data points along this coordinate, maximum variance of data is obtained (compared to any other vector in original space). Second principal component is the coordinate in the direction of the second greatest variance of the data, and so on.
For example, if we are mapping from 3 dimensional space to 2 dimensions, the first principal component is the 2 dimensional plane which has the maximum variance of data when each 3D data point is mapped to its nearest 2D point on the 2D plane.
Lets denote the basis of original space by
, , ... , . Our goal is to find the principal components (coordinate of the linear subspace), denoted by , , ... , in the hope that . First, we would like to obtain the first principal component or the components in the direction of maximum variance. This component can be treated as a vector in the original space and so is written as a linear combination of the basis in original space.
Vector
contains the weight of each basis in this combination.
Suppose we have n data points in the original space. We represent each data points by
, , ..., . Projection of each point on the is .Let
be the sample covariance matrix of data points in original space. The variance of the projected data points, denoted by is
we would like to maximize
over set of all vectors in original space. But, this problem is not yet welldefined, because for any choice of , we can increase by simply multiplying by a positive scalar greater than one. And the maximum will goes to infinity which is not the proper solution for our problem. So, we add the following constraint to the problem to bound the length of vector :
subject to :
This constraint makes
the unit vector in ddimensional euclidian space and as a result the optimization problem is no longer unconstrained.Using Lagrange Multiplier technique we have:
By taking derivative of
w.r.t. primary variable we have:
Note that
is symmetric so .From the above equation we have:
.
So
is the eigenvector and is the eigenvalue of the matrix .(Taking derivative of w.r.t. just regenerate the constraint of the optimization problem.)
By multiplying both sides of the above equation to and considering the constraint, we obtain:
The interesting result is that objective function is equal to eigenvalue of the covariance matrix. So, to obtain the first principle component, which maximizes the objective function, we just need the eigenvector corresponding to the largest eigenvalue of
. Subsequently, the second principal component is the eigenvector corresponding to the second largest eigenvalue, and so on.Principal component extraction using singular value decomposition
Singular Value Decomposition (SVD), is a wellknown way to decompose any kind of matrix (m by n) into three useful matrices.
.
where is m by m unitary matrix, , each column of is an eigenvector of
is m by n diagonal matrix, nonzero elements of this matrix are square roots of eigenvalues of .
is n by n unitary matrix, , each column of is an eigenvector of
Now, comparing the concepts of PCA and SVD, one may find out that we can perform PCA using SVD.
Let's construct a matrix p by n with our n data points such that each column of this matrix represent one data point in pdimensional space:
and make another matrix simply by subtracting the mean of data points from .
Then we will get a zeromean version of our data points for which
is the covariance matrix.So,
and give us the eigenvalues and corresponding eigenvectors of covariance matrix, respectively. We can then easily extract the desired principal components.MATLAB example
In this example we use different pictures of a man's face with different facial expressions. But, these pictures are noisy. So the face is not easily recognizable. The data set consists of 1965 pictures each 20 by 28. So dimensionality is 20*28= 560.
Our goal is to remove noise using PCA and of course by means of SVD function in matlab. We know that noise is not the main feature that makes these pictures look different, so noise is among least variance components. We first extract the principal components and then remove those which correspond to the least values of eigenvalues of covariance matrix. Here, eigenvalues are sorted in a descending order from column to column of matrix S. For example, the first column of matrix U, which corresponds to the first element in S, is the first principal component.
We then reconstruct the picture using first d principal components. If d is too large, we can not completely remove the noise. If it is too small, we will lose some information from the original data. For example we may lose the facial expression (smile, sadness and etc.). We can change d until we achieve a reasonable balance between noise reduction and information loss.
>> % loading the noisy data, file "noisy" stores our variable X which contains the pictures >> load noisy; >> >> >> % show a sample image in column 1 of matrix X >> imagesc(reshape(X(:,1),20,28)') >> >> % set the color of image to grayscale >> colormap gray >> >> % perform SVD, if X matrix if full rank, will obtain 560 PCs >> [U S V] = svd(X); >> >> d= 10; >> >> % reconstruct X using only the first d principal components and removing noise >> XX = U(:, 1:d)* S(1:d,1:d)*V(:,1:d)' ; >> >> % show image in column 1 of XX which is a noise reduced version >> imagesc(reshape(XX(:,1),20,28)')
PCA continued, Lagrange multipliers, singular value decomposition (SVD) (Lecture 2: Sept. 15, 2014)
Principal component analysis (continued)
PCA is a method to reduce dimensions in data or to extract features from data.
Given a data point in vector form
the goal of PCA is to map to where ^{d} and ^{p} such that p is much less than d. For example we could map a twodimensional data point onto any onedimensional vector in the original 2D space or we could map a threedimensional data point onto any 2D plane in the original 3D space.The transformation from ddimensional space to pdimensional space is chosen in such a way that maximum variance is retained in the lower dimensional subspace. This is useful because variance represents the main differences between the data points, which is exactly what we are trying to capture when we perform data visualization. For example, when taking 64dimensional images of handwritten digits, projecting them onto 2dimensional space using PCA captures a significant amount of the structure we care about.
In terms of data visualization, it is fairly clear why PCA is important. High dimensional data is sometimes impossible to visualize. Even a 3D space can be fairly difficult to visualize, while a 2D space, which can be printed on a piece of paper, is much easier to visualize. If a higher dimensional dataset can be reduced to only 2 dimensions, it can be easily represented by plots and graphs.
In the case of many data points
in ddimensional space:
There is an infinite amount of vectors in variance. In the images below, the data points mapped onto u in Mapping 1 have greater variance than in Mapping 2, and thus preserves more information in the data.
^{p} to which the points in X can be mapped. When mapping these data points to lower dimensions information will be lost. To preserve as much information as possible the points are mapped to a vector in ^{p} which will preserve as much variation in the data as possible. In other words, the data is mapped to the vector in ^{p} that will have maximumNotice(Figure 1.) the projection of a single data point x on u is p = u^{T}x (a scalar), thus the projection of multiple data points(Figure 2.) onto u is computed by Y = u^{T}X, where X is the matrix representing multiple data points. The variance of Y can be calculated easily as Y is a 1 by n vector.
Expand to higher dimensional space Covariance is used to find the variance of Y. The formula is given by: Var(u^{T}X) = u^{T}Su where S is the sample covariance matrix of X.
^{q} where q > 1,In finding the first principal component the objective is to find the direction u which will have the greatest variance of the points projected onto it. This can be expressed as the following optimization problem:
max_{u} u^{T}Su
s.t. u^{T}u = 1
The restriction u^{T}u = 1 is to bound the problem. If the length of u were not restricted, u^{T}Su would go to infinity as the magnitude of u goes to infinity. The selection of 1 as the bound is arbitrary.
Also from the three images above, we can see that the first principal component is the "line of best fit" that passes through the multidimensional mean of all data points, that minimizes the sum of distance squared from data points to the vector. Tho not illustrated but if the above example is doing dimensional reduction from R3 to R1, then the second principal component will have the same fashion as first component, with correlation between the first component and data points removed.
Lagrange multipliers
The Lagrange multiplier is a method for solving optimization problems. After constraining the principal component analysis output vector to an arbitrary size the task of maximizing variance in lower dimensional spaces becomes an optimization problem.
Lagrange multipler theorem
The method of Lagrange multipliers states that given the optimization problem:
The Lagrangian is defined by:
,where
term may be either added or subtracted.Which implies, where
is the gradient, that:
In practice,
is calculated by computing the three partial derivatives of with respect to , setting them to 0, and solving the resulting system of equations.
That is to say, the solution is given by taking the solution of:
which maximize the objective function f(x,y).
Intuitive explanation
The Lagrange multiplier utilizes gradients to find maximum values subject to constraints. A gradient
is a vector which collects all of the partial derivatives of a function. Intuitively, this can be thought of as a vector which contains the slopes of all of a graph's tangent lines, or the direction which each point on the graph points.Finding a point where the gradient of f(x,y) and g(x,y) differ by a factor lambda:
defines a point where our restraint and objective functions are both pointing in the same direction. This is important because if our objective function is pointing in a higher direction than g, increasing our objective function will increase our output, meaning we are not at a maximum. If the restraint is pointing in a higher direction than f, then reducing our input value will yield a higher output. The factor lambda is important because it is adjusted to ensure that the constraint is satisfied and maximized at our current location.Lagrange multiplier example
Consider the function
such that . We would like to maximize this objective function.Calculating the Lagrangian gives
This gives three partial derivatives:
This gives two solutions:
Both solutions give extreme values of our objective function. Because this is an optimization problem we are interested in the solution which gives the maximum output, which is solution 1.
Singular value decomposition revisited
Any matrix X can be decomposed to the three matrices:
where:is an m x m matrix of the eigenvector XX^{T}
is an m x n matrix containing singular values along the diagonal and zeroes elsewhere
is an orthogonal transpose of a n x n matrix of the eigenvector X^{T}X
Connection of Singular Value Decomposition to PCA
If
is centered (has a mean of 0) then is the covariance matrix.Recall that for a square, symmetric matrix
, we can have where columns of are eigenvectors of , and is a diagonal matrix containing the eigenvalues.Using the SVD of
we get . Then consider :
(By the definition of V)
Thus
is the matrix containing eigenvectors of the covariance matrix .It is important to remember that
and are unitary matricies, i.e.: and , each equality using the appropriate dimensional identity matrix. It is also interesting that the nonsquare matrix formed by taking a subset of the eigenvectors contained in or has the property that (the appropriate lower dimensional identity matrix).Matlab example of dimensionality reduction using PCA
The following example shows how principal component analysis can be used to identify between a handwritten 2 and a handwritten 3. In the example the original images are each 64 pixels and thus 64 dimensional. By reducing the dimensionality to 2D the 2's and 3's are visualized on a 2D plot and can it could be estimated as to which digit one was.
>> % load file containing 200 handwritten 2's and 200 handwritten 3's >> load 2_3; >> >> who? >> % displays that X is 64x400 matrix containing the handwritten digits >> >> imagesc(reshape(X(:,1),8,8)') >> colormap gray >> % displays first image, a handwritten 2 >> >> M = mean(X,2)*ones(1,400); >> Xb = XM >> % M is the matrix of the mean of column vectors repeated 400 times >> % Xb is data centred about mean >> >> % Now use SVD to find eigenvectors, eigenvalues of Xb >> [U S V] = svd(Xb); >> >> % 2D projection on points >> Y = U(:,1:2)'*X; >> >> % 2D projections of 2's >> plot(Y(1,1:200),Y(2,1:200),'.') >> >> % 2D projections of 3's >> plot(Y(1,201:400),Y(2,201:400),'.')
PCA Algorithm, Dual PCA, and Kernel PCA (Lecture 3: Sept. 17, 2014)
PCA Algorithm (Algorithm 1)
 Recover basis
 To find the basis for PCA we first center the data by removing the mean of the sample, then calculate the covariance:
 Let be the eigenvectors of corresponding to the top eigenvalues.
 Encode training data
 To encode our data using PCA we let
 where is an encoded matrix of the original data
 Reconstruct training data
 To project our data back to the higher dimension
 Encode test example
 where is an encoding of
 Reconstruct test example
Dual principal component analysis
 Motivation
In some cases the dimension of our data can be much larger than the number of data points
. For example, suppose we are interested in weekly closing rates of the Dow Jones Industrial Average, S&P 500 Index, and the NASDAQ Composite Index over the past forty years. In this example we only have three data points , but our dimension is over two thousand! Consequently, the matrix used to find the basis is , which is computationally heavy to calculate. As another example is studying genes for patients since the number of genes studied is likely much larger than the number of patients. When the data's dimensionality is high relative to the number of data points collected, a less computeintensive way of recovering the basis is desirable. Calculating dual PCA is more efficient than PCA when the number of data points is lower relative to the dimensionality of the data. This is intuitive because it allows us to model our optimization problem as a decomposition of our data, rather than our dimensions.
 Recall from Lecture 2 how we used singular value decomposition for PCA:
 Multiply both sides by V to get:
 Since (the identity matrix) the SVD can be rearranged as
 Then, multiply both sides by ^{1} to get:
 And so can be calculated in terms of and . is the eigenvectors of , so it is much easier to find than because .
 Algorithm
 We can replace all instances of in Algorithm 1 with the dual form, , to obtain the algorithm for Dual PCA.
 Recover Basis:
 For the basis of Dual PCA, we calculate
 then let be the eigenvectors of with respect to the top d eigenvalues. Finally we let be the diagonal matrix of square roots of the top d eigenvalues.
 Encode Training Data:
 Using Dual PCA to encode the data, let
 where is an encoded matrix of the original data
 Reconstruct Training Data:
 Project the data back to the higher dimension by
 Encode Test Example:
 where is an encoding of
 Reconstruct Test Example:
 Note that the steps of Reconstructing training data and Reconstructioning test example still depend on , and therefore still will
be impractical in the case that the original dimensionality of the data (
) is very largeKernel Principal Component Analysis
 Kernel methods
Kernel methods are a class of algorithms used for analyzing patterns and measuring similarity. They are used for finding relations within a data set. Kernel methods map data into a higherdimensional space to make it easier to find these relations. While PCA is designed for linear variabilities, a lot of higherdimensional data are nonlinear. This is where Kernel PCA can be useful to help model the data variability when it comes to a nonlinear manifold. Generally speaking, Kernels is a measure of "similarity" between data points without explicitly performing the dot product between data points pairs. This can be done using different methods. For example, the dot product looks at whether vectors are orthogonal. The Gaussian kernel looks at whether two objects are similar on a scale of 0 to 1. Kernels also measure similarity between trees. We will not go in too much detail at the moment.
 Some common examples of kernels include:
 Linear kernel:
 Polynomial kernel:
 Gaussian kernel:
Where
^{d}The kernel matrix
is the matrix where is the chosen kernel between ddimensional points and Other kernel examples can be found here:
 Kernel PCA
 With the Kernel PCA, we take the original or observed space and map it to the feature space and then map the feature space to the embedded space of a lower dimension.
 In the Kernel PCA algorithm, we can use the same argument as PCA and again use the Singular Value Decomposition (SVD) where:
 where contains the eigenvectors of
 The goal is to reduce the dimensionality from the space X to the dimensionality of space Y by passing through H without having to know exactly.
 The algorithm is similar to the Dual PCA algorithm except that the training and the test data cannot be reconstructed since is unknown.
 Algorithm
 Recover Basis:
 For the basis of Kernel PCA, we calculate
 using the kernel and let be the eigenvectors of with respect to the top eigenvalues. Finally we let be the diagonal matrix of square roots of the top eigenvalues.
 Encode Training Data:
 Using Kernel PCA to encode the data, let
 Recall :
 where is an encoded matrix of the image of the original data in the function . is computable without knowledge of via the kernel function.
 Reconstruct Training Data:
 However, is unknown so we cannot reconstruct the data.
 Encode Test Example:
 This is possible because we know that .
 Recall
 :
 is the transformation of the training data while is the transformation of the test example
 Reconstruct Test Example:
 Once again, since is unknown, we cannot reconstruct the data.
Centering for kernel PCA and multidimensional scaling (Lecture 4: Sept. 22, 2014)
Centering for kernel PCA
In the derivation of kernel PCA we assumed that
which is improbable for a random data set and an arbitrary mapping . We must be able to ensure this condition though we don't know .Let
Define the centered transformation (not computable)
and the centered kernel (computable)
Then
where
Here
and denotes the data set matrix and elements respectively and represent test elements at which the centered kernel is evaluated.In practice, with
as the data matrix, the Kernel Matrix, denoted by , will have elements .We evaluate
, and find the eigenvectors/eigenvalues of this matrix for Kernel PCA.Metric Multidimensional Scaling
Multidimensional Scaling (MDS) is another dimension reduction method that attempts to preserve the pairwise distance,
, between data points. MDS helps with the problem of making a configuration of points in Euclidean space by using the distance information between the patterns. For example, suppose we want to map a set of points X, , to a set of points Y, . We want to preserve the pairwise distances such that is approximately the same as .MDS example
We can map points in 2 dimensions to points in 1 dimension by projecting the points onto a straight line:
Note that the pairwise distances are relatively preserved. However, the triangle inequality is not preserved:
 , but
Another example of when it's crucial to preserve the distances between data points in lower dimension space (MDS) is when dealing with "distances between cities", as shown in here
Distance matrix
Given a
matrix, , the distance matrix is defined by:Where
is defined as the euclidean distance between points and inImportant properties:
 The matrix is symmetric
 ie
 The matrix is always nonnegative
 ie
 The diagonal of the matrix is always 0
 ie
 If 's don't overlap, then all elements that are not on the diagonal are positive
 Triangle inequality holds for all distance
 Triangle inequality: whenever are three different indices
Example:
For:
Or:
The distance matrix is:
For any mapping to matrix
, we can calculate the corresponding distance matrixWhere
is defined as the euclidean distance between points and inNote that a kernel matrix is a positive semidefinite matrix.
Finding the optimal Y
MDS minimizes the metric
Where
, andThe distance matrix
can be converted to a kernel matrix:Where
andHere, K is a kernel matrix implies that K is positive semidefinite, so
In fact, there is a onetoone correspondence between symmetric matrices and kernel matrices
Theorem
If
where and
Relation between MDS and PCA
It has been shown that the goal of MDS is to obtain
K is positive semidefinite so it can be written as .
Thus
and
From SVD we may make the decompositions
Let
It can be shown that for fixed that the minimum over occurs at , which implies that
Since
and are diagonal matrices containing the eigenvalues of respectively, and all said eigenvalues are positive, then the min is clearly obtained when the nonzero diagonal elements in are the largest eigenvalues inTherefore in Euclidean space, the solution of MDS is the same as that of PCA.
Approximation to MDS Algorithm <ref> http://www.cmap.polytechnique.fr/~peyre/cours/x2005signal/dimreduc_landmarks.pdf </ref>
The problem with the MDS algorithm is that the matrices FastMap, MetricMap, and Landmark MDS and unifies the mathematical foundation of the three presented MDS algorithms.
and are not sparse. It is therefore expensive to compute eigendecompositions when the number of data points is very large. To reduce the computational work required we can use Landmark Multidimensional Scaling (LMDS). LMDS is an algorithm designed to run classical MDS to embed a chosen subset of data, landmark points, and then located the remaining data points within this space given knowledge of its distance to the landmark points. The following paper has details onISOMAP and LLE (Lecture 5: Sept. 29, 2014)
ISOMAP
Isomap is a nonlinear dimensionality reduction method. In Multidimensional scaling we seek to find a low dimensional embedding
for some high dimensional data , minimizing where is a matrix of euclidean distances of points on (same with ). In Isomap we assume the high dimensional data in lie on a low dimensional manifold, and we'd like to replace the euclidean distances in with the distance on this manifold. We don't have information about this manifold, so we approximate it by forming the knearestneighbour graph between points in the dataset. The distance on the manifold can then be approximated by the shortest path between points. This is called the geodesic distance, which is a generalization of the notion of straight line to curved spaces and is denoted by . This is useful because it allows for nonlinear patterns to be accounted for in the lower dimensional space. From this point, the rest of the algorithm is the same as MDS. The approach of Isomap is capable of discovering nonlinear degrees of freedom that underlie the complex nature of the data. This algorithm efficiently computes a globally optimal solution which is guaranteed to converge asymptotically to the true underlying structure. (Further Reading: A Global Geometric Framework for Nonlinear Dimensionality Reduction, by Joshua B. Tenenbaum, Vin de Silva, John C. Langford)The Isomap Algorithm can thus be described through the following steps:
1. Construct the Neighbourhood Graph
Determine which points are neighbours on manifold M based on distances
between pairs of points i and j in the input space X. To do this, we can use one of the two methods below:
a) KNearest Neighbours: For each point i, select the k points with the minimum distance to point i as neighbours.
b) Radius Neighbourhood: For each point i, the set of neighbours is given by
The neighbourhood relations are represented as a weighted graph G over the data points, with edges of weight between the neighbouring points.
2. Compute shortest paths
Estimate geodesic distances
between all pairs of points on the manifold M by computing their shortest path distance in graph G.
3. Construct low dimensional embeddings (classical MDS)
Apply classical MDS to the matrix of graph distances
, constructing an embedding of the data in a ddimensional Euclidean space Y that best represents the manifolds estimated intrinsic geometry.
It is worth noting that as is not a Euclidean distance, the matrix is not guaranteed to be positive semidefinite. This is handled in the implementation [5] by considering only the positive value of the real part only, which can be considered as projecting the matrix onto the
cone of nearest positive semidefinite matrix.
It is important to note that k determines "locality". Isomap captures local structure and "unfolds" the manifold.
The swiss roll manifold is a commonly discussed example of this. Note that the Euclidean distance between points A and B drastically differs from the Geodesic distance.
ISOMAP Results
A Global Geometric Framework for Nonlinear Dimensionality Reduction is the paper that proposed ISOMAP, the below figures show some images for ISOMAP results. The first image shows the different orientation of the face images dataset, as shown in the figure the x axis captures the horizontal orientation, while the y axis captures the vertical orientation and the bar captures the lightning direction. The second image shows the 2 handwritten digits and ISOMAP results in this dataset.
Locally Linear Embedding (LLE)
Locally Linear Embedding (LLE) is another technique for nonlinear dimensionality reduction that tries to find a mapping from high dimensional space to low dimensional space such that local distances between points are preserved. LLE has some advantages over Isomap, including faster optimization and better result in many problems. LLE reconstructs each point based on its knearest neighbour.
E.g.
 neighbours.png
knearest neighbours for x, k = 6
The method optimizes the equation
Algorithm
 Find the knearest neighbour for each data point
 Compute
 Let is a neighbour of Let be all k weights of point (ie row i of W with all zeroes removed) Then:
 and hence we need to optimize
 This is a separable optimization problem which can be solved for each point independently, which can be simplified to
 Now we have
 Let . Therefore we can instead simplify
 Use Lagrange Multipliers to optimize the above equation:

 Note that we do not need to compute
 Finally, solve where w is known and y is unknown
Notes:
is invertible under some conditions depending on , as it must have rank so must be greater than or equal
LLE doesn't work for out of sample case there are extensions for it to handle this as in this [6]
If we want to reduce the dimensionality to p, we need to choose the p+1 smallest eigenvectors.
Pros of Isomap:
 Calculation is based on eigenvalue and eigenvectors, thus overall stability and optimality is maintained.
 Only need one predetermined parameter: k(nearest neighbours) or (neighbourhood radius)
 Able to determine intrinsic dimension of signal from residual.
Cons of Isomap
 It is slow  it requires comparing all possible paths between each N(N1)/2 pairs of points and determining which is shortest. (But this can be fixed via the Nystrom approximation which we will see later.)
 It is not obvious how to handle out of sample data
Pros of LLE:
 LLE is capable of retain the structure and characterstic of original data.
 LLE requires little parameters to calculate
 Combining 1 & 2, LLE is good for trouble shooting and error dignostics, since it can retain most from original data and needs minimal amount of parameter to calculate.
 Much faster than ISOMAP, often with comparable or better performance
Cons of LLE
 It is not obvious how to handle out of sample data
 There is no way of knowing if reducing the data to p dimensions is the optimal reduction.
LLE continued, introduction to maximum variance unfolding (MVU) (Lecture 6: Oct. 1, 2014)
LLE
Recall the steps of LLE can be summarized as follows:
 For each data point, find its nearest neighbours for some fixed .
 Find reconstruction weights
, minimizing  Reconstruct the low dimensional embedding with respect to these weights.
That is, find low dimensional minimizing
.
The previous lecture covered how to find the weights . Here we will discuss how to find the lowdimensional embedding using these weights.
First, we want to write the previous equation in a matrix form.
Consider that is a matrix and is the column ( ) in the identity matrix (like Matlab notation)
Then the sample,
, can be represented as follows:Note that:
where is the row from the identity matrixSimilarly,
So the objective is equivalent to minimizing
But first, to prevent the trivial solution of the 0matrix, we add the constraint that
. Now, the objective can be rewritten as minimizing:where the above norm is the Frobenius norm, which takes the square root of the sum of squares of all the entries in the matrix. Recall this norm can be written in terms of the trace operator:
By letting
, the objective function will becomeThis minimization has a wellknown solution: We can set the rows of
to be the lowest eigenvectors of , where is the dimension of the reduced space.To solve this minimization using the Lagrange multiplier, the Lagrangian is:
Setting this equal to zero and solving gives
where
If we want to reduce the dimensions to p, we need to choose p + 1 smallest eigenvectors. This is because the first eigenvalue of the matrix is equal to 1 since it is a Laplacian (the number of
eigenvalues of a Laplacian is the number of connected components).Note that for LLE, the bottom d+1 eigenvectors of the matrix M represents a free translation mode of eigenvalue 0. By discarding this eigenvector, we enforce the constraint of
since it means that the components of the other eigenvectors must sum to 0, because of orthogonality. The remaining d eigenvectors form the d embedding coordinates found by LLE.Comparison between Dimensionality Reduction Approaches (Unified Framework)
Approach  Solution \ Eigenvectors of 

PCA  or 
MDS  a.k.a. where 
ISOMAP  
LLE  or 
KPCA 
From the above table we can interpret all of these algorithms as variations of kernel PCA. More details are presented in this paper: A kernel view of the dimensionality reduction of manifolds.
Note that PCA, MDS, ISOMAP and KPCA choose the largest eigenvalue/eigenvector pairs for embedding while LLE chooses the smallest pair(For MDS, D is Euclidean distance). However, in LLE, if we let
be the largest eigenvalue of and replace the matrix with , the largest eigenvalue/eigenvector pairs will be chosen instead.Relationship between LLE and Kernel PCA on LLE kernel
The smallest eigenvalue of LLE Kernel
is 0. The eigenvector corresponding to this eigenvalue is .So when we do LLE we find eigenvectors corresponding to smallest p+1 eigenvalues and discard the first eigenvector which is
.Now, If we construct a new matrix, a socalled deflated matrix by multiplying
with a factor , we will get a new kernel with rank d1.i.e.
has the eigenvector removed.Next we only need to find eigenvectors corresponding to smallest p eigenvalues and do not need to discard the first one, since it has been moved already.
However, we realize that
equals the centering matrix .So what we do in LLE is similar to what we do in Kernel PCA.
In fact doing LLE is equivalent to perform Kernel PCA on LLE kernel up to a scaling factor.
Supervised Locally Linear Embedding
Dick et al (2003) proposed a supervised LLE. The big difference between LLE and SLLE is that in SLLE, the class information of data would be considered to get the nearest neighbors.
Maximum variance unfolding
In maximum variance unfolding (MVU), as in isomap and LLE, we try to reduce the dimensionality of a nonlinear data. We may assume that the data is sampled from a lowdimensional manifold that is embedded within a higherdimensional Euclidean space. The aim is to "unfold" this manifold in a lowerdimensional Euclidean space such that the local structure of each point is preserved. We would like to preserve the distance from any point to each of its nearest neighbours, while maximizing the pairwise variance for all points which are not its neighbours. The MVU algorithm uses semidefinite programming to find such a kernel.
Optimization constraints
For a kernel
such that , we have the following 3 constraints: must be centred, i.e. .
 should be positive semidefinite, i.e. .
 The local distance should be maintained between data and its embedding.
This means that for such that are neighbours, we require that satisfies
where is the kernel matrix.
We can summarize this third constraint as:
where is the neighbourhood indicator, that is, if are neighbours and otherwise.
Optimization problem
One option for finding the correct K would be to minimize the rank of K. The intuition behind this is that even though the data might be highdimensional, it lies on a manifold of low dimension. Since
is not a convex function, the objective function is illdefined and is replaced with since when K is semidefinite the trace is just equal to the sum of the singular values. However, in practice minimizing the trace does not result in a faithful lowdimensional representation. An explanation for this phenomenon is still an open problem.Recall that in PCA,
. To find a useful K, the objective of can still be used, but the problem has to be changed from a minimization to a maximization. The intuition here is that you are maximizing the total variance explained by the kernel. Thus the problem iswhich is a semidefinite program.
MVU (Maximum Variance Unfolding) Method
 Construct the neighbourhood graph using knearest neighbours
 Find the kernel K by solving the SDP problem: max
 Apply Kernel PCA on the learned kernel K
Maximum Variance Unfolding is a useful nonlinear interpretation of the PCA intuition. MVU is robust and will preserve the nearest neighbours of the original data in any mapping.
To categorize dimension reduction methods we have learnt so far
Locality Preserving methods  Globality Preserving methods  

Linear methods  LLE  MDS, PCA 
Nonlinear methods  ISOMAP, LPP(locality preserving projection)  MVU, KPCA 
Action Respecting Embedding and Spectral Clustering (Lecture 7: Oct. 6, 2014)
SDE Paper Notes
Unsupervised Learning of Image Manifolds by Semidefinite Programming introduced the maximum variance unfolding technique. The results of the paper compare the proposed technique against existing dimensionality reduction techniques. The first figure compares SDE against PCA in the "faces" dataset. To maintain the entire dataset variation we need 5 dimensions using SDE, while we need 10 dimensions using PCA. The second figure compares SDE against ISOMAP in a nonconvex dataset. To maintain the whole dataset variation we need only 2 dimensions using SDE, while we need more than 5 dimensions using ISOMAP. The two figures show that SDE can represent data variation using a much smaller number of eigenvectors than PCA or ISOMAP. Also, if two points take the same action, the distance between theses two points should not change( this is important ). Rotation and transaction matrix are the only way to remain the distance.
Results of SDE against PCA. The figure is taken from here
Results of SDE against ISOMAP. The figure is taken from here
SDE against ISOMAP showcases a known problem that ISOMAP can struggle against nonconvex data.
Action Respecting Embedding
When we are dealing with temporal data (data in which the order is important) we could consider a technique called Action Respecting Embedding. Examples include stock prices, which are time series, or a robot that moves around and takes pictures at different positions. Usually there are various actions that transform one data point into the next data point. For example, consider a robot taking pictures at different positions: actions include stepping forward, stepping backward, stepping right, turning, etc. After each action, the robot takes a picture at its current position, then undertakes another action, continuing until a set of data points is collected. Or consider a financial time series, where actions include stock splits, interest rate reduction, etc. Or consider a patient under a treatment, where actions include taking a specific step in the treatment or taking a specific drug, which could affect blood sugar level, cholesterol, etc.
Some information cannot be captured in the form of similarities.
Action Respecting Embedding takes a set of data points
along with associated discrete actions . The data points are assumed to be in temporal order, where was taken between and .Contain all actions to be distance preserving transformations in the feature space.
Now we realize that intuitively, if two points have the same action, then the distance between them should not change after the action. There are only three types of transformations that do not change the distance between points. They are rotation, reflection, and translation. Since rotation and reflection have the same properties, we will call both of these rotation for our purposes. Any other transformation does not preserve distances. Note that we are able to model any action in low dimensional space by rotation, translation or any affine combination of them.
Action Respecting Embedding has similar algorithm to SDE[7]
Algorithm
1) Construct a neighbourhood representation
2) Solve to find maximum variance subject to constraints
3) Construct lower dimensional embedding from eigenvectors of the kernel
Differences:
1) Constructing neighbourhoods based on action pairs
2) Adding action constraints
Optimization Problem
Recall that kernel function satisfies the property:
.We can formulate the idea of distance preserving mathematically as
, if are points that perform the same type of transformations.The maximum variance unfolding problem is
 ,such that
The Action Respecting Embedding problem is similar to the maximum variance unfolding problem, with another constraint:
For all i, j,
,A Simple Example
Consider the two examples presented in the ARE paper [8] of a robot moving around taking pictures with some action between them. The input is a series of pictures with actions between them. Note that there are no semantics for the actions, only labels
The first example is a robot that takes 40 steps forward then takes a 20 steps backward. The following figure (which is taken from ARE paper [9] shows the action taking by the robot and the results of ARE and SDE. We can see that ARE captures the trajectory of the robot. Although ARE doesn't have semantics for each action it captured an essential property of these actions that they are opposite in direction and equal in magnitude. We could look at the position on manifold vs. time for a robot that took 40 steps forward and 20 steps backward. We could also look at the same analysis for a robot that took 40 steps forward and 20 steps backward that are twice as big.
Another complicated action and the result of ARE and SDE are shown in the next figure (taken from ARE paper [10]), where we can see that ARE captured the trajectory of the robot.
Sometimes it's very hard for humans to recognize what happened by merely looking at the data, but this algorithm will make it easier for you to see exactly what happened.
Types of Applications
Actionrespecting embeddings are very useful for many AI applications, especially those that involve robotics, because actions can be defined as simple transformations that are easy to understand. Another common type of application is planning, that is, when we need to find a sequence of decisions which achieve a desired goal, e.g. coming up with an optimal sequence of actions to get from one point to another.
For example, what type of treatment should I give a patient in order to get from a particular state to a desired state?
For each action
, we have a collection of data points such that .Through the algorithm, we learn
: where is that rotation matrix for action , and is the translation vector vector for action . We can estimate and . These can be solved in closed form so that we will know what happens to any input if action is taken.Spectral Clustering
Clustering is the task of grouping data such that elements in the same group are more similar (or dissimilar) to each other compared to the elements in other groups. In spectral clustering, we represent the data as a graph first and then perform clustering.
For the time being, we will only consider the case where there are two clusters.
Given
and notation of similarity or dissimilarity, we can represent the data as a weight undirected graph where V represents the vertices (the data) and E represents the edges (similarity or dissimilarity between two data  for now we will assume simliarity).Using intuition, we can see that the goal is to minimize the sum of the weights of the edges that we cut (i.e. edges we remove from G until we have a disconnected graph)
where A and B are the partitions of the vertices in G made by a cut, and
is the weight between points i and j.Consider the following graphs of data:
Although "cut" may minimize the edge weights that we use to partition, note that using cut(A,B) may cause many edges to be cut, but which have smaller weights. For example, if there is one datum that is very dissimilar to the rest of the data, we may end up partitioning this one point to itself despite it resulting in many edges being cut. This is the case in Example 3 above. This occurrence may cause an imbalance in partition sizes. To avoid this problem, we minimize the ratiocut.
We define ratiocut as:
where is the cardinality of the set and is the complement of .This equation will be minimized if there is a balance between set
and . This, however, is an NP Hard problem; we must relax the constraints to arrive at a computationally feasible solution.Let
such that
We will show in the next lecture that minimizing
is equivalent to minimizing the ratiocut.Choices of
The neighbourhood graph
If the pairwise distance are smaller than
between two points , let , else let . This similarity graph is generally considered an unweighted graph.The nearest neighbourhood graph
If the
is among the nearest neighbours , then . This will result in a nonsymmetric matrix, so we can either let as long as is among the nearest neighbour of or is among the nearest neighbour of ; or let
 only when is among the nearest neighbour of and is among the nearest neighbour of .
The resulting matrix is symmetric.
The fully connected graph
As the graph is fully connected, the weight function should reflect the local neighbourhood relationships. A Gaussian similarity function is often chosen in this case.
Spectral Clustering (Lecture 8: Oct. 8, 2014)
Problem Formulation and Notations
Once we are given a set of points
and a sense of what it is to be similar or dissimilar, we can represent these data points as a weighted undirected graph. Let be a graph, where V is the set of vertices and E is the set of edges.In graph theory, partitioning the set of vertices of a graph into two disjoint sets is called a cut. Consequently the set of edges which has one end point in each subset (partition) is called a cutset, which crosses the cut. In a connected graph each cutset determines a unique cut. So, in some cases cuts are identified with their cutsets rather than with their vertex partitions.
The cut partitions from into two subsets and .
The cutset of a cut
is the set of edges that have one endpoint in and the other endpoint in . If and are specified vertices of the graph , an cut is a cut in which belongs to the set , and belongs to the set .The weight of a cut is different for different types of graphs:
 In an Unweighted Undirected graph, the size or weight of a cut is the number of edges crossing the cut, or 1 for each edges crossing the cut.
 In a Weighted graph, the value or weight is defined by the sum of the weights of the edges crossing the cut.
Cut can be minimized or maximized, as defined below:
 A cut is minimum if the size or weight of the cut is not larger than the size of any other cut. In other words, partitioning vertices of a graph into two disjoint subsets that are joined by at least one edge is minimum cut, which is 2 here in the following example.
 A cut is maximum if the size of the cut is not smaller than the size of any other cut, which is 5 here in the following example.
The total weight of the cut is
Minimizing this is a combinatorial problem. In order to result in a balanced partition, we consider ratio cut defined in terms of the size of each subgraph as the following:
Solving this problem is NP hard, and thus relaxation of the constraints is required to arrive at a suboptimal solution. Consider the following function :
The function
assigns a positive value to any point that belongs to , and a negative value to any point that belongs to .Ex. If
then the values for will be +1 and values for will be 1.Derivation
Problem: Prove that minimizing
is equivalent to minimizing ratiocutProof:
We have 4 cases:
 ,
 ,
 ,
since both
Note that
 and
Note that
 and
Therefore, our main equation becomes:
Since
, then:
Let ,
then:
Since n is always nonnegative, minimizing ratiocut is equivalent to minimizing .
Therefore, minimizing is equivalent to minimizing . This completes the proof.
Problem: Find an appropriate relaxation of the optimization problem.
Define
to be the matrix of all weights.
Define to be a diagonal nxn matrix such that is the sum of the elements in row of :
Define
is a matrix such that , is called the Laplacian of the graph.
Show:
Proof:
Note that can be written as
and can be written as
Recall that we want to reach to the quadratic form in the right hand side
By changing the in the second term into
By substituting the value of
Which means minimizing the R.H.S minimizes the L.H.S
Selection of L
In class we discussed the Laplacian, other selections for L include normalized Laplacians such as
orRelaxation of constraint
Observe that
Therefore our goal is to solve the following optimization problem
instead of solving the more difficult ratiocut optimization problem
Our new problem can be solved by using Lagrange multiplier:
Set
The equation above is clearly satisfied if and only if is an eigenvector of L, and is the corresponding smallest eigenvalue.
[11] Since a graph with one connected component has only one constant eigenvector that is associated with eigenvalue. And as we can see from the previous equation that is the eigenvector of . So is the eigenvector corresponding to the second smallest eigenvalue, with positive elements corresponds to one class and negative elements correspond to the other one.
Similarly to LLE, the eigenvector corresponding to the zero eigenvalue is ignored. It turns out that LLE computes a specific Laplacian for the graph.
Spectral Clustering Results
The following figures show [12]the results of the spectral clustering against other clustering algorithms. As clustering is considered as an illdefined problem it's hard to say that one algorithm groups the points better than other algorithms. However, in the figures below, spectral clustering groups the points as a person may.
Results of spectral clustering. The figure is taken from here
Results of spectral clustering. The figure is taken from here
The most notable issue with spectral clustering is its dependency on the definition of "similar" and/or "dissimilar". For example, suppose a class of students were to be cut into two groups. If weights were given to pairs of students according to the similarity of their gender, then the two groups would presumably be separated into males and females. However, this weighting doesn't account for other similarities the class might have such as colour of clothing worn that day, or less visible similarities such as number of hours slept the previous night. The definition of "similar" is therefore what drove the cut, and not the actual similarities between the students. Spectral clustering is a problem that only has subjective solutions due to the subjective nature of the problem.
Multiclass Spectral Clustering
Spectral clustering can be used in multiclass case by choosing the smallest
eigenvectors and ignoring the smallest one (associated with eigenvalue), this will result in a matrix of size which can be used as a covariate of other clustering algorithm.Alternatively, the k eigenvectors can be used to create
clusters where the the cluster to which a point is assigned is given by converting the signs of each element in the matrix to binary then reading off the cluster number of each row as the number whose binary representation is given by that row. This is akin to cutting the data into two parts times where each subsequent cut approximates the best ratiocut based cut when the previous cuts have already been made. This assigns points with the same sign in the matrix to the same cluster.Laplacian Eigenmap
Laplacian eigenmap [13] solves the problem of dimensionality reduction by obtaining the eigenvectors that correspond to the lowest eigenvalues and project the data to these eigenvectors. This is very similar to the spectral clustering.
We can interpret the multiclass spectral clustering as a dimensionality reduction step before applying a clustering technique.
Also spectral clustering can be interpreted from Random walk point of view as explained here [14]: "Random walk is a stochastic process on a graph which moves randomly from node to node. So spectral clustering tries to find a graph partition such that random walk stays long within a cluster and with seldom moves from one cluster to another."
A side note: The frobenius norm of the covariance matrix can measure the linear correlation between two variables, the larger the norm the larger the correlation. And this is true only for linear correlation as for the nonlinear correlation data should be mapped to Hilbert space first before applying the covariance operator and obtain the Schmidt norm to get a measure of dependence. Supervised PCA uses this concept.
The following is a matlab sample code that shows the norm of a covariance matrix for uncorrelated, linearly correlated, nonlinearly correlated random variables.
>> X1 = randn(1, 10000); >> X2 = randn(1, 10000); % x1 and x2 are independent >> X = [X1;X2]; >> cov(X') % 2 by 2 matrix. % Covariance is small since x1 and x2 and independent. So we look at the norm of covariance. >> norm(cov(X')) % small as X1 and X2 are independent. % This is a measure of correlation between these two, which is a measure of linear dependence. >> X2 = 2 * X1; >> X = [X1;X2]; >> cov(X') >> norm(cov(X')) % large as X1 and X2 are linearly dependent >> X2 = sin(X1); >> X = [X1;X2]; >> cov(X') >> norm(cov(X')) % not large as the dependency between X1 and X2 is not linear.
LE(Laplacian eigenmap) uses the similar approach with LLE. It is fast but sometimes can not provide an ideal outcome. It's difference with other DR(dimension reduction) methods is that LE maintains an ideal robustness while there are outliers in the dataset, which is a property that other DR methods do not process.
Supervised Principal Component Analysis (Lecture 9: Oct. 15, 2014)
Assignment 1 Solutions
Overall the first assignment was done quite well. The average mark for undergrads was approximately 70% and the average grade for graduates was approximately 75%. This lecture will focus on the details of the first assignment and going over the solutions. The solutions will not be posted online.
 Question 1 was done quite well. Some marks were deducted for unusual scaling of eigenvalues (division by (n1)). In general it is a good idea to scale the eigenvalues, however, it will usually not affect whether the PCA function works or not. Marks were given simply for having a picture for most of the questions. Note that the explanations for the questions should have some detail, oneword answers are not sufficient.
 Question 2 part(a) had some issues. Some students plotted too many eigenvectors. Also in general, the fewer PCs we keep, the better. When determining how many eigenvectors to keep, look for an "elbow" in the plot for when the variance represented by a given eigenvector drops off.
 Question 3 was very well done, as long as you had some pictures, you would have gotten most of the marks.
 Question 4 had some issues. These are very wellknown properties, so you should prove it using the definitions and show more detailed steps.
 Question 5 was marked based on how far a student correctly followed the solution. Take the partial derivatives according to , and according to , set equal to 0 and simplify. Substituting one into the other and further simplifying gives the answer.
 Question 6 part(a) as well as part(c) were done well. Part(b) required showing that all eigenvectors of are also eigenvectors of
Introduction to Supervised Principal Component Analysis
Motivation: There are several applications in which we would like to reduce the dimensionality of data to be able to deal with it easier, however we would like to keep some desired properties or features of data in low dimensional representation at the same time. In fact, we don't want to reduce the dimensionality blindly.
For example, suppose that we would like to rank undergrad students in CS departments in top universities in Canada according to their grades in different courses. Suppose that there are 2000 students and 20 common courses. Since the data set is large, we want to reduce the dimensionality (number of courses) to 3 dimensions to process faster. If we run a PCA algorithm blindly, the algorithm returns the 3 courses in which students' grades have most variation. Assume there is a course (for example, Data Visualization) that we would like to keep in our top three courses because its grade is important in our evaluation.
Sometimes there is a feature or dimension of the data that we don't want to omit during dimensionality reduction. This is where supervised PCA is used.
Principal component analysis (PCA) may not be the ideal way for processing multiple sets of data at once or data with different classes of characteristics (dataset with different modes of variation), since the principal component which maximizes variation may not preserve these characteristics/classes. As such, maximizing the variance sometimes is not the best way to reduce the dimensionality of the data. As shown in the below figure (taken from here [15]), we can see that projecting the data on the direction of maximum variation will not make the two classes discriminated in the lower dimensions compared to FDA.
Sometimes we will do clustering as a prealgorithm, or we can incorporate side data and use labels in PCA in order to create a better dimensionality reduction. In these cases, we are more interested in finding subspaces with respect to the labels of the original data instead of those that maximizes the variance.
In dimensionality reduction we might have some "side information" or labels. In low dimension data we want points with the same label to be close to each other. Consider if you have a set of images of faces. The different faces can be divided into glasses/no glasses classes or beards/no beards classes, or I have other information about which people are similar. In low dimensional representation I want these points to be close to each other. KPCA may be able to distinguish some prespecified features in the picture, while PCA could only give components that have max variance. In dimension reduction, we might have some "side information" or labels.
Problem Statement
Given a set of data
are pdimensional explanatory variates and ldimensional response variables, the goal is to estimate a lowdimension subspace S such that S contains as much predictive information about the response variable Y as the original explanatory variable X.Alternatively, given an i.i.d (independent and identically distributed) sample
, we are looking for an orthogonal projection of X onto such that Y depends mainly on .Related Work
There are many families of algorithms to solve this problem, some of which are briefly described below.
Classical Fisher's discriminant analysis (FDA)
This algorithm is very famous, however it will not be covered in this class. This[16] or [17] are brief introductions to FDA. A brief overview is also given below.
Introduction
Consider a twoclass problem, i.e.
= 1 in the above formulation. For each class, let and denote the mean and covariance matrix of each class, for = 0, 1. Using a generic vector , we transform each data point using . Thus, the new means and covariances are as below:We would like to minimize the separation within the individual classes (withinclass variance), and maximize the separation of the two classes of data (betweenclass variance), of the projected data.
Solving the Problem
Let
be the withinclass covariance matrix. Then, minimizing the withinclass variance is described by:Similarly, let
be the betweenclass covariance matrix. Then, maximizing the betweenclass variance is described by:We can combine these two optimization problems to simply solve the following:
This problem is equivalent to:
This is a generalized eigenvalue problem, where
can be computed as the eigenvector of with maximal eigenvalue.Metric Learning (ML)
The technique attempts to construct a Mahalanobis distance over the input space, which could then be used instead of Euclidean distances. ML methods search for a metric (or equivalently, a linear transformation) under which points in the same class (similar points) are near each other and simultaneously far from points in the other classes. We need to minimize the sum of distances between similar class samples and maximize the sum of distance between different class samples.
Euclidean Distance:
Mahalanobis Distance:
, where A is a positive, semidefinite matrixWrite
Thus, we obtain the mapping
More information can be found in Metric Learning.
This website introduces several strategies on metric selection Distance Metric Learning
Sufficient Dimension Reduction (SDR)
In sufficient dimension reduction, we attempt to find a linear subspace
such that is preserved.i.e
Supervised Principal Components (SPCA)
 This version of Supervised Principal Components is different from the the Supervised Principal Component Analysis that we are going to talk about in this lecture. It is proposed by Bair er. al., and the original paper can be found here[18].
 The idea is to perform PCA only on selected data with a strong estimated correlation to the response variable (Y). This removes some unseen factors that are unrelated to the response variable (Y).
 This method is commonly used in biostatistics, but it's different from the one we are covering this course.
Steps
 Recall that in linear regression the coefficient for feature j is computed as: Note that doesn't have to be centered. This is essentially a measure of similarity between feature j (row j) and output Y.
 Replace the data matrix so that it only consists of features for whom the absolute value of the calculated coefficients exceeds some determined threshold .
 Compute the first (few) principal components of the reduced data matrix.
 Use the principal component(s) calculated in step 3 in a regression model or classification algorithm to predict the outcome.
SPCA method focuses on a region of interest from the spatial domain as training set, thus it will be more suitable to extract the property of data that is at most interest.
HilbertSchmidt Independence Criterion (HSIC)
Two random variables
and are independent if and only if any bounded continuous function of them is uncorrelated.If we map these random variables to reproduce kernel Hilbert spaces (RFHS), we can test for correlation. Problem: Given
determines whether .To do so, we define a RKHS
containing all continuous bounded realvalued functions of x,and a RKHS
containing all continuous bounded realvalued functions of y.
Distance correlation can provide a new approach to testing the joint independence of random vectors. Measuring and testing dependence by correlation of distances is explored here.
Let
be n independent observations from .An estimator of HSIC is:
 where , is a kernel of X, is a kernel of Y,and is the centering matrix. Note that K and B are positive semidefinite.
The HSIC is 0 if and only if X and Y are independent. Thus the estimator described above can be used to test for independence. For more details, see [19].
How far are two distributions from each other? A measure of similarity of
and is where P and Q are probability distributions. We can calculate the distance between distributions and as follows:So does the mean capture all the characteristics of the distribution in the higher dimensional space. The idea of this measure of distance is counterintuitive at first glance, since it states that after mapping into high dimension space using
, we can measure the distance of distribution by using the mean of the sample data points. It's difficult to find an explicit , but we can use kernel trick to solve the problem.Note that, the mean in the original dimensions is not enough to capture the characteristics of the distribution. For example, the following figure shows a two different distributions with exact same mean but with different variance. So, say we have a point in 2 dimensions, the first dimension of the point is the mean and the second dimension of the point is the variance. So, this point captures these two characteristic of the distribution. So, in infinity dimension, a point can capture all momentum of the distribution and therefore, only mean in higher space can capture the characteristic of the distribution.
Now that we have a way to measure the distance between two distributions, we have a way to test for independence since:
independent
Hilbert Schmidt Independence Criterion (Lecture 10: Oct. 20, 2014)
Introduction to the Hilbert Schmidt Independence Criterion (HSIC)
Last lecture we reviewed the background and setup for SPCA. Given data, X, and corresponding labels, Y, if we want to maximize the dependency between
(the projected data) and , then we will need the Hilberts Schmidt Independence Criterion (HSIC). When data are mapped to a higher dimension space (e.g. a kernel), we hope that the data and dependency are both linear with respect to that space. However, some cases won't necessarily conform to these assumptions. This was explained last lecture with the example of two distributions with the same mean, but with different variances.Intuition: Two random variables
~ and ~ in are independent if and only if any bounded continuous function of them is uncorrelated[20]) measures distance between two distributions and . It also allows us to check independence since implies independency. We can rewrite the distance equation as where expanding gives the Hilbert Schmidt Independence Criterion (HSIC). This term is based on the norm of the covariance matrix in Hilbert space.
(which is called Maximum Mean Discrepancy (MMD)Derivation of HSIC
Goal: Given an
. sample, , and labels, , we want an orthogonal projection of the data onto such that depends on .An estimator of
is given by: where
 is a kernel over , positive semidefinite
 is a kernel over , positive semidefinite
 is the centering matrix given by
We then want to maximize maximize the dependency of
onThis is done by:
 Compute the linear kernel of
is a matrix
Notice the similarity to Kernel PCA, which finds , the linear kernel of
:  Compute a kernel of labels
is a matrix
:  According to HSIC, the dependency between and can be computed as
 where
 Since doesn't depend on , we only need to maximize the trace, where the only unknown is
 So we solve
 where
Since the trace of a matrix is invariant under cyclic permutations, this can be rewritten as:
 Letting
 There is no upper bound on this optimization problem, which means that increasing the elements of U infinitely will increase the objective function infinitely. So, our problem is illdefined. Thus, we need put such a constraint to restrict and solve our optimization problem under that consideration.
 Therefore, we are going to solve our objective function such that:
 To solve this problem, use the Lagrange Multipliers:
 The Lagrangian is given by:
(i.e. is an orthogonal matrix) as our restriction, just like with .
 Taking the derivative of this with respect to U and setting to zero yields
 Hence, adding this restriction implies we need to find the eigenvectors corresponding to the largest eigenvalues of as the columns of .
, the problem becomes .
 Letting
 is the projection of X on a lower dimension space, but in this context we are ensuring it has maximum dependence on , rather than trying to maximize captured variance as with .
Note that if
, then . Since this is now the sample covariance matrix of centered , the solution to will be identical to the solution to . Therefore is just with the condition that . If , then, since is the kernel over all the labels and so represents the pairwise similarity over labels, we are working only under the knowledge that each datum is similar to itself  we do not know how similar any datum is to another datum. Or we can interpret it as knowing that each sample is in a separate class; this is how PCA is conducted.
 SPCA is therefore PCA with additional knowledge regarding similarity between data. So if you don't have labels use PCA, but if you do have more information about the similarity of the points, then you can impose it using SPCA. Therefore, PCA can be seen as a special case of SPCA.
Algorithm for SPCA
 Recover basis
Calculate
Define based on the top eigenvectors of corresponding to the top p eigenvalues  Encode training data
Let where is a matrix  Reconstruct training data
 Encode test example
where is a dimensional encoding of  Reconstruct test example
Note that if
is the linear kernel over Y then the dimension of the embedding one can create with this algorithm is bounded above by the rank of since in this case the rank of is bounded above by rank of . Any additional eigenvectors returned if the algorithm were asked for more would be arbitrary vectors from the null space of .Dual Supervised Principal Component Analysis
Recall how Dual PCA led to Kernel PCA. Dual PCA was based on cases where dimensionality
of data is large compared to the number of samples , and hence the eigenvectors of are hard to find. Instead, singular value decomposition was used to decompose the data into where U is the eigenvectors of
 V is the eigenvectors of
rearranging shows
Similarly, we can decompose instead of just decomposing . First as and are positive semidefinite matrices we define
 (As it's Positive SemiDefinite (PSD) matrix
So
So can be calculated asThe solution for U can be decomposed from
. Note that DSPCA depends only on the number of points, and is independent of the dimensionality.Note that if
is the linear kernel over Y then the dimension of the embedding one can create with this algorithm is bounded above by the rank of since in this case the rank of is bounded above by rank of . Any additional eigenvectors returned if the algorithm were asked for more would be arbitrary vectors from the null space of .Kernel Supervised Principal Component Analysis (Kernel SPCA)
In Kernel SPCA, we maximize subject to . There is a trick to doing this: U can be written as a linear combination of all data.
This algorithm allows us to rewrite our problem using dot product. In matrix form, we assume that, for each column of U, there are n coefficients such that
where is d x p, is d x n, is n x p Our objective function can be rewritten in terms of as
 notice that and are our linear kernel:
This constraint term can be rewritten in terms of
as: which implies which implies
Therefore the new optimization problem is subject to
 Here we don't have
 this implies that
 Notice that if this was just , then Q would be the eigenvectors, and the eigenvalues of . However, since we instead have , we are solving for the generalized eigenvectors of
 Therefore the solution will be the top p eigenvectors of the generalized eigenproblem
, but rather we have . If we were to write the Lagrange multiplier, we can see that the solution to this problem is the generalized eigenvectors of :
In high dimensional space, X isn't explicitly available. However, by denoting , then , but we don't have X. Therefore we need to project to a lower dimension space and compute
where is p x n, is p x n, and is n x n.
To project an outofsample point, x, we have:

 is the training data
 is the test data
 is the kernel between the training and test data. (ie. it codes how similar the training and test data are)
where
Note: There is an alternative way to derive the Kernel SPCA, which will be shown on Assignment 2
Algorithm for Kernel SPCA
Input:
 the kernel matrix of the training data
 the kernel matrix of the test data
 kernel matrix of target variable
Note: in lecture note professor use
to represent the kernel of target variable. For consistency is used instead. testing example
 training data size
Output:
 dimension reduced training data
 dimension reduced test data
 Define: and
 Compute basis: , the top generalized eigenvectors of and
 Encode the training data:
 Encode the test example:
Note that if
is the linear kernel over Y then the dimension of the embedding created with this algorithm has an upper bound defined by the rank of . This is because the rank of is bounded above by rank of . Any additional eigenvectors returned by the algorithm would be arbitrary vectors generated from the null space of .MATLAB Example for Kernel Matrix
Polynomial degree 4 kernel with kernel function as defined in 'kernel.m'
>> load X % data to find kernel matrix of >> >> n = size(X,2); >> K = zeros(n); >> for i=1:n >> for j =1:n >> K(i,j) = kernel(poly,X(:,i),X(:,j),4,[]); >> end >> end
Recommended Kernels for Classification
Delta Kernel
When applying KPCA for classification purposes, the recommended kernel to use on Y is the delta kernel.
if and are equal, and 0 otherwise.
This kernel distinguishes between elements belonging to different classes.
RBF Kernel
A good kernel to use on X is the Radial Basis Function Kernel (RBF).
, where is a free parameter.
is the squared Euclidean distance between the two vectors. The value of the RBF kernel decreases as distance between the two vectors increases.
Since the value of the kernel ranges between 0 and 1, it can also be interpreted as a similarity measure.
Experimental Results Visualization
The following images are the 2dimensional projections to the Iris flower data set produced by (in order from left to right):
a. Supervised PCA with
b. Bair's SPC
c. Kernel Dimension Reduction
d. Kernel Supervised PCA
The following are two more examples to show the effects of the 4 projections on different data sets. We are going to use the Binary XOR dataset and the Concentric Ring dataset to demonstrate.
The Binary XOR dataset
Original form of the Binary XOR dataset
An 1dimensional Gaussian noise is added to the dataset before projected to 2dimensional space.
2dimensional projections of the Binary XOR dataset produced by (in order from left to right):
a. Supervised PCA with
b. Bair's SPC
c. Kernel Dimension Reduction
d. Kernel Supervised PCA
KSPCA is able to clearly seperate the data into the 2 categories while the other methods fail to do so
The Concentric Ring dataset
Original form of the Concentric Ring data set
2dimensional projections of the Concentric Ring dataset produced by (in order from left to right):
a. Supervised PCA with
b. Bair's SPC
c. Kernel Dimension Reduction
d. Kernel Supervised PCA
KSPCA is able to seperate the data into 2 distinct clusters while the other methods are unble to do so
SPCA continued and Metric Learning(Lecture 12: Oct. 22, 2014)
SPCA continued
SPCA compared to other methods
The following figure compares the performance of different dimensionality reduction techniques across 5 datasets. The onenearestneighbor classifier is used, as it doesn't have parameters to tune for each dimensionality reduction algorithm, which guarantees a fair comparison between the different algorithms. (Figure from Supervised principal component analysis: Visualization, classification and regression on subspaces and submanifolds by Barshana, Ghodsi, Azimifara, and Zolghadri Jahromia.)
We can only project up to k1 dimensions using FDA, where k is the number of classes. Increasing the number of dimensions beyond
has no effect and the error becomes flat.Methods like KDR([21]) do not work in all of the datasets above, however SKPCA did pretty well for most of them. The consistently good performance of SKPCA lets us use it as a preprocessing step for any supervised machine learning task, avoiding the curse of dimensionality.
KSPCA has the same runtime efficiency as SPCA because its optimization is based on the number of observations, rather than the dimensionality of the data.
Feature Selection
There are two main approaches for dimensionality reduction [22] [23]
 Feature selection: Selecting a subset of the existing features
 Feature extraction: Combining the existing features to produce a smaller number of features.
The formulation of SPCA can also be used for feature selection as follows:
Remember that we have
as our projection. We computed in a way that has maximum dependence with the label .Now consider only the first column of
, denote it as the vector . We can project the data into one dimensional space using this vector .If vector
is sparse. i.e. we multiply it with to get a linear combination of rows of . However, since we have a lot of zeros in the resulting matrix will only contain information of rows that is not multiplied by . If we remove the unused rows we will still have maximum dependence with the label . This is equivalent to saying that we don't need these features/variables, because they have no dependence with . (i.e for , the first few features of Y which corresponds to the first few zero entries of are redundant.)To compute sparse eigenvectors, [24]. The method was previously interpreted using linear programming in 1999. In 2009, a new efficient algorithm called coordinate descent [25] was introduced.
, we can use LASSOUsing this method we can project the data on only one dimension. If we want to project on
dimensions we will need to use a column matrix with blocksparsity property. Block sparsity can be achieved by minimizing the norm, defined byMetric Learning
Introduction
Metric learning computes the Mahalanobis distance instead of Euclidean distance over the input space. The Mahalanobis distance is defined as
Motivation and Explanation:
Metric Learning was proposed by Xing et al in 2002. They proposed to construct a new distance metric over the points to use in place of Euclidean distances. In this approach, some amount of side information is employed to learn a distance metric that captures the desired information.
Suppose we only have classrelated side information (not as strong as a label) about some data set. For example, we only know a pair of
is similar or dissimilar. We notate Set containing all similar point pairs and Set containing all dissimilar point pairs:, if and are similar;
, if and are dissimilar.
For square of Euclidean distance between
,we can generalize it by adding a matrix
 .
Matrix
should make satisfy the four properties of a metric: Nonnegativity:
 Coincidence:
 Symmetry:
 Triangular Inequality:
It can be proven that when Matrix
is positive semidefinite, must satisfy these four properties. The distance is called the Mahalanobis distance.Note that Mahalanobis distance is equivalent to Euclidean distance by doing Cholesky decomposition to Matrix
.thus, we have
This can be interpreted as calculating the Euclidean distance after transforming
using transformation matrix by .Distance Metric Learning
The main idea behind this method is to minimize the sum of distances between samples that should be similar while maximizing the sum of distances between samples that should be dissimilar. We need to find a matrix A that accomplishes this given that Matrix A is positive semidefinite, i.e.
.We can define an Objective Function where we want to minimize the sum of the distances between the pairs of points in Set S:
When
, this criterion becomes not useful. We define the following constraint:
(Notice, not squared). This constraint is necessary because, if points are dissimilar, we don't want them to collapse into a single point. So we set the criterion to be, arbitrarily, greater or equal to 1.
The matrix
changes the definition of distance/similarity. But we don't know A, so the problem is then reduced to finding .Xing et al proposed an approach to get Matrix A by NewtonRaphson method. However, this approach does not work well.
MCML approach (Maximally Collapsing Metric Learning)
A popular and efficient approach is Maximally Collapsing Metric Learning, proposed by Amir Globerson and Sam Roweis. The objective function here is a set of conditional probabilities . For each , the authors introduced a conditional distribution over points such that
In an ideal situation, all points in Set S (similar set) are mapped to a single point and are mapped infinitely far from the points in Set D (dissimilar set). The ideal distribution is then:
Our goal is to make the conditional as close as possible to the ideal case. When [27] is applied to solve it.
, then under A all points in the same class will be mapped to a single point, infinitely far from other class points. Thus our objective is to find a Matrix A such that is as close to as possible. KL divergenceKL divergence is a measure of the difference between two probability distributions. We denote the difference between probability distribution
and by .Thus, the objective becomes minimizing the KL divergence between
and , i.e.
Learning a Metric from ClassEquivalence Side Information
Ghodsi et al 's approach
Instead of minimizing the Mahalanobis distance, we define a loss function, which, when minimized, attempts to minimize the squared induced distance between similar points and maximize the squared induced distance between dissimilar points. So the new objective function becomes:
The optimization problem then becomes
such that and .The Matrix
should still be positive semidefinite, hence the first constraint.We also don't want to get a trivial solution of
being 0 matrix (i.e. all zero distances), hence the second constraint.
Consider , we can rewrite it as
The term inside the summation is a constant, therefore it can be written as
Since
is positive semidefinite, thereforewe can write the equation as
Again, since the term inside the trace is a constant, we can write it as
So the objective function becomes:
We can further simply it by letting
and rewriting the equation as
such that
or
Now we use Lagrange Multiplier:
We get:
It implies that
are eigenvectors of the matrixor the solution is the P smallest eigenvectors of the matrix
Hence we can find
easily.Advantages over Distance Metric Learning
Aside from its convenient form, this formulation has other advantages over that used by Xing et al., especially with respect to the side information they convey Ghodsi et al
Xing et al. require at least one dissimilar pair in order to avoid a solution where all distances are zero. On the other hand, the constraint employed on the trace in this approach means that no restrictions need to be placed on the pairings. Side information can consist of similar pairs only, dissimilar pairs only, or any combination of the two.
In the absence of specific information regarding dissimilarities, Xing et al. assume that all points not explicitly identified as similar are dissimilar. This may be misleading and may force the algorithm to separate points that are actually similar. The formulation in Ghodsi et al. approach uses only the side information one actually has, partitioning the pairings into similar, dissimilar, and unknown.
Embeddings
Two sets of informed embeddings were generated from the same data set, each with side information pertaining to two different characteristics. The embeddings were generated with and without a preprocessing step. The preprocessing step informs the preprocessed embeddings about the characteristic of interest. It uses the new distances calculated with the learned metric.
In this experiment, all similar pairs were identified but no dissimilar pairs (fivepairs beards and allpairs glasses omitted). The pairs were selected at random. The "EquivalenceInformed MDS" uses the preprocessor.
Inspection (of this and other Figures) reveals that, in general, the informed versions of the embeddings manage to separate the data based on the target property, whereas the uninformed versions are typically chaotic. Side information can be used to capture characteristics of interest where uninformed embedding techniques fail to automatically discover them. Moreover, two different characteristics (beards and glasses) have been captured within the same data set, even when using only small quantities of classequivalence side information.
Generally, preprocessing is effective with a wide range of embedding techniques.
Dimensionality issue with this approach
We showed previously that W satisfies
So W is composed of eigenvectors of the matrix
.However, all these eigenvectors have to correspond to the same eigenvalue
.This implies that the rank of the matrix
is 1.So the dimensionality of the reconstructed data is only of dimension 1.
This is due to the constraint
that we picked initially. We can modify this constraint so that the rank of the solution matrix is greater than 1.Solution with other constraints
In order to get more eigenvalues, possible new constraints were proposed.
One of them is to let
We follow similar procedure as the original approach.
Using the Lagrange Multiplier we get:
Since we are minimizing the objective function. The solution for , where is a diagonal matrix containing the coefficients. will be p smallest eigenvectors of
Now that we learned that changing our constraint will change our solution, we can think of this constraint more carefully. Another constraint we can have is:
Basically, rescale each direction by the variance of points in that direction.
Relation to FDA
Recall that FDA solution is the eigenvectors of
Where
is called within class covariance. is called between class covariance.
Recall MCML solution is the eigenvectors of MS  MD.
Conceptually FDA and MCML are similar as FDA evaluate eigenvectors of within class (similar to S similar objects in MCML) and between class (similar to D different objects in MCML).
Nonnegative Matrix Factorization Via Rankone Downdate(NMF) (Lecture 13: Oct 27, 2014)
The following was finished with reference to Nonnegative matrix factorization via rankone downdate (Ghodsi et al 2008) [28]. The video of the presentation for this paper can be found at this link [29]
Background
Several problems in information retrieval can be posed as lowrank matrix approximation. The seminal paper by Deerwester et al. (1990) on latent semantic indexing (LSI) [30] showed that approximating a termdocument matrix describing a corpus of articles via the SVD led to powerful query and classification techniques. A drawback of LSI is that the lowrank factors in general will have both positive and negative entries, and there is no obvious statistical interpretation of the negative entries.
Nonnegative matrix factorization might be useful for image segmentation, such that all the basis images are nonnegative. For example, below we see two basis images. The one on left contains only nonnegative entries. The one on right does not have this constraint.
We see the photo on the left looks more like a normal face, since faces do not have negative entries.
Another application is text analysis. We can use a term frequency matrix to represent the number of times a word appears in different pieces of text. A nonnegative matrix is suitable here because a word cannot appear in a given text a negative number of times. Other possible applications include DNA analysis and retrieving notes from polyphonic music.
History
Nonnegative matrix factorization has its roots in the work of Gregory and Pullman (1983), Paatero and Tapper (1994) and Cohen and Rothblum (1993). However, in the field of chemometrics  the study of using datadriven approaches to extract information from chemical systems  a similar method to NMF has been around for a while as something called "Self Modelling Curve Resolution". The vectors in the matrices are seen as continuous curves, as opposed to discrete vectors. Lee and Seung (1999) proposed nonnegative matrix factorization (NMF) as used commonly today: given a matrix A ∈ R^{m×n}, we want to find nonnegative matrix factors W ∈ R^{m×k} and H ∈ R^{k×n} such that , both W and H have nonnegative entries, and k ≤ min(m, n).
In a related work, Hofmann (1999) demonstrated an application of NMF to text retrieval. Suppose the original matrix M represents text data, such that each column represents the counts over words appearing in documents. Then when W and H have been found, the columns of W can be considered topics affecting the entire corpus, while the columns of H are mixing proportions specifying the intensity of each topic in each document.
Since the problem is NPhard (Vavasis, 2007), it is not surprising that no algorithm is known to solve NMF to optimality. Heuristic algorithms proposed for NMF have generally been based on incrementally improving the objective
in some norm using local moves.PerronFrobenius Theorem
PerronFrobenius Theorem
Leading singular vectors of a nonnegative matrix are nonnegative.
This immediately suggests that there is a trivial rankone NMF:
Compute; ; ;
Therefore,
.However this is a rank one approximation of the matrix
which is not useful in many casesPower Method for Singular Value Decomposition
Power method computes the leading singular vectors / values (or eigenvectors / eigenvalues) of a matrix using an iterative approach. First, it chooses a random solution and then iteratively tries to improve the answer until the answer converges.
In the algorithm below, we make a random guess for the vector u (for example, a vector of ones). The algorithm then repeats until u, σ, v converge.
Algorithm for SVD using power method input A ∈ R ^{m×n} and k ≤ min(m, n) output U, Σ, V 1: for µ = 1 to k do 2: Select a random nonzero u ∈ R^{m} 3: u = u/‖u‖ 4: repeat {power method} 5: v = A^{T}u/‖A^{T}u‖ 6: σ = ‖Av‖ 7: u = Av/‖Av‖ 8: until stagnation in u, σ, v 9: A = A − uσv^{T} 10: U(:, µ) = u 11: V (:, µ) = v 12: Σ(µ, µ) = σ 13:end for
Note that there is no guarantee that the entries of the residual matrix A in line 9 are still nonnegative, therefore this algorithm cannot be used directly as an NMF algorithm for k ≥ 2.
One intuitive idea is that we can replace all negative entries of the residual matrix A with 0, but this is obviously not a very good method.
First Attempt to NMF
Modify the above algorithm following our intuitive idea, we have the following algorithm for NMF based on power method:
Algorithm for NMF based on power method input A ∈ R ^{m×n} and k ≤ min(m, n) output W, H 1: for µ = 1 to k do 2: Select a random nonzero u ∈ R^{m} 3: u = u/‖u‖ 4: repeat {power method} 5: v = A^{T}u/‖A^{T}u‖ 6: σ = ‖Av‖ 7: u = Av/‖Av‖ 8: until stagnation in u, σ, v 9: A = A − uσv^{T} 10: Replace negative entries of A with 0 11: U(:, µ) = u 12: V (:, µ) = v 13: Σ(µ, µ) = σ 14:end for 15:W = UΣ 16:H = V^{T}
The problem is that when changing the negative entries of the residual matrix
to zero, we cannot assure that the modified matrix is similar to the original residual matrix.Similarity to Clustering
Given
, our second observation is that singular vectors can be used to cluster rows and columns of A. Namely, similar entries in u correspond to similar rows of A, and similar entries in v correspond to similar columns of A.For the reasoning behind, we can refer to spectral clustering, or the algorithm of power method, where we have
and . Note that , entries of u can be view as cosine similarity between rows of A with v. If we cluster u and v and replace the entries in the cluster with smaller mean with 0, then we are reconstructing only part of the matrix A.Algorithm for Modified Power Iteration 1: Find initial v using power method 2: while not converged 3: u = Av / ‖Av‖ 4: cluster u and remove the cluster with smaller mean 5: v = A^{T}u / ‖A^{T}u‖ 6: cluster v and remove the cluster with smaller mean 7: σ = ‖A^{T}u‖ 8: end
RankOne Downdate (R1D)
Objective: find M, N such that A(M, N) is large and approximately rankone: [31].
. You can find the reference paper of RankOne Downdate hereTherefore the objective function can be written as:
This optimization problem is separable, so for each row i:
If N and
are given, then M and can be computed via least square method, and vice versa.In the second term
, assume N and are known, we have by least square method.
Therefore,
where each item in the summation represents the contribution of a row of A in the objective function. Intuitively, we can go through each row of A, calculate its contribution, and keep those that have positive contribution and drop those that have negative contribution.
Algorithm R1D input A ∈ R^{m×n}, k > 0 output W ∈ R^{m×k}, H ∈ R^{n×k} 1: for µ = 1 to k do 2: [M, N, u, v, σ] = ApproxRankOneSubmatrix(A) 3: W(M, µ) = u(M) 4: H(N, µ) = σv(N) 5: A(M, N) = 0 6: end for
The ApproxRankOneSubmatrix procedure could be formulated as follows:
Algorithm ApproxRankOneSubmatrix input A ∈ R^{m×n}, parameter γ > 1 output M ⊂ {1,...,m}, N ⊂ {1,...,n}, u ∈ R^{m}, v ∈ R^{n}, σ ∈ R 1: Select j_{0} ∈ {1, . . . , n} to maximize ‖A(:, j_{0})‖ 2: M = {1,..., m} 3: N = {j_{0}} 4: σ = ‖A(:, j_{0})‖ 5: u = A(:, j_{0})/σ 6: repeat 7: Let v = A(M, :)^{T}u(M) 8: N = {j: γv(j)^{2}  ‖A(M, j)‖ > 0} 9: v(N) = v(N)/‖v(N)‖ 10: Let u = A(:, N)v(N) 11: M = {i: γu(i)^{2}  ‖A(i, N)‖ > 0} 12: σ = ‖u(M)‖ 13: u(M) = u(M)/σ 14:until stagnation in M, N, u, σ, v
The basic idea would be repeatedly select a subset of rows and columns, based on the criteria whether they could contribute (as positive) to the sum of the rows/columns.
The R1D algorithm is quite scalable, as it does not need all the data to be kept in memory at once.
Some Results
The results in this section comes from [32].
In Tables 1 and 2 result of LIS and R1D algorithms are presented on the TDT Pilot Study (TDT Study, 1997). The columns of each table are the leading columns of W, with the leading terms per column displayed.
The LSI results show that the topics are not properly separated and terms from different topics recur or are mixed.
The columns in the R1D table are clearly identifiable topics, and the terms in each columns are all correctly associated with the given topics.
The following figure shows a simple binary image dataset (a), where each of the images is composed of one or two basis "triangles". The leftmost column of (b) and (c) illustrates the four leading columns of W which can be interpreted as the learned features. For each feature the images with the largest entries in the corresponding columns in H is showed to the right of each feature. The below figure shows that R1D can learn the correct basis (triangles as features and assign the suitable image to that base while the results of LSI are can not be interpreted (This may be due to assigning negative weights).
Taken from [3]
The below figure shows the results of R1D and LSI on Freyface dataset where also figures (b) and (c) illustrates the eigenfaces (leading columns of W) of discovered by R1D and LSI respectively. And the images with max entries in the corresponding columns in H. This figure shows that R1D have interpretable results more than LSI.
Taken from [4]
Text/Image Biclustering, Nyström Approximation, and Clustering (Lecture 14: Oct 29, 2014)
Text and Image Biclustering
In this section, we discuss two different methods for clustering of images and text: LSI (Latent Semantic Indexing), and R1D.
LSI (Latent Semantic Indexing)
Latent Semantic Indexing is one method for grouping text or images together. In the case of text, the LSI algorithm takes as input a set of documents as a matrix. The
columns of this matrix can represent documents, and the rows of this matrix represent all of the words present in the entire document set. The entry of the input matrix generally represents the number of times that term appears in document . It is common to rescale the raw frequencies of the input matrix to combat noisy data.LSI proceeds to group the data by expressing the input matrix
as .is an matrix, and it can be viewed as a basis for the set of documents. is an matrix, and it can be viewed as the expression of the basis elements (i.e. the coefficients for each document expressed in the basis). is a parameter, and is taken as an integer that is much smaller than or .
Determining the
and matrices is done identically to PCA except that we do not need to assume that the matrix has zero mean. This is also equivalent to taking a Singular Value Decomposition of the data.In the case that the
matrix is considered to be text, we interpret the basis to be different topics that are present in the document corpus. We can then cluster documents based on the coefficients used to represent them in this basis. Conversely, we can cluster terms based on their similarity to the basis vectors.Unfortunately, this type of method does not do the best job of distinguishing topics, as seen in the example used in class.
R1D
R1D is another method for grouping text or images (or other sets of data). R1D is very similar to Nonnegative Matrix Factorization, and text clustering for R1D can be formulated similarly to the above.
As seen in the example in class, R1D does a very good job of extracting distinct, important features from a large set of data. In both the image segmentation and text clustering examples, R1D significantly outperformed LSI in interpretation and differentiation of features. R1D is scalable, and can easily be applied to large data sets. The implementation only requires one row or column of the data in memory at any given time.
Nyström Approximation
Many of the algorithms we have seen thus far are not scalable on large data sets. Finding the eigenvalue decomposition of an
by matrix for large is very computationally expensive. A solution to this is the Nyström Approximation. The aim of Nyström approximation is that for large scale of data set, we randomly choose a subset of data using the result of singular decomposition of this subset of kernel matrix to approximate the whole singular decomposition of its kernel matrix.Suppose that we have n data points
for and we select data points randomly ( ).Without loss of generality, we write this data set in this form:
Then, we let
andThe data set can be written as
.
Now we'll compute the kernel matrix
.
Let
, and where is an x matrix, is an x matrix, and is an x matrix. Then,.
Now, since is s.p.d (symmetric positive definite), we can apply SVD to matrix :
where .
Therefore, we can use this outcome to calculate :
U is orthonormal, so we can multiply both sides by and we get:
therefore
Note that
so the inverse matrix isThus, we have
Thus, knowing the first
rows of the Kernel matrix allows us to compute the rest of matrix using this method. Also, the approximation is exact if the rank of the Kernel matrix .Example: Apply Nyström approximation to MDS
Recall that MDS starts with an
Note: is a positive semidefinite matrix since it is Euclidean distance matrix.
To relax this into a Nyström problem we can write
as a block matrix as follows:
Here,
is and represents the pairwise distances between selected points, and is an matrix with the distances between the selected points and all other points. We do not need to compute the pairwise distance between all points as is normally required for MDS.Once we have
and , we can approximate and in using centering formula:
to calculate elements of A and B.
(note that the constant centering term for
is dropped, because it introduces an irrelevant shift of origin).We can compute
approximately using . The approximation is exact when is of rank or less.The eigenvalues and eigenvectors of
can be approximated by the eigendecomposition of , using the top eigenvectors according to the desired dimensions. We set from the eigendecomposition of , and then the coordinates corresponding to are . Taken together, and give the new dimensional coordinates of the transformed data.In this way, we do not need to compute the (computationally expensive) eigendecomposition of the full matrix, but only that of the subselected
.More details can be found in this paper: FastMap, MetricMap, and Landmark MDS are all Nystrom Algorithms. Current research is exploring which points when selected yield the best approximation. This paper also outlines the different calculation of the elements of B for Landmark MDS (LMDS):
This alternative is valid since the only change to B is a translation of all columns by a multiple of the vector of all ones. Since B is only used in dot products the shift does not effect the results.
Application of the Nyström Method to Image Segmentation
The Nystrom Method can be used for representing a full set of data by finding similarities between groups given a small sample of the full data. For example, Fowlkes, Belongie, Chung, and Malik (2004) utilised the Nystrom Method to segment images based on groupings rather than on pixels. Clustering by pixels is problematic for high resolution images. Instead, the Nystrom Method allows a small number of samples to represent the larger picture.
Clustering
Cluster analysis groups data into clusters such that data within a cluster is similar and different from the data in other clusters. Clusters are better and more distinct when the similarity within a cluster is greater and the difference between clusters is greater as well. Cluster analysis groups data based on information found in the data and relationships between data points. In this way, cluster analysis is a type of unsupervised classification.
Clustering is not a well defined problem in general.
The image above demonstrates how a set of points can be divided into clusters in three different ways. The shapes of the markers indicate cluster membership. "This illustrates that the definition of a cluster is imprecise and that the best definition depends on the nature of data and desired results." ref
We represent a clustering function as .
is the partition of points, is the distance metric.
Properties We Want
There are 3 desirable properties we want in a good clustering function or algorithm
 Scale invariant:
 Richness: Any possible partitioning
 Consistency: Assume . If D' is such that distances between points in the same cluster of D are reduced and distances between points in different clusters are increased, then .
Impossibility Theorem
It is impossible to satisfy all three above conditions at the same time for any
.Generally we deem certain aspects less important and cluster anyway.
3 Types of Clustering
 Combinatorial algorithm  Clustering itself is a combinatorial problem. An example of Combinatorial algorithm is Kmeans clustering.
 Mixture models  In mixture models we interpret data points as a sample from a mixed probability distribution model. An example of a "mixture model" is the Gaussian Mixture Model
 Mode seeking  This method assumes no specific distribution however it uses a nonparametric function to describe the data in order to compute something similar in nature to the data given. One this is done, then we look for the mode of the distribution. Clustering is combinatorial by nature. For example, given 3 points, there are many ways to make 2 clusters of data. Combinatorial problems are generally very difficult to solve on large data sets, so we apply heuristics and relaxations to solve the problem suboptimally. An example of "mode seeking" clustering is the Mean Shift Algorithm.
Clustering (Lecture 15: Nov 3, 2014)
Clustering
Clustering is the task of grouping objects into groups (clusters) such that objects between groups will be more similar than objects in the different clusters. To do clustering we will consider the similarity with the distance between points as follow :
 Suppose we have n data points that are indexed 1,...,n
 Suppose we need K clusters, k ∈ {1,2,...,K}
 We need to assign each point to one cluster k=C(i). Where C is an encoder
 The goal is to find a grouping of data such that the distances between points within a cluster tend to be small and distances between points in different clusters tend to be large.
Clustering can be roughly distinguished as:
 Hard clustering: each object belongs to a cluster or not;
 Soft clustering: each object belongs to each cluster to a certain degree;
 Strict partitioning clustering: here each object belongs to exactly one cluster;
 Strict partitioning clustering with outliers: objects can also belong to no cluster, and are considered outliers;
 Overlapping clustering: while usually a hard clustering, objects may belong to more than one cluster;
 Hierachical clustering: objects that belong to a child cluster also belong to the parent cluster;
 Subspace clustering: while an overlapping clustering, within a uniquely defined subspace, clusters are not expected to overlap.
Combinatorial Algorithms
Consider now the sum of all the distances between all of the points. Call this distance T.
where w(C) is the within cluster distance and b(C) is the between cluster distance. Note that this summation double counts, but it double counts everything so it is still valid.
Kmeans minimizes withincluster point scatter
We want to minimize w and maximize b. But since T is a constant and
, and vice versa. So from this we can conclude we either need to minimize W or maximize B. So our new goal is to minimize W. The number of distinct assignments (the complexity of the algorithm), for m points and k clusters is which is very large so clearly we need a method with which to cluster.KMeans Clustering
Kmeans is one of the most popular and widely used clustering algorithms, which minimizes the variance within each cluster.
Idea: Take a greedy descent approach.
First initialize C(i) to a starting value.
Then, continue in such a way that the criterion W(C) improve at each step.
In the case of Kmeans clustering this is done by finding the C(i) (cluster centre) that is closest to each given data point at each step. Each data point is assigned to this cluster. Once this is done for all data points, we take each cluster and recalculate W(C) by taking the mean of all the points assigned to the given cluster. This step improves W(C) and we have just redefined C(i). Now, we repeat the step with the new C(i). The idea is that the points that are not clustered correctly will be very far from the new mean C(i), and they are likely to be assigned to a different cluster as the newly calculated mean of another cluster may be closer. This process is repeated until no points are assigned to a new cluster. This indicates that the algorithm will have converged.
Where is the number of points in cluster k.
Aside: For any set of observations S:
Now,
(2)
A very important property of the above objective function is the role of . You may have seen some other objective functions in other sources (including wikipedia page of Kmeans Algorithm [33] and even MATLAB kmeans function) in which there is no such variable in the objective function. The role of this variable here, is basically preventing a cluster to have so many points while the other cluster has few ones. Of course there are many cases which this phenomenon happens inevitably but, if a data set has some how a symmetric structure, there should not be a cluster that has so many more points than the other(s).
Just for clarification, consider this exaggerated example. Suppose we have 1000 data points and we want to group them into two clusters. Assume that if we use the objective function without
coefficient, we would end up to two clusters, one of them with 980 points and the other 20 points. Assume again that the sum of squared distance to clusters' centers for cluster one is 500 and for cluster two is 10. Now, if we evaluate this instance of clustering with the above objective function with coefficient , it will be 980*500+20*10 = 490200. But, assume that if we take 180 points from cluster one which has the maximum distance from cluster one's center and minimum distance from cluster two'center (of course this maximum distance from cluster one's center is still less than the minimum distance to cluster two's center because they are grouped in cluster one), the value of sum squared distances will become 280 for cluster one and 240 for cluster two(including the effect of change in centers position). These 180 points are actually located some where in between two clusters' centers. Now the objective function will become 800*280+200*240 = 272000 which is much less than before. Finally, it is worthwhile to say that however adding to our objective function makes our clusters to share data points in a more fairer manner, it also makes this optimization problem even harder.Optimization
We can optimize (2) in two steps:
 Given C, (2) is minimized by letting to be the centroid of all data points in each cluster.
 Given ,￼￼ (2) is minimized by assigning each to the closest
 So
This is basically kmeans clustering.
Iterative algorithm
 Initialize: pick K random points as cluster centers
 Alternate:
 Assign data points to closest cluster center.
 Change the cluster center to the average of its assigned points.
 Stop when no points' assignments change.
This applet illustrates the steps of Kmeans
Implementation Details and Extensions
Selecting the number of classes and initial points to start with are very important. If a bad number of initial points are chosen, we may end up losing some clusters in the essence of combining two of more clusters into one and if too small a K is selected then we will create almost sparse clusters or clusters with just one member.
There are a bunch of techniques to choose the proper initial values to prevent this phenomena, and also some other techniques that converges much faster than the classical kmeans method as described above.
[<ref> http://www.cs.utah.edu/~suresh/web/2010/05/08/newdevelopmentsinthetheoryofclusteringtutorial/ </ref> <ref> http://theory.stanford.edu/~sergei/slides/kdd10thclust.pdf </ref>]
 The Iterative SelfOrganizing Data Analysis Technique (ISODATA) algorithm can help us to find the number of clusters to start with.
 The algorithm "Kmeans++" can be used to find proper initial points.
 Other techniques applies the kernel trick to Kmeans, and produced kernel kmeans algorithm.
 This paper discusses How Fast is the kmeans Method is.
KMeans++ Initialization
There are two major shortcomings to the kmeans algorithm. The first is that the wortcase running time of the algorithm is superpolynomial in the size of input (i.e. not bounded by any polynomial). Secondly, the kmeans result can be arbitrarily bad with respect to the objective function compared to optimal clustering. Kmeans++ combats the second problem by using a procedure to select the initial kmeans value. The solution is O(log k) competitive to the optimal kmeans result.
Kmeans++ attempts to spread out the initial k centres as much as possible. The algorithm to do this is given below, as in this Wikipedia page:
1. Choose the first centre uniformly at random from the data.
2. For each data point
, find the distance between and the nearest cluster centre that already has been initialized.3. Randomly choose a new data point to be a cluster centre, where each point is sampled with weight relative to its distance from the nearest cluster centre.
4. Repeat 2 and 3 until
centres have been chosen.After determining the centres, proceed with the usual kmeans algorithm.
ExpectationMaximization Algorithm
Motivation: Mixture Model
Another method of clustering is the usage of a mixture model. In general, when presented with a set of data points, one intuitive method to organize these data points is to fit a predetermined model or distribution to the data points then use the given data points to estimate the parameters of the model or distribution. One common distribution is the Gaussian, or Normal, distribution. However one major drawback of this is that the spread of the data has only one mode (unimodal), thus, since the reason for clustering is to organize the data into separate sections, the usage of only one mode gives little benefit. However, this concept can be extended by assuming that there exists a mixture of multiple Gaussian distributions in which an unknown parameter determines which one a certain data point is sampled from. This concept is known as the Mixture Model which is illustrated in the following figure, where the unknown parameter (Z) is used to select one of the Gaussian distributions X. For example if we using a mixture of two Gaussians Z will have a Bernoulli distribution and according to its result one of the two distributions can be selected.
It is more obvious to see as follows:
1) We get the observation and we assume these points are from Gaussian distribution
2) The cluster appears that points are close to other data
3) We can then fit a mixture of Gaussian distributions
For example, you roll a dice. The dire rent result correspond different Gaussian distribution.
An interesting extension to the method would be to learn the number of clusters in the data by considering the data as being made of k clusters or of j clusters where k is less than j. This allows the mixture of k normals to be a submodel of the mixture of j normals. As one is a submodel of the other, and the loglikelihoods of each are easy enough to compute, one may perform a Likelihood Ratio Test to determine if the model with fewer clusters is sufficient to explain the data relative to the model with more clusters. One may then work backwards from 1 cluster per datum (which has perfect fit and so the likelihood is 1) towards a more reasonable number of clusters, or to work forwards from 1 cluster to a more reasonable number of clusters.
Review of Maximum Likelihood Estimator
This note from New York University is a good tutorial on MLE.
If the form of the distribution P is known but its parameters
are unknown, we can write likelihood function and maximize it with respect to Consider a cluster of the data that only has one mode. One of the most popular methods to determine the parameters of a distribution used to fit these data points is Maximum Likelihood Estimation (MLE). Essentially the given data points are assumed to be samples from the distribution where denotes the parameters of the distribution. Assuming we know what is and that all sample points are independent from each other we calculate the total probability of sampling the given points given the distribution and parameters as follows:, and for the whole data set, . However, we usually calculate the log likelihood instead as it is much easier to work with. It is defined as
Example:
where
Thus we need to maximize only the last part as the rest is constant:
taking the derivative and setting it equal to zero gives the estimates:
,
Expectation Maximization (EM) Algorithm
Another way of looking at the method the data is generated by is to think about it where there is a hidden variable. You see the observed variables
, but there exists some latent variables that you never observe. So we have original data in the form of . A popular method to solve this kind of problem is called the EM algorithm.In short, the EM algorithm is a way to maximize the likelihood when a part of the data is missing.
 We can write which is likelihood of observation.
 We can also define
 We can write the likelihood of observation in terms of the complete likelihood :
Idea
is the likelihood of our optimization. Calculating the log of a sum is difficult, so we want to change it to a sum of logs.
Suppose this function is greater than or equal to another function; that is, the second function is the lower bound of the first. Then maximizing the first function is equivalent to maximizing the lower bound, or what we can call the auxiliary function. When we are maximizing the auxiliary function we can say that it has two terms. One of the terms is the expectation of
which we treat as random variable.The EM algorithm has two alternating steps:
 Estimate , which makes it is easy to optimize for because it separates the set into different subsets. Now you know the parameters of the components.
 Knowing the parameters, what is the best estimate for ? Then knowing z what is the best guess for the parameters.
Iterate between these two.
EM is more general than computing mixture of Gaussian.
Pros and Cons of EM algorithm
Pros:
 You don't need to choose a step size
 The algorithm is guaranteed to converge after a finite number of iterations
 The algorithm is fast
Cons:
 Sometimes the algorithm only results in a local maxima
 The algorithm is sensitive to the initialization of parameters
[34] Shows that KMeans is really just the EM (Expectation Maximization) algorithm applied to a particular naive bayes model.
EM algorithm and Mixture Model (Cont) (Lecture 16: Nov 5, 2014)
EM algorithm (Continue from last lecture)
Let's suppose variable X is observed and variable Z is missing. So our observed log likelihood for X is
. And our complete likelihood function is (where complete means it includes missing data). We can consider observed likelihood function as the marginal density function of the complete likelihood function that sum over (or integrate over, if the variable Z is continuous) Z. Therefore we can write . This formula has the form of log of sum, which is difficult to manipulate in optimization problem. Therefore, we seek a way to transform this formula in terms of sum of log function. We can use Jenson inequality to help us perform this transformation.Jensen Inequality
The most commonly seen Jensen inequality [35] is defined in term of a convex function. For our purpose, since we are going to apply Jenson inequality towards a concave log function, we need to write it in terms of a concave function.
Theorem: Let f be a convex function, and let X be a random variable. Then:
.
We randomly pick two points a and b within the domain of the concave function, and . Therefore is a point between and . By Jenson's inequality for concave function (Theorem), we have the following inequality correct:
.
Jenson inequality within EM Algorithm
To apply Jenson's inequality to the EM algorithm, we first rewrite the observed likelihood formula as:
, where is a any valid probability density function of z conditioning on x. Note that log is a concave function, hence by Jenson's inequality, we arrive at the following inequality:
.
We define the second part of inequality as auxiliary function
. Since the auxiliary function is a lower bound of the likelihood function we want to maximize, instead of maximizing the likelihood function, we can maximize the auxiliary function instead.EM Algorithm in General
The EM Algorithm consists of the E step and M step, which are defined as:
Estep (expectation): Using the current value of parameter
, find the distribution , for the latent variable z.
Mstep (maximization): Maximize the expected value of
with respect to , where the expectation is with respect to the distribution that found in the E step
The upper
and notation means the current arguments is at iteration.In EM algorithm, we iterate through these two steps, until they converge. They are guarantee to converge, since at each iteration we will increase the maximum. However there is no guarantee that they will converge to a global maximum, since the summation of log function may not be a global concave function.
We can expand auxiliary function as
.
Detail of Mstep
Since the second term of
expansion don't contains , it is constant with respect to . When we maximize in Mstep with respect to , we don't need to consider it.
which is the expectation of complete likelihood function with respect to q(zx).
Detail of Estep
Claim: if
is , then is maximize with respect to q.Proof: Substitute
in the auxiliary function.
Since
when . From the previous equation we know that is the upper bond of the auxiliary function Therefore we can conclude that is the optimal solution of E step.Trivial Example
assume
, where ,observed value of is missing.
The goal is to estimate
. This is trivial since we can just do log likelihood as follows:Log Likelihood Solution
>
EM Solution
Say instead we want to use EM algorithm. We can do so as follows:
Note:
since are independent.
%Matlab apporach for EM algorthm in example theta = 12; % initial theta_o = 11; while abs(theta  theta_o) ~= 0 % check if it converages theta_o = theta; theta = 2*theta/(1+5*theta); % theta for t+1 end theta %return theta
= 0.200
The value of
from EM algorithm is the same as the value when we do likelihood.Mixture of Gaussian
Gaussian Mixture Models (GMM) are using a multivariate Gaussian mixture density. Expectation Maximization (EM) algorithm is used to fit data and estimate parameters for GMMs. GMMs is often used for data clustering, and sometimes it is considered as a soft clustering method. Similar as kmeans clustering, GMMs classify different clusters by using an iterative algorithm that converges to a local optimal. If the clusters have different size and correlations within them, GMM should be more appropriate than kmeans clustering. GMMs are also commonly used as a parametric model of the probability distribution of continuous measurements or features in a biometric system.
A Gaussian mixture model is a weighted sum of M component Gaussian distributions,where
where
, i = 1,...,M, are the mixture weights, and , are component Gaussian distributions, since we have
with mean vector
and covariance matrix . The mixture weights must satisfy the constraint that .Then expectation maximization algorithm is applied to estimate parameters of training data.
For instance, assume data is generated as mixture of two Normal Distribution
where:
,
.
So:
If
then we sample from , with .If
then we sample from , with .Then we define the join distribution
Now we are going to maximize this expectation
More details will be taught in next lecture.
EM algorithm and Mixture Model (Cont) (Lecture 17: Nov 10, 2014)
Assignment 2 Solutions
Overall the second assignment was done better that the first one. The coding was done quite well (less crazy pictures than in the first assignment). More issues for the written questions.
Mixture of Gaussian
Suppose that we have a mixture of 2 Gaussian models, such that
. The general idea of this model is that we have a random variable Z which we can use to decide the distribution of the corresponding X. Let's suppose variable X is observed and variable Z is missing. We have or (Usually we assume Z has Bernoulli distribution with the probability function ).If
then we sample from
If then we sample from
We also choose an arbitrary distribution .
We can write the complete likelihood function :
So the loglikelihood is :
We can write the join distribution of this model :
Note that if
then and if thenNow we can compute the loglikelihood :
Our goal is to maximize the expectation of the loglikelihood but to do that we have to know the expectation of
corresponding to the distribution . The previous lecture concluded that the best choice for is . Assume we know this and that (This is the expectation of z under the distribution ).We now have :
MStep
We want to maximize the expectation of the loglikelihood with respect to
assuming that we know the expectation of (which is found in the previous E step). We do this by taking the derivative with respect to each component of and setting them equal to zero. This is case, since we are assuming we know the value of , it is considered to be a constant with respect to each component of . Find
Thus
If we know
, we can find the optimal . Find and
Recall :
So
Thus
We can do the same for
and we obtain Find and
Recall :
So
and
The MStep allows us to compute
if we assume that is known.EStep
Note that according to Bayes rule:
The EStep allows us to compute
if we assume that is known.Let's go back to the expectation of
. Recall or .
(by Bayes' theorem)
Note that
and .So
Termination
Iteration will end if
and are close enough.(Where t is the time of iteration.)Idea of EMalgorithm
The expectation–maximization (EM) algorithm is an iterative method for finding the maximum likelihood or maximum a posterior (MAP) estimates of parameters when missing data occurs. The E represent Expectation step and M represent Maximization step. The expectation step uses the current estimates for parameters to create a new function for the expectation of the loglikelihood evaluated. The maximization step uses the expected loglikelihood found in expectation step, and computes parameters which maximize the expected loglikelihood. Then these estimated parameters are used in the E step. The iteration ends until the solution converges. It could be done as follows:
 We choose random initial values and we alternatively iterate the Mstep and Estep.
 If we start we a random , we do the EStep and compute .
 Then we do the MStep and compute and so on until it converges (= parameters do not change).
(In each iteration, the value of parameters will change. But the form of distribution q will not change) You can also think of EM algorithm in terms of graphical models where random variable x depends on random variable z. if z = 0 (like flipping a coin) then x is generated from normal distribution of
and and if , then x is generated from normal distribution of and . The following figure shows the graphical model representation.
EM algorithm on Gaussian Mixture model is a popular tool for clustering purposes or density estimation. Given an ndimensional data set, one can start clustering with some reasonable initial guess about the number of clusters (which will be the number of Gaussian components) and relative covariance and mean for each of these clusters and then run the EM algorithm.
Convergence of EMalgorithm
As the number of iterations of the M step and E step increase, the results will converge which makes this algorithm valid. Intuitively the likelihood function monotonically increases, so it will always have the maximum value. In this paper, proof is given in detail.
Feature Selection (Lecture 18: Nov. 12, 2014)
Introduction
Feature selection (variable selection or attribute selection) is a process for selecting a subset of existing features which best represent the data in a lower dimensional space. This is a similar but more generalized version of the concept feature extraction. Feature extraction is the process of generating new features from the features of the original data where as feature selection simply returns a subset of features. Feature selection is mostly used when the number of feature is too high (sometimes more than number of samples, e.g. DNA microarrays) and computation becomes too expensive when processing the data. Feature selection assumes that some of the features are redundant and cannot add any further information to the data, or some of the features are irrelevant, i.e. they do not have any useful information about the data (for example when you are processing the data of patients with heart disease, the patient ID is irrelevant)
Mixture of Gaussian continued
Code for a small example of a mixed Gaussian model in Matlab:
x1 = mvnrnd([1;2]',eye(2),1000); x2 = mvnrnd([3;4]',5*eye(2),1000); x = [x1,;x2]; [p,mu,phi,Pxtr] = mdgEM(x,2,100,0.1)
This code forms a mixture of two Gaussian models, with 2 distinct means, then tries to use the EM algorithm to separate the points into two classes.
That was a more obvious example. What about the following circumstances?
Question: The data points are not Gaussian distributed.
Answer: Even if the data is not Gaussian distributed, we can still use mixture of Gaussian to do the clustering.
Question: The two Gaussian distributions have same center but with different variances, and the number of the data points in each are not the same
Answer: If the number of data points are not the same, the results of the EM method above will assume that the number of data in two clusters are the same. The algorithm works decently well as a density modelling function when the centers are the same, however this is not useful for clustering.
Definitions and Relation to Previous Works
A Feature is defined as a prominent characteristic. In the context of statistics, features are vectors representing aspects (attributes) of the data. Feature selection is the process of judging the importance of features and selecting features that are most relevant. Principal components can be associated with features of the data with most variation. This leads to the idea of PCAbased feature selection and sparse PCAbased feature selection, where we use PCA and sparse PCA to extract the principle components (features).
Unsupervised Feature Selection
A criterion for feature selection is to minimize the error of the reconstructed data matrix based on a set of selected features. This is called Unsupervised Feature Selection. The following has been summarized using the following pdf, [36].
Definition: Let A be a m X n matrix whose rows represent the set of data instances and whose columns represent the set of features, S a set of features, and
the projection of A onto the columns of SThe unsupervised feature selection criterion is
Problem 1
Find a subset of features
such that:
where
is the feature selection criterion and
is the projection matrix
Proof
Approximate
asGiven
solve the following least square problem for
Recursive Selection Criterion
A recursive formula for feature selection criterion can be developed. This formula is based on a recursive formula for the projection matrix
Lemma 1
Given a set of features, for any S, P a subset of S we have the following;
where,
is a projection matrix which projects the columns of E onto the span of the subset R of columns:
Proof
We define the following matrices:
B can also be written as:
 where
Let S be the Schur complement of
in B.Now we compute the following:
We can see that:
By substitution we can see that:
And so,
Corrollary 1
Given a matrix A and a subset of columns S. For any P⊂S
Proof
Using Lemma 1, we compute:
The first term is the lowrank approximation of A based on
.The second term is equal to
as .We can show this by multiplying
by E. This gives:
Using
, we can write:
This expression is equal to zero since
This means that
, thus proving that the second expression is equal to and so completing our proof.Theorem 1
Given a set of features, for any S, P a subset of S we have the following:
where,
Proof
We can rewrite this expression with the trace function,
As
, we can write the following expression,
This results in:
We just replace
with to prove the theorem.Greedy Selection Criterion
Problem 2
At iteration t, find feature l such that,
S is the set of the features selected during the first t1 iterations.
This implementation is of order
floatingpoint operations per iteration. This is not very efficient for large data sets. A more efficient approach is to use the recursive formula from Theorem 1 for calculating the reconstruction error.
where
Problem 3
At interation t, find feature l such that,
where
S is the set of the features selected during the first t1 iterations
Problems With Greedy Feature Selection
Greedy feature selection can be very calculation intensive at each iteration. This makes it memory intensive and computationally complex, making its run time extremely long
Problem 4 (Partition based Greedy Feature Selection)
See Efficient Greedy Feature Selection for Unsupervised Learning paper for more details.
At interaction t, find feature l such that,
where
S is the set of set of features selected during the first t1 iterations
is the partition of features into c groups
solution: Determine the best feature to represent the centroids of each of the group. Update the formulas for
andConclusion
This work presents a novel greedy algorithm for unsupervised feature selection. It shows a feature selection criterion which measures the reconstruction error of the data matrix based on the subset of selected features and forms a recursive formula for calculating the feature selection criterion.We compute an efficient greedy algorithm for the feature selection, and two memory and time efficient variants.
We have shown that the proposed algorithm achieves better clustering performance and is less computationally demanding than the methods that give comparable clustering performances.
Supervised Feature Selection
Like we introduced Supervised PCA based on PCA, there is also a supervised version of Feature Selection. Supervised Feature Selection also use the HilbertSchmidt independence criterion (HSIC) to measure the dependence between features and labels.Song et al proposed this method in 2007.
Gap statistics
Gap statistics [37] is a method of choosing the optimal number of clusters for a set of data. The idea is to divide the data in to n clusters and compute the within class distance of the n clusters kmeans. As the number of clusters (n) increases, the within class distance shrinks because points are clustered with more similar points.
At a certain point, increasing the number of cluster won't cause a noticeable decrease in the within class distances because there are enough clusters to accommodate the difference in data and there is no merit in adding more. To determine the optimal number of clusters, graph the number of clusters vs the within cluster sum of squares. Generally the point at which the elbow occurs in the plot is the number of clusters you should use.
Affinity Propagation
An Affinity propagation is a clustering algorithm based on the concept of "message passing" between data points. Initially, the number of clusters is not required in affinity propagation. Suppose S is a similarity measure. Affinity propagation minimizes the cost function:
The first sum tells us if c is a good measure based on the other measures and the second sum tells us if c is a good measure based on itself.
Essentially by minimizing the above cost function, we find the best "representative" of each cluster of data points by looking at which point does each individual point believe is the best representation for themselves. However the point that has been selected as the representative must also select itself. Otherwise the second term will severely punish the cost function.
Learning Affine Transformations for Nonlinear Dimensionality Recuction (Lecture 19: Nov. 27, 2014)
Affine Transformation
In PCA, it was quite easy to compute outofsample points (test set). However, this is not the case for nonlinear transformations, (e.g. LLE) and we can only approximate the transformation on the outofsample points. We address this problem by introducing a transformation that handles the nonlinear cases. Affine transformations preserve the local distances between neighbours, while pulling nonneighbours as far apart as possible. Affine transformations also do a good job dealing with outofsample points, unlike LLE and Isomap.
To unfold a nonlinear manifold, we define two different sets:
S = {(i,j)
and are neighbours}O = {(i,j)
and are non neighbours}, where W is our transformation matrix.
Since the distance between neighbouring points is fixed,
Normalize:
We try to minimize our Err function. Since
, we have
where
, and A = which is positive semidefinite.Let
, where and
, where u is in the space S and is in null space
then we want to pull nonneighbours as far away as possible,
, where
Kernelizing: (Let
and )
Embedding by Affine Transformation.
Given a training set of n ddimensional points
we wish to learn a d by d transformation matrix that will unfold the underlying manifold and embed into by:
Patch Alignment
2 clusters may overlap with each other in the original space. The end points, in the original dimensions, are connected and form a smooth curve. This property is not guaranteed when we map the data into lower dimensions using a kernel. To connect the end points, rotation and translation matrices are used. The goal is to find the rotation and translation matrices that minimize the matching error of the two patches. Sometimes when we plot the data, we may find a "hole" in it. The "hole" means the data set is not convex. This implies that you cannot apply the LLE or Isomap methods on it. At this point, we would like to consider Patch Alignment.
The intuition behind Patch Alignment is that by preprocessing data into overlapping clusters, dimensionality reduction techniques can be sped up by calculating the results on each cluster. Then, various techniques are used to smooth the edges between clusters, creating a final clean output.