regression on Manifold using Kernel Dimension Reduction: Difference between revisions
| Line 7: | Line 7: | ||
===Sufficient Dimension Reduction=== | ===Sufficient Dimension Reduction=== | ||
The purpose of Sufficient Dimension Reduction (SDR) is to find a linear subspace S such that the response vector Y is conditionally independent of the covariate vector X. More specifically, let <math>( | The purpose of Sufficient Dimension Reduction (SDR) is to find a linear subspace S such that the response vector Y is conditionally independent of the covariate vector X. More specifically, let <math>(X,B_X)</math> and <math>(Y,B_Y)</math> be measurable spaces of covariates X and response variable Y. SDR aims to find a linear subspace <math>S \subset X </math> such that <math>S</math> contains as much predictive information about the response <math>Y</math> as the original covariate space. As seen before in (Fukumizu, K., Bach, F. R., & Jordan, M. I. (2004))<ref>Fukumizu, K., Bach, F. R., & Jordan, M. I. (2004):Kernel Dimensionality Reduction for Supervised | ||
Learning</ref> this can be written more formally as a conditional independence assertion. | Learning</ref> this can be written more formally as a conditional independence assertion. | ||
<math>S \ | == | ||
<math>Y independent (X-B^T X) | B^T X </math>. == | |||
The above statement says that <math>S \subset X</math> such that the conditional probability density function <math>p_{Y|X}(y|x)\,</math> is preserved in the sense that <math>p_{Y|X}(y|x) = p_{Y|B^T X}(y|b^T x)\,</math> for all <math>x \in X \,</math> and <math>y \in Y \,</math>, where <math>B^T X\,</math> is the orthogonal projection of <math>X\,</math> onto <math>S\,</math>. The subspace <math>S\,</math> is referred to as a ''dimension reduction subspace''. Note that <math>S\,</math> is not unique. | |||
We can define a minimal subspace as the | |||
===Kernel Dimension Reduction=== | ===Kernel Dimension Reduction=== | ||
Revision as of 19:36, 20 July 2009
An Algorithm for finding a new linear map for dimension reduction.
Introduction
This paper <ref>[1] Jen Nilsson, Fei Sha, Michael I. Jordan, Regression on Manifold using Kernel Dimension Reduction, 2007 - cs.utah.edu </ref>introduces a new algorithm for for discovering a manifold that best preserves the information relevant to a non-linear regression. The approach introduced by the authors involves combining the machinery of Kernel Dimension Reduction (KDR) with Laplacian Eigenmaps by optimizing the cross-covariance operators in kernel feature space.
Two main challenges that we usually come across in supervised learning are making a choice of manifold to represent the covariance vector and to choose a function to represent the boundary for classification (i.e. regression surface). As a result of these two complexities, most of the research in supervised learning has been focused on learning linear manifolds. The authors introduce a new algorithm that makes use of methodologies developed in Sufficient Dimension Reduction (SDR) and Kernel Dimension Reduction (KDR). The algorithm is called Manifold Kernel Dimension Reduction (mKDR).
Sufficient Dimension Reduction
The purpose of Sufficient Dimension Reduction (SDR) is to find a linear subspace S such that the response vector Y is conditionally independent of the covariate vector X. More specifically, let [math]\displaystyle{ (X,B_X) }[/math] and [math]\displaystyle{ (Y,B_Y) }[/math] be measurable spaces of covariates X and response variable Y. SDR aims to find a linear subspace [math]\displaystyle{ S \subset X }[/math] such that [math]\displaystyle{ S }[/math] contains as much predictive information about the response [math]\displaystyle{ Y }[/math] as the original covariate space. As seen before in (Fukumizu, K., Bach, F. R., & Jordan, M. I. (2004))<ref>Fukumizu, K., Bach, F. R., & Jordan, M. I. (2004):Kernel Dimensionality Reduction for Supervised Learning</ref> this can be written more formally as a conditional independence assertion.
== [math]\displaystyle{ Y independent (X-B^T X) | B^T X }[/math]. ==
The above statement says that [math]\displaystyle{ S \subset X }[/math] such that the conditional probability density function [math]\displaystyle{ p_{Y|X}(y|x)\, }[/math] is preserved in the sense that [math]\displaystyle{ p_{Y|X}(y|x) = p_{Y|B^T X}(y|b^T x)\, }[/math] for all [math]\displaystyle{ x \in X \, }[/math] and [math]\displaystyle{ y \in Y \, }[/math], where [math]\displaystyle{ B^T X\, }[/math] is the orthogonal projection of [math]\displaystyle{ X\, }[/math] onto [math]\displaystyle{ S\, }[/math]. The subspace [math]\displaystyle{ S\, }[/math] is referred to as a dimension reduction subspace. Note that [math]\displaystyle{ S\, }[/math] is not unique.
We can define a minimal subspace as the
Kernel Dimension Reduction
(this section will be updated shortly)
Manifold Learning
(this section will be updated shortly)
Manifold Kernel Dimension Reduction
(this section will be updated shortly)
Examples
(this section will be updated shortly)
SUmmary
(this section will be updated shortly)