independent Component Analysis: algorithms and applications: Difference between revisions

From statwiki
Jump to navigation Jump to search
Line 16: Line 16:
In this section we show that '''ICA''' cannot resolve independent components which have Gaussian distributions.
In this section we show that '''ICA''' cannot resolve independent components which have Gaussian distributions.


To see this, assume that the two source signals <math>s_1 \,</math> and <math>s_2 \,</math> are Gaussian and the mixing matrix <math>A\,</math> is orthogonal. Then the observed signals <math>x_1 \,</math> and <math>x_2 \,</math> will have joint density given by <math>p(x_1,x_2)=\frac{1}{2 \pi}\exp(-\frac{x_1^2+x_2^2}{2})</math>, which is rotationally symmetric. In other words, for '''any''' orthogonal mixing matrix, the joint density would be the same. This means that in the case of Gaussian variables, '''ICA''' can only determine the mixing matrix up to an orthogonal transformation.
To see this, assume that the two source signals <math>s_1 \,</math> and <math>s_2 \,</math> are Gaussian and the mixing matrix <math>A\,</math> is orthogonal. Then the observed signals <math>x_1 \,</math> and <math>x_2 \,</math> will have joint density given by <math>p(x_1,x_2)=\frac{1}{2 \pi}\exp(-\frac{x_1^2+x_2^2}{2})</math>, which is rotationally symmetric. In other words, the joint density is be the same for '''any''' orthogonal mixing matrix. This means that in the case of Gaussian variables, '''ICA''' can only determine the mixing matrix up to an orthogonal transformation.


==Measures of non-Gaussianity==
==Measures of non-Gaussianity==

Revision as of 15:30, 5 July 2009

Motivation

Imagine a room where two people are speaking at the same time and two microphones are used to record the speech signals. Denoting the speech signals by [math]\displaystyle{ s_1(t) \, }[/math] and [math]\displaystyle{ s_2(t)\, }[/math] and the recorded signals by [math]\displaystyle{ x_1(t) \, }[/math] and [math]\displaystyle{ x_2(t) \, }[/math], we can assume the linear relation [math]\displaystyle{ x = As \, }[/math], where [math]\displaystyle{ A \, }[/math] is a parameter matrix that depends on the distances of the microphones from the speakers. The interesting problem of estimating both [math]\displaystyle{ A\, }[/math] and [math]\displaystyle{ s\, }[/math] using only the recorded signals [math]\displaystyle{ x\, }[/math] is called the cocktail-party problem, which is the signature problem for ICA.

Introduction

ICA shows, perhaps surprisingly, that the cocktail-party problem can be solved by imposing two rather weak (and often realistic) assumptions, namely that the source signals are statistically independent and have non-Gaussian distributions. Note that PCA and classical factor analysis cannot solve the cocktail-party problem because such methods seek components that are merely uncorrelated, a condition much weaker than independence.

ICA has a lot of applications in science and engineering. For example, it can be used to find the original components of brain activity by analyzing electrical recordings of brain activity given by electroencephalogram (EEG). Another important application is to efficient representations of multimedia data for compression or denoising.

Definition of ICA

The ICA model assumes a linear mixing model [math]\displaystyle{ x = As \, }[/math], where [math]\displaystyle{ x \, }[/math] is a random vector of observed signals, [math]\displaystyle{ A \, }[/math] is a square matrix of constant parameters, and [math]\displaystyle{ s \, }[/math] is a random vector of statistically independent source signals. Note that the restriction of [math]\displaystyle{ A \, }[/math] being square matrix is not theoretically necessary and is imposed only to simplify the presentation. Also keep in mind that in the mixing model we do not assume any distributions for the independent components.

Ambiguities of ICA

Because both [math]\displaystyle{ A \, }[/math] and [math]\displaystyle{ s \, }[/math] are unknown, it is easy to see that the variances, the sign or the order of the independent components cannot be determined. Fortunately such ambiguities are often insignificant in practice and ICA can as well just fix the sign and assume unit variance of the components.

Why Gaussian variables are forbidden

In this section we show that ICA cannot resolve independent components which have Gaussian distributions.

To see this, assume that the two source signals [math]\displaystyle{ s_1 \, }[/math] and [math]\displaystyle{ s_2 \, }[/math] are Gaussian and the mixing matrix [math]\displaystyle{ A\, }[/math] is orthogonal. Then the observed signals [math]\displaystyle{ x_1 \, }[/math] and [math]\displaystyle{ x_2 \, }[/math] will have joint density given by [math]\displaystyle{ p(x_1,x_2)=\frac{1}{2 \pi}\exp(-\frac{x_1^2+x_2^2}{2}) }[/math], which is rotationally symmetric. In other words, the joint density is be the same for any orthogonal mixing matrix. This means that in the case of Gaussian variables, ICA can only determine the mixing matrix up to an orthogonal transformation.

Measures of non-Gaussianity

kurtosis

negentropy