http://wiki.math.uwaterloo.ca/statwiki/api.php?action=feedcontributions&user=Mderakhs&feedformat=atom statwiki - User contributions [US] 2022-07-03T18:56:34Z User contributions MediaWiki 1.28.3 http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10352 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T23:04:49Z <p>Mderakhs: /* Numerical Example */</p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref name=&quot;self&quot;&gt;[http://faculty.washington.edu/mfazel/nucnorm_acc_final.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> [[Image:NumExFig2-025.png|thumb|400px|right|The top plot shows the Rank degree obtained by the heuristic. The bottom plot shows the corresponding values of the heuristic objective function. A modified image borrowed from &lt;ref name=&quot;self&quot;&gt;&lt;/ref&gt;]]<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10351 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T23:04:26Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref name=&quot;self&quot;&gt;[http://faculty.washington.edu/mfazel/nucnorm_acc_final.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> [[Image:NumExFig2-025.png|thumb|200px|right|The top plot shows the Rank degree obtained by the heuristic. The bottom plot shows the corresponding values of the heuristic objective function. A modified image borrowed from &lt;ref name=&quot;self&quot;&gt;&lt;/ref&gt;]]<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10350 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T23:02:13Z <p>Mderakhs: /* Numerical Example */</p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[http://faculty.washington.edu/mfazel/nucnorm_acc_final.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> [[Image:NumExFig2-025.png|thumb|200px|right|The top plot shows the Rank degree obtained by the heuristic. The bottom plot shows the corresponding values of the heuristic objective function. A modified image borrowed from &lt;ref&gt;[http://faculty.washington.edu/mfazel/nucnorm_acc_final.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=File:NumExFig2-025.png&diff=10349 File:NumExFig2-025.png 2010-11-29T23:01:28Z <p>Mderakhs: </p> <hr /> <div></div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10348 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T23:01:07Z <p>Mderakhs: /* Numerical Example */</p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[http://faculty.washington.edu/mfazel/nucnorm_acc_final.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> [[Image:NumExFig2-025.png|thumb|200px|right|The top plot shows the Rank degree obtained by the heuristic. The bottom plot shows the corresponding values of the heuristic objective function. borrowed from &lt;ref&gt;[http://faculty.washington.edu/mfazel/nucnorm_acc_final.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10347 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T23:00:47Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[http://faculty.washington.edu/mfazel/nucnorm_acc_final.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> [[Image:NumExFig2-025.png.png|thumb|200px|right|The top plot shows the Rank degree obtained by the heuristic. The bottom plot shows the corresponding values of the heuristic objective function. borrowed from &lt;ref&gt;[http://faculty.washington.edu/mfazel/nucnorm_acc_final.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10346 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:58:13Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[http://faculty.washington.edu/mfazel/nucnorm_acc_final.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10345 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:58:01Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[faculty.washington.edu/mfazel/nucnorm_acc_final.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10344 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:55:58Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[http://www.stanford.edu/~boyd/papers/pdf/rank_min_heur_sys_approx.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10343 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:54:52Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10342 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:49:32Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.111.7751&amp;rep=rep1&amp;type=pdf Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10341 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:49:16Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.111.7751&amp;rep=rep1&amp;type=pdf|Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10340 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:47:35Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body|http://faculty.washington.edu/mfazel/acc04-tutorial.pdf]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.111.7751&amp;rep=rep1&amp;type=pdf|Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10339 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:46:31Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf|A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.111.7751&amp;rep=rep1&amp;type=pdf|Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10338 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:43:38Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf|A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://citeseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.111.7751%26rep%3Drep1%26type%3Dpdf&amp;rct=j&amp;q=A%20rank%20minimization%20heuristic%20with%20application%20to%20minimum%20order%20system%20approximation%2C%20M.%20Fazel%2C%20H.%20Hindi%2C%20and%20S.%20Body&amp;ei=wB30TMXBHoHAsAO8z_zpCw&amp;usg=AFQjCNE0TDEO6CrMIR6hxleuVJ6IHrfGYg&amp;sig2=JlWUtQTg2uosst_rQKJqOA&amp;cad=rja|Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10337 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:42:53Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf|A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;[http://www.google.com/url?sa=t&amp;source=web&amp;cd=5&amp;ved=0CDkQFjAE&amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.111.7751%26rep%3Drep1%26type%3Dpdf&amp;rct=j&amp;q=A%20rank%20minimization%20heuristic%20with%20application%20to%20minimum%20order%20system%20approximation%2C%20M.%20Fazel%2C%20H.%20Hindi%2C%20and%20S.%20Body&amp;ei=wB30TMXBHoHAsAO8z_zpCw&amp;usg=AFQjCNE0TDEO6CrMIR6hxleuVJ6IHrfGYg&amp;sig2=JlWUtQTg2uosst_rQKJqOA&amp;cad=rja|Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10336 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:42:02Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[http://faculty.washington.edu/mfazel/acc04-tutorial.pdf|A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10335 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:41:44Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body|http://faculty.washington.edu/mfazel/acc04-tutorial.pdf]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10334 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:41:19Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;[A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body|faculty.washington.edu/mfazel/acc04-tutorial.pdf]&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10333 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:39:59Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;A rank minimization heuristic with application to minimum order system approximation, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10332 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-29T21:39:37Z <p>Mderakhs: </p> <hr /> <div>== Introduction ==<br /> The Rank Minimization Problem (RMP) has applications in a variety of areas including control theory, system identification, statistics and signal processing. Except in some special cases the RMP is known to be computationally hard. <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C,<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\, X \in \mathbb{R}^{m \times n}&lt;/math&gt; is the decision variable and &lt;math&gt;\, C &lt;/math&gt; is a convex set.<br /> <br /> If the matrix is symmetric and positive semidefinite, trace minimization is a very effective heuristic for solving the rank minimization problem. Trace minimization results in a semidefinite programming problem which can be easily solved.<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This paper&lt;ref&gt;A rank minimization heuristic with application to minimum order system approximation&lt;/ref&gt; focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for general non-square matrices.<br /> #Showing that the new heuristic can be reduced to a SDP, and hence effectively solved.<br /> #Applying the method to the minimum order system approximation problem.<br /> <br /> == Nuclear Norm Heuristic: A Generalization Of The Trace Heuristic ==<br /> Solving the rank minimization problem using the trace norm heuristic is not possible when the matrix under consideration is non-symmetric, or non-square. In this case, the authors develop a new heuristic.<br /> This heuristic minimizes the sum of the singular values of the matrix &lt;math&gt;X \in \real^{m\times n}&lt;/math&gt;, which is known as the ''nuclear norm'' or ''Ky-Fan n-norm'' of &lt;math&gt;\,X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;, the singular values of &lt;math&gt;\,X&lt;/math&gt;<br /> <br /> The nuclear norm is the dual of the spectral norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \},&lt;/math&gt; where &lt;math&gt;\|Y\|&lt;/math&gt; represents the spectral norm of &lt;math&gt;\,Y&lt;/math&gt; defined as the maximum singular value of &lt;math&gt;\,Y&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;\,X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\,\mbox{Tr } X&lt;/math&gt;, so the nuclear norm heuristic reduces to the trace minimization heuristic.<br /> <br /> The authors suggest using the nuclear norm as a heuristic for solving the rank minimization problem because the nuclear norm is the convex envelope of the rank function on the set of matrices with norm less than one and thus the nuclear norm minimization problem can be seen as a relaxation of the rank minimization problem. <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;\,f&lt;/math&gt; (on &lt;math&gt;\,C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;\,g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\,\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\textrm{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;\,M&lt;/math&gt;, that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\,\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> Note that this implies that &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;. Particularly, that means if &lt;math&gt;\,p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;\,p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt; and by solving the nuclear norm minimization problem we obtain a lower bound on the optimal value of the rank minimization problem.<br /> <br /> == Expressing as an SDP Problem ==<br /> <br /> To solve the nuclear norm minimization problem we can convert the problem into a SDP problem. We begin be transforming the problem into the following form<br /> <br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> We can then use the following lemma to transform the constraints into linear matrix inequalities (LMIs). <br /> &lt;br /&gt;&lt;br /&gt;<br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;\|X\|_*\leq t&lt;/math&gt; if and only if there exists matrices &lt;math&gt;Y \in \real^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\succeq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> The nuclear norm minimization problem can now be expressed as <br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \succeq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> where &lt;math&gt;\,Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;\,C&lt;/math&gt; can be expressed as linear matrix inequalities then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a rational matrix &lt;math&gt;\,H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \Complex^{m\times n}&lt;/math&gt; and &lt;math&gt;\,p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;\,p_i&lt;/math&gt; its complex conjugate is also a pole, and whenever &lt;math&gt;p_i = \bar{p_j}&lt;/math&gt; we have &lt;math&gt;R_i=\bar{R_j}&lt;/math&gt;.<br /> <br /> We want to describe the system as simply as possible. That is to say, we are looking for &lt;math&gt;\,H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with accuracy &lt;math&gt;\,\epsilon &gt; 0&lt;/math&gt;.<br /> The matrices &lt;math&gt;\,G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;\,H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> &lt;br /&gt;&lt;br /&gt;<br /> &lt;center&gt;&lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;&lt;/center&gt;<br /> <br /> This optimization problem can be expressed as a SDP problem as discussed in the previous section.<br /> <br /> ==Numerical Example==<br /> <br /> Consider a [http://en.wikibooks.org/wiki/Control_Systems/MIMO_Systems MIMO system] (i.e. multiple inputs and outputs); in particular, we will analyze one with 2 inputs and 2 outputs. We have a normalized transfer matrix &lt;math&gt;F&lt;/math&gt; (i.e. &lt;math&gt;\|F\|_{\infty} = 1&lt;/math&gt;). Additionally, assume this matrix is of order 8 so that we have poles &lt;math&gt;p_1,\dots,p_8&lt;/math&gt; which come in pairs (i.e. complex conjugates). The goal is to reduce the order while minimizing the information lost.<br /> <br /> Solving the SDP representation of the problem discussed in the previous section (&lt;math&gt;\epsilon = 0.05&lt;/math&gt;) gives an approximation of order 6. Additionally, the 8th order and 6th order representations are similar and are the same in many cases.<br /> <br /> ==References==<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=sign_up_for_your_presentation&diff=10169 sign up for your presentation 2010-11-25T20:36:56Z <p>Mderakhs: /* Set A */</p> <hr /> <div>Sign up for your presentation in the following table. Put your name and a link to the paper that you are going to present. Chose a date between Nov 16 and Dec 2 (inclusive).<br /> <br /> == Set A ==<br /> <br /> {| class=&quot;wikitable&quot;<br /> <br /> {| border=&quot;1&quot; cellpadding=&quot;3&quot;<br /> |-<br /> |width=&quot;100pt&quot;|Date<br /> |width=&quot;200pt&quot;|Presentation (1)<br /> |width=&quot;200pt&quot;|Presentation (2)<br /> |-<br /> |-<br /> |Nov 16|| Sepideh Seifzadeh [http://www.cs.berkeley.edu/~jordan/papers/lacoste-sha-jordan-nips08.pdf] [[DiscLDA: Discriminative Learning for Dimensionality Reduction and Classification|Summary]] || Manda Winlaw [http://www-stat.stanford.edu/~tibs/Correlate/pmd.pdf], [[A Penalized Matrix Decomposition, with Applications to Sparse Principal Components and Canonical Correlation Analysis|Summary]]<br /> |-<br /> |-<br /> |Nov 18||Rahul [http://dsp.rice.edu/files/cs/baraniukCSlecture07.pdf][[Compressive Sensing| Summary]]||Fatemeh Dorri [http://www.cs.berkeley.edu/~jordan/papers/daspremont-siam-review.pdf],[[A Direct Formulation For Sparse PCA Using Semidefinite Programming|Summary]]<br /> |-<br /> |-<br /> |Nov 23 || Pouria Fewzee ||Lisha Yu[http://books.nips.cc/papers/files/nips21/NIPS2008_0197.pdf][[Deflation_Methods_for_Sparse_PCA|Summary]]<br /> |-<br /> |-<br /> |Nov 25 ||Laleh Ghoraie [http://arxiv.org/abs/0903.3131][[Matrix_Completion_with_Noise| Summary]] || Mehrdad Gangeh [http://pami.uwaterloo.ca/~mgangeh/SupervisedDictionaryLearning.pdf][[Supervised Dictionary Learning|Summary]]<br /> |-<br /> |-<br /> |Nov 30 || Greg D'Cunha [http://www.stanford.edu/~hllee/icml07-selftaughtlearning.pdf] [[Self-Taught Learning| Summary]]|| Mohammad Derakhshani [http://faculty.washington.edu/mfazel/nucnorm_acc_final.pdf],<br /> [[A Rank Minimization Heuristic with Application to Minimum Order System Approximation|Summary]]<br /> |-<br /> |Dec 2 || Yongpeng Sun [http://www.google.ca/url?sa=t&amp;source=web&amp;cd=1&amp;ved=0CBUQFjAA&amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.76.5871%26rep%3Drep1%26type%3Dpdf&amp;rct=j&amp;q=%22Uncovering%20shared%20structures%20in%20multiclass%20classification.%22&amp;ei=5We3TJTVOYH98Abi3czpCQ&amp;usg=AFQjCNF6MSD0BNolGNe4z6d1RKeR7ZWJsw&amp;sig2=B9YHZC8V9q6nIPa4vJKr-g&amp;cad=rja] [http://www.wikicoursenote.com/wiki/Uncovering_Shared_Structures_in_Multiclass_Classification Summary] || Ryan Case [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.92.4570&amp;rep=rep1&amp;type=pdf]<br /> |-<br /> |-<br /> |}<br /> |}<br /> <br /> == Set B ==<br /> <br /> <br /> {| class=&quot;wikitable&quot;<br /> <br /> {| border=&quot;1&quot; cellpadding=&quot;2&quot;<br /> |-<br /> |width=&quot;100pt&quot;|Name<br /> |width=&quot;900pt&quot;|Second paper (The paper that you are going to write a critic on it. This is different from the paper that you have chosen for presentation.) <br /> |-<br /> |Greg D'Cunha || Probabilistic matrix factorization [http://www.google.ca/url?sa=t&amp;source=web&amp;cd=1&amp;ved=0CBkQFjAA&amp;url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.127.6198%26rep%3Drep1%26type%3Dpdf&amp;rct=j&amp;q=Probabilistic%20matrix%20factorization&amp;ei=Crm9TPDpDIiosAOHhqmYDQ&amp;usg=AFQjCNHdgX8UXD5fthc85O4lJxH-bRD86Q&amp;sig2=OgSnMU_ax6PT0XeY3UGS9A&amp;cad=rja] [[Probabilistic Matrix Factorization | Summary]]<br /> |-<br /> |Fatemeh Dorri||Optimal Solutions forSparse Principal Component Analysis[http://www.princeton.edu/~aspremon/OptSPCA.pdf]<br /> |-<br /> |Lisha Yu|| Maximum‐margin matrix factorization[http://ttic.uchicago.edu/~nati/Publications/MMMFnips04.pdf]<br /> |-<br /> |Ryan Case|| Probabilistic non-linear principal component analysis with Gaussian process latent variable models [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.66.8581&amp;rep=rep1&amp;type=pdf]<br /> |-<br /> |-<br /> |Yongpeng Sun|| Multi‐Task Feature Learning [http://books.nips.cc/papers/files/nips19/NIPS2006_0251.pdf]<br /> |-<br /> |Manda Winlaw|| Consistency of trace norm minimization [http://www.di.ens.fr/~fbach/bach08a.pdf]<br /> |-</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10046 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:29:16Z <p>Mderakhs: /* Minimum Order System Approximation */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a ratioanl matrix &lt;math&gt;H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in C^{m\times n}&lt;/math&gt; and &lt;math&gt;p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;p_i&lt;/math&gt; its compelx conjucate is also a pole, and whenever &lt;math&gt;p_i = p_j^*&lt;/math&gt; we have &lt;math&gt;R_i=R_j^*&lt;/math&gt;.<br /> <br /> We want the simpler description for the system. That is to say we are looking for &lt;math&gt;H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with athe accuracy of &lt;math&gt;\epsilon&lt;/math&gt;.<br /> So the &lt;math&gt;G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This optimization problem can be expressed in SDP representation using above discussion.<br /> <br /> <br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10045 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:28:51Z <p>Mderakhs: /* Minimum Order System Approximation */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a ratioanl matrix &lt;math&gt;H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in C^{m\times n}&lt;/math&gt; and &lt;math&gt;p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;p_i&lt;/math&gt; its compelx conjucate is also a pole, and whenever &lt;math&gt;p_i = p_j^*&lt;/math&gt; we have &lt;math&gt;R_i=R_j^*&lt;/math&gt;.<br /> <br /> We want the simpler description for the system. That is to say we are looking for &lt;math&gt;H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with athe accuracy of &lt;math&gt;\epsilon&lt;/math&gt;.<br /> So the &lt;math&gt;G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This optimization problem can be expressed in SDP representation using the heuristic.<br /> <br /> <br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10044 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:28:19Z <p>Mderakhs: /* Minimum Order System Approximation */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a ratioanl matrix &lt;math&gt;H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in C^{m\times n}&lt;/math&gt; and &lt;math&gt;p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;p_i&lt;/math&gt; its compelx conjucate is also a pole, and whenever &lt;math&gt;p_i = p_j^*&lt;/math&gt; we have &lt;math&gt;R_i=R_j^*&lt;/math&gt;.<br /> <br /> We want the simpler description for the system. That is to say we are looking for &lt;math&gt;H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with athe accuracy of &lt;math&gt;\epsilon&lt;/math&gt;.<br /> So the &lt;math&gt;G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> \end{array}<br /> <br /> This optimization problem can be expressed in SDP representation using the heuristic.<br /> &lt;/math&gt;<br /> <br /> <br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10043 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:26:43Z <p>Mderakhs: /* Minimum Order System Approximation */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a ratioanl matrix &lt;math&gt;H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in C^{m\times n}&lt;/math&gt; and &lt;math&gt;p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;p_i&lt;/math&gt; its compelx conjucate is also a pole, and whenever &lt;math&gt;p_i = p_j^*&lt;/math&gt; we have &lt;math&gt;R_i=R_j^*&lt;/math&gt;.<br /> <br /> We want the simpler description for the system. That is to say we are looking for &lt;math&gt;H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with athe accuracy of &lt;math&gt;\epsilon&lt;/math&gt;.<br /> So the &lt;math&gt;G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, \quad k=1,\dots,K<br /> <br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> <br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10042 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:26:22Z <p>Mderakhs: /* Minimum Order System Approximation */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a ratioanl matrix &lt;math&gt;H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in C^{m\times n}&lt;/math&gt; and &lt;math&gt;p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;p_i&lt;/math&gt; its compelx conjucate is also a pole, and whenever &lt;math&gt;p_i = p_j^*&lt;/math&gt; we have &lt;math&gt;R_i=R_j^*&lt;/math&gt;.<br /> <br /> We want the simpler description for the system. That is to say we are looking for &lt;math&gt;H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with athe accuracy of &lt;math&gt;\epsilon&lt;/math&gt;.<br /> So the &lt;math&gt;G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp;\sum_{i=1}^N \mbox{Rank}(R_i) \\<br /> \mbox{subject to: } &amp; \|H(j\omega_k)-G_k\| \leq \epsilon, k=1,\dots,K<br /> <br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> <br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10041 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:23:20Z <p>Mderakhs: /* Minimum Order System Approximation */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a ratioanl matrix &lt;math&gt;H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in C^{m\times n}&lt;/math&gt; and &lt;math&gt;p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;p_i&lt;/math&gt; its compelx conjucate is also a pole, and whenever &lt;math&gt;p_i = p_j^*&lt;/math&gt; we have &lt;math&gt;R_i=R_j^*&lt;/math&gt;.<br /> <br /> We want the simpler description for the system. That is to say we are looking for &lt;math&gt;H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> The transfer matrix is measured for a few frequencies, &lt;math&gt;\omega_1,\dots,\omega_K\in \real&lt;/math&gt; with athe accuracy of &lt;math&gt;\epsilon&lt;/math&gt;.<br /> So the &lt;math&gt;G_k&lt;/math&gt; are given as the measured approximation of &lt;math&gt;H(j\omega_k)&lt;/math&gt;.<br /> Therefore we have the following minimization problem:<br /> <br /> <br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10040 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:19:02Z <p>Mderakhs: /* Minimum Order System Approximation */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a ratioanl matrix &lt;math&gt;H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in C^{m\times n}&lt;/math&gt; and &lt;math&gt;p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;p_i&lt;/math&gt; its compelx conjucate is also a pole, and whenever &lt;math&gt;p_i = p_j^*&lt;/math&gt; we have &lt;math&gt;R_i=R_j^*&lt;/math&gt;.<br /> <br /> We want the simpler description for the system. That is to say we are looking for &lt;math&gt;H&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \deg(H) = \sum_{i=1}^N \mbox{Rank}(R_i)<br /> &lt;/math&gt;<br /> is minimized.<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10039 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:16:20Z <p>Mderakhs: /* Minimum Order System Approximation */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a ratioanl matrix &lt;math&gt;H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in C^{m\times n}&lt;/math&gt; and &lt;math&gt;p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;p_i&lt;/math&gt; its <br /> compelx conjucate is also a pole, and whenever &lt;math&gt;p_i = p_j^*&lt;/math&gt; we have &lt;math&gt;R_i=R_j^*&lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10038 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:16:10Z <p>Mderakhs: /* Minimum Order System Approximation */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a ratioanl matrix &lt;math&gt;H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \complex^{m\times n}&lt;/math&gt; and &lt;math&gt;p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex &lt;math&gt;p_i&lt;/math&gt; its <br /> compelx conjucate is also a pole, and whenever &lt;math&gt;p_i = p_j^*&lt;/math&gt; we have &lt;math&gt;R_i=R_j^*&lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10037 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:15:36Z <p>Mderakhs: /* Minimum Order System Approximation */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a ratioanl matrix &lt;math&gt;H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in C{m\times n}&lt;/math&gt; and &lt;math&gt;p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex $p_i$ its <br /> compelx conjucate is also a pole, and whenever &lt;math&gt;p_i = p_j^*&lt;/math&gt; we have &lt;math&gt;R_i=R_j^*&lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10036 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:15:16Z <p>Mderakhs: /* Minimum Order System Approximation */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a ratioanl matrix &lt;math&gt;H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \complex^{m\times n}&lt;/math&gt; and &lt;math&gt;p_i&lt;/math&gt; are the complex poles of the system with the property that for each complex $p_i$ its <br /> compelx conjucate is also a pole, and whenever &lt;math&gt;p_i = p_j^*&lt;/math&gt; we have &lt;math&gt;R_i=R_j^*&lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10035 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:14:57Z <p>Mderakhs: /* Expressing as an SDP */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> <br /> == Minimum Order System Approximation ==<br /> <br /> The rank minimization heuristic can be used in the minimum order system approximation problem.<br /> In system theory, the effect of a system can be modeled using a ratioanl matrix &lt;math&gt;H(s)&lt;/math&gt;:<br /> &lt;math&gt; H(s) = R_0 +\sum_{i=1}^N \frac{R_i}{s-p_i}&lt;/math&gt; where &lt;math&gt;R_i \in \complex^{m\times n}&lt;/math&gt; and $p_i$ are the complex poles of the system with the property that for each complex $p_i$ its <br /> compelx conjucate is also a pole, and whenever &lt;math&gt;p_i = p_j^*&lt;/math&gt; we have &lt;math&gt;R_i=R_j^*&lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10031 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:01:20Z <p>Mderakhs: /* A Generalization Of The Trace Heuristic */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === Nuclear Norm Heuristic A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> <br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10030 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:00:30Z <p>Mderakhs: /* Expressing as an SDP */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^T&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10029 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T04:00:11Z <p>Mderakhs: /* Expressing as an SDP */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as <br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> where &lt;math&gt;Y=Y^T, Z=Z^&lt;/math&gt; are new variables. If the constraint set &lt;math&gt;C&lt;/math&gt; can be expressed as linear matrix inequalityes then the problem is an SDP, and can be solved using available SDP solvers.<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10027 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:57:48Z <p>Mderakhs: /* Expressing as an SDP */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10025 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:57:37Z <p>Mderakhs: /* Expressing as an SDP */</p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> can be expressed as<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr}(Y) + \mbox{Tr}(Z) \\<br /> \mbox{subject to: } &amp; \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right] \leq 0 \\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10024 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:56:24Z <p>Mderakhs: </p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> <br /> Using this lemma the nuclear norm minimization problem<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; t \\<br /> \mbox{subject to: } &amp; \|X\|_*\leq t\\<br /> &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10023 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:50:30Z <p>Mderakhs: </p> <hr /> <div>=== Introduction ===<br /> Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10022 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:49:42Z <p>Mderakhs: /* Expressing as an SDP */</p> <hr /> <div>Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in \real&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10021 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:49:21Z <p>Mderakhs: /* Expressing as an SDP */</p> <hr /> <div>Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in R&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0, \quad \mbox{Tr}(Y) + \mbox{Tr}(Z) \leq 2t<br /> &lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10020 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:48:17Z <p>Mderakhs: /* Expressing as an SDP */</p> <hr /> <div>Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in R&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \left[\begin{array}{cc}<br /> Y &amp; X\\<br /> X^T &amp; Z<br /> \end{array}\right]\geq 0<br /> &lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10019 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:47:54Z <p>Mderakhs: </p> <hr /> <div>Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in R&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \begin{array}{cc}<br /> Y &amp; X\\<br /> X^T Z<br /> \end{array}\geq 0<br /> &lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10018 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:47:25Z <p>Mderakhs: /* Expressing as an SDP */</p> <hr /> <div>Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in R&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \begin{array}<br /> Y &amp; X\\<br /> X^T Z<br /> \end{array}\geq 0<br /> &lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10017 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:47:10Z <p>Mderakhs: /* Expressing as an SDP */</p> <hr /> <div>Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in R&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> <br /> &lt;math&gt;<br /> \begin{array}<br /> Y &amp; X<br /> X^T Z<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10016 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:46:38Z <p>Mderakhs: /* Expressing as an SDP */</p> <hr /> <div>Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in R&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n}&lt;/math&gt; such that<br /> &lt;math&gt;<br /> \begin{array}<br /> Y &amp; X<br /> X^T Z<br /> \end{array}\geq 0, <br /> <br /> &lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10015 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:46:10Z <p>Mderakhs: /* References */</p> <hr /> <div>Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> === Expressing as an SDP ===<br /> <br /> To express the relaxed version as a SDP we need to express the constraints by linear matrix inequalityes (LMIs). <br /> <br /> '''Lemma 1''' For &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt; and &lt;math&gt;t\in R&lt;/math&gt;, we have &lt;math&gt;R^{m\times m}&lt;/math&gt; and &lt;math&gt;Z\in \real^{n\times n} such that<br /> &lt;math&gt;<br /> \begin{array}<br /> Y &amp; X<br /> X^T Z<br /> \end{array}\geq 0, <br /> <br /> &lt;/math&gt;<br /> <br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs http://wiki.math.uwaterloo.ca/statwiki/index.php?title=a_Rank_Minimization_Heuristic_with_Application_to_Minimum_Order_System_Approximation&diff=10011 a Rank Minimization Heuristic with Application to Minimum Order System Approximation 2010-11-24T03:41:14Z <p>Mderakhs: /* A Generalization Of The Trace Heuristic */</p> <hr /> <div>Rank Minimization Problem (RMP) has application in a variety of areas such as control, system identification, statistics and signal processing. Except in some special cases RMP is known to be computationaly hard. <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Rank } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> If the matrix is symmetric and positive semidifinite, trace minimization is a very effective heuristic for rank minimization problem. The trace minimization results in a semidefinite problem which can be easily solved.<br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \mbox{Tr } X \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> This paper focuses on the following problems:<br /> #Describing a generalization of the trace heuristic for genaral non-square matrices.<br /> #Showing that the new heuristic can be reduced to an SDP, and hence effictively solved.<br /> #Applying the mothod on the minimum order system approximation.<br /> <br /> === A Generalization Of The Trace Heuristic ===<br /> This heurisitic minimizes the sum of the singular values of the matrix &lt;math&gt;X\in \real^{m\times n}&lt;/math&gt;, which is the nuclear norm of &lt;math&gt;X&lt;/math&gt; denoted by &lt;math&gt;\|X\|_*&lt;/math&gt;.<br /> <br /> &lt;math&gt;<br /> \begin{array}{ l l }<br /> \mbox{minimize} &amp; \|X\|_* \\<br /> \mbox{subject to: } &amp; X \in C<br /> \end{array}<br /> &lt;/math&gt;<br /> <br /> According to the definition of the nuclear norm we have &lt;math&gt;\|X\|_*=\sum_{i=1}^{\min\{m,n\} }\sigma_i(X)&lt;/math&gt; where &lt;math&gt; \sigma_i(X) = \sqrt{\lambda_i (X^TX)}&lt;/math&gt;.<br /> <br /> The nuclear norm is dual of the spectrial norm <br /> &lt;math&gt;\|X\|_* =\sup \{ \mbox{Tr } Y^T X | \|Y\| \leq 1 \}&lt;/math&gt;. So the relaxed version of the rank minimization problem is a convex optimization problem.<br /> <br /> When the matrix variable &lt;math&gt;X&lt;/math&gt; is symmetric and positive semidefinite, then its singular values are the same as its eigenvalues, and therefore the nuclear norm reduces to &lt;math&gt;\mbox{Tr } X&lt;/math&gt;, and that means the heuristic reduces to the trace minimization heuristic.<br /> <br /> ==== Nuclear Norm Minimization vs. Rank Minimization ====<br /> <br /> [[Image:Convex Envelope.png|thumb|200px|right|convex envelope of a function, borrowed from &lt;ref&gt;Rank Minimization and Applications in System Theory, M. Fazel, H. Hindi, and S. Body&lt;/ref&gt;]]<br /> ''' Definition:''' Let &lt;math&gt;f:C \rightarrow\real&lt;/math&gt; where &lt;math&gt;C\subseteq \real^n&lt;/math&gt;. The convex envelope of &lt;math&gt;f&lt;/math&gt; (on &lt;math&gt;C&lt;/math&gt;) is defined as the largest convex function &lt;math&gt;g&lt;/math&gt; such that &lt;math&gt;g(x)\leq f(x)&lt;/math&gt; for all &lt;math&gt;x\in X&lt;/math&gt;.<br /> <br /> '''Theorem 1''' The convex envelope of the function &lt;math&gt;\phi(X)=\mbox{Rank }(X)&lt;/math&gt;, on &lt;math&gt;C=\{X\in \real^{m\times n} | \|X\|\leq 1\} &lt;/math&gt; is &lt;math&gt;\phi_{\mbox{env}}(X) = \|X\|_*&lt;/math&gt;.<br /> <br /> <br /> Suppose &lt;math&gt;X\in C&lt;/math&gt; is bounded by &lt;math&gt;M&lt;/math&gt; that is &lt;math&gt;\|X\|\leq M&lt;/math&gt;, then the convex envelope of &lt;math&gt;\mbox{Rank }X&lt;/math&gt; on &lt;math&gt;\{X | \|X\|\leq M\}&lt;/math&gt; is given by &lt;math&gt;\frac{1}{M}\|X\|_*&lt;/math&gt;. <br /> <br /> &lt;math&gt;\mbox{Rank } X \geq \frac{1}{M} \|X\|_*&lt;/math&gt;<br /> That means if &lt;math&gt;p_{\mbox{rank}}&lt;/math&gt; and &lt;math&gt;p_{*}&lt;/math&gt; are the optimal values of the rank minimization problem and dual spectrial norm minimization problem then we have<br /> &lt;math&gt;p_{\mbox{rank}}\geq \frac{1}{M} p_{*}&lt;/math&gt;<br /> <br /> ===References===<br /> &lt;references /&gt;</div> Mderakhs