• Non-Diagonalizable/Defective Matrix - a matrix that is not diagonalizable
• Diagonalizable/Non-Defective Matrix - a matrix that is diagonalizable

Eigen Decomposition

• is the factorization of a square matrix into a canonical form, whereby the matrix is represented in terms of its eigenvalues and eigenvectors)
• for the eigendecomposition of rectangular matrices see Singular Value Decomposition
• only diagonalizable matrices can be factorized this way
• when the matrix being factorized is a normal or real symmetric matrix, the decomposition is called “spectral decomposition”, derived from the spectral theorem
• diagonalizing a matrix 𝐴 is the same process as finding 𝐴‘s eigenvalues and eigenvectors

A 𝑛✕𝑛 matrix 𝐴 is diagonalizable iff 𝐴 has 𝑛 linearly independent eigenvectorsA 𝑛✕𝑛 matrix 𝐴 is diagonalizable iff 𝐴 can be decomposed into:

• 𝐴 = 𝑃𝐷𝑃^-1

where:

• 𝐷 - diagonal 𝑛✕𝑛 matrix of 𝑛 eigenvalues as diagonal
• 𝑃 - nonsingular 𝑛✕𝑛 matrix of 𝑛 eigenvectors of 𝐴 as columns

Eigen Decomposition - Examples

Eigen Decomposition - Computation

• compute the eigenvalues (𝜆𝑠) of matrix 𝐴 (i.e. 0 = 𝑑𝑒𝑡𝑒𝑟𝑚𝑖𝑛𝑎𝑛𝑡(𝐴 - 𝜆𝐼) = 𝑐ℎ𝑎𝑟𝑎𝑐𝑡𝑒𝑟𝑖𝑠𝑡𝑖𝑐-𝑝𝑜𝑙𝑦𝑛𝑜𝑚𝑖𝑎𝑙_𝐴(𝜆))
• compute the eigenvectors (𝑣𝑠) of matrix 𝐴 (i.e. solve (𝐴 - 𝜆𝐼)𝑣 = 0 for each eigenvalue 𝜆)
• assemble the eigenvectors as column vectors of matrix 𝑃
• assemble the eigenvalues as a diagonal of matrix 𝐷
• thus 𝐴 is diagonalized to 𝑃𝐷𝑃 ^-1

a property of eigenvalues is that multiplying a 𝑛✕𝑛 matrix 𝐴 by its eigenvector 𝑣𝑖 is the same as multiplying that eigenvector by its eigenvalue 𝜆𝑖. In both cases it scales the eigenvector:

Indent

𝐴𝑣 = 𝜆𝑣

now combine all 𝑛 eigenvectors and eigenvalues of 𝐴 into a single equation:

Indent

𝐴𝑃 = 𝑃𝐷
𝐴𝑃𝑃 ^-1 = 𝑃𝐷𝑃 ^-1
𝐴𝐼 = 𝑃𝐷𝑃 ^-1
𝐴 = 𝑃𝐷𝑃 ^-1

Eigen Decomposition - Change of Basis Understanding

Transformations 𝐷 and 𝐴 are the same but expressed in different basis vectors:

• 𝐴 is expressed as the standard basis
• 𝐷 is expressed as the eigenbasis (i.e. the eigenvectors of 𝐴)

𝑃^-1𝐴𝑃 = 𝐷

• 𝑃 is a change of basis matrix that does a transformation from the eigenbasis to the standard basis
• 𝐴 does the transformation expressed in the standard basis
• 𝑃^-1 is a change of basis matrix that does a transformation from the standard basis to the eigenbasis

𝐴 = 𝑃𝐷𝑃^-1

• 𝑃^-1 is a change of basis matrix that does a transformation from the standard basis to the eigenbasis
• 𝐷 does the transformation expressed in the eigenbasis
• 𝑃 is a change of basis matrix that does a transformation from the eigenbasis to the standard basis

Eigen Decomposition - Orthogonally Diagonalizable

• if matrix 𝐴 is symmetric/positive semi-definite, then:
  • 𝐴 is orthogonally diagonalizable.
  • the normalized eigenvectors of 𝐴 are mutually orthonormal, thus: 𝑃 ^𝑇𝑃 = 𝑃𝑃 ^𝑇= 𝐼
• thus: 𝑃 ^-1= 𝑃 ^𝑇
• thus: 𝐴 = 𝑃𝐷𝑃 ^-1becomes 𝐴 = 𝑃𝐷𝑃 ^𝑇

see Non-Defective Matrix

Eigen Decomposition - Uniqueness

If a matrix is diagonalized, its diagonal form is unique, up to a permutation of the diagonal entries. This is because the entries on the diagonal must be all the eigenvalues. For instance

Indent

are examples of 2 different ways to diagonalize the same matrix

Eigen Decomposition - Ease of Other Computations

Diagonalizable matrices and maps are especially easy for computations, once their eigenvalues and eigenvectors are known

• 𝑘^𝑡ℎ power of 𝐴 - one can raise a diagonal matrix 𝐷 to the 𝑘^𝑡ℎ power by simply raising the diagonal entries to that power, then compute 𝑃𝐷^𝑘𝑃^-1
  • 𝐴^𝑘 = 𝑃𝐷^𝑘𝑃^-1
• determinant of 𝐴 - same as determinant of a diagonal matrix 𝐷 which is simply the product of all diagonal entries
• given a diagonalizable matrix 𝐴 and a vector 𝑥̅, we could solve: 𝐴^𝑘𝑥̅, by:
  • decomposing the vector 𝑥̅ as a linear combination of eigenvectors of 𝐴:
    • 𝑥̅ = 𝑐₁𝑣̅₁ + … + 𝑐_𝑛𝑣̅_𝑛
  • 𝐴^𝑘𝑥̅ = 𝑐₁(𝜆₁)^𝑘𝑣̅₁ + … + 𝑐₁(𝜆_𝑛)^𝑘𝑣̅_𝑛

Eigen Decomposition - Resources

• Gilbert Strang
• 3Blue1Brown