Gaussian distributions have the nice algebraic property of being closed under conditioning and marginalization. Being closed under conditioning and marginalization means that the resulting distributions from these operations are also Gaussian

Problem

Given a Multivariate Normal (MVN) distribution, calculate its marginal and conditional distribution

Solution

Marginalizing an MVN Distribution

Given a multivariate gaussian distribution 𝐏(𝑋,𝑌) defined below

Indent

Marginalizing the set of random variables 𝑋 from 𝐏(𝑋,𝑌) yields:

• 𝐏(𝑋) = 𝒩(𝜇_𝑋,𝛴_𝑋𝑋)

Marginalizing the set of random variables 𝑌 from 𝐏(𝑋,𝑌) yields:

• 𝐏(𝑌) = 𝒩(𝜇_𝑌,𝛴_𝑌𝑌)

Conditioning an MVN Distribution

Given a multivariate gaussian distribution 𝐏(𝑋,𝑌) defined below

Indent

Conditioning 𝐏(𝑋,𝑌) to yield 𝐏(𝑋|𝑌):

• 𝐏(𝑋|𝑌) = 𝒩(𝜇_𝑋 + 𝛴_𝑋𝑌𝛴_𝑌𝑌^-1(𝑌 - 𝜇_𝑌), 𝛴_𝑋𝑋 - 𝛴_𝑋𝑌𝛴_𝑌𝑌^-1𝛴_𝑌𝑋)

Conditioning 𝐏(𝑌,𝑋) to yield 𝐏(𝑌|𝑋):

• 𝐏(𝑌|𝑋) = 𝒩(𝜇_𝑌 + 𝛴_𝑌𝑋𝛴_𝑋𝑋^-1(𝑋 - 𝜇_𝑋), 𝛴_𝑌𝑌 - 𝛴_𝑌𝑋𝛴_𝑋𝑋^-1𝛴_𝑋𝑌)

This operation is the cornerstone of Gaussian process Regression since it allows Bayesian inference

Note that the new mean only depends on the conditioned variable, while the covariance matrix is independent of this variable

Proof

TODO

Resources

• Marginal_cond_for_multivariate_normal.pdf
• https://stats.stackexchange.com/questions/30588/deriving-the-conditional-distributions-of-a-multivariate-normal-distribution