Marginal Independence (Independence)

Two random variables are independent, statistically independent, marginally independent, or stochastically independent if the realization of one does not affect the probability distribution of the other

the following implies each other:

1. 𝐴 and 𝐵 are Independent
2. 𝐴 ⊥ 𝐵
3. 𝐏(𝐴,𝐵) = 𝐏(𝐴)𝐏(𝐵)
4. 𝐏(𝐴|𝐵) = 𝐏(𝐴)
5. 𝐏(𝐵|𝐴) = 𝐏(𝐵)

with equation 3 we can derive equation 2 and equation 4 as shown below:

𝐏(𝐴|𝐵)          = 𝐏(𝐴)      # equation 3
𝐏(𝐴|𝐵)𝐏(𝐵)      = 𝐏(𝐴)𝐏(𝐵)
𝐏(𝐴,𝐵)          = 𝐏(𝐴)𝐏(𝐵)  # equation 2
𝐏(𝐵|𝐴)𝐏(𝐴)      = 𝐏(𝐴)𝐏(𝐵)
𝐏(𝐵|𝐴)          = 𝐏(𝐵)      # equation 4

Conditional Independence

The following implies each other:

1. 𝐴 and 𝐵 are Conditionally Independent when given 𝐶
2. 𝐴 ⊥ 𝐵 | 𝐶
3. 𝐏(𝐴,𝐵|𝐶) = 𝐏(𝐴|𝐶)𝐏(𝐵|𝐶)
4. 𝐏(𝐴,𝐵|𝐶) = 𝐏(𝐴|𝐵,𝐶)𝐏(𝐵|𝐴,𝐶)
5. 𝐏(𝐴|𝐵,𝐶) = 𝐏(𝐴|𝐶)
6. 𝐏(𝐵|𝐴,𝐶) = 𝐏(𝐵|𝐶)

with equation 2 we can derive equation 4 as shown below:

𝐏(𝐴,𝐵|𝐶) = 𝐏(𝐴,𝐵,𝐶)          / 𝐏(𝐶)
𝐏(𝐴,𝐵|𝐶) = 𝐏(𝐴|𝐵,𝐶) * 𝐏(𝐵,𝐶) / 𝐏(𝐶)
𝐏(𝐴,𝐵|𝐶) = 𝐏(𝐴|𝐵,𝐶) * 𝐏(𝐵|𝐶)
𝐏(𝐴,𝐵|𝐶) = 𝐏(𝐴|𝐶)   * 𝐏(𝐵|𝐶)
therefore
𝐏(𝐴|𝐵,𝐶) = 𝐏(𝐴|𝐶)

Formal Definition

two random events 𝐴 and 𝐵 are conditionally independent given a third event 𝐶 precisely if the occurrence of 𝐴 and the occurrence of 𝐵 are independent events in their conditional probability distribution given 𝐶. In other words, 𝐴 and 𝐵 are conditionally independent given 𝐶 if and only if, given knowledge that 𝐶 occurs, knowledge of whether 𝐴 occurs provides no information on the likelihood of 𝐵 occurring, and knowledge of whether 𝐵 occurs provides no information on the likelihood of 𝐴 occurring

Independence ⟸|⟹ Conditional Independence

independence ⇏ conditional independence

say we have the following

𝐴 = 3 𝐵 = 3 𝐴 and 𝐵 = 1 not (𝐴 or 𝐵) = 4 Ω (sample space) = 9 𝐶 = 𝐴 or 𝐵 = 5

Show 𝐴 and 𝐵 are independent (i.e. 𝐏(𝐴|𝐵) = 𝐏(𝐴))

• 𝐏(𝐴|𝐵) = 𝐏(𝐴,𝐵)/𝐏(𝐵) = (1/9)/(3/9) = 1/3
• 𝐏(𝐴) = (3/9) = 1/3

clearly 𝐏(𝐴|𝐵) = 𝐏(𝐴), therefore 𝐴 and 𝐵 are independent (with respect to the sample space)

Show 𝐴 and 𝐵 are NOT conditionally independent with respect to C (i.e. 𝐏(𝐴|𝐶,𝐵) ≠ 𝐏(𝐴|𝐶))

• 𝐏(𝐴|𝐶,𝐵) = 𝐏(𝐴,𝐵,𝐶)/𝐏(𝐵,𝐶) = (1/9)/(1/9) = 1
• 𝐏(𝐴|𝐶) = 𝐏(𝐴,𝐶)/𝐏(𝐶) = (3/9)/(5/9) = 3/5

clearly 𝐏(𝐴|𝐶,𝐵) ≠ 𝐏(𝐴|𝐶), therefore 𝐴 and 𝐵 are not conditionally independent

conditional independence ⇏ independence

say we have the following

𝐴 = 4 𝐵 = 4 𝐶 = 4 𝐴∩𝐶 = 2 𝐵∩𝐶 = 2 𝐴∩𝐵 = 2 𝐴∩𝐵∩𝐶 = 1 Ω (sample space) = 7

Show that 𝐴 and 𝐵 are conditionally independent with respect to C (i.e. 𝐏(𝐴|𝐵,𝐶) = 𝐏(𝐴|𝐶))

• 𝐏(𝐴,𝐶) = 2/7
• 𝐏(𝐶) = 4/7
• 𝐏(𝐴,𝐵,𝐶) = 1/7
• 𝐏(𝐵,𝐶) = 2/7
• 𝐏(𝐴|𝐶) = 𝐏(𝐴,𝐶)/𝐏(𝐶) = (2/7)/(4/7) = 1/2
• 𝐏(𝐴|𝐵,𝐶) = 𝐏(𝐴,𝐵,𝐶)/𝐏(𝐵,𝐶) = (1/7)/(2/7) = 1/2

clearly 𝐏(𝐴|𝐵,𝐶) = 𝐏(𝐴|𝐶), therefore 𝐴 and 𝐵 are conditionally independent with respect to 𝐶

Show that 𝐴 and 𝐵 are NOT independent (i.e. 𝐏(𝐴|𝐵) ≠ 𝐏(𝐴))

• 𝐏(𝐴|𝐵) = 𝐏(𝐴,𝐵)/𝐏(𝐵) = (2/7)/(4/7) = 1/2
• 𝐏(𝐴) = 4/7

clearly 𝐏(𝐴|𝐵) ≠ 𝐏(𝐴), therefore 𝐴 and 𝐵 are NOT independent