／var／logmarcus chiu

Fisher Information - Fisher Information Matrix

Created on · Last Modified on

Fisher Information
  • is a way of measuring the amount of information an observation of a random variable𝑋 has about an unknown parameter 𝜃 of a parametric-distribution 𝐏(𝑋|𝜃) that models 𝑋 (note: 𝐿(𝑋|𝜃) = 𝐏(𝑋|𝜃))

Fisher Information Matrix
  • generalizes the Fisher Information to a vector parameter 𝜽

Fisher Information - Formulas

Fisher Information 𝐼(𝜃) is the variance of the score function 𝑠(𝜃) (i.e. 𝑠(𝜃) = (𝛿/𝛿𝜃) 𝑙𝑜𝑔𝐿(𝑋|𝜃) = [1/𝐿(𝑋|𝜃)] · (𝛿/𝛿𝜃)𝐿(𝑋|𝜃))

  • 𝐼(𝜃) = 𝑉𝑎𝑟~𝑋[𝑠(𝜃)]
  • 𝐼(𝜃) = 𝐄~𝑋[𝑠(𝜃)2] - (𝐄~𝑋[𝑠(𝜃)])2
  • 𝐼(𝜃) = 𝐄~𝑋[𝑠(𝜃)2] - 0 # 𝐄~𝑋[𝑠(𝜃)] = 0 see section in Score Function

1st formula
  • 𝐼(𝜃) = 𝐄~𝑋[𝑠(𝜃)2]
  • 𝐼(𝜃) = 𝐄~𝑋[((𝛿/𝛿𝜃) 𝑙𝑜𝑔𝐏(𝑋|𝜃))2]
  • 𝐼(𝜃) = 𝐄~𝑋[((𝛿/𝛿𝜃) 𝑙𝑜𝑔𝐿(𝑋|𝜃))2] # where 𝑙𝑜𝑔𝐿 is the log-likelihood-function

2nd formula
  • 𝐼(𝜃) = - 𝐄~𝑋[(𝛿/𝛿𝜃) 𝑠(𝜃)]
  • 𝐼(𝜃) = - 𝐄~𝑋[(𝛿2/𝛿𝜃2) 𝑙𝑜𝑔𝐏(𝑋|𝜃)]
  • 𝐼(𝜃) = - 𝐄~𝑋[(𝛿2/𝛿𝜃2) 𝑙𝑜𝑔𝐿(𝑋|𝜃)] # where 𝑙𝑜𝑔𝐿 is the log-likelihood-function

Fisher Information - Example Computations

Fisher Information - Resources