Maxent - NLP PoS Example

let’s say we have the following event space:

• NN NNS NNP NNPS VBZ VBD

and the following empirical distribution:

• 3 5 11 13 3 1

maximize entropy 𝐻 (un-normalized distribution):

• 1/𝑒 1/𝑒 1/𝑒 1/𝑒 1/𝑒 1/𝑒

maximize entropy 𝐻 with respect to normalized probability distribution. let’s add a constraint feature 𝑓₀ = {NN, NNS, NNP, NNPS, VBZ, VBD} with 𝐄[𝑓₀] = 1:

• 1/6 1/6 1/6 1/6 1/6 1/6

from the empirical distribution we see that 𝑁* are more common 𝑉*. let’s add another constraint feature 𝑓₁ = {NN, NNS, NNP NNPS} with 𝐄[𝑓₁] = 32/36:

• 8/36 8/36 8/36 8/36 2/36 2/36

we also see that proper nouns are more frequent than common nouns. let’s add another constraint feature 𝑓₂ = {NNP NNPS} with 𝐄[𝑓₂] = 2/3:

• 4/36 4/36 12/36 12/36 2/36 2/36

we could keep refining the model (e.g. by adding a feature to distinguish singular vs plural nouns, or verb types)