Nadaraya-Watson Estimator - Formula

• 𝑦ˆ(𝑥) = [𝛴_{1≤𝑖≤𝑛}𝑘_ℎ(𝑥-𝑥_𝑖)𝑦_𝑖] / [𝛴_{1≤𝑖≤𝑛}𝑘_ℎ(𝑥-𝑥_𝑖)]

where:

• 𝑘_ℎ() - kernel
• ℎ - bandwidth

Nadaraya-Watson Estimator - Derivation

• 𝐄[𝑌|𝑋=𝑥] = ∫𝑦·𝑓(𝑦|𝑥)·𝑑𝑦
• 𝐄[𝑌|𝑋=𝑥] = ∫𝑦·𝑓(𝑦,𝑥)/𝑓(𝑥)·𝑑𝑦

Using the kernel density estimation for the joint distribution 𝑓(𝑦,𝑥) and 𝑓(𝑥) with a kernel 𝑘_ℎ()

• 𝑓ˆ(𝑦,𝑥) = (1/𝑛)·𝛴_{1≤𝑖≤𝑛}𝑘_ℎ(𝑥-𝑥_𝑖)𝑘_ℎ(𝑦-𝑦_𝑖)
• 𝑓ˆ(𝑥) = (1/𝑛)·𝛴_{1≤𝑖≤𝑛}𝑘_ℎ(𝑥-𝑥_𝑖)

We get

• 𝐄ˆ[𝑌|𝑋=𝑥] = ∫𝑦·𝑓(𝑦,𝑥)/𝑓(𝑥)·𝑑𝑦
• 𝐄ˆ[𝑌|𝑋=𝑥] = ∫𝑦·𝛴_{1≤𝑖≤𝑛}𝑘_ℎ(𝑥-𝑥_𝑖)𝑘_ℎ(𝑦-𝑦_𝑖)/𝛴_{1≤𝑗≤𝑛}𝑘_ℎ(𝑥-𝑥_𝑗)·𝑑𝑦
• 𝐄ˆ[𝑌|𝑋=𝑥] = 𝛴_{1≤𝑖≤𝑛}𝑘_ℎ(𝑥-𝑥_𝑖)∫𝑦·𝑘_ℎ(𝑦-𝑦_𝑖)·𝑑𝑦/𝛴_{1≤𝑗≤𝑛}𝑘_ℎ(𝑥-𝑥_𝑗)
• 𝐄ˆ[𝑌|𝑋=𝑥] = 𝛴_{1≤𝑖≤𝑛}𝑘_ℎ(𝑥-𝑥_𝑖)𝑦_𝑖/𝛴_{1≤𝑗≤𝑛}𝑘_ℎ(𝑥-𝑥_𝑗)

Nadaraya-Watson Estimator - Other

• Gaussian Process Regression vs Nadaraya-Watson Kernel Regression

Resources

• Justin Esarey - Kernel Density Estimation & Kernel Regression