• Feature is an interesting part of the larger raw data
• Feature Function is a function that extracts a particular feature from the larger raw data
• Feature Engineering is the process of using domain knowledge to extract features from raw data and creating feature functions
• Feature-Based Model is a model constructed out of features

Features in Specific Domains

• Computer Vision’s Image Features - Feature Detection - Feature Detectors
• NLP’s Information Retrieval (IR) - Information Extraction (IE)

Feature Functions 𝑓_𝑖(𝑦,𝒙)

𝑓_𝑖: 𝑿×𝑌 → scalar or nominal

• inputs:
  • 𝑿=𝒙: observed data
  • 𝑌=𝑦: target value
• outputs either:
  • scalar variable
  • nominal variable (e.g. {0,1}, {0, 1, 2}, etc)

feature function properties:

feature function expectation

empirical expectation of a feature function 𝑓_𝑖:

• 𝐄_{𝑒𝑚𝑝𝑖𝑟𝑖𝑐𝑎𝑙}[𝑓_𝑖] = 𝛴_{(𝑦,𝒙)∊𝑑𝑎𝑡𝑎}[𝑓_𝑖(𝑦,𝒙)]

model expectation of a feature function 𝑓_𝑖:

• 𝐄_{𝑚𝑜𝑑𝑒𝑙}[𝑓_𝑖] = 𝛴_{(𝑦,𝒙)∊𝑿×𝑌}[𝐏(𝑦,𝒙)·𝑓_𝑖(𝑦,𝒙)]

Feature Conversions - To Numerical

• Embedding

Feature-Based Models

feature-based models:

• have a set of 𝑛 feature functions {𝑓₁, …, 𝑓_𝑛}
• will assign a weight 𝜃_𝑖 to each feature function 𝑓_𝑖:
  • a POSITIVE weight says a particular observation 𝒙 MATCHES a particular class 𝑦
  • a NEGATIVE weight says a particular observation 𝒙 DOES NOT MATCH a particular class 𝑦

some feature-based model types:

• Logistic (Logit) Regression Model
• Linear Regression (LR) Models

Feature-Based Models - Choosing the Right Features

see: Dimensionality Reduction

Feature-Based Linear Models - As Probabilistic Model

make a probabilistic model from the linear combination:

• 𝛴_{1≤𝑖≤𝑛}[𝜃_𝑖𝑓_𝑖(𝑦,𝒙)]

since 𝛴_{1≤𝑖≤𝑛}[𝜃_𝑖𝑓_𝑖(𝑦,𝒙)] can be negative, we need to constrain it above 0. Thus we take the exponential:

• 𝑒^{𝛴_{1≤𝑖≤𝑛}[𝜃_𝑖𝑓_𝑖(𝑦,𝒙)]}

next we normalize it

• 𝐏(𝑌=𝑦|𝑿=𝒙) = 𝑒^{𝛴_{1≤𝑖≤𝑛}[𝜃_𝑖𝑓_𝑖(𝑦,𝒙)]}/ [ 𝛴_{𝑦’∊𝑌}[𝑒^{𝛴_{1≤𝑖≤𝑛}[𝜃_𝑖𝑓_𝑖(𝑦’,𝒙)]}] ]

Resources

• NLP - YouTube