objective function or criterion - the function we want to optimize (either: minimize or maximize)

cost function or loss function or error function:
- an objective function we want to MINIMIZE
- is evaluated against a SINGLE training example:
  - 𝑙(𝜃|𝑥_𝑖,𝑦_𝑖)
expected cost or expected loss or expected error:
- is evaluated against the ENTIRE training set 𝒟:
  - 𝐿(𝜃|𝒟) = 𝐄_{(𝑥,𝑦)~𝒟}[𝑙(𝜃|𝑥,𝑦)]

Example Functions

Resources