Mathematical Optimization Strategies

strategies for solving the mathematical optimization problem:

• Grid Search or Brute Force Approach - TODO
• Random Search - TODO
• Hill Climbing - assumes you optimize discrete problems (e.g. traveling salesman problem) and can evaluate a cost function for a given state
• Ascent - assumes you optimize a continuous function and can compute its gradient/slope in a given state
• Bayesian Optimization - TODO

Mathematical Optimization Strategies - Comparison

given a simple 1 hyperparameter and the corresponding function:

• whiter points correspond to parameter/hyperparameter values generated earlier in the process
• redder points correspond to parameter/hyperparameter values generated later on in the process

Resources

• https://thenextweb.com/podium/2019/11/11/machine-learning-algorithms-and-the-art-of-hyperparameter-selection/amp/