General ML Regularization Techniques
- Early Stopping
- L1／L2 Regularization
- Max Norm Constraints／Regularization
- Regularization - Parameter Weight Decay
ANN Specific Regularization Techniques
Regularization Methods Comparisons on MNIST
|
Method |
Test Classification Error % |
|---|---|
|
L2 |
1.62 |
|
L2 + L1 applied towards the end of training |
1.60 |
|
L2 + KL-sparsity |
1.55 |
|
1.35 | |
|
Dropout + L2 |
1.25 |
|
1.05 |
From Dropout: A Simple Way to Prevent Neural Networks from Overfitting