Overfitting, L1 and L2 regularization, gradient descent variants, learning rates, Adam, early stopping and dropout.
39 cards · basic cards · AI-written, checked twice. Edit anything.
- What is overfitting?
Show answer
A model learns the training data too well, including noise, so it performs poorly on new unseen data.- How does overfitting typically appear in a learning curve?
Show answer
Training error continues to decrease while validation error starts increasing.- Name one common cause of overfitting.
Show answer
The model is too complex relative to the amount of training data.- Why is a validation set critical for detecting overfitting?
Show answer
It estimates performance on unseen data independent of the training set.- What term does L1 regularization add to the loss function?
Show answer
Lambda times the sum of absolute values of weights.- What is L1 regularization also called?
Show answer
Lasso regularization.- Which norm does L1 regularization minimize, the L1 or L2 norm?
Show answer
The L1 norm (sum of absolute values).- What feature selection property does L1 regularization have?
Show answer
It can drive some weights exactly to zero, effectively removing features.- How does L1 regularization penalize small weights versus large weights?
Show answer
It penalizes all weights equally per unit of magnitude.- What term does L2 regularization add to the loss function?
Show answer
Lambda times the sum of squared weights.- What is L2 regularization also called?
Show answer
Ridge regularization.- Which norm does L2 regularization minimize, the L1 or L2 norm?
Show answer
The L2 norm (square root of sum of squares).- How does L2 regularization affect weight magnitudes?
Show answer
It encourages all weights to be smaller but rarely drives them exactly to zero.- In what scenario might L2 regularization be preferred over L1?
Show answer
When all features are potentially useful and you want to keep them all but reduce their magnitude.- What is batch gradient descent?
Show answer
Computes gradients on the entire dataset before updating weights once per epoch.
FlashKeepers