Question

Which of the following statements is true for Ridge Regression?

• A. The regularizer in the objective function of Ridge Regression is used to guard against scenarios where the model works well for the test data, but poorly for the training data.
• B. The regularizer of Ridge Regression uses \(L_1\) norm.
• C. Ridge Regression aims to reduce the number of parameters that have negative values.
• D. The regularizer of Ridge Regression may increase the bias of the model, but it helps in reducing the variance in predictions.

Hint:

Ridge Regression's penalty is the squared L2 norm; think about what shrinking weights does to bias and variance.

Answer 1

The Correct Option is D

Ridge Regression modifies ordinary least squares by adding a penalty on the size of the weights: $\text{Cost} = \text{RSS} + \lambda \sum w_i^2$. We check each statement against this definition.

1. The regularizer guards against the model working well on test data but poorly on training data: This gets overfitting backwards. Overfitting means good performance on training data and poor performance on test data, not the reverse, so this statement is false.
2. The regularizer uses $L_1$ norm: Ridge's penalty term is $\sum w_i^2$, the squared $L_2$ norm. The $L_1$ norm penalty ($\sum |w_i|$) is what Lasso Regression uses instead, so this is false for Ridge.
3. Ridge Regression reduces the number of parameters with negative values: Ridge shrinks all weights toward zero by roughly the same proportion, but it does not target the sign of the weights or force any of them to zero, so this statement misrepresents what the penalty does.
4. The regularizer may increase bias but reduces variance: Since the penalty shrinks weights away from their unconstrained least-squares fit, it introduces a small, controlled bias. In exchange, the model becomes far less sensitive to noise in the training set, so its variance across different samples drops. This trade-off is precisely the mechanism Ridge Regression uses to reduce overfitting, which makes this statement true.

The correct option is the fourth one: Ridge Regression's $L_2$ penalty increases bias slightly while reducing variance, which is the standard bias-variance trade-off behind regularization.

Let's summarize:

• Ridge Regression uses an $L_2$ (squared-weight) penalty, not $L_1$.
• It shrinks weights but does not zero them out or specifically target negative values.
• Its main effect is trading a bit of bias for a large drop in variance, which reduces overfitting.

So the statement in option (D) is the one that correctly describes Ridge Regression.