Linked to λ: a larger budget t allows larger weights and less regularization. Both sliders describe the same optimum.
Increase λ to pull the optimum toward zero.
The equivalent budget t = R(w*) is calculated at the optimum for λ. The linked sliders show the matching constraint R(w) ≤ t.
Trackpad: click-drag to pan · Two-finger swipe to rotate · Pinch to zoom.
Linear height preserves the curve shape. Use vertical stretch to emphasize the sides.
What is Regularization?
Regularization adds a penalty term to the loss function that discourages large weights. This prevents overfitting by constraining model complexity.
L1 vs L2: The Key Insight
L1 (Lasso) adds |w0| + |w1| — the constraint region is a diamond. Loss contours hit the diamond at its corners (on the axes), pushing weights to exactly zero. This produces sparse models.
L2 (Ridge) adds w0² + w1² — the constraint region is a circle. Loss contours hit the circle at a smooth tangent point, shrinking weights toward zero but rarely reaching it.
What identifies the optimum? In the constraint view, find the lowest point on the red part of the loss surface above the allowed weights. The budget shown is equivalent to the current λ. In the combined view, minimize the sum of loss and penalty. In 2D, the highlighted loss contour touches the equivalent constraint boundary at the optimum.
How to Use
- Drag the 3D surface to rotate and see the loss landscape from any angle
- Switch L1/L2/Elastic/None to see how the penalty reshapes the surface
- Adjust λ to control regularization strength
- Watch the contour view to see the classic textbook diagram update live
- Change loss eccentricity to see how elongated loss contours interact with constraint shapes
- Click the combined surface to drop a ball and watch gradient descent converge
Objective Functions
min L(w) + λ · R(w)
| Type | Penalty R(w) | Constraint Shape | Effect |
|---|---|---|---|
| L1 (Lasso) | Σ|wi| | Diamond | Sparse weights |
| L2 (Ridge) | Σwi² | Circle | Small weights |
| Elastic Net | α·L1 + (1-α)·L2 | Rounded diamond | Both |
Why L1 Produces Sparsity
Geometrically: the diamond corners of the L1 constraint protrude along the axes. Elliptical loss contours are most likely to first touch the diamond at a corner, where one weight is exactly zero. The more elongated the loss contours, the stronger this effect.
Analytically: the L1 gradient is ±1 regardless of weight magnitude, providing a constant force toward zero. L2's gradient is 2w, which weakens as w approaches zero.
Data space
Drag the line to shift it · Shift-drag to rotate it · Click to add a point · Shift-click to remove · Both lines refit immediately; axes stay fixed. Soft blue dashed: unregularized fit · Light violet: optimal fit · Soft amber: selected line
Model: y = b + mx. The landscape uses these points’ mean squared error. By default, both intercept b and slope m are penalized, producing a bowl or pyramid. Uncheck the intercept option to penalize only the slope, producing a trough.
Live math
View settings
Drag to orbit 360° · Scroll to zoom · Shift-drag or right-drag to pan · Touch: pinch and two-finger pan