Interactive Linear Regression

This interactive demo helps you understand how linear regression works with different loss functions. Adjust the sliders to modify the regression line and observe how it affects both MSE (Mean Squared Error) and MAE (Mean Absolute Error) in real-time.

Interactive Elements in This Demo

Adjust the slope and intercept sliders to see how your regression line changes
Generate random data with different sizes and noise levels
Add outliers to see how they affect each loss function differently
Try different example datasets to explore various data patterns
Click directly on the plot to add points; double-click to remove them

Connection to Parameter Space

As seen in Demo 1, every point in parameter space (w₀, w₁) corresponds to a line in feature space. The optimal parameters are found at the lowest point of the loss surface.

Each point on the parameter space surfaces represents a possible model with specific slope and intercept values. The height of the surface shows the loss value for that model. The MSE surface is smooth and bowl-shaped with a unique minimum, while the MAE surface has sharper edges and can have multiple optimal solutions along a line.

Summary and Explanations

Linear Regression Model

Linear regression finds a linear relationship between variables: \( \hat{y} = w_0 + w_1 x \), where \(w_0\) is the intercept and \(w_1\) is the slope.

Loss Functions

Mean Squared Error (MSE)

\[ \text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2 \]

MSE squares the errors, giving more weight to large errors. It produces a smooth, bowl-shaped loss surface with a unique global minimum.

The analytical solution for minimizing MSE is:

\[ w_1 = \frac{\sum(x_i - \bar{x})(y_i - \bar{y})}{\sum(x_i - \bar{x})^2} = \frac{Cov(x, y)}{Var(x)} \] \[ w_0 = \bar{y} - w_1\bar{x} \]

Mean Absolute Error (MAE)

\[ \text{MAE} = \frac{1}{n}\sum_{i=1}^{n}|y_i - \hat{y}_i| \]

MAE uses the absolute value of errors, treating all error magnitudes more uniformly. It produces a more angular loss surface and is more robust to outliers.

The solution for minimizing MAE often involves:

\[ w_1 = \text{median of pairwise slopes} \] \[ w_0 = \text{median of } (y_i - w_1 x_i) \]

Visual Interpretations

Residuals Plot: Shows how each loss function measures error. MSE creates areas (squares) while MAE uses lengths (line segments).
Loss Curves: Demonstrate how the loss changes as you adjust the slope parameter. Notice the smooth parabola for MSE versus the V-shape for MAE.
Parameter Space: 3D visualizations of the loss surface over all possible combinations of slope and intercept. MSE creates a smooth bowl, while MAE creates a more angular surface.

Key Differences

Aspect	MSE	MAE
Sensitivity to Outliers	High (squares errors)	Low (linear penalty)
Loss Surface	Smooth, differentiable everywhere	Angular, not differentiable at zero error
Computational Complexity	Simple closed-form solution	Requires median calculations
Optimal Solution	Mean-centered	Median-centered

Linear Regression: MSE vs MAE Comparison

Interactive Elements in This Demo

Interactive Regression Model

Loss Functions Comparison

Geometric Interpretation of Residuals

Parameter Space Comparison