Discriminant Functions: Least Squares

Applying least squares regression to the multi-class classification problem: a closed-form discriminant with appealing simplicity but well-known pathologies for classification tasks.

1. Setup: One Class Per Discriminant

We assign each of $K$ classes its own linear discriminant $y_k(\mathbf{x}) = \mathbf{w}_k^\top \widetilde{\mathbf{x}}$, where $\widetilde{\mathbf{x}} = (1, \mathbf{x}^\top)^\top$ includes the bias. Stacking the $K$ weight vectors as columns of $\widetilde{\mathbf{W}}$, all predictions are obtained at once:

$$\mathbf{y}(\mathbf{x}) = \widetilde{\mathbf{W}}^\top \widetilde{\mathbf{x}} \in \mathbb{R}^K.$$

We assign $\mathbf{x}$ to class $C_k$ where $y_k(\mathbf{x})$ is largest.

2. One-Hot Targets and the Sum of Squared Errors

Use one-hot encoding: the target for a class-$k$ example is the vector $\mathbf{t}$ with $t_k = 1$ and all other entries 0. Collect all $N$ targets row-wise into an $N \times K$ target matrix $\mathbf{T}$, and all inputs row-wise into $\widetilde{\mathbf{X}}$ (the $N \times (d{+}1)$ design matrix with a leading column of 1s). The sum of squared errors is

$$E(\widetilde{\mathbf{W}}) = \tfrac{1}{2} \operatorname{tr}\!\Bigl\{\bigl(\widetilde{\mathbf{X}}\widetilde{\mathbf{W}} - \mathbf{T}\bigr)^\top \bigl(\widetilde{\mathbf{X}}\widetilde{\mathbf{W}} - \mathbf{T}\bigr)\Bigr\}.$$

The trace sums the squared errors for all $N$ data points and all $K$ class outputs simultaneously.

3. Closed-Form Solution

Setting $\partial E / \partial \widetilde{\mathbf{W}} = \mathbf{0}$ gives the multi-output version of the normal equations:

4. Problems with Least Squares for Classification

Despite the elegant closed-form solution, least squares classification has three serious drawbacks.