The figure compares error values from various loss functions when the actual output label $y$ is 1, focusing on the discrepancies between predicted $\hat{y}$ and true values $y$.

In the figure, the subfigure on the left denotes Mean Squared Error (MSE) loss, while the subfigure on the right denotes the Negative Log-Likelihood (NLL) loss.

$$ \text{MSE}(\hat{y}, y) = (y-\hat{y})^2 $$

$$ \text{NLL}(\hat{y}, y) = y\log(\hat{y}) + (1-y)\log(1-\hat{y}) $$

Code

The figure shows that while the NLL curve increases more steeply, both curves display a similar shape.

Why not MSE for Logistic Regression?

This raises the critical issue of whether MSE is appropriate for logistic regression classification tasks, which can be examined from multiple angles.

• Optimization Difficulty: When using the MSE as the loss function for logistic regression, the optimization problem becomes non-convex. This non-convexity arises because logistic regression outputs probabilities through the sigmoid function, which introduces a non-linear transformation. This behavior makes the optimization process challenging, hindering the effective training of the logistic regression model. In contrast, using a steeper loss function like cross-entropy ensures a convex loss surface with a unique global minimum, facilitating a more straightforward and reliable optimization process.

  

• Probabilistic Interpretation: MSE is typically associated with regression tasks in which continuous outcomes are predicted, and it provides a clear probabilistic interpretation, primarily focusing on the error distribution following a Gaussian Distribution. However, logistic regression deals with the probabilities of categorical outcomes, and its standard loss function, the NLL, offers a direct probabilistic interpretation for observing the appearance of different categories. Using MSE in this context might obscure the probabilistic meanings and relationships inherent in logistic regression analysis.

The choice of loss function significantly impacts how we interpret data and errors. Generally, a loss function is pivotal in establishing the relationship between the predicted values and the actual labels for each sample. Additionally, the chosen loss function reflects the underlying assumptions of our prediction model.

Critical Analysis of MSE Design Choices

In the conception of the Mean Squared Error (MSE) loss function, certain design choices are not arbitrary but rather the result of deliberate and nuanced reasoning:

• Choice of Squared Errors in MSE: The adoption of squaring errors, as opposed to raising them to other powers such as four, warrants scrutiny. While the squaring process inherently accentuates larger errors, thereby ensuring a robust sensitivity to outliers, it also introduces a level of moderation absent in higher powers. The selection of the square function strikes a deliberate balance: it is severe enough to penalize larger errors substantially yet avoids the disproportionate escalation associated with higher exponents, which could render the model overly sensitive and unstable.
• Preference for Difference Over Ratio: The decision to utilize differences $y-\hat{y}$ rather than ratios in error calculation embodies a strategic rationale $\frac{\hat{y}}{y}$. By focusing on absolute discrepancies, the model adheres to a linear perspective of error magnitude, providing a clear, consistent error scale. In contrast, ratios could introduce erratic behavior in the loss landscape, particularly when the actual values approach zero, potentially skewing the model evaluation and undermining its reliability.
• Aggregation by Summation: The aggregation of individual squared errors through summation into a total loss is another critical decision point, i.e., $\sum_i{(y^{(i)}-\hat{y}^{(i)})^2}$. This methodology ensures an equitable contribution from each observation, thereby democratizing the influence of individual data points on the model's learning trajectory. The choice of summation, as opposed to alternative aggregation methods like multiplication $\prod_i{(y^{(i)}-\hat{y}^{(i)})^2}$, avoids the undue dominance of outlier errors and furnishes a loss metric that is both informative and representative of the dataset's overall error landscape.

In synthesizing these elements into the MSE framework, the design reflects a calculated balance of mathematical rigor, practical utility, and conceptual clarity, culminating in a loss function that is both robust and interpretable in its facilitation of model optimization.

Loss ≠ Performance

It is worth noting that the loss function is usually reflect our need. Although we can theoretically define any loss function, optimizing it is often fraught with difficulties. This is exemplified when attempting to optimize specific performance metrics derived from the Confusion Matrix—like precision, sensitivity, and specificity.