eric.ness.net

1. Introduction

What Are Ablation Studies?

Ablation studies are a fundamental technique in machine learning and statistics used to analyze the contribution of individual features, model components, or preprocessing steps. By systematically removing or altering certain elements of a model, researchers can evaluate their importance and assess the robustness of predictive systems. This method is widely used in feature selection, model interpretability, and domain-specific applications where understanding individual feature impact is critical.

However, ablation studies often yield unexpected results, one of the most notable being the occurrence of negative r² values. While conventionally seen as a sign of poor model performance, negative r² can also indicate deeper insights about model stability, feature redundancy, or fundamental limitations in learning patterns from data.

Why Focus on Negative r²?

The coefficient of determination, r², is a standard metric for evaluating regression models. It measures how well a model explains the variance in a target variable, with values ranging from 1 (perfect prediction) to 0 (equivalent to predicting the mean of the target variable). However, when r² is negative, it suggests that the model is performing worse than simply predicting the mean—a situation that arises in cases of severe overfitting, poor feature selection, or excessive model constraints.

Negative r² is often observed in ablation studies when:

• Key predictive features are removed, leaving the model with little useful signal.
• Models struggle with generalization due to limited or noisy data.
• Regularization techniques (e.g., Lasso, ElasticNet) overly constrain the model’s complexity.

These observations lead to important questions:

• Are some models more prone to negative r² than others?
• Does negative r² always indicate a failure, or can it offer useful insights?
• How do different feature removal strategies influence the prevalence of negative r²?

Empirical Investigation

To explore these questions, we conducted an experiment using synthetic data across multiple regression models, including linear models (Linear Regression, Ridge, Lasso, ElasticNet), tree-based models (Random Forest, Gradient Boosting), and nonlinear models (KNN, RANSAC). Each model was evaluated over 100 trials to measure how frequently it produced negative R2R^2R2 values.

Key Findings:

• RANSAC exhibited the highest occurrence of negative r² (100% of trials), suggesting extreme sensitivity to feature removal.
• Linear models (Linear Regression, Ridge, Bayesian Ridge) showed negative r² in ~88% of cases, indicating that they rely heavily on all available features for predictive performance.
• Lasso and ElasticNet had slightly lower negative r² rates (~83%), likely due to their built-in feature selection mechanisms.
• Tree-based models (Random Forest, Gradient Boosting) and KNN still produced negative r² values in ~85-93% of cases, implying that even more flexible models can suffer significant performance drops with feature ablation.

What This Blog Post Covers

In the following sections, we will:

1. Break down the mathematical intuition behind negative r² and why it occurs.
2. Analyze the results from our empirical study, comparing how different models behave under feature removal.
3. Discuss practical takeaways for machine learning practitioners—when negative r² should be a red flag, when it can be informative, and what alternative metrics to consider.

By the end of this post, you’ll have a clearer understanding of when negative r² indicates genuine model failure versus when it serves as a valuable diagnostic tool in ablation studies.

2. Understanding Negative r2 in Model Evaluation

Mathematical Definition of r2

The coefficient of determination, r2, is a widely used metric for evaluating the performance of regression models. It quantifies how well a model explains the variance in the target variable compared to simply predicting the mean. The formula for r2 is:

$r^2=1–\frac{\sum (y_i–{\hat{y}}_i{)}^2}{\sum (y_i–\overline{y}{)}^2}$

where:

• yiy_iyi​ represents the actual values,
• y^i\hat{y}_iy^​i​ represents the predicted values from the model,
• yˉ\bar{y}yˉ​ is the mean of the target variable.

In an ideal case:

• r2 = 1 means the model perfectly predicts all target values.
• r2 = 0 means the model performs no better than predicting the mean of the target variable.
• r2 < 0 means the model is performing worse than simply predicting the mean.

This last case—negative r2—is often surprising to practitioners, as it suggests that the model is introducing more error than it would if it had made no attempt to learn relationships in the data.

Common Causes of Negative r2

Negative r2 values can emerge for several reasons, particularly in the context of ablation studies, where key features are removed to analyze their importance. Below are the main causes:

1. Poor Generalization Due to Small or Noisy Datasets

When a dataset is small or contains significant noise, a model may struggle to capture meaningful patterns. This can lead to situations where the model overfits the training data but fails to generalize to unseen data. Since r2 is calculated based on test data, an overfitted model may perform worse than simply predicting the mean, resulting in a negative r2.

2. Overfitting and Underfitting

• Overfitting occurs when a model is too complex relative to the dataset, capturing noise rather than true patterns. When tested on new data, such a model can produce erratic predictions, increasing error and leading to negative r2.
• Underfitting happens when a model is too simple to capture meaningful relationships, resulting in poor predictive performance even on training data. This can also cause negative r2 values, especially when strong signals from important features are removed.

3. Feature Removal Leading to Significant Loss of Predictive Power

Ablation studies systematically remove features to assess their impact on a model’s performance. If a removed feature contained critical information for predicting the target variable, the model may no longer be able to make meaningful predictions, leading to worse-than-baseline performance and negative r2 values.

Certain models are particularly sensitive to feature removal:

• Linear models (Linear Regression, Ridge, Bayesian Ridge) rely on all available features, so removing a key feature can significantly degrade performance.
• Regularized models (Lasso, ElasticNet) are designed to handle feature selection better, but if too many informative features are removed, their predictive power collapses.
• Tree-based models (Random Forest, Gradient Boosting) can sometimes compensate for missing features, but extreme feature removal still results in negative r2.
• KNN and RANSAC tend to be highly sensitive to feature loss, as seen in the experiment results.

Misconceptions About Negative r2

Negative r2 is often misunderstood as an outright indicator of a failed model, but this is not always the case. Below are some key clarifications:

1. Negative r2 Does Not Always Mean the Model is Completely Broken

A model with negative r2 is performing worse than predicting the mean, but this may be due to external factors such as feature removal, small sample sizes, or mismatched model complexity rather than an inherently flawed model. If a model performed well before feature ablation but shows negative r2 afterward, the issue is likely related to feature importance rather than model choice.

2. Negative r2 Can Indicate Weak Signal Rather Than Model Failure

If negative r2 appears consistently, it may suggest that the dataset lacks a strong predictive signal. In some cases, data preprocessing, feature engineering, or increasing dataset size can improve performance. However, if a model frequently produces negative r2 across multiple trials and datasets, it might indicate that the problem setup itself needs reconsideration.

3. Empirical Study: How Often Do Models Produce Negative r2?

In theory, negative r2 values arise when a model performs worse than a simple baseline that predicts the mean of the target variable. But how often does this happen in practice, and which models are most prone to it? To answer these questions, we conducted an empirical study by testing multiple regression models on randomly generated datasets.

Experiment Setup

To systematically investigate negative r2, we designed an experiment using synthetic data and a range of regression models, including linear models, tree-based models, and non-linear models. The key steps are as follows:

1. Data Generation:

   • We generated 200 samples with 10 features, drawn from a standard normal distribution.
   • The last column of the dataset was used as the target variable, while the remaining columns served as input features.
   • Since the dataset was randomly generated, there was no strong underlying relationship between the features and the target.

2. Model Selection:

   • We tested nine regression models:
     • Linear Models: Linear Regression, Ridge, Lasso, ElasticNet, Bayesian Ridge
     • Robust Regression: RANSAC
     • Non-Linear Models: K-Nearest Neighbors (KNN)
     • Tree-Based Models: Random Forest, Gradient Boosting

3. Training and Evaluation:

   • We performed 100 independent trials, each with a new random dataset.
   • Each model was trained on 160 samples and tested on the remaining 40.
   • We computed the r2 score on the test set for each model.
   • We recorded the percentage of times r2 was negative across the 100 trials.

Code Implementation

Below is the Python implementation of the experiment:

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet, BayesianRidge, RANSACRegressor
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import r2_score
from sklearn.preprocessing import StandardScaler
import numpy as np
import pandas as pd

# List of models to test
models = {
    "LinearRegression": LinearRegression(),
    "Ridge": Ridge(alpha=1.0),
    "Lasso": Lasso(alpha=0.1),
    "ElasticNet": ElasticNet(alpha=0.1, l1_ratio=0.7, max_iter=2000, tol=1e-4, random_state=42),
    "BayesianRidge": BayesianRidge(),
    "RANSACRegressor": RANSACRegressor(),
    "KNN": KNeighborsRegressor(n_neighbors=5),
    "RandomForest": RandomForestRegressor(n_estimators=100, random_state=42),
    "GradientBoosting": GradientBoostingRegressor(n_estimators=100, random_state=42)
}

results = {model: [] for model in models.keys()}

# Run multiple trials
num_trials = 100

for _ in range(num_trials):
    # Generate synthetic data
    data = np.random.normal(size=(200, 10))

    # Train-test split
    X_train, X_test = data[:160, :-1], data[160:, :-1]
    y_train, y_test = data[:160, -1], data[160:, -1]

    # Standardize features
    scaler = StandardScaler()
    X_train_scaled = scaler.fit_transform(X_train)
    X_test_scaled = scaler.transform(X_test)

    # Train and evaluate each model
    for name, model in models.items():
        model.fit(X_train_scaled, y_train)
        y_test_pred = model.predict(X_test_scaled)
        test_score = r2_score(y_test, y_test_pred)
        results[name].append(test_score)

# Summarizing results
summary = {
    "Model": [],
    "Negative R2 Percentage": [],
    "Mean R2": [],
    "Median R2": []
}

for name, scores in results.items():
    summary["Model"].append(name)
    summary["Negative R2 Percentage"].append(np.mean(np.array(scores) < 0) * 100)
    summary["Mean R2"].append(np.mean(scores))
    summary["Median R2"].append(np.median(scores))

# Convert to DataFrame for display
df_summary = pd.DataFrame(summary)
from IPython.display import display
display(df_summary)

Summary of Findings

After running the experiment, we obtained the following results:

Model	Negative r2 Percentage (%)	Mean r2	Median r2
LinearRegression	88.0	-0.107	-0.086
Ridge	88.0	-0.106	-0.085
Lasso	83.0	-0.041	-0.025
ElasticNet	83.0	-0.049	-0.035
BayesianRidge	88.0	-0.038	-0.019
RANSACRegressor	100.0	-1.059	-0.966
KNN	91.0	-0.240	-0.224
RandomForest	85.0	-0.130	-0.145
GradientBoosting	93.0	-0.293	-0.278

Key Observations

1. r2 was negative in nearly all models a significant portion of the time.

   • Even powerful models like Random Forest and Gradient Boosting saw 85-93% negative r2 occurrences, suggesting that negative r2 is a general phenomenon when working with random features.

2. Linear models (Linear Regression, Ridge, Bayesian Ridge) exhibited negative r2 in ~88% of cases.

   • This confirms that linear models struggle when there is no clear feature-target relationship.
   • The mean r2 values for these models hovered around -0.10, indicating performance only slightly worse than the mean baseline.

3. Lasso and ElasticNet had slightly better performance (~83% negative r2).

   • Their built-in feature selection mechanisms may prevent them from overfitting to noise as aggressively as standard linear regression.

4. RANSAC was the worst performer, producing negative r2 in 100% of cases.

   • This suggests that RANSAC fails completely when feature importance is not well defined, as it relies on finding strong inlier relationships.

5. KNN and Gradient Boosting showed particularly poor mean and median r2 values (~-0.24 to -0.29).

   • These models tend to overfit in small datasets, which might explain their poor generalization performance in this setup.

Does Regularization Help?

• Lasso and ElasticNet performed slightly better than standard Linear Regression, suggesting that regularization helps reduce extreme negative r2 values.
• However, even with regularization, the frequency of negative r2 remained high (~83%), meaning it does not completely eliminate the issue when there is no strong predictive signal.

Conclusion

Our findings show that negative r2 values are common across models when working with arbitrary feature sets and synthetic data. While some models handle the absence of strong predictive relationships better than others, negative r2 is not exclusive to linear models—it also occurs frequently in tree-based and non-linear models.

In the next section, we’ll discuss practical implications of negative r2, including when it should raise red flags, when it is simply an artifact of data limitations, and what alternative metrics can provide better insights.

4. Implications for Machine Learning Practitioners

The empirical results from our experiment demonstrate that negative r2 values occur frequently across different models, particularly when feature relationships are weak or absent. This raises important considerations for practitioners who rely on r2 as a primary evaluation metric.

In this section, we explore when negative r2 is a concern, when it can be informative, and alternative metrics to consider for robust model evaluation.

When Negative r2 Is a Concern

Negative r2 is often seen as a red flag, and in many cases, it should prompt further investigation into model performance. Below are scenarios where a negative r2 suggests potential issues:

1. Poor Model Generalization

• If a model consistently produces negative r2 on test data but performs well on training data, it indicates overfitting.
• This is especially problematic in real-world applications where generalization is critical.

How to address it?

• Increase training data size if possible.
• Use regularization techniques (e.g., Ridge, Lasso, ElasticNet).
• Reduce model complexity (e.g., fewer features, pruning in tree models).

2. Feature Importance Misinterpretation

• A high negative r2 in ablation studies suggests that the removed feature was critical for prediction.
• If key features consistently cause large drops in performance, this signals strong feature dependency.

How to address it?

• Consider feature selection techniques to assess true importance.
• Evaluate if the removed features contain leakage or if the model is overly reliant on them.
• Use alternative models that may generalize better to different feature subsets.

3. Model Selection Issues

• Some models, such as RANSAC and KNN, showed particularly poor r2 scores in the experiment.
• Choosing the wrong model architecture for the data distribution can lead to high variance and instability.

How to address it?

• Test multiple models, especially robust and regularized regressors.
• Tune hyperparameters to find optimal performance.
• Use cross-validation rather than relying on a single train-test split.

When Negative r2 Is Informative

Although negative r2 is often a sign of poor model performance, in some cases, it can provide useful insights about data and feature importance rather than simply indicating failure.

1. Identifying Redundant or Weak Features

• If removing a feature does not significantly impact r2, the feature may be redundant.
• If r2 drops drastically, it suggests that the feature is highly informative.

Takeaway: Negative r2 can help determine which features contribute the most to model performance.

2. Understanding Data Limitations

• If multiple models produce negative r2 scores across different configurations, the dataset may lack meaningful predictive patterns.
• This is common in cases where data is highly noisy or where randomness dominates.

Takeaway: Negative r2 across models suggests a need for better feature engineering, larger sample sizes, or alternative problem formulations.

3. Evaluating Regularization Effects

• Our experiment showed that regularized models (Lasso, ElasticNet) had slightly lower occurrences of negative r2 than unregularized linear models.
• This suggests that some negative r2 results can be mitigated by controlling model complexity.

Takeaway: Regularization can help stabilize r2 scores, though it won’t eliminate negative values when feature relationships are weak.

Beyond r2: Alternative Metrics to Consider

Since negative r2 is often misleading or uninformative in some cases, it’s useful to consider alternative evaluation metrics that provide a more complete picture of model performance.

1. Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE)

• Why use it? These metrics directly measure prediction error without comparison to a baseline.
• Use case: Suitable when absolute prediction accuracy is more important than variance explanation.

$RMSE=\sqrt{\frac{1}{n}\sum _{i=1}^n(y_i–{\hat{y}}_i{)}^2}$

$MAE=\frac{1}{n}\sum _{i=1}^n|y_i–{\hat{y}}_i|$

2. Mean Squared Log Error (MSLE)

• Why use it? Handles small-scale variations better than r2 and prevents large penalties for minor prediction errors.
• Use case: Useful in regression problems where relative differences matter more than absolute errors.

3. F-score and IoU (for classification-like regression problems)

• Why use it? In cases where the problem is closer to classification (e.g., predicting categories instead of continuous values), F-score and Intersection over Union (IoU) may be more relevant.
• Use case: Helpful for segmentation problems, threshold-based decision models, and feature selection analysis.

5. Conclusion and Future Directions

Summary of Key Takeaways

• Negative r2 is common in ablation studies, particularly when removing important features.
• Certain models (RANSAC, KNN) are more prone to negative r2 than others, indicating their sensitivity to feature removal.
• Regularization (Lasso, ElasticNet) helps reduce extreme negative r2 values but does not eliminate the issue entirely.
• Negative r2 should not always be dismissed as model failure—it can indicate weak feature relationships, excessive complexity, or data limitations.
• Alternative metrics such as RMSE, MAE, and MSLE should be considered when negative r2 is misleading.

Open Questions for Future Research

While this study provides insights into the prevalence of negative r2, several important research questions remain:

• How does dataset size impact the frequency of negative r2 occurrences?
  • Would increasing training data mitigate this issue, or do some models inherently struggle with negative r2?
• Can specific model tuning strategies reduce negative r2 in ablation studies?
  • Would adjusting hyperparameters like depth in tree models or number of neighbors in KNN improve results?
• Are there better ways to quantify model failure in feature removal experiments?
  • Should alternative baseline comparisons be used instead of the mean-based r2 calculation?

Final Thoughts

Negative r2 values often alarm practitioners, but they are not always an indication of failure. In ablation studies, negative r2 can reveal important insights about feature importance, dataset limitations, and model behavior. By combining multiple evaluation metrics and considering the context in which negative r2 appears, we can make more informed decisions about model performance and robustness.