Model Evaluation: How Good is Your Model?

You can't just trust that a trained model will work well. You need to measure its performance rigorously.

The Fundamental Problem: Train-Test Split

The biggest mistake beginners make is testing a model on the same data it was trained on. This leads to overfitting - the model memorizes the training data but fails on new, unseen data.

python

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train on training data ONLY
model = RandomForestClassifier()
model.fit(X_train, y_train)

# Test on unseen testing data
predictions = model.predict(X_test)

Classification Metrics: Beyond Just "Accuracy"

When dealing with classification problems (cat vs dog, spam vs not spam), accuracy alone can be misleading.

Confusion Matrix: The Foundation

A table that shows the four possible outcomes of predictions:

text

                Actual
            Positive    Negative
Predicted
Positive    TP (True Positive)   FP (False Positive)
Negative    FN (False Negative)   TN (True Negative)

python

from sklearn.metrics import confusion_matrix, classification_report

cm = confusion_matrix(y_true, y_pred)
print(cm)
# Output: [[TN, FP],
#          [FN, TP]]

Key Metrics Derived from Confusion Matrix:

1. Accuracy: Overall, how often is the model correct?

   • Formula: (TP + TN) / Total

   • When to use: When classes are balanced

2. Precision: When it predicts "positive", how often is it correct?

   • Formula: TP / (TP + FP)

   • Interpretation: "How reliable are positive predictions?"

   • Use case: Spam detection (minimize false positives - don't want good emails marked as spam)

3. Recall (Sensitivity): How many actual positives did it catch?

   • Formula: TP / (TP + FN)

   • Interpretation: "How many sick people did we identify?"

   • Use case: Disease detection (minimize false negatives - don't want to miss sick people)

4. F1-Score: The harmonic mean of Precision and Recall

   • Formula: 2 * (Precision * Recall) / (Precision + Recall)

   • When to use: When you need a balance between precision and recall

python

from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score

print(f"Accuracy: {accuracy_score(y_test, predictions):.2f}")
print(f"Precision: {precision_score(y_test, predictions):.2f}")
print(f"Recall: {recall_score(y_test, predictions):.2f}")
print(f"F1-Score: {f1_score(y_test, predictions):.2f}")

# Or get everything at once:
print(classification_report(y_test, predictions))

Regression Metrics

For predicting continuous values (house prices, temperatures):

• Mean Absolute Error (MAE): Average of absolute errors

• Mean Squared Error (MSE): Average of squared errors (punishes large errors more)

• R² Score: How much variance is explained by the model (0-1, higher is better)

python

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

print(f"MAE: {mean_absolute_error(y_test, predictions):.2f}")
print(f"MSE: {mean_squared_error(y_test, predictions):.2f}")
print(f"R²: {r2_score(y_test, predictions):.2f}")

Model Optimization: Making Your Model Better

Cross-Validation: More Robust Evaluation

Train-test split only gives you one performance estimate. Cross-validation gives you multiple, more reliable estimates.

K-Fold Cross-Validation

Splits data into K folds, uses each fold as test set once:

text

Fold 1: [Test][Train][Train][Train][Train]
Fold 2: [Train][Test][Train][Train][Train]
Fold 3: [Train][Train][Test][Train][Train]
...

python

from sklearn.model_selection import cross_val_score

# 5-fold cross-validation
scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
print(f"Cross-val scores: {scores}")
print(f"Mean accuracy: {scores.mean():.2f} (+/- {scores.std() * 2:.2f})")

Stratified K-Fold

Preserves the percentage of samples for each class (important for imbalanced datasets).

Hyperparameter Tuning: Finding the Best Model Settings

Models have "knobs" you can adjust (hyperparameters). Finding the right settings dramatically improves performance.

Grid Search

Systematically tries all combinations of hyperparameters:

python

from sklearn.model_selection import GridSearchCV

# Define parameter grid
param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [3, 5, 7, None],
    'min_samples_split': [2, 5, 10]
}

# Search for best parameters
grid_search = GridSearchCV(
    RandomForestClassifier(),
    param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1
)

grid_search.fit(X_train, y_train)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best cross-val score: {grid_search.best_score_:.2f}")

# Use the best model
best_model = grid_search.best_estimator_

Randomized Search

More efficient for large parameter spaces - samples random combinations.

python

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_dist = {
    'n_estimators': randint(50, 500),
    'max_depth': randint(3, 20),
    'min_samples_split': randint(2, 20)
}

random_search = RandomizedSearchCV(
    RandomForestClassifier(),
    param_dist,
    n_iter=50,  # Number of random combinations to try
    cv=5,
    scoring='accuracy',
    n_jobs=-1,
    random_state=42
)

Putting It All Together: Complete Workflow

python

from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix
import pandas as pd

# 1. Load and prepare data
# df = pd.read_csv('data.csv')
# X = df.drop('target', axis=1)
# y = df['target']

# 2. Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 3. Define and tune model
param_grid = {
    'n_estimators': [100, 200],
    'max_depth': [5, 10, None]
}

grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,
    scoring='f1'
)

# 4. Train and optimize
grid_search.fit(X_train, y_train)
best_model = grid_search.best_estimator_

# 5. Evaluate final model
final_predictions = best_model.predict(X_test)

print("Best Parameters:", grid_search.best_params_)
print("\nClassification Report:")
print(classification_report(y_test, final_predictions))

# 6. Cross-validation score for final estimate
cv_scores = cross_val_score(best_model, X, y, cv=5, scoring='f1')
print(f"\nCross-val F1 scores: {cv_scores}")
print(f"Average F1: {cv_scores.mean():.3f} (+/- {cv_scores.std() * 2:.3f})")

Key Takeaways

1. Never test on training data - always use proper train-test splits

2. Choose metrics wisely - accuracy isn't always the right choice

3. Use cross-validation for more reliable performance estimates

4. Hyperparameter tuning is essential for optimal performance

5. Different problems need different metrics:

   • Medical diagnosis: Focus on Recall (minimize false negatives)

   • Spam detection: Focus on Precision (minimize false positives)

   • Balanced classification: Accuracy or F1-score

   • Regression: MAE, MSE, or R²

This process turns machine learning from "throwing models at data" into a systematic, measurable engineering discipline where you can confidently say "this model works well because..."