AI & Machine Learning Roadmap

Master data processing, modeling, and deep learning algorithms.

Overall Progress0%

Model Evaluation

Explanation

Model evaluation answers a single question: how well does this model actually perform? A model that looks perfect on the data it trained on can fail completely on new data, so evaluation is always about performance on unseen examples and about choosing the right metric for the task. This lesson covers the training/validation/test split, regression metrics (MAE, MSE, RMSE, R²), and classification metrics built from the confusion matrix (accuracy, precision, recall, F1, ROC/AUC). You will calculate metrics step by step on a concrete example. The calculations use pure Python so you can see exactly where each number comes from.

Why Evaluation Matters

Performance on training data is not performance in the real world.
The right metric reveals weaknesses (e.g., missing the rare class).
Evaluation is how you compare models and decide what to ship.

Train / Validation / Test

Train on the training set; tune on validation; judge once on test.
The test set is held out until the very end.
This separation prevents fooling yourself about generalization.

Generalization

Generalization is performance on data the model has not seen.
Overfitting is the failure to generalize: memorizing instead of learning.
Good evaluation detects overfitting by comparing train and test performance.

Regression Metrics

Metric	What it measures	Notes
MAE	Average absolute error	Same units as the target
MSE	Average squared error	Penalizes large errors heavily
RMSE	Square root of MSE	Interpretable, still penalizes big errors
R²	Variance explained (0–1)	Higher is better; 0 = predicting the mean

The Confusion Matrix

	Predicted positive	Predicted negative
Actual positive	True Positive (TP)	False Negative (FN)
Actual negative	False Positive (FP)	True Negative (TN)

TP, TN, FP, FN

TP: correctly predicted the positive class.
TN: correctly predicted the negative class.
FP: predicted positive but it was negative (false alarm).
FN: predicted negative but it was positive (missed).

Accuracy

Accuracy = (TP + TN) / total.
The share of all predictions that are correct.
Misleading when classes are imbalanced.

Precision and Recall

Precision = TP / (TP + FP): of the predicted positives, how many are real?
Recall = TP / (TP + FN): of the actual positives, how many did we catch?
There is a trade-off: raising one often lowers the other.

F1 Score

F1 = 2 · (precision · recall) / (precision + recall).
The harmonic mean of precision and recall.
Useful when you want a single balanced number.

Specificity, ROC, and AUC (Concept)

Specificity = TN / (TN + FP): how well we identify negatives.
ROC plots true-positive rate against false-positive rate across thresholds.
AUC summarizes the whole curve: 1.0 perfect, 0.5 random.

Thresholds

Classifiers output a probability; a threshold turns it into a class.
Lower threshold → more predicted positives → higher recall, lower precision.
Choose the threshold to match the cost of false positives vs false negatives.

Class Imbalance

With 1% positives, 99% accuracy is achieved by always predicting negative.
Use precision, recall, F1, or AUC — not accuracy — on imbalanced data.

Cross-Validation (Concept)

Split data into k folds; train on k-1, test on the remaining, and rotate.
Averages performance over many splits for a more reliable estimate.
Reduces the luck of a single train/test split.

Baseline Models

A baseline is a trivial model (predict the majority class or the mean).
A real model must beat the baseline to be worth anything.
Always compare against a baseline before celebrating.

Choosing the Right Metric

Goal	Metric
Regression error in original units	MAE or RMSE
Balanced classes	Accuracy
Minimize false alarms	Precision
Catch as many positives as possible	Recall
Balance precision and recall	F1

Common Evaluation Mistakes

Evaluating on the training set.
Using accuracy on imbalanced data.
Leaking information between training and test.
Ignoring the baseline model.
Reporting one metric when the task needs another.

Syntax

Metrics in pure Pythonpython

1TP, FP, FN, TN = 50, 10, 5, 1002 3accuracy = (TP + TN) / (TP + FP + FN + TN)4precision = TP / (TP + FP)5recall = TP / (TP + FN)6f1 = 2 * (precision * recall) / (precision + recall)

These formulas are what scikit-learn computes. Pure Python makes each term explicit.

Examples

Confusion matrix worked example

A fraud detector on 1000 transactions (50 real frauds).

text

1TP=40  FP=20  FN=10  TN=9302Accuracy  = (40+930)/1000 = 0.973Precision = 40/(40+20)    = 0.674Recall    = 40/(40+10)    = 0.805F1        = 2*0.67*0.80/(0.67+0.80) = 0.73

Output

(97% accurate, yet it misses 20% of fraud — accuracy hides this)

Regression MAE example

Three predictions vs actuals.

text

1Actual:   100, 200, 3002Predicted: 90, 220, 2803Errors:    10,  20,  204MAE = (10+20+20)/3 = 16.7

Output

(on average, predictions are off by 16.7 units)

Key Takeaways

Evaluate on unseen test data, never the training set.
The confusion matrix is the basis of classification metrics.
Precision minimizes false alarms; recall minimizes misses; F1 balances them.
Accuracy misleads on imbalanced data — use precision/recall/F1/AUC.
For regression, MAE and RMSE measure error size; R² measures explained variance.
Always compare against a baseline model.

← PreviousUnsupervised Learning Next →Deep Learning Basics