AI & Machine Learning Roadmap

Master data processing, modeling, and deep learning algorithms.

Overall Progress0%

Data Preprocessing

Explanation

Data preprocessing is the step between raw data and a working machine-learning model: it transforms messy, incomplete, inconsistent records into clean, consistent input. A model learns exactly the patterns in the data you give it, so bad or unprocessed data produces bad results no matter how clever the algorithm is. This lesson walks through the whole pipeline: handling missing values, removing duplicates, treating outliers, encoding categorical variables, scaling features, splitting data, and avoiding the subtle trap of data leakage. You will also see feature engineering and how to handle imbalanced data. The running example is a small, messy customer dataset that you clean step by step.

Raw vs Prepared Data

	Raw data	Prepared data
Missing values	Present	Handled (imputed or dropped)
Types	Mixed, inconsistent	Consistent, numeric where needed
Scale	Different ranges	Comparable (scaled)
Categorical	Text labels	Encoded numerically

Most real datasets arrive in a form a model cannot consume directly. Preprocessing is the translation layer.

Data Quality

Completeness: are values missing?
Consistency: do values follow one convention (e.g., "NY" vs "New York")?
Validity: are values in the right range and type?
Accuracy: are the values correct in the first place?

Missing Values

Missing data is common and must be handled deliberately.
First ask why it is missing: at random, by design, or systematically.
The answer determines whether dropping or imputing is safe.

Dropping vs Imputing

Approach	What it does	When to use
Drop rows	Remove records with missing values	When few rows are missing and data is plentiful
Drop columns	Remove a mostly-empty column	When a column is almost entirely missing
Impute	Fill in a sensible value	When you cannot afford to lose the data

Mean, Median, and Mode Imputation

Mean imputation: fill with the column average (sensitive to outliers).
Median imputation: fill with the middle value (robust to outliers).
Mode imputation: fill with the most common value (for categorical data).

Duplicate Records

Duplicates can skew counts and give a model a false sense of frequency.
Detect exact duplicates and near-duplicates (same person, two spellings).
Decide whether duplicates are truly the same record before removing.

Outliers

An outlier is a value far from the rest of the distribution.
Outliers can be genuine (a real high spender) or errors (age 250).
Handle with care: investigate before removing or capping.

Categorical vs Numerical Data

Type	Example	Typical treatment
Numerical	Age = 34	Scale to a common range
Categorical (nominal)	City = "Accra"	One-hot or label encode
Categorical (ordinal)	Size = S/M/L	Encode preserving order

Encoding Categorical Variables

	Label encoding	One-hot encoding
Result	One column of integers	One binary column per category
Order	Implies order (0 < 1 < 2)	No implied order
Use for	Ordinal categories	Nominal categories

Feature Scaling

	Normalization (Min-Max)	Standardization (Z-score)
Formula	(x - min) / (max - min)	(x - mean) / std
Range	0 to 1	Centered on 0, unit variance
Sensitive to	Outliers	Less sensitive to outliers

Scaling matters for distance-based and gradient-based models, where a feature with a huge range can dominate others.

Train / Validation / Test Split

Train: the data the model learns from.
Validation: used to tune choices during development.
Test: held out until the end, used once to judge final performance.
Split randomly and reproducibly so results are comparable.

Data Leakage

Leakage is when information from the test set (or the future) influences training.
Examples: scaling using the whole dataset before splitting, or including the target in a feature.
Leakage makes a model look better than it will perform in reality.

Feature Engineering and Selection

Feature engineering creates new, more useful inputs (age from birth date).
Feature selection keeps the most informative features and drops noise.
Better features usually beat a fancier model.

Imbalanced Datasets

Imbalance means one class is far rarer than another (fraud, disease).
A model can look accurate by always predicting the majority class.
Address it with resampling, class weights, and the right metric (not accuracy).

Preprocessing Pipelines

A pipeline chains the same steps in the same order every time.
Pipelines prevent leakage and make results reproducible.
The same pipeline that prepares training data must prepare new data in production.

Common Preprocessing Mistakes

Imputing without asking why data is missing.
Scaling before splitting (leakage).
One-hot encoding a high-cardinality column blindly.
Removing outliers without investigating them.
Forgetting to apply the same preprocessing to new data.

Syntax

Preprocessing with pure Pythonpython

1# Mean imputation2data = [10, 20, None, 40, 50]3known = [v for v in data if v is not None]4mean = sum(known) / len(known)5imputed = [mean if v is None else v for v in data]6 7# Min-max normalization8lo, hi = min(imputed), max(imputed)9scaled = [(v - lo) / (hi - lo) for v in imputed]

These operations are what pandas/Sklearn do under the hood. Pure Python makes the math visible; real workflows use the optimized libraries.

Examples

A messy dataset, before and after

Cleaning a small customer table.

text

1BEFORE:2  {name: "Ada",  age: 34,   city: "Accra"}3  {name: "Ada",  age: 34,   city: "Accra"}   # duplicate4  {name: "Ben",  age: null, city: "kumasi"}  # missing + case5  {name: "Cleo", age: 250,  city: "Tema"}    # outlier6 7AFTER:8  {name: "Ada",  age: 34,  city: "Accra"}9  {name: "Ben",  age: 34,  city: "Kumasi"}   # imputed, normalized10  {name: "Cleo", age: 34,  city: "Tema"}     # capped/corrected

Output

(duplicates removed, missing imputed, case normalized, outlier treated)

Key Takeaways

Preprocessing translates raw data into clean, consistent model input.
Handle missing values deliberately: drop or impute based on why data is missing.
Encode nominal categories with one-hot; scale numerical features.
Split first, then fit scaling/imputation on training only to avoid leakage.
Better features matter more than a fancier model.
Use pipelines so the same steps apply to training and production data.

← PreviousData Visualization Next →Supervised Learning