Data Preprocessing: Cleaning & Preparing Your Data

This is about fixing problems in your data so it's consistent and reliable.

1. Handling Missing Values

Data often has gaps (NaN, null, or blank values). You need to handle them:

Strategies:

• Deletion: Remove rows/columns with missing values (simple but can lose information)

• Imputation: Fill missing values with:

  • Mean/Median/Mode (for numerical data)

  • Forward-fill/Backward-fill (for time series)

  • Predictive imputation (using ML to predict missing values)

python

import pandas as pd
import numpy as np

# Example dataset with missing values
data = {'age': [25, 30, np.nan, 35, 40],
        'salary': [50000, np.nan, 70000, 80000, 90000]}
df = pd.DataFrame(data)

# Fill missing age with median, salary with mean
df['age'].fillna(df['age'].median(), inplace=True)
df['salary'].fillna(df['salary'].mean(), inplace=True)

2. Handling Outliers

Outliers are extreme values that can skew your model.

Detection Methods:

• Z-score method (values beyond ±3 standard deviations)

• IQR method (Interquartile Range)

• Visualization (box plots, scatter plots)

Handling:

• Remove them

• Cap them (set min/max thresholds)

• Transform them (using log transforms)

Feature Engineering: Creating & Transforming Features

This is the art of creating new features or transforming existing ones to help your model perform better.

3. Encoding Categorical Variables

ML models only understand numbers, not text. You need to convert categories to numbers.

Types:

• Label Encoding: Categories → Numbers (0, 1, 2, 3...)

  • Good for ordinal data (e.g., "low", "medium", "high")

• One-Hot Encoding: Creates binary columns for each category

  • Good for nominal data (e.g., "red", "blue", "green")

python

from sklearn.preprocessing import LabelEncoder, OneHotEncoder

# Label Encoding
le = LabelEncoder()
df['city_encoded'] = le.fit_transform(df['city'])

# One-Hot Encoding
df_encoded = pd.get_dummies(df, columns=['city'], prefix='city')

4. Feature Scaling/Normalization

When features have different scales (e.g., age: 0-100, salary: 0-200,000), models can become biased toward larger numbers.

Common Methods:

• Standardization (Z-score Normalization):

  • Scales to mean=0, standard deviation=1

  • Formula: (x - mean) / std

  • Good for most cases

• Min-Max Scaling:

  • Scales to range [0, 1]

  • Formula: (x - min) / (max - min)

  • Good for neural networks

• Robust Scaling:

  • Uses median and IQR

  • Good when you have outliers

python

from sklearn.preprocessing import StandardScaler, MinMaxScaler

# Standardization
scaler = StandardScaler()
df_scaled = scaler.fit_transform(df[['age', 'salary']])

# Min-Max Scaling
minmax_scaler = MinMaxScaler()
df_minmax = minmax_scaler.fit_transform(df[['age', 'salary']])

5. Creating New Features

Sometimes the most powerful features are ones you create:

Examples:

• From dates: day_of_week, month, is_weekend, season

• From text: word count, sentiment score, topic

• Mathematical transformations: log(x), x², √x

• Interaction features: price_per_sqft, BMI (weight/height²)

python

# Creating new features from existing ones
df['price_per_sqft'] = df['price'] / df['square_footage']
df['age_squared'] = df['age'] ** 2
df['income_to_debt_ratio'] = df['income'] / df['debt']

The Complete Pipeline

In practice, you'd use Scikit-learn's pipelines to automate this process:

python

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer

# Define preprocessing for numerical vs categorical columns
numeric_features = ['age', 'salary']
categorical_features = ['city', 'education']

numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
])

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)
    ])

Why This Matters So Much

1. Garbage In, Garbage Out: No matter how advanced your model, bad data → bad predictions

2. Model Performance: Proper preprocessing can improve accuracy by 20-50%

3. Training Stability: Scaled features help models converge faster and more reliably

4. Model Interpretation: Well-engineered features are easier to interpret

Rule of Thumb: Data scientists spend 60-80% of their time on data preprocessing and feature engineering. The actual model training is often the quick part!

This process turns your raw data from a messy reality into a clean, structured format that machine learning algorithms can effectively learn from.