Supervised Learning: A Quick Refresher

In supervised learning, we have a dataset containing input-output pairs (features and labels). The goal is to learn a mapping function from the inputs to the outputs so that we can predict the output for new, unseen inputs.

• Input (X): Features (e.g., square footage, number of bedrooms)

• Output (Y): Label (e.g., house price, "spam" or "not spam")

1. Linear Regression

What it is: A model used to predict a continuous, numerical value. It assumes a linear (straight-line) relationship between the input features and the output.

• Use Case: Predicting house prices, stock prices, sales revenue, temperature.

How it Works:
The model learns a "line of best fit" through the data. The equation for a simple linear regression with one feature is:

y = wX + b

• y = predicted output

• X = input feature

• w = weight (slope of the line)

• b = bias (y-intercept)

With multiple features, it becomes: y = w₁X₁ + w₂X₂ + ... + wₙXₙ + b

The Math Connection:

• Linear Algebra: The entire prediction y is a dot product between the weight vector (w₁, w₂, ... wₙ) and the feature vector (X₁, X₂, ... Xₙ), plus the bias. The entire dataset is represented as a matrix X.

• Calculus: How does it learn? We define a loss function—almost always Mean Squared Error (MSE). To find the w and b that minimize MSE, we use gradient descent. Calculus computes the gradient of the loss with respect to w and b and guides us downhill to the minimum.

• Statistics: The model's performance is evaluated using statistical measures like R-squared (how much variance is explained) and Root Mean Squared Error (RMSE). The very idea of fitting a line of "best fit" comes from statistics (Ordinary Least Squares).

2. Logistic Regression

What it is: Despite its name, it is a model used for classification, not regression. It predicts the probability that an input belongs to a particular category (e.g., spam vs. not spam).

• Use Case: Email spam detection, fraud detection, medical diagnosis (sick/healthy).

How it Works:
Instead of fitting a straight line, logistic regression passes the linear combination of inputs (z = wX + b) through a special S-shaped "sigmoid" function. This function squashes any input value into a range between 0 and 1, which we interpret as a probability.

sigmoid(z) = 1 / (1 + e^{-z})

If the output probability is > 0.5, we classify it as Class A; if it's < 0.5, we classify it as Class B.

The Math Connection:

• Linear Algebra: Just like linear regression, the core computation z = wX + b is a linear algebra operation.

• Calculus: The learning process is again driven by gradient descent, but the loss function is different. For logistic regression, we use Log Loss (or Cross-Entropy Loss). This function heavily penalizes confident but wrong predictions (e.g., predicting a probability of 0.99 for a spam email that is actually not spam). Calculus is used to minimize this log loss.

• Probability: This is the heart of logistic regression. The model's output is a direct probability score. The sigmoid function transforms a linear output into a valid probability. We are essentially modeling the probability that Y belongs to a certain class given X.

3. Decision Trees

What it is: A non-linear model that predicts a value or class by learning a series of simple, hierarchical "if-else" questions based on the input features. It mimics human decision-making.

• Use Case: Both classification (Classification Tree) and regression (Regression Tree). Customer segmentation, loan approval, medical diagnosis.

How it Works:
The algorithm builds a tree structure:

1. It starts at the root (the top).

2. It asks a question about a feature (e.g., "Is age < 30?").

3. Based on the answer (Yes/No), it splits the data and goes down a branch.

4. It repeats this process at each new node, creating a tree-like structure.

5. It stops according to stopping criteria (e.g., maximum depth, or a node has only one class), creating leaf nodes. The leaf node gives the final prediction (a class label for classification, or an average value for regression).

The Math Connection:

• Statistics & Probability: This is the core math behind decision trees. The key is how the algorithm chooses which feature to split on at each node. It uses statistical measures to find the split that creates the "purest" sub-groups.

  • For Classification: It uses metrics like Gini Impurity or Information Gain (which is based on Entropy from information theory). These measure the "disorder" in a node. The goal is to minimize disorder—a pure node has only one class.

  • For Regression: It uses metrics like Variance or Standard Deviation Reduction. The goal is to create subsets of data where the target values are as similar as possible (low variance).

• Calculus: While the tree-building process itself is based on greedy algorithms and doesn't use calculus directly for its splits, calculus can be involved in advanced optimization for pruning the tree or in Gradient Boosting frameworks (like XGBoost) that use decision trees as "weak learners" and optimize them using gradient descent.

Summary Table

These three algorithms form the essential toolkit for a data scientist. Linear and Logistic Regression are powerful, interpretable linear models, while Decision Trees are intuitive, non-linear models that are the building blocks for more complex algorithms like Random Forests and Gradient Boosted Machines.