Unsupervised Learning: The Goal

The algorithm is given a dataset without any instructions on what to predict. Its job is to:

• Discover inherent groupings (Clustering).

• Find a simpler representation of the data (Dimensionality Reduction).

1. Clustering (K-Means)

What it is: An algorithm that partitions data into K distinct, non-overlapping groups (or clusters). The goal is to group data points so that points in the same cluster are as similar as possible, and points in different clusters are as dissimilar as possible.

• Use Case: Customer segmentation, image compression, grouping genes, document organization.

How it Works (The Intuition):

1. Initialize: Choose the number of clusters, K. Randomly place K "centroids" (the center of a cluster) in the data space.

2. Assignment Step: Assign each data point to the closest centroid. This forms K preliminary clusters.

3. Update Step: Calculate the actual center (mean) of all the points in each cluster. This becomes the new centroid for that cluster.

4. Repeat: Repeat steps 2 and 3 until the centroids no longer move significantly (they have "converged").

The Math Connection:

• Linear Algebra: This is central to K-Means. The "closeness" between a data point and a centroid is almost always measured by the Euclidean Distance. If you think of data points and centroids as vectors in space, the Euclidean distance is the straight-line distance between them. The entire algorithm is about manipulating these vectors in a high-dimensional space.

• Calculus: The K-Means algorithm can be shown to be optimizing a specific objective function: the Within-Cluster Sum of Squares (WCSS). This is the sum of squared distances from each point to its assigned centroid. The algorithm iteratively minimizes this WCSS. While K-Means itself doesn't use gradient descent, the process of reassigning points and moving centroids is a highly efficient way to find a local minimum for this function.

• Statistics & Probability: The concept of a "centroid" is the multivariate mean of the points in a cluster. The entire algorithm is based on minimizing variance within clusters and maximizing variance between clusters.

2. Dimensionality Reduction (Principal Component Analysis - PCA)

What it is: A technique to reduce the number of features (dimensions) in a dataset while preserving as much of the important information (variance) as possible. It transforms the data into a new, lower-dimensional coordinate system.

• Use Case: Visualizing high-dimensional data in 2D/3D, noise reduction, speeding up other ML algorithms by reducing the number of features, feature engineering.

How it Works (The Intuition):

1. Standardize the Data: Scale the data so that each feature has a mean of 0 and a standard deviation of 1. This ensures all features are treated equally.

2. Find the Directions of Maximal Variance: PCA calculates a new set of axes, called Principal Components.

   • The first principal component is the direction in the data that captures the most variance (the spread of the data).

   • The second principal component is the direction that captures the next most variance, with the crucial constraint that it is orthogonal (perpendicular) to the first.

   • This continues for all subsequent components.

3. Project the Data: You then project your original data points onto these new, fewer principal components to create a new, lower-dimensional dataset.

The Math Connection:

• Linear Algebra: PCA is pure linear algebra. It's essentially an eigen decomposition of the data's covariance matrix.

  • The Covariance Matrix describes how the features relate to each other.

  • The Eigenvectors of this covariance matrix are the Principal Components—they are the directions of maximum variance.

  • The Eigenvalues tell you how much variance is captured by each corresponding eigenvector. You choose how many components to keep based on the magnitude of the eigenvalues.

• Statistics: The core idea PCA optimizes for is variance. It finds the new directions that preserve the maximum statistical variance of the original data. The covariance matrix itself is a fundamental statistical object.

• Calculus: While the final algorithm uses linear algebra, one way to derive the first principal component is by solving an optimization problem using calculus: "Find the unit vector (direction) such that the variance of the projections of the data points onto it is maximized." This can be solved using Lagrange multipliers.

Summary Table

How They Work Together: A Scenario

Imagine you have a dataset of customer purchasing habits with 50 features (e.g., money spent on groceries, electronics, frequency of visits, etc.).

1. You could use PCA first:

   • You run PCA on the 50 features and find that the first 5 principal components capture 90% of the variance in your data.

   • You now have a new dataset with only 5 features (the principal components) that is much easier to work with.

2. Then, you could use K-Means:

   • You run K-Means on this new 5-dimensional dataset to group your customers into distinct segments (e.g., "budget shoppers," "luxury buyers," "bulk purchasers").

   • Because you reduced the dimensions first, K-Means runs faster, is less affected by the "curse of dimensionality," and might find cleaner clusters.

In summary, K-Means and PCA are two sides of the unsupervised learning coin: one finds structure by grouping, and the other finds structure by simplifying.