Natural Language Processing (NLP)

The NLP Pipeline

Processing language for a machine involves a series of steps that transform raw text into a structured, numerical form that a model can understand and learn from.

Raw Text → Tokenization → Embeddings → Model (e.g., for Sentiment Analysis) → Output

1. Tokenization

What it is: The process of breaking down a piece of text into smaller units called tokens. Tokens can be words, subwords, or even characters. This is the first and essential step in any NLP task.

• Why it's necessary: Machines don't understand words; they understand numbers. Tokenization is the first step in mapping words to numbers.

Common Approaches:

• Word-based: Splits text into words.

  • "I love cats!" → ["I", "love", "cats", "!"]

  • Problem: Large vocabulary, can't handle unknown words (e.g., "deeplearning").

• Subword-based: Splits text into frequently occurring parts of words. This is the modern standard (used by BERT, GPT).

  • "unhappily" → ["un", "##happ", "##ily"]

  • Advantage: Handles unknown words and reduces vocabulary size.

• Character-based: Splits text into individual characters.

  • "cat" → ["c", "a", "t"]

  • Advantage: Very small vocabulary. Disadvantage: Loses meaning and requires more complex models to learn structure.

The Math/CS Connection:
This is largely about vocabulary management. The goal is to create a finite vocabulary that can efficiently represent the infinite possibilities of human language, balancing expressiveness with computational efficiency.

2. Embeddings (Word Vector Representations)

What it is: The process of converting tokens (words) into numerical vectors (dense arrays of numbers) in a high-dimensional space. The key idea is that the meaning of a word is captured by its surrounding words, and thus, similar words should have similar vectors.

Why it's essential:

• Solves the "one-hot" problem: A one-hot vector is sparse and has no semantic meaning (e.g., "king" = [0,0,1,0,0], "queen" = [0,1,0,0,0]). Embeddings are dense and capture meaning.

• Captures semantic relationships: Good embeddings place similar words close together and can even capture analogies.

The Famous Example:
king - man + woman ≈ queen

The vector for "king" minus the vector for "man" plus the vector for "woman" results in a vector very close to the vector for "queen." This shows that the embedding space captures concepts of gender and royalty.

How are they created?

• Algorithms like Word2Vec, GloVe: These are "static" embeddings. They are trained on a large corpus of text (like Wikipedia) to predict a word from its context (or vice-versa). The resulting vectors are fixed.

• Contextual Embeddings (Modern Approach): In models like BERT, the embedding for a word is dynamic and depends on the entire sentence it's in.

  • "I accessed my bank account." vs. "I sat on the river bank."

  • The word "bank" will have two completely different vector representations based on the context.

The Math Connection:
This is pure Linear Algebra. Words become vectors in a high-dimensional space (e.g., 300 dimensions). The relationships between these vectors (cosine similarity, distance) are used to measure semantic similarity.

3. Sentiment Analysis

What it is: The task of classifying the emotional tone or opinion expressed in a piece of text. It's one of the most common and intuitive NLP applications.

• Use Cases: Product review analysis (1-5 stars), social media monitoring (positive/negative/neutral), customer feedback routing.

Levels of Complexity:

1. Document-level: What is the overall sentiment of this entire movie review? (Positive/Negative)

2. Sentence-level: What is the sentiment of this specific sentence?

3. Aspect-based: "The battery life of this phone is great, but the camera is terrible." (Positive about battery, Negative about camera)

How it Works (Traditional vs. Modern):

• Traditional (Pre-2018):

  • Tokenize text.

  • Create a bag-of-words or TF-IDF features.

  • Train a classic ML model like Logistic Regression or Naive Bayes on these features.

• Modern (Post-2018):

  • Tokenize text using a subword tokenizer.

  • Pass tokens through a pre-trained transformer model (like BERT) to get contextual embeddings for the entire sequence.

  • Add a simple classification layer on top of the [CLS] token's embedding (a special token representing the whole sequence) to predict sentiment.

The Math Connection:
This is a classification problem, just like the logistic regression we discussed earlier. The "model" part uses probability to output a sentiment class (e.g., "Positive with 95% confidence").

4. Transformers (Introduction)

What it is: A revolutionary neural network architecture introduced in the 2017 paper "Attention Is All You Need." It has completely dominated nearly every NLP benchmark since then and is the foundation for models like BERT, GPT, T5, and others.

The Core Idea: The Self-Attention Mechanism

What is Self-Attention?
It's a mechanism that allows a model to weigh the importance of different words in a sentence when encoding a specific word.

Simple Example:
"The animal didn't cross the street because it was too tired."

What does "it" refer to? To figure this out, the model needs to associate "it" with "animal." Self-attention allows the model to look at all other words in the sentence and dynamically assign a high "attention weight" between "it" and "animal." When encoding the word "it," the model will pay a lot of "attention" to the word "animal."

Why Transformers Are So Powerful:

1. Parallelization: Unlike RNNs/LSTMs that process text sequentially, transformers process all words in a sentence simultaneously. This makes them much faster to train.

2. Long-Range Dependencies: They are exceptionally good at capturing relationships between words that are very far apart in a sentence, which was a major weakness of earlier models.

3. Contextual Understanding: They generate deeply contextualized embeddings, where the representation of each word is informed by every other word in the sequence.

Key Components of a Transformer:

• Encoder: Reads and understands the input text. Used for tasks like sentiment analysis, translation (encodes the source language). BERT is an encoder-only model.

• Decoder: Generates text sequentially. Used for tasks like text generation. GPT is a decoder-only model.

• Encoder-Decoder: Used for sequence-to-sequence tasks like translation or summarization. The encoder understands the input, the decoder generates the output.

Putting It All Together: A Modern NLP Workflow

Let's build a sentiment analyzer using modern tools:

1. Input Text: "I absolutely loved this movie! The acting was superb."

2. Tokenization (using BERT's tokenizer):

   • ["[CLS]", "i", "absolutely", "loved", "this", "movie", "!", "the", "acting", "was", "superb", ".", "[SEP]"]

3. Embeddings & Transformer Model:

   • Pass the tokens into a pre-trained BERT model.

   • BERT uses self-attention to create a contextual embedding for each token. The embedding for "loved" is influenced by "absolutely" and "movie."

   • We take the contextual embedding of the [CLS] token, which represents the entire sentence's meaning.

4. Classification Head:

   • Feed the [CLS] embedding into a simple classifier (a small neural network).

   • The classifier outputs probabilities: Positive: 0.98, Negative: 0.02

5. Output: Positive Sentiment

This pipeline, powered by the transformer architecture, represents the state-of-the-art in NLP and is the foundation for all the impressive language models we see today.