AI: Natural Language
Processing & Semantic Analysis

Natural Language Processing sits at the centre of today's technology revolution — the breakthroughs behind ChatGPT, Bard and their successors are just the starting line. Over the next few years NLP and text analysis will permeate every sector, reshaping data-driven decisions and letting us decode human communication in ways we never thought possible. This course turns that promise into skill: it walks the full arc from rule-based and statistical methods through machine learning to the deep-learning and transformer architectures that power modern language models, always pairing theory with hands-on practice so you can apply these tools to real data-science problems.

Tokenization

Splitting text into units (words, subwords or characters). The first step of almost every pipeline.

Stemming vs Lemmatization

Stemming chops suffixes (jumping→jump); lemmatization maps to dictionary forms using POS (was→be).

Stopwords

Very common words (the, of) often removed because they carry little distinguishing signal.

n-gram

A contiguous sequence of n tokens; the basis of count-based language models under the Markov assumption.

Regular expression

A formal pattern language for matching and extracting text — emails, dates, URLs, etc.

Bag of words

A representation that counts word occurrences while ignoring order.

TF-IDF

Term frequency × inverse document frequency — weights words by how distinctive they are to a document.

Vector space model

Representing text as vectors so that geometric closeness reflects semantic similarity.

Cosine similarity

The cosine of the angle between two vectors; a length-independent similarity measure.

Information retrieval

Finding documents relevant to a query from a large collection (the science behind search).

Naive Bayes

A generative classifier applying Bayes' theorem with a conditional-independence assumption between features.

Logistic regression

A discriminative linear classifier whose sigmoid output is a class probability.

Sentiment analysis

Classifying the opinion/emotion of text as positive, negative or neutral.

Word embedding

A dense, learned vector for a word where geometry encodes meaning (e.g. Word2Vec).

Word2Vec

Skip-gram/CBOW model that learns embeddings from context; famous for analogy arithmetic.

Softmax

Turns a vector of scores into a probability distribution; the standard output for classification.

Backpropagation

Computing gradients of the loss via the chain rule to train neural networks.

POS tagging

Labelling each word with its part of speech (noun, verb, adjective…).

Viterbi algorithm

Dynamic programming to find the most probable hidden-state (tag) sequence in an HMM.

Named Entity Recognition

Detecting and classifying entity spans — PERSON, LOCATION, ORGANIZATION, DATE, MONEY.

RNN

A network with a recurrent hidden state for processing sequences one step at a time.

LSTM

A gated RNN with a protected cell state that captures long-range dependencies.

Self-attention

A mechanism where each token weighs all others via softmax(QKᵀ/√d)V to build its representation.

Transformer

An attention-based architecture (no recurrence) that parallelises training; basis of modern LLMs.

Positional encoding

Information added to embeddings so a transformer knows word order.

Pre-trained language model

A model (BERT, GPT) trained on large corpora, then adapted to tasks.

Transfer learning

Reusing knowledge from a pre-trained model to solve a new task with little data.

BERT vs GPT

BERT is a masked-LM encoder (understanding); GPT is an autoregressive decoder (generation).

Seq2Seq

An encoder–decoder model mapping an input sequence to an output sequence (e.g. translation).

Beam search

Decoding that keeps the k best partial sequences at each step instead of just the top one.

Question answering

Returning a precise answer to a question, by extracting a span or generating text.

BLEU / ROUGE

n-gram precision (BLEU) and recall (ROUGE) metrics for generation tasks.

Corpus

A structured collection of text used to train or evaluate models; its size and quality cap what a model can learn.

Markov assumption

The simplifying claim that the next token depends only on a fixed window of recent tokens — the basis of n-gram models.

Laplace (add-one) smoothing

Adding a pseudo-count to every event so unseen n-grams or words get non-zero probability instead of zeroing a product.

Perplexity

$\exp$ of average per-word cross-entropy; the standard language-model score — the model's effective "branching factor" (lower is better).

Cross-entropy loss

The training objective for classifiers and LMs: penalises low probability assigned to the correct label/token.

Sigmoid

$\sigma(z)=1/(1+e^{-z})$ — squashes a real score to a probability; the activation behind logistic regression and LSTM gates.

Activation function

A non-linearity (ReLU, tanh, sigmoid) that gives a network expressive power; without it, depth collapses to a linear map.

Generative vs discriminative

Generative models $P(x,y)$ how data is produced (Naive Bayes); discriminative models the boundary $P(y\mid x)$ directly (logistic regression).

SVD / LSA

Singular Value Decomposition factorises a matrix; applied to a term–document matrix it yields Latent Semantic Analysis topics.

PCA

Principal Component Analysis — projects high-dimensional data onto its directions of greatest variance for visualisation/compression.

Curse of dimensionality

In very high-dimensional sparse spaces, distances concentrate and data becomes sparse — a motivation for dense embeddings.

Distributional hypothesis

"You shall know a word by the company it keeps" — meaning is inferred from co-occurrence context.

Skip-gram / CBOW

Word2Vec's objectives: predict context from a word (skip-gram) or a word from its context (CBOW).

Inverted index

A map from each term to the documents containing it — the data structure that makes large-scale search fast.

BM25

A refined TF-IDF ranking with term-frequency saturation and length normalisation; the standard lexical search baseline.

HMM

Hidden Markov Model — transition + emission probabilities over hidden states; classic model for POS tagging, decoded by Viterbi.

BIO tagging

Encoding spans as Begin/Inside/Outside per token, turning entity detection into per-token classification.

Vanishing/exploding gradient

Gradients shrinking to zero or blowing up through many layers/time-steps; the core RNN weakness LSTMs address.

BPTT

Backpropagation Through Time — unrolling a recurrent net across steps and back-propagating to train shared weights.

Query / Key / Value

The three projections each token produces in attention; query·key sets the weight, value is what gets mixed.

Multi-head attention

Running several attention maps in parallel so different heads capture different relations (syntax, coreference, …).

Masked LM (MLM)

BERT's pre-training: predict randomly masked tokens from both sides, yielding bidirectional representations.

Autoregressive LM

GPT's pre-training: predict the next token left-to-right, $P(x_{1:n})=\prod_t P(x_t\mid x_{

Fine-tuning

Continuing training of a pre-trained model on a small task dataset; lightweight variants include adapters and LoRA.

Prompting (zero/few-shot)

Steering a frozen model via its input — instructions alone (zero-shot) or with in-context examples (few-shot).

RAG

Retrieval-Augmented Generation — retrieve relevant documents, then read/generate the answer, grounding it in evidence.

Hallucination

A model producing fluent but false content, because it optimises for plausibility, not truth.

Precision / Recall / F1

Core classification metrics: correctness of positives (precision), coverage of positives (recall), and their harmonic mean (F1).

Beam search vs greedy

Decoding strategies: greedy takes the top token each step; beam search keeps the k best partial sequences for a better overall path.