Course presentation

Objective: define the field, fix the vocabulary, and survey what is and isn't possible today.

• Definition & relationship with other disciplines — Computer Vision recovers semantic and geometric information from images and video; it is the inverse of computer graphics, which renders images from a known 3-D model. It overlaps image processing (which transforms images into images), photogrammetry (precise 3-D measurement from photos), pattern recognition and machine learning. Concretely: graphics asks "given this cube and light, what pixels result?"; CV asks the far harder reverse — "given these pixels, what cube and light produced them?".
• Main concepts — a grayscale image is a function $I(x,y)$ mapping pixel coordinates to intensity (colour adds channels, video adds time $t$); the canonical pipeline runs acquisition → pre-processing → feature/representation → analysis → interpretation. Recovering the 3-D scene from a 2-D image is ill-posed: infinitely many scenes project to the same picture, so extra information (priors, geometry, learned statistics) is always required.
• Historical background — from Larry Roberts' "blocks world" (1963) and David Marr's three levels of analysis (computational / algorithmic / implementational) in the 1980s, through the feature-engineering era (SIFT, HOG) of the 2000s, to the deep-learning inflection point when AlexNet won ImageNet in 2012 and halved the error rate of hand-designed pipelines.
• State of the art — robust object detection and segmentation, monocular and multi-view depth, photorealistic generation (diffusion), and large vision-language models that caption, answer questions about and reason over images — the applications that frame the rest of the course.

• Szeliski, Computer Vision: Algorithms and Applications — Ch. 1 (Introduction): read for the field map and the historical timeline.
• Forsyth & Ponce, Computer Vision: A Modern Approach — Ch. 1: complementary framing of what CV is and isn't.

Computer Vision Engineering (I)

Objective: understand the optical and illumination physics that determine image quality.

• Basic illumination techniques & sources — front lighting reveals surface texture, back lighting produces clean silhouettes for measuring outlines, and structured lighting projects a known pattern to extract 3-D shape. In industrial CV, choosing the right lighting is often cheaper and far more reliable than engineering an algorithm to cope with bad lighting — a controlled scene turns a hard recognition problem into an easy thresholding one.
• Camera elements & optics fundamentals — aperture controls light intake and depth of field, focal length sets magnification and field of view, and the thin-lens relation $\tfrac{1}{f}=\tfrac{1}{z_o}+\tfrac{1}{z_i}$ ties object distance $z_o$, image distance $z_i$ and focal length $f$ together: only objects at the conjugate distance focus sharply, everything else blurs into a circle of confusion.
• Fisheye & omnidirectional cameras — ultra-wide lenses sacrifice the rectilinear (straight-lines-stay-straight) perspective model for coverage approaching or exceeding 180°, introducing strong radial distortion that bends straight world lines into curves; modelling and undistorting this is essential before any geometric reasoning.
• Sensors & example applications — CCD sensors shift charge across the chip for low noise; CMOS sensors digitise per-pixel for speed, low power and lower cost (now dominant). Both follow the photon → charge → voltage → digital number path, and the choice of lens + sensor is dictated by the application: machine vision, automotive, microscopy, astronomy.

• Szeliski — Ch. 2.1–2.3 (image formation, photometric & optical models): the physics behind every formula above.
• Livingstone, Vision and Art: The Biology of Seeing — light & the eye, the biological motivation for engineering an artificial vision front end.

Computer Vision Engineering (II)

Objective: represent colour, encode images efficiently, and start coding with OpenCV.

• Color perception — human vision is trichromatic: three cone types (roughly L/M/S, "red/green/blue") sample the spectrum, so any colour collapses to three numbers. This is why three channels suffice — and why metamerism arises: two physically different spectra that excite the cones identically look the same, which is exactly what lets RGB displays fool the eye with just three primaries.
• Color spaces — RGB mixes brightness and colour together; HSV separates hue, saturation and value so you can threshold on colour regardless of lighting; YCbCr splits luminance from chrominance (the basis of compression and broadcast); Lab is perceptually uniform so Euclidean distance approximates perceived colour difference. Pick the space that isolates the property your task depends on.
• Image compression — natural images are highly redundant (neighbouring pixels correlate). Lossy JPEG exploits this in three steps — split into 8×8 blocks, apply the DCT, then quantize (coarsely round) the high-frequency coefficients the eye barely notices, then entropy-code — while lossless PNG uses prediction + DEFLATE to compress without discarding any data.
• First steps with OpenCV — imread/imshow, the surprising BGR channel order (not RGB), uint8 vs float32 dtypes and the overflow/clipping bugs they cause, and basic array-slicing manipulation of the image-as-NumPy-array.

• Szeliski — Ch. 2.3 (colour models) & Ch. 2.3.3 (compression): how the spaces and JPEG are defined.
• Forsyth & Ponce — colour chapter: perception-first treatment of why three channels work.

Camera configuration & streaming

Objective: operate a real camera and deliver a live video stream — the engineering glue of a CV system.

• Acquisition methods — continuous frame grabbing vs hardware/software triggering (capture exactly when an event occurs, e.g. a part passes a sensor) and multi-camera synchronisation so frames share a timestamp — essential for stereo and motion analysis where mis-timed frames ruin correspondence.
• Focus / Iris — focus sets the conjugate distance that appears sharp; the iris (aperture) trades light intake against depth of field. A wide aperture (small f-number) gathers light and isolates a subject with shallow DoF; a narrow aperture keeps a wide depth in focus but demands more light or longer exposure.
• Exposure, white balance, gamma — exposure time and gain both brighten the image but gain amplifies noise, so prefer exposure when motion allows; white balance rescales channels to achieve colour constancy under different illuminants (so white looks white); gamma encoding $V_{out}=V_{in}^{\gamma}$ allocates more code values to dark tones, matching the eye's roughly logarithmic response (encode ≈1/2.2, decode ≈2.2).
• Setting up an RTSP server — the Real-Time Streaming Protocol packetises and timestamps frames for low-latency networked delivery, letting downstream CV nodes consume a live feed — the engineering glue between a camera and a vision service.

• Szeliski — Ch. 2.3.1 (sampling, aliasing, exposure): the signal-processing view of acquisition. OpenCV video I/O docs — practical capture & streaming.

Introduction to projective geometry

Objective: adopt homogeneous coordinates so projection and perspective become linear.

• Projective space ℙⁿ — represent points by $(n{+}1)$-vectors defined only up to a non-zero scale, i.e. as rays through the origin in $\mathbb{R}^{n+1}$. This makes points at infinity first-class citizens (a final coordinate of 0), so parallel lines "meet at infinity" and perspective foreshortening becomes algebraically clean.
• Projective space ℙ² — the image plane. Beautifully, both points and lines are 3-vectors and they are dual: a point lies on a line iff $\mathbf{x}^{\top}\mathbf{l}=0$, the line through two points is their cross product $\mathbf{l}=\mathbf{x}_1\times\mathbf{x}_2$, and the intersection of two lines is $\mathbf{x}=\mathbf{l}_1\times\mathbf{l}_2$ — one operation does join and meet.
• Projective space ℙ³ — the world space of scene points, represented by 4-vectors, that the camera projects down to ℙ² via a single $3\times4$ matrix.

• Hartley & Zisserman, Multiple View Geometry — Ch. 1–2 (projective geometry of 2-D and 3-D): the definitive, careful treatment — read Ch. 2 closely for point/line duality.

Image transformations

Objective: classify and apply the 2-D transformation hierarchy on images.

• Scaling & Translation — scaling multiplies coordinates by $s$ (uniform) or $s_x,s_y$ (anisotropic); translation adds an offset. Translation is not linear in 2-D coordinates, which is precisely why we lift to homogeneous coordinates — there it becomes the extra column $t$ of a $3\times3$ matrix.
• Rotation — an orthonormal matrix $R$ ($R^{\top}R=I$, $\det R=1$) that preserves lengths and angles; rotation + translation is a rigid/Euclidean motion (3 DOF in 2-D), the transform a moving rigid object or camera undergoes.
• Affine — a 6-DOF map (a general $2\times2$ linear part + translation) combining rotation, anisotropic scale and shear; it preserves parallelism and ratios of areas but not angles or lengths — a good model for weak-perspective or a distant planar surface.
• Perspective (homography) — the full 8-DOF $3\times3$ projective map; it preserves straight lines and the cross-ratio but not parallelism, so railway tracks converge. It exactly models a planar surface viewed from any angle, or any two views related by pure rotation.

• Szeliski — Ch. 2.1.1 (2-D transforms) & 3.6 (image warping/resampling): the practical mechanics of applying a transform to pixels.
• Hartley & Zisserman — Ch. 2: the full transformation hierarchy and what each preserves.

Image acquisition model

Objective: write the complete pixel-from-world projection equation.

• Lens models: pinhole, thin, thick — the pinhole is the ideal: every ray passes through one point, giving exact perspective with infinite depth of field but no light. The thin-lens model adds focusing and finite aperture; the thick-lens (and full compound-lens) model adds the principal planes and aberrations real optics introduce.
• Projection models — perspective (objects shrink with distance, the realistic model) versus orthographic and weak-perspective approximations that drop the depth division and are valid when the scene's depth is small relative to its distance — far simpler algebra, used in early SfM.
• Intrinsic & extrinsic parameters — intrinsics $K$ encode focal lengths $f_x,f_y$ (in pixels), the principal point $c_x,c_y$ and skew $s$; extrinsics $[R\,|\,t]$ encode the camera's orientation and position in the world. Together $P=K[R\,|\,t]$ is the $3\times4$ projection matrix.
• Coordinate systems involved — a point travels world → camera (apply $[R|t]$) → image plane (perspective division by depth) → pixel grid (apply $K$); knowing which frame you are in is half of solving any geometry problem.

• Hartley & Zisserman — Ch. 6 (camera models): the full $P=K[R|t]$ decomposition and its degrees of freedom.
• Forsyth & Ponce — geometric camera models: a gentler first pass on the same equations.

Three-dimensional vision I & II

Objective: survey the cues and devices that recover depth from images.

• Human 3-D & stereoscopic vision — two eyes/cameras see a point at slightly different image positions; this disparity $d$ is inversely proportional to depth, giving $Z = f\,B/d$ for focal length $f$ and baseline $B$. Wider baselines resolve far objects better but shrink the overlapping field of view.
• Structured light (projection) — actively project a known pattern (stripes, dots, Gray code) and read its deformation on the surface; because the projected pattern is the correspondence, dense depth is recovered without solving the matching problem — the principle behind the original Kinect.
• Time of flight — emit modulated light and measure round-trip time (or phase) per pixel: $Z = c\,\Delta t/2$. Works in textureless scenes where stereo fails, but resolution and multipath are limiting.
• Depth from focus / shape from shading & texture — monocular cues: which depth is in focus reveals distance, shading gradients reveal surface orientation (shape-from-shading), and the foreshortening of a known texture reveals slant. Each is weak alone but informative as a prior.
• Optical flow & motion — the brightness-constancy assumption (a moving point keeps its intensity) linearises to $I_x u + I_y v + I_t = 0$, one equation in two unknowns $(u,v)$ — the aperture problem. Lucas–Kanade resolves it by assuming constant flow in a small window.

• Szeliski — Ch. 11–12 (stereo & 3-D reconstruction) and Ch. 9 (dense motion / optical flow): the geometry and the algorithms.
• Hartley & Zisserman — stereo & reconstruction chapters: the projective view of two-view depth.

Multiple view geometry

Objective: relate two or more views algebraically.

• Single & dual-camera models — a single projection matrix $P$ maps the world to one image; a dual (stereo) model adds a second camera with a fixed relative pose $(R,t)$, and the pair's geometry is what lets us triangulate depth.
• Homography — a $3\times3$ invertible $H$ relating two images of the same plane, or two views sharing the same centre (pure rotation). Eight DOF, estimable from four point correspondences — the workhorse of rectification, mosaicing and augmented-reality marker tracking.
• Epipolar geometry — for a general (non-planar) scene, the fundamental matrix $F$ encodes the two-view geometry: any point $\mathbf{x}$ in image 1 must lie on the epipolar line $F\mathbf{x}$ in image 2, expressed by $\mathbf{x}'^{\top}F\mathbf{x}=0$. $F$ has rank 2 and 7 DOF.
• Structure from motion — given many overlapping views, jointly recover all camera poses and a 3-D point cloud: match features, estimate pairwise geometry, triangulate, then globally refine — the engine behind photogrammetry and 3-D scanning apps.

• Hartley & Zisserman — Ch. 9–11 (epipolar geometry, the fundamental matrix, the trifocal tensor and SfM): the core reference for this and the next session.

Multiple view geometry (continued)

Objective: deepen the two-view tools and complete the SfM pipeline.

• Single & dual-camera models — revisited as an estimation problem: normalise coordinates (translate/scale so points are centred with unit average distance) for numerical conditioning, and wrap the fit in RANSAC so a few wrong matches don't corrupt the model.
• Homography — the Direct Linear Transform stacks each correspondence into two rows of a system $Ah=0$ and solves by SVD (the null-space vector); $\geq4$ matches suffice. Applications: stereo rectification, image mosaicing/panoramas, planar AR.
• Epipolar geometry — the normalised 8-point algorithm estimates $F$ linearly; with known intrinsics, $F$ becomes the essential matrix $E=K'^{\top}FK$, which factors as $E=[t]_\times R$ and decomposes (via SVD) into the relative rotation $R$ and translation direction $t$ between the two cameras.
• Structure from motion — from relative pose, triangulate 3-D points, then run bundle adjustment: a large non-linear least-squares refinement minimising total reprojection error over all camera and point parameters simultaneously.

• Hartley & Zisserman — Ch. 11 (estimating $F$, the normalised 8-point algorithm) & the bundle-adjustment appendix.
• Szeliski — Ch. 11 (SfM pipelines end to end).

Depth cameras

Objective: configure and capture from real depth-sensing rigs.

• Stereoscopic camera configuration & capture — physically mount two calibrated cameras with a known baseline, then rectify the pair (warp both images so epipolar lines become corresponding horizontal rows). After rectification a block-matching or semi-global-matching algorithm scans each row for the disparity, and $Z=fB/d$ converts it to depth.
• Structured-light camera configuration & capture — replace the second camera with a projector: emit a coded pattern (e.g. Gray-code stripes or a speckle), capture the deformed pattern, decode which projector column each pixel saw, and triangulate — effectively a stereo pair where one "camera" is an active emitter, robust on textureless surfaces.

• Szeliski — Ch. 12 (depth sensing, rectification, active illumination): the practical pipeline for both rigs.

Camera calibration I

Objective: understand how intrinsics and distortion are estimated.

• Introduction & tools — calibration recovers the intrinsics $K$ and lens-distortion coefficients so pixel measurements become metric geometry; without it, every projection, stereo depth and pose estimate carries a systematic bias. OpenCV's calibrateCamera implements the standard toolchain.
• Zhang's method — show a planar checkerboard (known geometry) at several orientations. Each view yields a homography between the board plane and the image; because a homography of a calibration plane constrains $K$ via the image of the absolute conic, ~3+ views over-determine the 5 intrinsic unknowns, which are then refined non-linearly together with distortion.
• Calibration process issues — pattern coverage (corners must fill the frame, including edges/corners where distortion is strongest), avoiding degenerate near-parallel board poses, and choosing the right distortion model (radial $k_1,k_2,k_3$ + tangential) — too many terms over-fits, too few leaves residual bias.

• Zhang (2000), "A flexible new technique for camera calibration" — the original method, well worth reading; Szeliski Ch. 6.3 for the textbook treatment.

Camera calibration II

Objective: run an end-to-end calibration and validate it.

• Pattern image acquisition — capture 15–30 sharp, well-distributed checkerboard views spanning depth, tilt and frame position; blurry or motion-smeared frames give sub-pixel corner errors that propagate into $K$.
• Running the calibration process — detect and sub-pixel-refine the corners, build the correspondence between board points and image points, solve the linear system for an initial $K$ and per-view poses, then jointly refine $K$, distortion and all poses by minimising reprojection error (Levenberg–Marquardt).
• Checking reprojection error — re-project the known 3-D corners with the estimated parameters and measure the RMS pixel distance to the detected corners; a good calibration lands below ~0.5 px. Inspect per-view error to spot and drop a bad capture.

• Szeliski — Ch. 6.3 (calibration in practice); OpenCV calibration tutorial for the exact API and corner-refinement steps.

Mid-term exam

Objective: assess Sessions 1–13 — formation, optics, colour, projective geometry, the camera model, multi-view geometry and calibration.

• Mid-term exam — solving Computer Vision problems on the foundations and geometry material covered so far: image formation and optics, colour and compression, projective geometry and the transformation hierarchy, the full camera model $K[R|t]$, multi-view geometry ($H$, $F$, $E$, SfM) and calibration.

Pixelwise operations

Objective: transform each pixel independently via an intensity map $s = T(r)$.

• Pixelwise operations & pixel transformations — apply an intensity map $s=T(r)$ identically to every pixel: the negative $s=L-1-r$ (invert), log $s=c\log(1+r)$ (compress bright, expand dark), gamma $s=c\,r^{\gamma}$ (perceptual remap) and contrast stretching (linearly rescale a narrow intensity band to the full range). All are 1-D lookup tables.
• Histogram equalization — use the image's own cumulative distribution as the transform: $s=\text{round}((L-1)\cdot\text{cdf}(r))$. This spreads frequent intensities apart and squeezes rare ones together, flattening the histogram and maximising global contrast — automatic, parameter-free contrast enhancement.
• Binarization — collapse to a 0/1 image by thresholding $s=\mathbb{1}[r>t]$; the gateway to morphology, connected-component labelling and shape analysis. The whole game is choosing $t$ (Otsu, Session 19, picks it automatically).

• Szeliski — Ch. 3.1 (point operators) & 3.1.4 (histogram equalization): the canonical curves and the CDF derivation.
• Forsyth & Ponce — early image-processing chapter for intuition.

Local operations

Objective: process each pixel using its neighbourhood via convolution.

• Introduction — a local (neighbourhood) operator computes each output pixel from a window of input pixels. The linear case is convolution: slide a kernel $K$ over the image and take the weighted sum, $(I*K)(x,y)=\sum_{i,j} I(x-i,y-j)K(i,j)$. Boundary handling (zero-pad, reflect, replicate) matters at the edges.
• Separable filters — a kernel is separable if $K=\mathbf{u}\mathbf{v}^{\top}$; then 2-D convolution becomes a 1-D horizontal pass followed by a 1-D vertical pass, dropping the cost per pixel from $O(k^2)$ to $O(2k)$. The Gaussian and the box filter are separable — a large speedup for big kernels.
• Linear filtering examples — box and Gaussian blur (low-pass, denoising), Sobel (first-derivative, edges), Laplacian (second-derivative, fine detail), and unsharp masking (sharpen by adding back the high-pass residual).
• Band-pass & orientable filters; other local operators — Difference-of-Gaussians as a band-pass/blob detector, steerable filters that synthesise any orientation from a basis, and non-linear neighbourhood ops: the median filter (removes salt-and-pepper noise, preserves edges) and the bilateral filter (edge-preserving smoothing).

• Szeliski — Ch. 3.2 (linear filtering & separability) and 3.3 (median/bilateral and other non-linear filters): work through the separability proof.

Global operations

Objective: analyse and filter images in the frequency domain.

• Fourier transform & the 2-D (bidimensional) DFT — express the image as a sum of 2-D sinusoids; the DFT $F(u,v)$ gives the amplitude and phase of each spatial frequency $(u,v)$. The 2-D DFT is separable, so it is computed as 1-D FFTs along rows then columns in $O(MN\log MN)$.
• DFT interpretation — the magnitude spectrum says how much of each frequency is present (centre = low frequencies = smooth gradients; periphery = high frequencies = edges and texture), while the phase says where that structure sits — and phase carries most of the recognisable image content (swap two images' phases and the picture follows the phase).
• Other global transformations — the DCT (real-valued, energy-compacting — the heart of JPEG from Session 3) and wavelets (joint space-frequency, multi-scale — the basis of image pyramids and JPEG-2000).

• Szeliski — Ch. 3.4 (Fourier transforms & the convolution theorem) and 3.5 (pyramids/wavelets).
• Forsyth & Ponce — frequency-domain chapter for the signal-processing grounding.

Salient feature detection

Objective: find repeatable, distinctive image points.

• Introduction & definitions — a good feature is repeatable (found again under viewpoint/lighting change), distinctive (its descriptor is unique enough to match), local (robust to occlusion and clutter) and ideally invariant to rotation, scale and illumination. These properties are what make features matchable across images.
• Surface curvature & curvature-based methods — treat intensity as a 2-D surface $z=I(x,y)$; corners are points of high curvature in two directions, edges high curvature in one. Curvature of the level curves gives an early, principled corner measure.
• Harris & Stephens method — form the second-moment (structure) matrix $M$ by summing gradient outer products over a window; its two eigenvalues describe how intensity varies locally. Rather than compute eigenvalues, Harris scores cornerness cheaply as $R=\det(M)-k\,\mathrm{tr}(M)^2$ ($k\approx0.04$–0.06): large positive $R$ = corner, large negative = edge, small = flat.
• Gradient-based methods — build detectors and descriptors from the image gradient $\nabla I=(I_x,I_y)$: gradient magnitude marks edges, orientation histograms (the idea behind SIFT/HOG) give rotation-tolerant descriptors.

• Harris & Stephens (1988) — the original four-page paper, very readable; Szeliski Ch. 7.1 (feature detectors & their invariances).

Intro to object recognition

Objective: separate objects from background with classical segmentation.

• Introduction — fix the vocabulary: classification says what an image contains, detection says what and where (boxes), segmentation labels every pixel. Classical recognition tackles the simplest version — separate a known object from its background.
• Edge detection-based techniques — find boundaries (Canny: gradient + non-max suppression + hysteresis thresholding) then close them into object contours; works when objects have crisp, complete edges but struggles with gaps and texture.
• Image thresholding — global thresholding splits the histogram at one level; adaptive thresholding varies the level per region for uneven lighting. Otsu's method picks the global level automatically by maximising the between-class variance $\sigma^2_{between}(t)=w_0w_1(\mu_0-\mu_1)^2$ — equivalently minimising within-class variance — assuming a bimodal histogram.
• Region-based algorithms — group pixels by similarity: region growing (expand from seeds while a homogeneity test holds), split-and-merge (quadtree partition then merge similar neighbours), and connected-component labelling of a binary mask.

• Otsu (1979) — the original automatic-thresholding paper; Szeliski Ch. 5 (segmentation, active contours, graph cuts); Forsyth & Ponce segmentation chapters.

Image matching techniques

Objective: locate and align content across images.

• Template matching — slide a reference patch $T$ over the image and score each location: SSD (sum of squared differences, fast but brightness-sensitive) or normalised cross-correlation (NCC, which subtracts the mean and divides by the norm, making it invariant to linear brightness/contrast changes). The best-scoring location is the match.
• Feature matching + homography — detect features, match their descriptors (nearest neighbour with a ratio test), then fit a homography $H$ with RANSAC: repeatedly sample 4 matches, fit $H$, count inliers, keep the best — discarding the inevitable wrong matches. The result aligns or stitches the images.

• Szeliski — Ch. 7.1–7.2 (feature matching & the ratio test) and Ch. 8 (image alignment, RANSAC & stitching).

Convolutional neural networks

Objective: motivate and define the CNN.

• Introduction — a fully-connected MLP on a $224\times224\times3$ image would need ~150k weights per neuron and ignore that nearby pixels relate. A convolutional layer instead slides a small shared kernel, so it has few parameters and respects spatial structure — the right model for images.
• Applications — the same convolutional backbone, with different output heads, powers classification, detection, semantic/instance segmentation, depth, pose and generation — the unifying architecture of modern CV.
• Functional perspective — a CNN is a composition $f=f_L\circ\dots\circ f_1$ of differentiable layers; training minimises a loss by gradient descent, with gradients computed by backpropagation (the chain rule applied layer by layer from the loss backward).

• Drummy et al., Dive into Deep Learning — the CNN chapters (convolutions, padding/stride, channels); Szeliski Ch. 5.3–5.4 for the CV framing.

Convolutional neural networks II

Objective: build a CNN in practice.

• Deep learning frameworks — tensors (n-D arrays on CPU/GPU), autograd (the framework records operations and differentiates automatically), and the canonical training loop: forward pass → compute loss → loss.backward() → optimizer.step() → zero gradients. GPUs accelerate the underlying matrix multiplies by 1–2 orders of magnitude.
• Implementing a CNN from scratch — stack conv → activation (ReLU) → pooling blocks to extract features, flatten, then fully-connected layers to a class score; pick a loss (cross-entropy for classification), an optimiser (SGD/Adam) and a learning rate, then train and monitor train/validation curves.

• Dive into Deep Learning — the modern-training chapters and the PyTorch framework tutorials: build and train a small CNN end to end.

Convolutional neural networks III — advanced architectures

Objective: trace the landmark architectures and the ideas each contributed.

• AlexNet (2012) — the breakthrough that started the deep era: 8 layers, ReLU activations (faster, no saturation), dropout regularisation, and training split across two GPUs. Halved the ImageNet error and ended the hand-crafted-feature era.
• VGG (2014) — showed that depth drives accuracy: uniform stacks of small $3\times3$ convolutions, where two stacked 3×3s match one 5×5's receptive field with fewer parameters and more non-linearity.
• GoogLeNet / Inception (2014) — parallel multi-scale branches per module, and $1\times1$ convolutions to mix channels and cut depth cheaply before expensive 3×3/5×5 convs — far fewer parameters at the same accuracy.
• ResNet (2015) — the residual block $y=F(x)+x$: the layer learns a residual on top of the identity, so adding depth never hurts. Enabled 50–152+ layer networks and remains the default backbone.
• U-Net (2015) — a symmetric encoder–decoder with skip connections that copy high-resolution encoder features across to the decoder, giving sharp per-pixel output — the standard for segmentation and the structural ancestor of diffusion denoisers.

• He et al. (ResNet, 2016) and Simonyan & Zisserman (VGG, 2015) — read both for the depth/residual story; Dive into Deep Learning — the modern-CNN chapter walks through each architecture in code.

Recurrent neural networks

Objective: model sequences and connect vision to language.

• Introduction — an RNN carries a hidden state forward, $h_t=f(Wx_t+Uh_{t-1}+b)$, reusing the same weights at every step so it handles variable-length sequences (video frames, word sequences). Trained by backpropagation-through-time.
• GRU networks — gated recurrent units use update and reset gates to decide how much past state to keep, mitigating vanishing gradients with fewer parameters and gates than the LSTM — often the practical default.
• LSTM networks — add an explicit cell state plus input/forget/output gates; the forget gate lets information persist or be erased across many steps, capturing long-range dependencies a plain RNN forgets.
• Image captioning with attention — a CNN encodes the image into a grid of feature vectors; an RNN decoder generates words one at a time and, at each step, computes an attention distribution $\alpha=\text{softmax}(\text{score}(h,\text{features}))$ over the grid to look at the relevant region for the next word.

• Dive into Deep Learning — the RNN/GRU/LSTM and attention chapters (with the captioning case study); Bishop, Pattern Recognition and ML for the probabilistic sequence-model background.

Detection & segmentation using Deep Learning

Objective: map each dense-prediction task to its network design.

• Semantic segmentation — label every pixel with a class via a fully-convolutional encoder–decoder (U-Net/FCN), trained with per-pixel cross-entropy; it tells what is at each pixel but does not separate two instances of the same class.
• Classification + localization — one object: a shared backbone feeds a classification head (softmax over classes) and a regression head (4 box coordinates), trained with a combined classification + box-regression loss.
• Object detection / instance segmentation — many objects: two-stage (Faster R-CNN: propose regions then classify/refine) or one-stage (SSD, YOLO: dense predictions in one pass); Mask R-CNN adds a per-instance mask head. IoU measures box overlap and non-max suppression removes duplicate boxes for the same object.

• Szeliski — Ch. 6 (recognition & detection); Dive into Deep Learning — the object-detection and semantic-segmentation chapters (anchors, IoU, NMS, FCN).

Detection & segmentation using Deep Learning II

Objective: train a one-stage detector end-to-end.

• Training YOLO for object detection — divide the image into an $S\times S$ grid; each cell predicts a few anchor-based boxes with coordinates, an objectness confidence and class probabilities, all in one forward pass. The multipart loss sums box-localization, objectness and classification terms (with $\lambda_{coord}$ up-weighting localization and down-weighting empty cells); at inference, confidence thresholding + NMS yield the final detections. In practice you fine-tune a pretrained YOLO on your own annotated dataset.

• Redmon et al. (YOLO, 2016) — the original one-look formulation; Ultralytics YOLO docs for the practical training workflow; Dive into Deep Learning for anchors and loss design.

Generative models I

Objective: learn distributions over images (latent-variable & autoregressive).

• Unsupervised learning — learn structure without labels. An autoencoder compresses an image to a bottleneck code and reconstructs it; PCA is the linear special case (the optimal linear bottleneck), so an autoencoder is "PCA with non-linear encoder/decoder".
• Variational Autoencoders (VAE) — the encoder outputs a distribution $q(z|x)$ (mean + variance) rather than a point; training maximises the ELBO = reconstruction term − KL($q\,\|\,$prior), and the reparameterization trick $z=\mu+\sigma\odot\epsilon$ keeps it differentiable. Sampling $z$ from the prior and decoding generates new images.
• PixelRNN / PixelCNN — autoregressive models that factor the image likelihood exactly as $p(x)=\prod_i p(x_i\mid x_{<i})$, predicting each pixel conditioned on the ones already generated — sharp and likelihood-exact, but slow because generation is sequential.

• Kingma & Welling (VAE, 2014) — the original ELBO + reparameterization paper; Bishop, Pattern Recognition and ML — latent-variable models & the EM/variational background; Dive into Deep Learning for the code.

Generative models II

Objective: contrast adversarial and diffusion generation.

• Generative Adversarial Networks (GANs) — two networks compete: a generator $G$ turns noise $z$ into fake images, a discriminator $D$ tries to tell real from fake, and $G$ is trained to fool $D$. At the min-max equilibrium $G$'s samples are indistinguishable from real data — yielding very sharp images, at the cost of unstable training and mode collapse (the generator emits only a few varieties).
• Diffusion models — define a fixed forward process that gradually adds Gaussian noise until an image becomes pure noise, then train a network to reverse one noising step (in practice, to predict the noise $\epsilon$). Generation starts from noise and denoises iteratively to a sample — stable to train, high quality and diverse, the basis of modern text-to-image systems.

• Goodfellow et al. (GAN, 2014) — the adversarial game; Ho et al. (DDPM, 2020) — the noise-prediction diffusion formulation; Dive into Deep Learning — GAN chapter for the training loop.

Transformer architecture

Objective: understand attention and Vision Transformers.

• Self-attention & transformers — every token emits a query, key and value; attention weights each token's value by the softmax-normalised similarity of its query to all keys, $\text{softmax}(QK^{\top}/\sqrt{d_k})V$. The $\sqrt{d_k}$ scaling keeps the dot products from saturating the softmax. Multi-head attention runs several in parallel; no recurrence or convolution is involved, so it parallelises fully.
• Vision Transformers (ViT) — cut the image into fixed patches (e.g. 16×16), linearly embed each as a token, add positional encodings (since attention is permutation-invariant), prepend a class token, and run a standard transformer encoder. With enough data it matches or beats CNNs.
• Hybrid models — combine a CNN feature extractor (strong local inductive bias, data-efficient) with transformer reasoning over those features (global context), e.g. DETR for end-to-end detection without hand-designed anchors/NMS.
• Applications — one attention-based backbone now spans classification, detection and generation, increasingly unifying the architectures of the whole field.

• Vaswani et al. (Attention Is All You Need, 2017) — the transformer; Dosovitskiy et al. (ViT, 2021) — images as patch sequences; Dive into Deep Learning — attention & transformers, with code.

Synthesis & wrap-up

Objective: integrate the course — from image formation to transformers — and close the project work.

• End-to-end view — trace one real pipeline through every stage you have learned: acquire and calibrate (Modules 1–2) → pre-process and filter (Module 4) → detect features or segment (Module 5) → feed a deep model (Modules 6–7) → produce the output. At each stage you now choose deliberately between a classical, interpretable, data-light tool and a learned, powerful, data-hungry one.
• Project consolidation & outlook — final-project guidance (scope, baseline-first, honest evaluation) and the open frontiers: foundation and vision-language models, 3-D and video understanding, and efficiency/deployment on constrained hardware.

• Szeliski — concluding chapters & applications; revisit each module's readings for the final exam, prioritising the formulas on your one-page sheet.