From-Scratch Build · HPC Curriculum
A book I'm writing from scratch to teach myself the foundations of high-performance computing — how supercomputers are built, why parallelism is hard, and how to make code run fast across many cores and machines. My own chapters, examples and explanations.
What it is
This is the first of two volumes I'm authoring to learn high-performance computing — the field of squeezing huge amounts of computation out of clusters of processors working together. Rather than just reading about it, I decided the best way to understand it was to write it: each chapter forces me to explain a concept clearly enough that it survives being put on a page.
The foundations volume covers everything you need before you can do anything clever: what HPC hardware looks like, how parallel work is split up, where memory bottlenecks come from, how jobs are scheduled on shared clusters, and the two programming models — shared-memory threading and message passing — that almost everything else is built on.
The core idea I wanted to learn: a faster computer is rarely one fast chip — it's many ordinary chips coordinated well. Performance comes from the architecture, the way work is divided, and how data moves through memory, not from raw clock speed. This volume is my attempt to internalise that.
The toolkit
Each chapter is built around a tool or idea that was new to me. Here is what each one actually does in the HPC picture.
The job scheduler. On a shared cluster you don't run programs directly — you submit batch jobs, and Slurm queues them and parcels out nodes, cores and time fairly.
Compiler pragmas that turn a serial loop into a multi-threaded one across the cores of a single machine. The gentlest on-ramp to parallel programming.
The Message Passing Interface — how separate processes on separate machines cooperate by explicitly sending and receiving data. The backbone of cluster computing.
Getting started with GPUs: thousands of tiny cores for data-parallel work. OpenACC adds directives; CUDA gives lower-level control.
You can't optimise what you can't measure. Profiling and debugging tools show where time and communication actually go.
When hundreds of processes read and write at once, the filesystem becomes the bottleneck. Parallel I/O and the right data layout keep it fed.
The chapters
The foundations volume walks from "what even is a supercomputer" up to writing genuinely parallel code. I've grouped the chapters into the arc below — the last stage is where I'm still drafting.
What HPC is, why it exists, and how machines grew from single CPUs to massive parallel clusters.
Inside an HPC system: nodes, interconnects, performance metrics, and getting practical access to a cluster.
Shared vs. distributed memory — the single most important distinction that shapes how you write parallel code.
OpenMP multithreading and a deep dive into MPI: the two workhorse models for splitting computation.
An introduction to GPU computing and the high-performance libraries that save you reinventing the wheel.
Performance analysis, parallel I/O, cloud HPC and good software practice. The chapters I'm still actively writing.
How I work on it
Every chapter follows the same loop, which is really a loop for learning a topic, not just writing about it:
The chapters live as notebooks so the prose, the code and the output sit together — you read the idea, then watch it run with a single sbatch or mpirun.
Reflection