Inference Engineering

Inference Engineering — Philip Kiely

Read: November 2026

A model’s life has two phases. Training is where it learns, and inference is where it gets used to answer real requests. This book is about the second phase: making AI models faster, cheaper and more reliable once people depend on them. It works in three layers: the runtime (one model on one GPU, the graphics processing unit chips these models run on), the infrastructure (scaling across clusters, regions and clouds) and the tooling engineers use across both.

What makes it easy to follow is the order. Each chapter assumes only the ones before it, going from “why would I do this” to models, hardware, software, optimization techniques, other modalities and finally running it all in production. These notes follow the same path.

1. Prerequisites: why and when. Calling a model through an API is the easy start. Running it yourself on dedicated GPUs makes sense when traffic is high enough that paying per GPU beats paying per million tokens, when you run a custom or fine-tuned model, or when your product chains several models and the delays between them add up. This chapter also sets up the measuring stick. Time to first token (TTFT) is how long before the answer starts appearing, and tokens per second (TPS) is how fast it streams after that. Both are reported as percentiles, because an average hides the slow requests: “P99 is 2 seconds” (P99 being the 99th percentile) means 99 out of 100 requests were faster, and the slowest one is what users remember. On picking a model, benchmarks are good for shortlisting but get saturated or gamed (Goodhart’s Law: once a measure becomes a target, it stops being a good measure). Distillation, which trains a small model to imitate a bigger one, is one way to get a cheaper model that behaves similarly.

2. Models: what is actually being computed. An LLM (large language model) writes one token (a word or word fragment) at a time, and each new token is chosen by looking at everything before it. The part that does the looking is attention, which works out which earlier words matter for the next one. Image models work differently: they start from random noise and clean it up step by step, and they’re a pipeline of several models, not one big one. Video is the same idea with far more data. The chapter ends with the key skill, working out what is slowing a model down. A GPU has two limits, compute (calculations per second) and memory bandwidth (how fast it can pull data in). Knowing which one you’re stuck on tells you which fix will help.

3. Hardware: where it runs. GPUs are built for huge numbers of simple tasks in parallel, unlike CPUs (central processing units), which are good at complex sequential work. Their onboard memory (VRAM, video random access memory) caps how big a model can fit, so large models are spread across the eight GPUs in a node, linked by fast connections (NVLink, NVSwitch), and across nodes by InfiniBand. The opposite case exists too: a GPU can be split into up to seven slices for small models. The chapter walks through NVIDIA’s generations (Hopper, Ada Lovelace, Blackwell, Rubin), other accelerators, and local inference on laptops and phones. The author’s view is that the future is local and cloud working together: small, quick queries on your device, heavy ones in the datacenter.

4. Software: what talks to the hardware. CUDA (Compute Unified Device Architecture) is NVIDIA’s platform for writing code that runs on its GPUs, and PyTorch sits above it for describing the math. Model weights are stored in formats like safetensors (weights only) or ONNX (Open Neural Network Exchange, which bundles weights plus the execution graph). On top sit inference engines, which are the programs that actually serve a model: vLLM and SGLang (fast to start) and TensorRT-LLM (the most tunable, but it compiles itself for specific hardware first). NVIDIA Dynamo works above all three to orchestrate large deployments. The chapter closes with how to measure honestly: shadow real production traffic onto the system under test so it sees real load without affecting real users, and profile it. A benchmark tells you how the system performs, and profiling tells you why.

5. Techniques: making it faster. Five families of optimization, and they build on the earlier chapters.

Quantization: store the model’s numbers at lower precision so it’s smaller and faster. Risk is low for the weights, moderate for the KV (key-value) cache and high for attention.
Speculative decoding: a small, quick model guesses several tokens ahead and the big model checks them all in one pass. When the guesses are right, you get several tokens for the price of one.
Caching: while generating, the model keeps notes on what it has already read so it doesn’t redo that work for every token. That’s the KV cache. Reusing it across requests saves a lot of compute, which is why some routers send your request to the server that already holds your context.
Parallelism: splitting one model across many GPUs, either for lower latency or for higher throughput.
Disaggregation: reading the prompt (prefill) and writing the answer (decode) are different kinds of work, so run them on separate machines. It pays off for something like a coding assistant, where a lot of code goes in and comparatively little comes out.

These don’t always combine. Bigger batches use up the spare compute that speculation needs, for example. The skill is picking a set that works together, and the book keeps returning to one principle: the more constraints you can introduce, the faster the system gets.

6. Modalities: the same playbook on different models. Everything above was mostly about text, and this chapter applies it to the rest. Vision language models read images or video along with text. Embedding models turn text into searchable vectors, which is what powers search, RAG (retrieval-augmented generation) and agent memory. ASR (automatic speech recognition) models like Whisper turn speech into text, and TTS (text-to-speech) models turn text into speech, with streaming and speech-to-speech variants for real-time use. Image generation gets kernel-level tuning, and video generation is the most demanding of all, best run on the newest GPUs with caching tricks and the work split across GPUs. Each modality has its own bottleneck, so the same techniques get applied differently.

7. Production: keeping it running at scale. This is where a fast model becomes a dependable service, and it’s a hard problem of its own, not a footnote. Containers package the model so it runs the same everywhere. Autoscaling adds and removes replicas using both traffic (a leading signal) and utilization (a lagging one), and cold starts matter because a new replica can take minutes to load a big model. A router decides where a request should go, while a load balancer decides where it could go. Beyond one cluster, a control plane handles deployments and global scaling while workload planes handle the traffic. GPU capacity is a blend of reserved instances for the baseline with on-demand and spot for peaks. GPUs also fail often, so the book’s stance is to expect it, cordon bad nodes and keep going. Changes roll out as canaries to a small slice of traffic, because a full duplicate fleet for blue-green deploys costs too much in GPUs. The chapter finishes with observability (latency percentiles, queue depth, utilization), client code and streaming protocols.

“The mental shift from per-token pricing to owning your unit economics is harder than any of the technical parts, and the book says so directly.”