Lesson 4 of 7
Serving engines
6 min read
You can run a model in a few lines of Python. But that naive loop leaves the GPU mostly idle — so how do real deployments serve thousands of users at once?
The layer between your code and the GPU
A serving engine is purpose-built software that sits between requests and the GPU and runs the model efficiently. The naive way handles one request at a time and wastes most of the hardware. A serving engine keeps the expensive GPU saturated: it batches many users' requests together, reuses the memory of tokens already computed instead of redoing them, and streams tokens back as they're produced.
A serving engine's whole job is keeping a costly GPU busy — batching requests and reusing work so you get far more throughput from the same chip.
Pick the engine for the job
Different engines optimise for different situations. vLLM is built for high-throughput serving — many concurrent users on a server, squeezing the maximum requests out of each GPU. Ollama is built for ease: run an open-weights model on your own laptop with one command, great for local development and privacy. Others sit in between. Behind an API, a provider is running an engine like this for you; self-host, and choosing and tuning one becomes your job.
There's no single best engine. vLLM leans throughput-on-a-server; Ollama leans run-it-locally. Match the engine to whether you're serving crowds or working solo.
You rarely need to write GPU code yourself. Learning to configure a serving engine — batch sizes, memory limits, which model to load — gets you most of the performance with none of the low-level maths.
The shape of it
- —A serving engine runs a model efficiently, keeping the GPU busy across many requests.
- —It batches requests and reuses computed tokens so one GPU serves far more traffic.
- —vLLM targets server throughput; Ollama targets easy local runs — pick by your situation.
You want to serve one model to thousands of concurrent users and maximise requests per GPU. Which fits best?
Continue in the app
Take the whole Inference & Hardware course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.