Lesson 4 of 7
Streaming & latency
6 min read
Two apps make the identical API call and wait the identical few seconds. One feels sluggish, the other feels alive. The only difference is when they start showing the answer.
Don't wait for the whole reply
By default your app asks for a reply and waits for the entire thing before showing anything — so the user stares at a spinner while the model generates. Streaming flips that: the API sends each token the moment it's produced, so you paint the answer as it arrives, word by word, the way the chat apps you know already do.
Streaming doesn't make the model faster — it makes the wait feel short by showing the answer as it's written. Time-to-first-token is the number your users feel.
Latency is a product decision
Total time depends on things you can tune: a bigger model is slower, a longer reply takes longer, and more input to read adds delay. Streaming attacks the perceived wait; picking a right-sized model and capping the reply length attack the real one. For a chat UI, stream. For a background job that just needs the finished JSON, don't bother — wait for the whole thing.
Two clocks matter: how long the answer takes, and how long it feels. Streaming fixes the feeling; model choice and reply length fix the clock.
Streaming isn't free complexity for its own sake — reach for it in interactive UIs, skip it for batch or tool-call responses your code parses whole. Match the technique to the surface.
The shape of it
- —Streaming sends tokens as they're generated instead of waiting for the whole reply.
- —It cuts perceived latency — time-to-first-token is what users feel.
- —Real latency scales with model size, reply length, and input length.
- —Stream interactive UIs; wait for the full response in background jobs.
Users say your chat feature 'feels slow', though your API calls return in a normal few seconds. What's the highest-leverage fix?
Continue in the app
Take the whole Building AI Apps with the API course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.