Lesson 2 of 7
Reading the leaderboards
6 min read
Somebody posts 'the new #1 model just dropped' with a chart. Should you switch? First ask: who decided it's #1, and did they test your kind of question?
Where the rankings come from
Two things produce those league tables. Benchmarks are fixed exam papers — the model answers a set of set questions and gets a score. Arenas are popularity contests — people vote on which of two anonymous answers they prefer, and the wins pile up into a ranking. Both are useful; both are averages over someone else's tasks.
A leaderboard is an average over a fixed set of tasks. Change the task and the order reshuffles — the #1 for coding may sit mid-table for writing.
A rank is not 'best for you'
The chart can't know what you do all day. It also misses things scores don't capture — how it feels to talk to, whether it refuses too much, the price, the speed. Treat a leaderboard as a shortlist, not a verdict: it tells you who's roughly in the running, then you test them on your own work.
Use rankings to build a shortlist of two or three, then judge them on your own real tasks. Your workflow is the only benchmark that counts.
Benchmarks get gamed. Once a test is famous, labs train to ace it, so scores creep up without the model getting better at your job. A number that looks decisive is often the least interesting thing on the page.
The shape of it
- —Benchmarks are fixed exams; arenas are head-to-head votes — both are averages.
- —The order depends on the task, and misses feel, refusals, price, and speed.
- —Use a leaderboard as a shortlist, then test the finalists on your own work.
A model tops the coding leaderboard. You mostly write marketing emails. What should you conclude?
Continue in the app
Take the whole Which AI should I use? course — tracked
Get your personalized path, progress and streaks in the app — this lesson and every next one, in order.