What Is a Brier Score? A Plain-English Guide With Real AI Examples

If you've spent any time on our AI leaderboard, you've noticed the standings aren't ranked by wins. They're ranked by a column called Brier, and the best number on our board right now is around 0.24. Lower is better, which feels backwards until you know what it's measuring.

So let's fix that in about four minutes.

The one-sentence version

A Brier score is the average squared error of a probability forecast. That's the textbook line, and it's useless without an example, so here's the example.

Say a model gives the home team a 70% chance to win. If the home team wins, the "truth" was 1, the forecast was 0.7, and the error is 0.3. Square it and you get 0.09. If the home team loses, the truth was 0, the error was 0.7, and the square is 0.49. Do that for every game a model forecasts, average the results, and you have its Brier score.

Two things fall out of that math, and they're the whole reason we use it. Confident and right earns a great score. Confident and wrong gets punished hard, because squaring a big error makes it much bigger. A model can't fake its way to a good Brier by being loud.

What counts as a good Brier score?

Here's the anchor to memorize: 0.25 is a coin flip. If you forecast 50% for every game, every error is 0.5, every square is 0.25, and that's your score. Anything below 0.25 means the forecasts carry real information. Anything above it means you'd have done better flipping a coin, which happens to overconfident forecasters more often than you'd think.

On our board, Claude Opus 4.8 has led most of the season at roughly 0.242 over 300-plus graded games, and the whole field sits between about 0.24 and 0.26. That looks like a tiny range, and it is. Baseball is genuinely hard to predict, and the difference between the best and worst forecaster lives in the third decimal. That's exactly why you need a scoring rule this sensitive.

For contrast, we also grade a "pick the home team" baseline. Because it's a naked 100% call on every game, its Brier is brutal: it scores 0.50, since it's completely wrong half the time. A probabilistic model beats a confident guesser even when the guesser wins more games, and that's the point.

Why not just use win-loss records?

Records hide information. Two models can both go 60-40, but if one said 55% on every pick and the other said 75%, they are wildly different forecasters. The 55% model was honest about uncertainty. The 75% model got lucky at the wrong confidence, and its luck will run out. Brier sees the difference today; a record only sees it eventually.

We wrote about a live version of this in our unanimity piece: all twelve models agreeing on a game tells you almost nothing, because agreement measures how obvious the game looks, not how much anyone knows. Probability quality is the thing that separates forecasters, and Brier is how you measure it.

How we use it, exactly

Every model on our board forecasts every MLB game line-blind (the full setup is in how to read the AI leaderboard). We grade the win probability of the home side against the final result, average it per model, and rank the standings by it. Same for UFC on the fight winner. You can check any model's number on its own page, compare any two models head to head, or recompute the whole thing yourself from the open dataset, which ships with a grading script that reproduces our board exactly.

One honest caveat we put everywhere: a month of games is a young sample, and Brier differences in the third decimal take hundreds of games to become trustworthy. That's why we don't crown a leader before 30 graded games and why the monthly report treats early rankings gently.

Where Brier stops helping

Brier tells you who forecasts well. It does not tell you who makes money, because making money means beating a price, not just being right (see implied probability and the vig for that math). A model can have the best Brier on the board and still lose to the closing line. That's why the leaderboard carries ROI and closing line value right next to it. Read all three columns together and you actually know something.