My new side project: trying to find out if AI can beat Vegas
I love building, I can’t stop, and for the past few months my nights have belonged to a side project called Predicted Sports. The pitch is simple: AI models predict sports, and every single prediction gets graded in public. No deleting the misses, no highlight reel of the hits. A scoreboard.
It started, like a lot of good rabbit holes, with the World Cup.
How it started
I think like pretty much everyone else, the world around, I was captivated by the World Cup. Leading up to the first game I thought… “hey, why not try to build a FiveThirtyEight style model?”
That model got built (it predicted an Argentina / France final), but ultimately gave way to something I thought could be a little bit more interesting: “Can AI Predict Sports?”
We have benchmarks galore for AI. How do they do on math? How do they do on writing? I know plenty of people have tried to sports bet using frontier models too, but as far as I can tell, no one has line-blind run as many models as possible to see if one can glean more insight from the data than the others. So, that’s what I built.
The experiment
Before every MLB game, each frontier model (at the time of writing we have 11 on the board… the Claudes, the GPTs, Gemini, and friends) gets an identical, point-in-time data packet about three hours before first pitch. Ratings, recent form, matchups, park and umpire, situational splits. Two things are deliberately missing: the betting line is stripped out, and web search is off. The model can’t peek at the market and dress the answer up as analysis. It has to build its own probabilities from the same data everyone else got.
Every model also gets the exact same prompt. Same instructions, same wording, same output format. There has been a little A/B testing along the way, but we’ve stuck with the “A” path that you can see on the site today.
Then the call gets locked, the game gets played, and we grade it two ways: against the result (accuracy, Brier score) and against the closing line (ROI on a one-unit bet).
It’s all run through OpenRouter, and I’m trying to make sure to keep my own budget in check while simultaneously check as many models as possible. It’s been an interesting balancing act.
What’s happened so far
MLB isn’t the only sport I’ve run this on, I’ve also tried UFC. I’ll be adding NFL / College Football too. The most interesting run so far was UFC 329.
At UFC 329, the two best models went 13-1 across the card, and a $100 bet at the closing line on fight and method would have returned $1,210. Baseball has been a bit more of a tossup, but we are seeing models like Claude Opus 4.8 reliably predicting winners while Grok 4.3 and Deepseek v4 continue to dominate the totals side despite averaging ~4 runs off the mark.
We’re gauging this stuff as well against various ‘strategies.’ Like, for example, just betting on favorites. At the time of writing, just betting on MLB favorites would have returned +1.2%, we’re tracking 3 models better than that (Claude Opus 4.8, GPT-5.6 Sol Pro, Claude Sonnet 5) and the rest are missing the mark.
Fun fact, if you go with the consensus of the two Claude models the ROI has been 13.6% throughout the month of July.
The nerd paragraph
For the fellow developers: the site is a Rails 8 app on Postgres, with a separate data pipeline that scrapes, models, and grades every morning, then syncs the results over. There’s a season simulator, an AI leaderboard, fighter databases, fantasy tools, the works. It has been an absurd amount of fun to build, and it’s the first side project in years where the scope creep felt like a feature.
What’s next
NFL season. The season simulator is already live and running all 272 games ten thousand times a day. Football is where the models meet the deepest, sharpest betting market in the world so it’s going to be fun to see that one play out.
Can AI beat Vegas? I honestly don’t know yet. That’s the fun of it. But we’re going to find out the only way that counts: one graded pick at a time, receipts and all.
If you want to watch the experiment unfold, it’s all at predictedsports.com.