Every few months, people gather on a forecasting platform known as Metaculus, each hoping their predictions on global events will snag a share of a five thousand dollar prize. The questions themselves are daunting, like whether Thailand might see military upheaval before next September or if Israel could launch another military strike on Iran within the year.
Instead of simple answers, participants weigh out probabilities, creating what sometimes look like eerily sharp insights. In the past, Metaculus users pegged the Russian invasion of Ukraine almost exactly two weeks ahead of time, and they saw the end of Roe versus Wade long before the Supreme Court weighed in.
AI Joins the Ranks of Top Forecasters
But this summer’s twist was hard to ignore: one of the top ten finishers was an artificial intelligence. The AI, named Mantic, soared above expectations. “It’s actually kind of mind blowing,” said Toby Shevlane, the chief executive at Mantic.
Coming into the contest, most believed the best bot would still trail the veterans. Instead, Mantic’s scores hit more than eighty percent of the top humans’ average, an unprecedented leap.
Forecasting isn’t some dusty niche; it’s everywhere, according to Nathan Manzotti, a data specialist who has helped several government agencies sharpen their predictive edge. Agencies use these skills to anticipate—and sometimes deflect—surprises that could send policy spinning in unpredictable directions.
According to Anthony Vassalo of RAND, forecasting allows policymakers to steer the outcomes before events cement themselves as history. If an impending disaster seems likely, forecasters can highlight a course correction.
But that sort of predictive power doesn’t come easy or cheap. Questions can take days to analyze, sometimes with hefty costs attached, especially when institutions like RAND need to watch multiple regions at once.
Here’s where the new breed of AI excels. Unlike other data-heavy fields, forecasting geopolitics forces machines to grapple with messiness and uncertainty. Large language models, trained on enormous sets of previous questions, adapt their techniques just as humans do—only faster and across much broader scopes.
Ben Turtel, chief executive of LightningRod, says, “Our main insight was actually predicting the future tends to be a verifiable problem, because that’s like, how humans learn, right?”
During June’s contest, Metaculus’s own AI reached the twenty fifth spot out of hundreds. This time, Mantic jumped to eighth among nearly five hundred and fifty, outpacing all previous machines in the cup’s history.
That said, engineer Ben Wilson at Metaculus urges some caution. The contest had just sixty questions and is filled with competitors at many experience levels. And there is one more catch: the scoring system rewards not only accuracy but also the number of predictions and how quickly they are updated. A tireless machine can adjust estimates almost instantly, something impossible for a human competitor.
Still, for institutions looking for broad, high quality coverage, these new results are a tantalizing taste of what is possible. Vassalo says, “I actually don’t need it to be able to get to the level of a superforecaster. I need it to be as good as the crowd.” The aggregated prediction of the Metaculus community—the so-called wisdom of the crowd—remains consistently strong and tough for any single competitor, human or digital, to beat.
If artificial intelligence can sort through hundreds of questions in tandem, it frees up flesh and blood experts to focus on the toughest, most sensitive topics. Yet a final hurdle remains, as Manzotti puts it with a hint of irony: “A lot of leadership will throw the data out the window if they have a gut feeling in a different direction.” Not even machines can fix that.
