Researchers test if Generative AI can make forecasts
Markets are amazed by the potential of generative artificial intelligence. Well trained, it can give correct search answers in a wink. But can it make forecasts? A test made by researchers indicates that when Open AI’s GenAI-tool GPT-4 is asked about the future, it is mostly guessing. “When it comes to looking into the crystal ball, humans still have the upper hand for now”, the researchers write in a blog post.
“In order to make effective long-term plans, it is crucial to accurately forecast and plan for future scenarios. Our finding that GPT-4 has particularly poor forecasting capabilities bolsters the case that the threat of an AI system planning in the long term against human interests remains, thankfully, quite low, conclude the researchers, Dr. Philipp Schoenegger, Research Officer at the Behavioural Lab at the London School of Economics and Dr. Peter S. Park, MIT AI Existential Safety Postdoctoral Fellow and Director of the non-profit StakeOut.AI.
“The results of our study? In simple terms, GPT-4 was no Nostradamus. Not only did it underperform compared to the human crowd’s median predictions, but its forecasts were also indistinguishable from just guessing 50% for every question. This suggests that while GPT-4 might be an intellectual heavyweight in many areas, when it comes to looking into the crystal ball, humans still have the upper hand for now.”
Nostradamus was a 16th century French astronomer famous for prophetic predictions.
The researchers write that the logic for the test was straightforward: if GPT-4 can predict future events, it demonstrates a deeper understanding of how the world works beyond just regurgitating memorised data.
“This makes real-world forecasting tournaments an ideal testbed for evaluating the generalised reasoning and prediction capabilities of AI systems.”
The stage for the evaluation of GPT-4’s predictive capabilities was the Metaculus platform (a platform for building forecasts), where over the course of three months, GPT-4 and 843 human participants tried to predict various future events about Big Tech, US politics, viral outbreaks, and the Ukraine war.
“One possible reason for GPT-4’s subpar performance is its inability to keep up with real-time information. While human forecasters can adjust their predictions based on new information and current events, GPT-4’s knowledge has a fixed cut-off point. Even with background information fed into the model, it can’t fully account for the dynamic, evolving nature of world events.”
The blog post says that the results cast doubt on the immediate prospects of AI taking over jobs that rely on predictive decision-making.
“OpenAI’s mission is to create “highly autonomous systems that outperform humans at most economically valuable work.” Whether this mission is on track to occur will be largely determined by how capable LLMs—and AI systems in general—turn out to be at economically relevant tasks.”
“Predicting the future is a task of especially high economic relevance, especially given that many white-collar jobs in business, policy, and law rely on the ability to make accurate predictions in various domains.”
“Our results suggest that even state-of-the-art AI systems would not yet be competitive with human expertise in occupations that heavily rely on accurate future predictions”, the researchers write.
However the researchers note that the field of AI is constantly evolving:
“Today’s shortcomings might become tomorrow’s breakthroughs. Future research might look into creating ensembles of LLM forecasters, or perhaps into designing models that can actively access and learn from the internet. Another potential research direction is hybrid forecasting models, which aims to combine the comparative strengths of both humans and AI systems and is more akin to most currently deployed AI solutions.”
They write that it remains vital to constantly monitor AI capabilities: including, but not limited to how their forecasting capabilities evolve over time.
“Such a forward-thinking approach can help us ensure that the development of these AI systems will be a boon, rather than a bane, to us humans.”
Moonshot News is an independent European news website for all IT, Media and Advertising professionals, powered by women and with a focus on driving the narrative for diversity, inclusion and gender equality in the industry.
Our mission is to provide top and unbiased information for all professionals and to make sure that women get their fair share of voice in the news and in the spotlight!
We produce original content, news articles, a curated calendar of industry events and a database of women IT, Media and Advertising associations.
