Forecasting GDP Growth in Q4 2023 (with LLMs)

By Wojciech Gryc on January 25, 2024

Introduction

The vision behind Emerging Trajectories is that if we have a semi-competent analyst with access to all of the world's information. We already explored forecasting the New Hampshire Republican Primaries, and this post looks at predicting Q4 2023 GDP growth.

Experiment and Results

The results for this experiment are similar to those of the New Hampshire Primary, so we won't go into as much detail. In this case, the Bureau of Economic Analysis (BEA) releases a quarterly press release with a statement similar to the one below...

Real gross domestic product (GDP) increased at an annual rate of ___ percent in the fourth quarter of 2023 (table 1), according to the "advance" estimate released by the Bureau of Economic Analysis.

In this case, we were predicting what the "___" would be. In reality, the number was 3.3%, as per the announcement on the morning of January 25.

We ran two types of forecasting strategies. The first was asking ChatGPT for a prediction, encouraging it to use its Bing web search for information. The second approach ("PhaseLLM/GPT-4") was using a Google search agent that obtained search results for GDP growth projections and passed this information to GPT-4.

Daily forecasts versus the actual result are below.

Analysis

ChatGPT performed much better here than the PhaseLLM/GPT-4 approach. Both approaches, however, were not close in terms of predicting the actual GDP forecast. Fortunately for the US and us, the performance of the US economy was a surprise for many.

Between this and the New Hampshire Primary experiment, it's clear that LLMs can provide a decent narrative for forecasts, and our basic experiment can be expanded in a few ways.

• Day-to-day inconsistency. Forecasts can be inconsistent day-to-day, even when results do not change and when LLM temperature settings are low. I believe this is happening because prompts do not reference prior forecasts. It's worth taking a more Bayesian approach (similar to superforecasting), where the LLM has a view into its prior forecasts and updates those with new information.
• Forecasting versus predicting/analysis. The LLMs are not doing any analysis, but rather summarizing information from the web. This is a good start, but it's not clear how much of the forecast is coming from the LLM versus the information it's provided. It's possible that the LLMs are simply summarizing the information they are provided, and not doing any analysis themselves.
• Complementing the LLM with RAG or other (non-web search) data. The workflows today are very basic and were meant to be a proof-of-concept the next step here would be to provide the LLM with broader capabilities around web browsing, developing a knowledge base, and referencing it to generate forecasts.
• External tests and guardrails. Many State-of-the-Art (SOTA) use LLMs to explore ideas, but further validate those ideas with more formal models. AlphaGeometry does this for mathemtatical proofs, and Voyager does with a Minecraft agent. Ensuring that the predictions our LLMs make are consistent with other facts or ideas in a broader database or world model could make the forecasts more stable and accurate.

Conclusion and Next Steps

We will be running more experiments, and building out the roadmap mentioned above, in the coming weeks and months. Stay tuned as we develop a more thorough framework and modeling approach!

If you have any questions or if you want to get involved, please email us at hello --at-- phaseai --dot-- com.