By Wojciech Gryc on February 13, 2024
This morning, the US Bureau of Labor Statistics announced that the consumer price index (CPI) increased 0.3 percent in January. This was higher than expected, leading to subsequent drops in stock market indices.
We've been using LLMs to monitor and forecast inflation for Janaury 2024 as part of our early experiments. Our Emerging Trajectories Agent (ETA) forecast a 0.4% for the month, higher than reported but also close to what was observed. Read on for our analysis.
We used three types of agents or LLMs for this research:
Results for the three agents are below...
The biggest concern with ChatGPT and Gemini forecasts are the big swings in predictions. This seems to be happening because the LLMs are conflating two types of predictions: those for January '24 CPI and CPI estimates for the entire year. This is less of a problem with the ET Agent, though it's unclear why.
What is incredibly exciting is that our agent was consistently predicting a result of 0.4% for January 2024, very close to the 0.3% that was actually announced. This is particularl impressive as the ET Agent converged on these results before the end of January, and was one of the few "experts" that predicted higher-than-expected inflation.
This was the last experiment from our initial manual forecasts. Since we launched this experiment in late January, we've release v0.2.3 of the Emerging Trajectories package, and forecasts are now logged directly in our platform. Agents are also significantly more complex, and can use different knowledge bases. This is outlined in our product roadmap.
The biggest challenges we're still seeing here are two-fold. First is consistency of forecasting — we often see relatively large swings day-to-day when making forecasts, even when controlling for the confusion between annual and monthly CPI estimates. The second issue is effectively using past information. Our more recent experiments around Boeing, Aurora Innovation, and climate change use a new type of agent that builds on prior forecasts. This seems to make forecasts more consistent and less "jumpy", but it's unclear how accuracy will be affected by this.
Remember, our forecasting approach is open source, so feel free give it a try! We're always lookiong for collaborators, too, so please email us at hello --at-- phaseai --dot-- com if you want to get involved.
Below are the prompt we used with ChatGPT.
You are an economic researcher working to predict what the US Bureau of Labor Statistics will say is the Consumer Price Index (CPI) for January 2024. Please do the following: 1. Use your ability to search the internet to find relevant facts and information. 2. Provide a set of facts and information you will use to make your prediction. 3. Provide a prediction that can be used to fill in this specific note: "The Consumer Price Index for All Urban Consumers (CPI-U) increased ____ percent in January 2024 on a seasonally adjusted basis." Please be very clear what the specific individual number for the percentage prediction is above; do not give a range. I realize there is lots of uncertainty to this, and I am OK with that.
Here is the prompt we used with Gemini. Note that we initially tried to use the same prompt as that for ChatGPT, but found Gemini did not consistently use its web search capabilities and also refued to give a concrete forecast.
Please help us predict what the US Bureau of Labor Statistics will say is the Consumer Price Index (CPI) for January 2024. I realize this is a very uncertain prediction you will be making. Feel free to do research and explore scenarios, describing your work to me, but I ultiamtely need a prediction.
Provide a prediction that can be used to fill in this specific note: "The Consumer Price Index for All Urban Consumers (CPI-U) increased ____ percent in January 2024 on a seasonally adjusted basis."
Please be very clear what the specific individual number for the percentage prediction is above; do not give a range. I realize there is lots of uncertainty to this, and I am OK with that.