Product Roadmap for Q1 2024

By Wojciech Gryc on January 28, 2024

Introduction

Now that we have a few forecasts under our belts, we're starting to think about how to formalize the forecast writing process. This roadmap article is a brief overview of the logic we're using to structure forecasting agents. It will inform our work on the website, product, and API.

Mental Model

The mental model for our forcasting platform is below.

Statements are general "fill in the blank" observations about a world at a specific time. An example could be "The price of Bitcoin will be $_____ USD on January 1, 2025." or "The probability that President Trump will be re-elected on November 6, 2024 is ____." These statements need to be filled in by the agents we are building.

A Forecast is a statement where our agent fills in the blank with its best estimate. Agents do this "filling in" with the help of a Knowledge Base. A Knowledge Base could be any source of information. Our most basic one is a Google search query where information is explored and included in the agent's prompt. Other knowledge base elements could include past forecasts, knowledge graphs, or a RAG-based system.

Types of Agents

We'll begin by focusing on three types of agents.

• Basic Agent: an agent that uses web searches to inform a forecast.
• Priors Agent: an agent that uses its prior forecasts to determine if new information was found, and then update its forecast accordingly. This should address forecast stability concerns we saw in early models.
• Facts Agent: an agent that uses priors (as above) and also generates a list of facts from provided content to more rigorously and systematically update its forecasts.

Types of Knowledge Bases

Knowledge bases will all be text-based at first. They will focus on the following content.

• Web Searches: using web searches to find content to confirm/update information.
• Web Browsing: visiting web pages and determining if links should be followed and/or new searches should be run.
• Manual Content Stores: content collections added manually to the agents' list of knowledge, such as government reports or new articles.

Assumptions

With the above in mind, we're making a few assumptions to simplify the forecasting process.

• Forecasts and Statements are numerical in nature. We only make numerical forecasts at this time.
• Knowledge Bases are tied to individual Agents. To simplify the data handling, storage, and querying process, we assume there is a 1:1 mapping between agents and knowledge bases.
• Agents do not judge the veracity of statements in their Knowledge Base. We assume whatever is in the knowledge base is accurate or true, at this time.
• Forecasts will use textual data and prompts as inputs, initially. To facilitate a faster POC build, we'll begin by including all relevant information in prompts. Our aim is to move to more complex strategies after Q1.
• No evaluations of forecasts. We are prioritizing better forecasts via more complex agents before we begin formally building evaluation tools.

Conclusion

The abose is how we'll be building out our API and platform. We'll stirve to update the documents and code examples as we do so!