Emerging Trajectories v0.2.8: News Agents, Crawlers, and RAG

By Wojciech Gryc on March 26, 2024

We're happy to announce the latest release of Emerging Trajectories (ET). This version supports the most sophisticated knowledge and forecasting agents we've built to date, and also provides a framework for how we'll be thinking about knowledge in the future.

News Agents, Crawlers, and Other "Knowledge Workers"

This version provides support for new types of GenAI knowledge "workers" or research agents.

• News crawlers. We provide support for using news APIs, so that you can query and integrate your preferred news monitoring solutions.
• Abstracted website crawlers. Website crawling and scraping is an art. Earlier versions of ET depends on PhaseLLM's crawlers, but we've now made it possible to build your own. We support Playwright crawlers, which are more amenable to robotic process automation, both in headless and non-headless mode.
• Abstracted text extraction from websites. Similarly, crawlers can be set up to extract information/text from websites in different ways. This should enable more robust and clean information extraction, and thus better (and more auditable!) forecasts.

Fact Databases (and Retrieval Augmented Generation, RAG)

One of the biggest switches we've made over the last few versions is disconnecting the underlying knowledge base from the forecasting agent's actual work of forecsasting. With this approach, we build a general knowledge base that an be used for anything -- forecasting, search, or other knowledge intenseive tasks.

In v0.2.8, we introduce three parts to this process...

FactRAGFileCache: this is a fact-based knowledge base. Various LLM-powered research tools (our news bot, Google search, etc.) log information into this knowledge base, which can then be used for future work. All the original sources and content associated with facts are logged, so you can review and audit them as needed.

FactsRAGForecastingAgent: this is a fact-based forecasting agent. It uses the knowledge base above to choose which facts to incorporate into its forecasts and forecast updates. It also cites these facts — when combined with the FactRAGFileCache above, you are then able to audit all forecasts and underlying source materials.

FactBot: a "helper" bot that converts citations made by forecasters into user-friendly text and content.

What's Next?

We're really excited about the progress made on the back-end knowledge bases, fact extraction, and forecasting solutions we've made. We also completely understand that our front-end has been... suboptimal.

Our goal over the coming weeks is to revamp our front-end to enable easier use of our research agents, tools, forecasts, and more. Stay tuned for updates and demo videos!