15 May 2019

Post-mortem on Version 0.1 of the Stochastic Futures Platform

by Wojciech Gryc

A few weeks ago, I wrote a piece on using AI to generate forecasts for geopolitical and world events. Since then, I’ve spent time exploring the idea and building the alpha version of the software. I’m not releasing the software to the public yet but did get some initial feedback on forecasting and features within the software. This post is meant be a journal entry on what I’ve learned so far and where I’m going next with the product.

Features and Screenshots

Below are a few screenshots of the Stochastic Futures Platform (SFP) version 0.1. The major components to the platform are (a) the ability to log statements and forecasts, (b) track and review news associated with the forecasts, and (c) get scored on the accuracy of your forecasts.

A list of statements that are being tracked and forecasted against.

News feed based on keywords, sorted by the number of keywords appearing in content.

Brier scores calculated for a user, including estimates against control predictions.

Lessons Learned

After using this platform for about eight weeks, here are some initial thoughts for the next iteration.

This has become my primary news reader. First and foremost, I’m glad to have built this iteration of SFP as it has, so far, become my most regular news reading service. I’m able to track about 10,000 articles per day across my favored news sources and these are tied to major investment decisions or ideas that I am tracking.

Workflow automation is a killer feature. After building the ability to track, read, and document news within the platform, I began wondering how I could automate my workflows around news and current events. For example, news around funding rounds for startups would affect my forecasts; rather than manually reading these things, I could eventually automate forecasts to change when such news is observed. Furthermore, the first iteration of the news product was focused on keyword searches (e.g., track certain keywords that might be relevant to certain forecasts). I found that over time, as I interacted with news sources and read specific articles, the platform generated enough data to learn which content might be relevant to my goals or forecasts.

Writing a narrative around predictions is critical. In particular, I find that putting together a decision tree¹ around what it would take to generate an increase or decrease in forecast estimates is helpful. This allows you to be rational about the future circumstances around a forecast, rather than responding to new developments with emotion when they actually occur. I also find that building such a decision tree helps me understand the actual context of the forecast I am making, and the various actors or entities that the forecast affects.

New interfaces for reading news. I think there is a major opportunity to build bots or algorithms that generate forecasts based on news content… Or at the very least, algorithms that cluster news content in new ways. Google News, Techmeme, and other news sites typically cluster articles based on similar content, but this isn’t what makes sense to me as a news consumer – building a newsfeed of articles focused on a similar topic but with content diverse enough to provide new ideas or facts, for example, would be more interesting to me.

Brier scores need to be weighted. While Brier scores are a useful way to measure how effective your forecasting is, you also need to control for when forecasts are actually made – the further out you are with making correct predictions, the more valuable your approach tends to be.

Features for Version 0.2

With the above in mind, I’ve been working on the next iteration of the platform. I’ve made a few major changes already.

Support for notes. Rather than limiting the platform to forecasts, I’ve also provided the ability to create notes. You can make notes public; the idea here is to create content based on your own analysis that might be related to forecasts or current events, or alternatively is simply worth publishing.
Workflow automation. This new version uses Postgres and Airflow for the backend, and the website is built directly on top of these tools. It’s now extremely easy to build workflows based on published articles, content, and so on. The goal here is to begin experimenting with news-based workflows and automation.
AI support. To push further on the above, this version also has support for Python-based AI frameworks. As a proof-of-concept, I’ve built a workflow that tracks new articles, classifies them for funding round announcements, and then sends emails to notify users when such announcements are made².

If you’re interested in providing feedback on the above, please let me know! I’m excited to keep iterating on this.

I’d like to extend a thank you to those who tried this version and gave me feedback, including Andrea Yip, Marcin Kozakowski, Rawy Samman, and Ray Cha.

Notes:

See Decision Trees for Decision Making as an example, though lots has been written on the topic over the last 50 years.
If you’d like to see what this looks like, or even subscribe to the AI-driven news feed, just let me know.