by Wojciech Gryc
A few weeks ago, I wrote a piece on using AI to generate forecasts for geopolitical and world events. Since then, I’ve spent time exploring the idea and building the alpha version of the software. I’m not releasing the software to the public yet but did get some initial feedback on forecasting and features within the software. This post is meant be a journal entry on what I’ve learned so far and where I’m going next with the product.
Below are a few screenshots of the Stochastic Futures Platform (SFP) version 0.1. The major components to the platform are (a) the ability to log statements and forecasts, (b) track and review news associated with the forecasts, and (c) get scored on the accuracy of your forecasts.
After using this platform for about eight weeks, here are some initial thoughts for the next iteration.
This has become my primary news reader. First and foremost, I’m glad to have built this iteration of SFP as it has, so far, become my most regular news reading service. I’m able to track about 10,000 articles per day across my favored news sources and these are tied to major investment decisions or ideas that I am tracking.
Workflow automation is a killer feature. After building the ability to track, read, and document news within the platform, I began wondering how I could automate my workflows around news and current events. For example, news around funding rounds for startups would affect my forecasts; rather than manually reading these things, I could eventually automate forecasts to change when such news is observed. Furthermore, the first iteration of the news product was focused on keyword searches (e.g., track certain keywords that might be relevant to certain forecasts). I found that over time, as I interacted with news sources and read specific articles, the platform generated enough data to learn which content might be relevant to my goals or forecasts.
Writing a narrative around predictions is critical. In particular, I find that putting together a decision tree1 around what it would take to generate an increase or decrease in forecast estimates is helpful. This allows you to be rational about the future circumstances around a forecast, rather than responding to new developments with emotion when they actually occur. I also find that building such a decision tree helps me understand the actual context of the forecast I am making, and the various actors or entities that the forecast affects.
New interfaces for reading news. I think there is a major opportunity to build bots or algorithms that generate forecasts based on news content… Or at the very least, algorithms that cluster news content in new ways. Google News, Techmeme, and other news sites typically cluster articles based on similar content, but this isn’t what makes sense to me as a news consumer – building a newsfeed of articles focused on a similar topic but with content diverse enough to provide new ideas or facts, for example, would be more interesting to me.
Brier scores need to be weighted. While Brier scores are a useful way to measure how effective your forecasting is, you also need to control for when forecasts are actually made – the further out you are with making correct predictions, the more valuable your approach tends to be.
With the above in mind, I’ve been working on the next iteration of the platform. I’ve made a few major changes already.
If you’re interested in providing feedback on the above, please let me know! I’m excited to keep iterating on this.
I’d like to extend a thank you to those who tried this version and gave me feedback, including Andrea Yip, Marcin Kozakowski, Rawy Samman, and Ray Cha.