Three Reasons Data Scientists Might Prevent The Next Market Collapse

Kevin Petrie's picture

Remember Richard Fuld?

Next month Fuld, a lightning rod for Wall Street criticism, will give his first major public speech since Lehman Brothers collapsed on his watch as CEO in September 2008. As we all remember, that event contributed to a surprise implosion in global markets, jobs and our 401K plans.

Fuld, who will speak to investors about Lehman’s prior strategy, the financial crisis and keys to success today, will surely note that the world is brighter now. Silicon Valley is booming and the US economy is steadily creating jobs. The Dow Jones Industrial Average has nearly tripled since bottoming out in early 2009.

But are we better able to predict and avoid panics? I believe Data Science is our best hope.

There is no better judge of the need for better forecasting than Alan Greenspan, whose failure (along with almost everyone) to anticipate the 2008 crash prompted many criticisms of the long-celebrated US Federal Reserve Chairman. What went wrong? Blind Curve Ahead

“I have come to see that an important part of the answers to those questions is a very old idea: “animal spirits,” wrote Greenspan in a Foreign Affairs article that coincided with the publication of his book “The Map and the Territory: Risk, Human Nature, and the Future of Forecasting” in October 2013.

Like they sound, “animal spirits” mean that emotion – in this case risk aversion and panic – overcame rationality in the crash of 2008. And no conventional measure predicted the powerful result.

Greenspan is optimistic that we can improve our predictions.

I have come to believe that people, especially during periods of extreme economic stress, act in ways that are more predictable than economists have traditionally understood… If economists better integrate animal spirits into our models, we can improve our forecasting accuracy.

Here are three reasons I believe Data Science, especially sentiment analysis, might get us closer to this point.

1.    We’re getting better at tracking emotions via social media

Social media tracking is becoming a primary staple of market analysis for the private and public sector. For example, IBM says it has helped more than 100 organizations monitor Twitter streams to make better business decisions. For example, they found that phone carrier churn is correlated to weather outages. And in the retail industry, the most valuable customers were most likely to respond negatively to a change in sales associate. While the findings are not surprising, they can help enterprises focus on the right things.

Social media analysis has its share of black eyes as we continue to learn: overly simplistic algorithms often miss cultural, linguistic and contextual nuances. But the innovative energy focused on this is raising the bar all the time, as shown by the representation at an upcoming Sentiment Analysis Innovation conference in San Francisco.

2.    Wall Street firms are starting to invest in data science

Hedge funds and other investment firms, no strangers to quantitative analysis or innovative trading strategies, are testing the waters with data science as well. DCM Capital was an early false start in 2013, putting itself up for auction one month after launching a trading venue whose subscribers could track Facebook and Twitter feeds to predict market sentiment. A more recent and serious venture touching on this opportunity is Two Sigma Investments, which tasks data scientists rather than MBAs to mine headlines, weather reports and Tweets in search of investment ideas. Whether or not they succeed, the money and innovation is starting to apply the right queries to financial market analysis.

3.    It's about more than trading: sentiment analysis has the attention of economists and policymakers

Researchers at Oxford University and TheySay Analytics recently outlined a new approach in their report Predicting Economic Indicators from Web Text Using Sentiment Composition. "We can achieve high predictive accuracy" for the non-farm payroll index, they wrote. While this particular indicator might not correlate directly to market crashes, the study shows that the right minds are broadening the economic focus of sentiment analysis.

The US federal government might have an interesting role to play in all this. Policymakers at all levels continue to worry about systemic financial risk. They would do well to seek the advice of DJ Patil, who moved from Greylock Partners to become the US’ first Chief Data Scientist at the White House. Patil has experience in all the right areas: finance, data science and now public policy.

Markets will never be predictable. But increasingly interconnected economies and increasingly concentrated financial institutions demand that we get more rational about preventing panic.