Data Mining the News to Predict Equity Premiums

Investors can forecast an equity premium by data mining the news, according to a new study.  

Researchers from the Helmut Schmidt University and the University of Rostock in Germany created a database of roughly 700,000 newspaper articles published in the New York Times and the Washington Post between 1980 and 2018.  

They then used a machine learning algorithm to determine whether news stories can forecast monthly equity premiums. Their findings? News, especially when it covers geopolitical issues, links to returns.  

According to the researchers, this paper is the latest of several that demonstrate how investors can use data mining to improve their predictions of equity premiums. 

While data mining the news to predict returns has been researched before, the studies prioritized quantitative measures, according to the paper, which was published online on April 26. This paper, by Philipp Adämmer and Rainer A. Schüssler, focused more on tracking qualitative measures using machine learning.  

“Due to this characteristic and complexity, statistical machine learning methods are natural candidates to handle qualitative data such as newspaper articles,” economists Adämmer and Schüssler wrote.

Here’s how it worked: the researchers assigned importance to topics covered by the media based on how much coverage those topics received. Those topics were averaged on a monthly basis, the paper said. 

The researchers focused on 100 topics, then created a regression to relate those topics to equity premiums. The program the researchers created allows one to focus on just a single topic, or to average all the topics together, depending on which is more accurate in the moment, according to the paper.  

The researchers found that the news could predict equity premiums that other established forecasts missed.  

[II Deep Dive: Big Data: Too Popular for its Own Good?] 

The findings from this modeling showed that data mining the news could be especially effective in down markets when investors most need to produce returns, the researchers said.

Data mining the news to forecast equity returns was also most effective when the topics covered geopolitics, rather than simply economic news, the paper said.