Related Topics

I'm currently building a somewhat similar neural network based on Twitter data, and with all respect to Johan Bollen, Huina Mao and Xiao-Jun Zeng, there is simply no way to empirically tie the team's 6 dimensions of mood (based on GPOMS) to the stock market. Their mood dimension were: Calm, Alert, Sure, Vital, Kind and Happy. Could somebody please tell me how the heck a strongly Kind set of tweets has anything to do with the stock market?

My research shows that a good percentage of tweets are essentially teenagers asking Justin Bieber to follow their account (you would be astonished at how many hundreds of thousands of these I see each day). If these requests start turning more Kind then the stock market will swing? Don't believe the headline from this paper. The authors essentially chose the mood dimensions that best matched the data and declared victory.

There was also a very odd detail in the paper that many people missed. Their training set was made up of 9 months of data, which then predicted one month of actual data. This is an extremely impractical approach, due to the time required to train the model, but it also begs the question, why wasn't a larger set of data predicted against? They had data from Feb-Dec, and they could only predict for that final month?

The approach that I've seen in the field, which I think is much more realistic, is to test for changes in sentiment and use that as an alert to a human trader. The guys over at DataMinr have an incredible n-gram based sentiment approach that alerts stocks traders when sentiment changes (they use 30 day moving averages, tests for subject frequency, etc.). Those guys identified the Osama event after 15 tweets came through on the subject. They are quietly rocking the Twitter sentiment space while Bollen is going for headlines.

But don't get me wrong, Bollen is a must cite player in this field. He has two papers out on the subject and one of the few actually using a neural network (the other big neural network shop is over at Stanford, where they did an amazing Twitter sentiment paper, using SVM and ANN). I would give my right arm just to have a conversation with Johan. So Johan, if you use Quora, please help me understand the science behind the selective mood dimension thing. Did you actually ignore some of your dimensions and is that OK? And did you ever attempt to predict months other than December? Why the 9 month train set? There are lowly researchers in Palo Alto losing sleep over this stuff!