Predicting the Stock Market with Twitter

Given at JavaScript North East, Monday 5th October 2013. http://jsnortheast.co.uk/

Transcript

1. Predicting the stock market with Twitter Nabil Freeman @nfrmn
2. Hello •  Nabil Freeman •  Co-founder @ Matchchat •  www.matchchat.co.uk •  www.twitter.com/nfrmn
3. About this project •  Hacked together in a weekend •  Research paper by Tushar Rao & Saket Srivastava in Delhi, India •  Soon to be open sourced
4. Theory •  Twitter sentiment reflects investor behavior on the stock market (EMH) •  Real-time thanks to the Streaming API •  Until recently, implementations have been pretty dumb
5. Anne Hathaway •  Berkshire Hathaway prices spike spuriously when Anne is in the media
6. Sentiment analysis Know what the Twitter hivemind thinks
7. Guts of the project •  Mine tweets for large technology stocks (MSFT, AMZN, AAPL, etc.) •  Calculate sentiment for each tweet, and create a trend •  Compare against historical closing price data for each company and find correlation
8. Technology •  Node.js •  ntwitter – github:AvianFlu •  SentiMental – github:thinkroth •  Node Language Detect – github:FGRibreau •  Yahoo Finance API •  MySQL
9. Scan.js •  Tracks an array of symbols on Twitter Streaming API using the $ hashtag •  Strips out any hyperlinks, retweets, copypasta & foreign tweets •  Stores the tweet with metadata in DB
10. Finance.js •  Extract dataset from DB •  Run sentiment analysis on data collection •  Score each Tweet’s sentiment •  Pull CSV from Yahoo Finance and compare the two trend lines
11. :-(
12. Why? •  Scan.js crashed too often •  No dataset = no trends •  Later version with forever.js improved problem, but still no dice
13. Improvements •  Try again with better technologies •  CouchDB (?) •  Understand node.js garbage collection •  Further down the line… •  Optimize sentiment tool for stock market language (bull, bear, dog, etc.) •  Write my own sentiment tool
14. Call for help •  Going to open source it and try again •  Still a Javascript neophyte •  Any help would be appreciated •  Maybe we can all make some $$$
15. Thank you, goodnight, much love nabil@matchchat.co.uk