Machine learning: boldly going where Twitter's APIs don't

MetaBeerTalks given by James Kember, Engineer, at MetaBroadcast on March 12th, 2014

1. Machine learning: boldly going where Twitter's APIs don't
2. Twitter • Lots of data! • Limited resources (180 calls per 15 mins) on /statuses/show:id • We can get more resource by using our users API allowance, but that still isn't enough • API doesn't give all we want :( (Favourites)
3. Favourites • Stream api doesn't give favourite count :( • Polling individual tweets does • We just want the totals per tweet • So we need to pick the most active tweets • Most tweets have no favourites on them
4. Metric gathering • We need metrics! • Sometimes you aren’t sure what the metrics are (limited resources) • Random sampling for a baseline
5. 1st Attempt • Hand crafted • Requires good domain knowledge • Difficult to get all the rules in place (required quite a few tweaks based on looking at results)
7. Machine Learning • Works well in a big data domain with random elements (Twitter..) • Possible to learn trends that you would not think of given appropriate data set • Neural Network + Backpropagation • After picking the dataset and the right algorithm it does all the work for you!
8. Results
9. Get some. • https://github.com/mbst/neural-network • https://engage.metabroadcast.com