Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PyData Berlin 2015 - Processing Hotel Reviews w...

PyData Berlin 2015 - Processing Hotel Reviews with Python

PyData Berlin 2015 - Processing Hotel Reviews with Python
Video https://www.youtube.com/watch?v=eIEsdluhoxY

Miguel Cabrera

May 30, 2015
Tweet

More Decks by Miguel Cabrera

Other Decks in Programming

Transcript

  1. Processing Hotel Reviews with Python Miguel Cabrera @mfcabrera & Friends

    https://www.flickr.com/photos/18694857@N00/5614701858/
  2. •  Colombian •  Neuberliner •  Work for TrustYou as Data

    (Scientist|Engineer|Juggler)™ •  Python around 2 years •  Founder and former organizer of Munich DataGeeks About Me
  3. •  Crawling •  Natural Language Processing / Semantic Analysis • 

    Record Linkage / Deduplication •  Ranking •  Recommendation •  Classification •  Clustering Tasks
  4. Batch Layer •  Hadoop •  Python •  Pig* •  Java*

    Service Layer •  PostgreSQL •  MongoDB •  Redis •  Cassandra DATA DATA Hadoop Cluster Application Machines Stack
  5. Hadoop Streaming hadoop jar contrib/streaming/hadoop-*streaming*.jar \ -file /home/hduser/mapper.py -mapper /home/hduser/mapper.py

    \ -file /home/hduser/reducer.py -reducer /home/hduser/reducer.py \ -input /user/hduser/text.txt -output /user/hduser/gutenberg-output
  6. Luigi •  Dependency definition •  Hadoop / HDFS Integration • 

    Object oriented abstraction •  Parallelism •  Resume failed jobs •  Visualization of pipelines •  Command line integration
  7. Minimal Bolerplate Code class WordCount(luigi.Task): date = luigi.DateParameter() def requires(self):

    return InputText(date) def output(self): return luigi.LocalTarget(’/tmp/%s' % self.date_interval) def run(self): count = {} for f in self.input(): for line in f.open('r'): for word in line.strip().split(): count[word] = count.get(word, 0) + 1 f = self.output().open('w') for word, count in six.iteritems(count): f.write("%s\t%d\n" % (word, count)) f.close()
  8. class WordCount(luigi.Task): date = luigi.DateParameter() def requires(self): return InputText(date) def

    output(self): return luigi.LocalTarget(’/tmp/%s' % self.date_interval) def run(self): count = {} for f in self.input(): for line in f.open('r'): for word in line.strip().split(): count[word] = count.get(word, 0) + 1 f = self.output().open('w') for word, count in six.iteritems(count): f.write("%s\t%d\n" % (word, count)) f.close() Task Parameters
  9. class WordCount(luigi.Task): date = luigi.DateParameter() def requires(self): return InputText(date) def

    output(self): return luigi.LocalTarget(’/tmp/%s' % self.date_interval) def run(self): count = {} for f in self.input(): for line in f.open('r'): for word in line.strip().split(): count[word] = count.get(word, 0) + 1 f = self.output().open('w') for word, count in six.iteritems(count): f.write("%s\t%d\n" % (word, count)) f.close() Programmatically Defined Dependencies
  10. class WordCount(luigi.Task): date = luigi.DateParameter() def requires(self): return InputText(date) def

    output(self): return luigi.LocalTarget(’/tmp/%s' % self.date_interval) def run(self): count = {} for f in self.input(): for line in f.open('r'): for word in line.strip().split(): count[word] = count.get(word, 0) + 1 f = self.output().open('w') for word, count in six.iteritems(count): f.write("%s\t%d\n" % (word, count)) f.close() Each Task produces an ouput
  11. class WordCount(luigi.Task): date = luigi.DateParameter() def requires(self): return InputText(date) def

    output(self): return luigi.LocalTarget(’/tmp/%s' % self.date_interval) def run(self): count = {} for f in self.input(): for line in f.open('r'): for word in line.strip().split(): count[word] = count.get(word, 0) + 1 f = self.output().open('w') for word, count in six.iteritems(count): f.write("%s\t%d\n" % (word, count)) f.close() Write Logic in Python
  12. class WordCount(luigi.hadoop.JobTask): date = luigi.DateParameter() def requires(self): return InputText(date) def

    output(self): return luigi.hdfs.HdfsTarget(’%s' % self.date_interval) def mapper(self, line): for word in line.strip().split(): yield word, 1 def reducer(self, key, values): yield key, sum(values) Luigi + Hadoop/HDFS
  13. Luigi •  Minimal bolierplate code •  Programmatically define dependencies • 

    Integration with HDFS / Hadoop •  Task Syncronization •  Can wrap anything
  14. Before •  Bash scripts + Cron •  Manual cleanup • 

    Manual failure recovery •  Hard(er) to debug
  15. We use it for… •  Standalone executables •  Dump data

    from databases •  General Hadoop Streaming •  Bash Scripts / MRJob •  Pig* Scripts
  16. SQL SELECT f3, SUM(f2), AVG(f1) FROM relation WHERE f1 >

    500 GROUP BY f3 rel = LOAD 'relation' AS (f1: int, f2: int, f3: chararray); rel = FILTER rel f1 > 500 by_f3 = GROUP rel BY f3; result = FOREACH by_f3 GENERATE group, SUM(by_f3.f2), AVG(by_f3.f1) Pig Latin Python def map(r): if r['f1'] > 500: yield r['f3'], [r['f1'], r['f2']] def reduce(k, values): avg = 0 summ = 0 l = len(values) for r in values: summ += r[1] avg += r[0] avg = avg/float(l) yield k, [summ, avg]
  17. Pig + Python •  Data loading and transformation in Pig

    •  Other logic in Python •  Pig as a Luigi Task •  Pig UDFs defined in Python
  18. Snippets from Reviews “Hips don’t lie” “Maid was banging” “Beautiful

    bowl flowers” “Irish dance, I love that” “No ghost sighting” “One ghost touching” “Too much cardio, not enough squats in the gym” “it is like hugging a bony super model”
  19. from gensim.models.doc2vec import Doc2Vec class LearnModelTask(luigi.Task): # Parameters.... blah blah

    blah def output(self): return luigi.LocalTarget(os.path.join(self.output_directory, self.model_out)) def requires(self): return LearnBigramsTask() def run(self): sentences = LabeledClusterIDSentence(self.input().path) model = Doc2Vec(sentences=sentences, size=int(self.size), dm=int(self.distmem), negative=int(self.negative), workers=int(self.workers), window=int(self.window), min_count=int(self.min_count), train_words=True) model.save(self.output().path)
  20. Takeaways •  It is possible to use Python as the

    primary language for doing large data processing on Hadoop. •  It is not a perfect setup but works well most of the time. •  Keep your ecosystem open to other technologies. •  Products reviews contain much more information than just facts.