Share
Embed
Europython 2016 - Things I wish I knew before using Python for Data Processing
30 minute talk in Europyhthon 2016 based on the a lighting talk.
Miguel Cabrera
July 20, 2016
Transcript
123 slides
-
Things I wish I Knew Before Using Python for Data
Processing Miguel Cabrera @mfcabrera 20 July 2016 -
Hello! I am Miguel! Data Engineer / Scien6st @
TrustYou Python ~ 2 years Berlin @mfcabrera mfcabrera at gmail hAp://mfcabrera.com 2 -
• (Rela6vely) New to Python, mostly Scien6fic stack
• You have used things like Numpy, Scikit-‐ Learn, Gensim, etc… • Your job 6tle includes either the word Data or “Machine Learning”. • Not necessarily a trained SoWware Engineer 3 Priors! -
• Data Scien6st? • Data Analyst? • Data
Engineer? • Machine Learning Developer? • SoWware Developer? • Other? 4 Who is who? -
• Basic Concepts and prac6ces • Some goodies of
the collec6on module • Iterators and Iterables • Conclusion 5 Agenda -
• Recent university grad • Mostly R and Matlab
• Writes niWy code to classify documents using Jupyter Notebooks • Mostly NltK and Scikit-‐Learn 6 David’s Story -
Code does not necessarily… • Have tests • Follow
conven6ons • Have documenta6on • Follow processes 14 -
“Software is the programming text that is part of a
deliverable” 15 -
You want to build Software… • Maintainable •
Testable • Deployable 16 -
“ Python is an interpreted, interac8ve, object-‐oriented programming
language. It incorporates modules, excep8ons, dynamic typing, very high level dynamic data types, and classes. 17 Python is… Source: https://docs.python.org/3/faq/general.html#what-is-python -
“ (OOP) is a programming paradigm based on the
concept of "objects", which may contain data, in the form of fields, oGen known as a0ributes; and code, in the form of procedures, oGen known as methods. 18 Object Oriented Programming (OOP) Source: https://en.wikipedia.org/wiki/Object-oriented_programming -
22 class Cookie(object): def __init__(self, sugar=5): self.sugar
= sugar def eat(self): pass def split(self): pass -
23 class Cookie(object): def __init__(self, sugar=5): self.sugar
= sugar def eat(self): pass def split(self): pass -
24 class Cookie(object): def __init__(self, sugar=5): self.sugar
= sugar def eat(self): pass def split(self): pass -
25 class Cookie(object): def __init__(self, sugar=5): self.sugar
= sugar def eat(self): pass def split(self): pass -
26 class Cookie(object): def __init__(self, sugar=5): self.sugar
= sugar def eat(self): pass def split(self): pass c = Cookie(3) -
27 class Alfajor(Cookie): def __init__(self, chocolate=10, sugar=10):
super(Alfajor, self).__init__(sugar=sugar) self.chocolate = chocolate a = Alfajor(chocolate=20, sugar=30) -
28 from sklearn import svm data = #
multiple lines to load the data X = # multiples lines extract the features y = # ... clf = svm.SVC() clf.fit(X, y) clf.predict(...) # multiples lines store the results -
How do I write good OO code? • DRY
• KISS • SOLID 30 -
“ Every piece of knowledge must have a single,
unambiguous, authorita8ve representa8on within a system 31 Source: https://en.wikipedia.org/wiki/Don%27t_repeat_yourself DRY: Don’t Repeat Yourself -
“ Simplicity is the ul8mate sophis8ca8on 32 Leonardo
Davicni KISS: Keep it Simple Stupid -
• Single responsibility principle • Open/closed principle •
Liskov subs6tu6on principle • Interface segrega6on principle • Dependency inversion principle 33 Be SOLID “Principles Of OOD”, Robert C. Martin Source: https://es.wikipedia.org/wiki/SOLID -
• Single responsibility principle • Open/closed principle •
Liskov subs6tu6on principle • Interface segrega6on principle • Dependency inversion principle 34 Be SOLID “Principles Of OOD”, Robert C. Martin Source: https://es.wikipedia.org/wiki/SOLID -
• “Readability counts” (PEP20) • Spaces vs. Tabs
• Indenta6on rules • Code organiza6on • PEP-‐8 is the de-‐facto style 37 Coding Conventions -
• Project structure • Tes6ng (Check out py.test!)
• Versioning and branching • Code Reviews • SoWware Development Life Cycle 40 Other Topics -
Talks @ Europython 42 • Clean Code in Python
by Mariano Anaya (Today at 15:45 Barria 2) • What’s the point of Object Orienta6on? by Iwan Vosloo (Thursday 11:15 A2) -
47 items = ["a", "b", "a", "x", "x", "y", "c",
"c", "a"] item_counts = {} for i in items: if i in items: item_counts[i] = item_counts[i] + 1 else: item_counts[i] = 1 Using dicts! -
48 items = ["a", "b", "a", "x", "x", "y", "c",
"c", "a"] item_counts = {} for i in items: try: item_counts[i] = item_counts[i] + 1 except KeyError: item_counts[i] = 1 Using dicts (EAFP version ) -
51 from collections import defaultdict items = ["a",
"b", "a", "x", "x", "y", "c", "c", "a"] item_counts = defaultdict(int) for i in items: item_counts[i] = item_counts[i] + 1 defaultdict(int, {'a': 3, 'b': 1, 'c': 2, 'x': 2, 'y': 1}) Using defaultdict -
53 from collections import Counter items = ["a",
"b", "a", "x", "x", "y", "c", "c", "a"] item_counts = Counter(items) print(item_counts) Using Counter Counter({'a': 3, 'x': 2, 'c': 2, 'b': 1, 'y': 1}) -
55 Extra goodies >>> c = Counter(a=3, b=1) >>>
d = Counter(a=1, b=2) >>> c.most_common() >>> c.values() >>> c + d >>> c -‐ d >>> c & d # intersection: min(c[x], d[x]) >>> c | d # union: max(c[x], d[x]) -
58 Counter based PMF from collections import Counter
class PMF(Counter): def normalize(self): total = float(sum(self.values())) for key in self: self[key] /= total -
59 Counter based PMF from collections import Counter
class PMF(Counter): def normalize(self): total = float(sum(self.values())) for key in self: self[key] /= total def __init__(self, *args, **kwargs): super(PMF, self).__init__(…) self.normalize() -
• Use the and extend Counter class •
Awesome ar6cle from @TreyHunner on Coun6ng [1]. 60 On Counting and Python [1] http://treyhunner.com/2015/11/counting-things-in-python/ -
• Code around the dict, tuples or lists
• Never know what to expect • Code becomes hard to read 62 Named Tuples -
63 Example from math import sqrt pt1 = (1.0,
5.0) pt2 = (2.5, 1.5) line_length = sqrt((pt1[0] -‐ pt2[0])**2 + (pt1[1] -‐ pt2[1])**2) Source: hAp://stackoverflow.com/ques6ons/2970608/what-‐are-‐named-‐tuples-‐in-‐python -
64 Example line_length = sqrt((pt1[0]
-‐ pt2[0])**2 + (pt1[1] -‐ pt2[1])**2) Source: hAp://stackoverflow.com/ques6ons/2970608/what-‐are-‐named-‐tuples-‐in-‐python -
65 Example from collections import namedtuple from math import
sqrt Point = namedtuple('Point', 'x y') pt1 = Point(1.0, 5.0) pt2 = Point(2.5, 1.5) line_length = sqrt((pt1.x -‐ pt2.x)**2 + (pt1.y -‐ pt2.y)**2) Source: hAp://stackoverflow.com/ques6ons/2970608/what-‐are-‐named-‐tuples-‐in-‐python -
66 It has cool methods _asdict Return a new OrderedDict
which maps field names to their values _make(iterable) Class method that makes a new instance from an exis6ng sequence or iterable. -
67 Extending NamedTuple from collections import namedtuple _HotelBase
= namedtuple( 'HotelDescriptor', ['cluster_id', 'trust_score', 'reviews_count', 'category_scores', 'intensity_factors'], ) class HotelDescriptor(_HotelBase): def compute_prior(self): if not self.trust_score or not self.reviews_count: raise NotEnoughDataForRanking("…") return _compute_prior(self.trust_score,…) -
69 l = [1, 2, 3, 4] for i
in x: print(x) -
70 an iterator comprehension (an) iterable produces
typically is iter() always is a generator expression a generator funcCon is is a generator container (list, dict, etc) next() Lazily produce the next value By Vincent Driessen - Source: http://nvie.com/posts/iterators-vs-generators/ -
71 an iterator (an) iterable iter() always
is next() Lazily produce the next value -
72 an iterator (an) iterable iter() always
is next() Lazily produce the next value comprehension -
73 an iterator (an) iterable iter() always
is next() Lazily produce the next value comprehension produces -
74 an iterator (an) iterable iter() always
is next() Lazily produce the next value comprehension produces container (list, dict, etc) -
75 an iterator (an) iterable iter() always
is next() Lazily produce the next value comprehension produces container (list, dict, etc) assert 1 in [1, 2, 3] -
76 an iterator (an) iterable iter() always
is next() Lazily produce the next value comprehension produces container (list, dict, etc) assert 1 in {1, 2, 3} -
77 an iterator (an) iterable iter() always
is next() Lazily produce the next value comprehension produces typically is container (list, dict, etc) -
78 an iterator (an) iterable iter() always
is next() Lazily produce the next value comprehension produces typically is container (list, dict, etc) l = [1, 2, 3, 4] x = iter(l) y = iter(l) -
79 an iterator (an) iterable iter() always
is next() Lazily produce the next value comprehension produces typically is container (list, dict, etc) l = [1, 2, 3, 4] x = iter(l) y = iter(l) type(l) >> <class 'list'> type(x) >> <class 'list_iterator'> -
80 an iterator (an) iterable iter() always
is next() Lazily produce the next value comprehension produces typically is container (list, dict, etc) l = [1, 2, 3, 4] x = iter(l) y = iter(l) type(l) >> <class 'list'> type(x) >> <class 'list_iterator'> next(x) >> 1 next(y) >> 1 next(y) >> 2 -
81 an iterator (an) iterable iter() always
is next() Lazily produce the next value comprehension produces typically is container (list, dict, etc) l = [1, 2, 3, 4] for e in l: print(e) -
• An iterable is any object that can return
an iterator • Containers, files, sockets, etc. • Implement __iter__(). • Some of them may be infinite • The itertools contain many helper func6ons 82 Iterables -
83 class InverseReader(object): def __init__(self):
with open('file.txt') as f: self.lines = f.readlines() self.index = len(self.lines) -‐ 1 def __iter__(self): return self def next(self): # Python 3 __next__ self.index -‐= 1 if self.index < 0: raise StopIteration return self.lines[self.index] -
84 ir = InverseReader() for line in ir:
print line -
85 an iterator (an) iterable iter() always
is next() Lazily produce the next value comprehension produces typically is container is is a generator always is a generator expression a generator funcCon -
86 an iterator (an) iterable iter() always
is next() Lazily produce the next value a generator expression -
87 an iterator (an) iterable iter() always
is next() Lazily produce the next value a generator expression a generator funcCon -
88 an iterator (an) iterable iter() always
is next() Lazily produce the next value a generator expression a generator funcCon -
89 an iterator (an) iterable iter() always
is next() Lazily produce the next value is is a generator a generator expression a generator funcCon -
90 an iterator (an) iterable iter() always
is next() Lazily produce the next value is is a generator always is a generator expression a generator funcCon -
91 is a generator funcCon is
a generator expression a generator -
92 is a generator expression numbers
= [x for x in range(1, 10)] squares = [x * x for x in numbers] type(squares) # list a generator -
93 is a generator expression numbers
= [x for x in range(1, 10)] squares = [x * x for x in numbers] type(squares) # list lazy_squares = (x * x for x in numbers) lazy_squares # <generator object <genexpr> at 0x104c6da00> a generator -
94 is a generator expression numbers
= [x for x in range(1, 10)] squares = [x * x for x in numbers] type(squares) # list lazy_squares = (x * x for x in numbers) lazy_squares # <generator object <genexpr> at 0x104c6da00> next(lazy_squares) # 1 next(lazy_squares) # 4 x a generator -
95 is a generator expression numbers
= [x for x in range(1, 10)] squares = [x * x for x in numbers] type(squares) # list lazy_squares = (x * x for x in numbers) lazy_squares # <generator object <genexpr> at 0x104c6da00> next(lazy_squares) # 1 next(lazy_squares) # 4 lazy_squares = (x * x for x in numbers) for x in lazy_squares: print x a generator -
96 a generator is a generator
funcCon -
97 a generator def fib(): prev, curr
= 0,1 while True: yield curr prev,curr = curr, prev+curr is a generator funcCon -
98 a generator def fib(): prev, curr
= 0,1 while True: yield curr prev,curr = curr, prev+curr f = fib() next(f) # 1 next(f) # 1 next(f) # 2 is a generator funcCon -
99 a generator def fib(): prev, curr
= 0,1 while True: yield curr prev,curr = curr, prev+curr is a generator funcCon for x in islice(fib(), 0,3): print x # 1 # 1 # 2 -
100 class HdfsLineSentence(object): def __init__(self, source):
self.source = source def __iter__(self): stream = self.source.open('r') for line in stream: cid, s = line.split('\t') # decode and do some work with s yield s sentences = HdfsLineSentence(...) for s in setences: print(s) -
101 an iterator (an) iterable iter() always
is next() Lazily produce the next value comprehension produces typically is container is is a generator always is a generator expression a generator funcCon -
108 class HdfsLineSentence(object): def __init__(self, source):
self.source = source def __iter__(self): stream = self.source.open('r') for line in stream: cid, s = line.split('\t') # decode and do some work with s yield s sentences = HdfsLineSentence(...) for s in sentences: print s -
109 class FilterComment(object): def __init__(self, source):
self.source = source def __iter__(self): for s in self.source: if s[0] != "#": yield s -
110 class FilterComment(object): def __init__(self, source):
self.source = source def __iter__(self): for s in self.source: if s[0] != "#": yield s sents = FilterComment(HdfsLineSentence(source)) for s in sents: print s -
111 class FilterComment(object): def __init__(self, source):
self.source = source def __iter__(self): for s in self.source: if s[0] != "#": yield s sents = FilterComment(HdfsLineSentence(source)) for s in sents: print s -
112 class FilterComment(object): def __init__(self, source):
self.source = source def __iter__(self): for s in self.source: if s[0] != "#": yield s sents = FilterComment(HdfsLineSentence(source)) for s in sents: print s -
113 def filter_comment(source): for s in
source: if s[0] != "#": yield s sents = filter_comment(HdfsLineSentence(source)) for s in sents: print s -
Talks @ Europython 114 • Itera6on, itera6on, itera6on by
John Sutherland (Friday 15:45 Barria 1) -
121 Spaguetti: https://www.flickr.com/photos/129610671@N02/16633987421/ (CC BY- NC-ND 2.0) vision.communicate Autovification: Credit:
AV Dezign https://www.flickr.com/photos/91345457@N07/22666878846/ (CC BY-NC-ND 2.0) Iterators and Iterables based on work of Vincent Driessen : http://nvie.com/posts/iterators-vs-generators/ Ideas from Iterables taken from RaRe Technologies blog; http://rare-technologies.com/data-streaming- in-python-generators-iterators-iterables/ Credits Counter image: Dean Hochman Source: https://www.flickr.com/photos/17997843@N02/24061690099/“ (CC BY-NC-ND 2.0) PMF Class based on Vik Paruchuri’s https://www.dataquest.io/blog/python-counter-class/ Cookies: Source Wikipedia https://en.wikipedia.org/wiki/File:R%C5%AFzn%C3%A9_druhy_cukrov%C3%AD_(2).jpg (CC BY 3.0) Counting things in Python: http://treyhunner.com/2015/11/counting-things-in-python/ -
Thanks! Any quesCons? You can find me at:
@mfcabrera [email protected] 123