Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Sofia Heisler - No More Sad Pandas: Optimizing ...

Sofia Heisler - No More Sad Pandas: Optimizing Pandas Code for Speed and Efficiency

When I first began working with the Python Pandas library, I was told by an experienced Python engineer: "Pandas is fine for prototyping a bit of calculations, but it's too slow for any time-sensitive applications." Over multiple years of working with the Pandas library, I have realized that this was only true if not enough care is put into identifying proper ways to optimize the code's performance. This talk will review some of the most common beginner pitfalls that can cause otherwise perfectly good Pandas code to grind to a screeching halt, and walk through a set of tips and tricks to avoid them. Using a series of examples, we will review the process for identifying the elements of the code that may be causing a slowdown, and discuss a series of optimizations, ranging from good practices of input data storage and reading, to the best methods for avoiding inefficient iterations, to using the power of vectorization to optimize functions for Pandas dataframes.

https://us.pycon.org/2017/schedule/presentation/628/

PyCon 2017

May 21, 2017
Tweet

More Decks by PyCon 2017

Other Decks in Programming

Transcript

  1. What's Pandas? • Open-source library that offers data structure support

    and a great set of tools for data analysis • Makes Python a formidable competitor to R and other data science tools • Widely used in everything from simple data manipulation to complex machine learning bit.ly/2rCVVUD
  2. Why optimize Pandas? • Pandas is built on top of

    NumPy and Cython, making it very fast when used correctly • Correct optimizations can make the difference between minutes and milliseconds bit.ly/2rCVVUD
  3. Our working dataset All hotels in New York state sold

    by Expedia Source: http://developer.ean.com/database/property-data
  4. Our example function def normalize(df, pd_series): pd_series = pd_series.astype(float) #

    Find upper and lower bound for outliers avg = np.mean(pd_series) sd = np.std(pd_series) lower_bound = avg - 2*sd upper_bound = avg + 2*sd # Collapse in the outliers df.loc[pd_series < lower_bound , "cutoff_rate" ] = lower_bound df.loc[pd_series > upper_bound , "cutoff_rate" ] = upper_bound # Finally, take the log normalized_price = np.log(df["cutoff_rate"].astype(float)) return normalized_price bit.ly/2rCVVUD
  5. Magic commands • “Magic” commands available through Jupyter/ IPython notebooks

    provide additional functionality on top of Python code to make it that much more awesome • Magic commands start with % (executed on just the line) or %% (executed on the entire cell) bit.ly/2rCVVUD
  6. • Use IPython's %timeit command • Re-runs a function repeatedly

    and shows the average and standard deviation of runtime obtained • Can serve as a benchmark for further optimization Timing functions with %timeit bit.ly/2rCVVUD
  7. %timeit df['hr_norm'] = normalize(df, df['high_rate']) 2.84 ms ± 180 µs

    per loop (mean ± std. dev. of 7 runs, 100 loops each) Timing functions with %timeit bit.ly/2rCVVUD
  8. Our practice function: Haversine distance def haversine(lat1, lon1, lat2, lon2):

    miles_constant = 3959 lat1, lon1, lat2, lon2 = map(np.deg2rad,\ [lat1, lon1, lat2, lon2]) dlat = lat2 - lat1 dlon = lon2 - lon1 a = np.sin(dlat/2)**2 + np.cos(lat1) *\ np.cos(lat2) * np.sin(dlon/2)**2 c = 2 * np.arcsin(np.sqrt(a)) mi = miles_constant * c return mi bit.ly/2rCVVUD
  9. Crude iteration, or what not to do • Rookie mistake:

    “I just wanna loop over all the rows!” • Pandas is built on NumPy, designed for vector manipulation - loops are inefficient • The Pandas iterrows method will provide a tuple of (Index, Series) that you can loop through - but it's quite slow bit.ly/2rCVVUD
  10. Running function with iterrows %%timeit haversine_series = [] for index,

    row in df.iterrows(): haversine_series.append(haversine(40.671, -73.985,\ row['latitude'], row['longitude'])) df['distance'] = haversine_series 184 ms ± 6.65 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) bit.ly/2rCVVUD
  11. Nicer looping: using apply • apply applies a function along

    a specified axis (rows or columns) • More efficient than iterrows, but still requires looping through rows • Best used only when there is no way to vectorize a function bit.ly/2rCVVUD
  12. Timing looping with apply %%timeit df['distance'] =\ df.apply(lambda row: haversine(40.671,

    -73.985,\ row['latitude'], row['longitude']), axis=1) 78.1 ms ± 7.55 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) bit.ly/2rCVVUD
  13. The scoreboard Methodology Avg.  single  run   3me  (ms) Marginal

     performance   improvement Looping  with  iterrows 184.00 Looping  with  apply 78.10 2.4x bit.ly/2rCVVUD
  14. Apply is doing a lot of repetitive steps %lprun -f

    haversine \ df.apply(lambda row: haversine(40.671, -73.985,\ row['latitude'], row['longitude']), axis=1) bit.ly/2rCVVUD
  15. Doing it the pandorable way: vectorize • The basic units

    of Pandas are arrays: • Series is a one-dimensional array with axis labels • DataFrame is a 2-dimensional array with labeled axes (rows and columns) • Vectorization is the process of performing the operations on arrays rather than scalars bit.ly/2rCVVUD
  16. Why vectorize? • Many built-in Pandas functions are built to

    operate directly on arrays (e.g. aggregations, string functions, etc.) • Vectorized functions in Pandas are inherently much faster than looping functions bit.ly/2rCVVUD
  17. Vectorizing significantly improves performance %%timeit df['distance'] = haversine(40.671, -73.985,\ df['latitude'],

    df['longitude']) 1.79 ms ± 230 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) bit.ly/2rCVVUD
  18. The function is no longer looping %lprun -f haversine haversine(40.671,

    -73.985,\ df['latitude'], df['longitude']) bit.ly/2rCVVUD
  19. The scoreboard Methodology Avg.  single  run  3me   (ms) Marginal

     performance   improvement Looping  with  iterrows 184.00 Looping  with  apply 78.10 2.4x Vectoriza3on  with   Pandas  series 1.79 43.6x bit.ly/2rCVVUD
  20. Why NumPy? • NumPy is a “fundamental package for scientific

    computing in Python” • NumPy operations are executed “under the hood” in optimized, pre-compiled C code on ndarrays • Cuts out a lot of the overhead incurred by operations on Pandas series in Python (indexing, data type checking, etc.) bit.ly/2rCVVUD
  21. Converting code to operate on NumPy arrays instead of Pandas

    series %%timeit df['distance'] = haversine(40.671, -73.985,\ df['latitude'].values, df['longitude'].values) 370 µs ± 18 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) bit.ly/2rCVVUD
  22. Optimizing with NumPy arrays • We've gotten our runtime down

    from 184 ms to 370 µs • That's more than 500-fold improvement! Methodology Avg.  single   run  3me   Marginal  performance   improvement Looping  with  iterrows 184.00 Looping  with  apply 78.10 2.4x Vectoriza<on  with  Pandas  series 1.79 43.6x Vectoriza3on  with  NumPy  arrays 0.37 4.8x bit.ly/2rCVVUD
  23. Okay, but I really want to use a loop… •

    There are a few reasons why you might actually want to use a loop: • Your function is complex and does not yield itself easily to vectorization • Trying to vectorize your function would result in significant memory overhead • You're just plain stubborn bit.ly/2rCVVUD
  24. Speeding up code with Cython • Cython language is a

    superset of Python that additionally supports calling C functions and declaring C types • Almost any piece of Python code is also valid Cython code • Cython compiler will convert Python code into C code which makes equivalent calls to the Python/C API. bit.ly/2rCVVUD
  25. Re-defining the function in the Cython compiler %load_ext cython %%cython

    cpdef haversine_cy(lat1, lon1, lat2, lon2): miles_constant = 3959 lat1, lon1, lat2, lon2 = map(np.deg2rad,\ [lat1, lon1, lat2, lon2]) dlat = lat2 - lat1 dlon = lon2 - lon1 a = np.sin(dlat/2)**2 + np.cos(lat1) *\ np.cos(lat2) * np.sin(dlon/2)**2 c = 2 * np.arcsin(np.sqrt(a)) mi = miles_constant * c return mi bit.ly/2rCVVUD
  26. Re-defining the function in the Cython compiler %%timeit df['distance'] =\

    df.apply(lambda row: haversine_cy(40.671, -73.985,\ row['latitude'], row['longitude']), axis=1) 76.5 ms ± 6.42 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) bit.ly/2rCVVUD
  27. Scoreboard Methodology Avg.  single   run  3me  (ms) Marginal  performance

      improvement Looping  with  iterrows 184.00 Looping  with  apply 78.10 2.4x Running  row-­‐wise  func3on   through  Cython  compiler 76.50 1.0x Vectoriza<on  with  Pandas  series 1.79 43.6x Vectoriza<on  with  NumPy  arrays 0.37 4.8x bit.ly/2rCVVUD
  28. Evaluating results of conversion to Cython Adding the -a option

    to %%cython magic command shows how much of the code has not actually been converted to C by default… and it's a lot! bit.ly/2rCVVUD
  29. Speeding up code with Cython • As long as Cython

    is still using Python methods, we won't see a significant improvement • Make the function more Cython-friendly: • Add explicit typing to the function • Replace Python/NumPy libraries with C-specific math libraries bit.ly/2rCVVUD
  30. Better cythonizing through static typing and C libraries %%cython -a

    from libc.math cimport sin, cos, acos, asin, sqrt cdef deg2rad_cy(float deg): cdef float rad rad = 0.01745329252*deg return rad cpdef haversine_cy_dtyped(float lat1, float lon1, float lat2, float lon2): cdef: float dlon float dlat float a float c float mi lat1, lon1, lat2, lon2 = map(deg2rad_cy, [lat1, lon1, lat2, lon2]) dlat = lat2 - lat1 dlon = lon2 - lon1 a = sin(dlat/2)**2 + cos(lat1) * cos(lat2) * sin(dlon/2)**2 c = 2 * asin(sqrt(a)) mi = 3959 * c return mi bit.ly/2rCVVUD
  31. Timing the cythonized function %%timeit df['distance'] =\ df.apply(lambda row: haversine_cy_dtyped(40.671,

    -73.985,\ row['latitude'], row['longitude']), axis=1) 50.1 ms ± 2.74 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) bit.ly/2rCVVUD
  32. Scoreboard Methodology Avg.  single   run  3me  (ms) Marginal  performance

      improvement Looping  with  iterrows 184.00 Looping  with  apply 78.10 2.4x Running  row-­‐wise  func<on  through   Cython  compiler 76.50 1.0x Looping  with  Cythoninzed  func3on 50.10  1.6x   Vectoriza<on  with  Pandas  series 1.79  28x   Vectoriza<on  with  NumPy  arrays 0.37 4.8x bit.ly/2rCVVUD
  33. The scoreboard Methodology Avg.  single  run  3me   (ms) Marginal

     performance   improvement Looping  with  iterrows 184.00 Looping  with  apply 78.10  2.4x   Looping  with  Cython 50.10  1.6x   Vectoriza<on  with   Pandas  series 1.79  28x   Vectoriza<on  with   NumPy  arrays 0.37  4.8x   bit.ly/2rCVVUD
  34. The zen of Pandas optimization • Avoid loops • If

    you must loop, use apply, not iteration functions • If you must apply, use Cython to make it faster • Vectorization is usually better than scalar operations • Vector operations on NumPy arrays are more efficient than on native Pandas series bit.ly/2rCVVUD
  35. A word of warning… “Premature optimization is the root of

    all evil” Source: https://xkcd.com/1691/
  36. References • http://cython.readthedocs.io/en/latest/ • http://cython.org/ • http://pandas.pydata.org/pandas-docs/stable/ • http://www.nongnu.org/avr-libc/user-manual/group__avr__math.html •

    https://docs.python.org/2/library/profile.html • https://docs.scipy.org/doc/numpy/user/whatisnumpy.html • https://ipython.org/notebook.html • https://penandpants.com/2014/09/05/performance-of-pandas-series-vs-numpy-arrays/ • https://www.datascience.com/blog/straightening-loops-how-to-vectorize-data- aggregation-with-pandas-and-numpy/