vigneshwer dhinakaran - Pumping up Python modules using Rust

If you’ve spent much time writing (or debugging) Python performance problems, you’ve probably had a hard time managing memory with its limited language support.

In this talk, we venture deep into the belly of the Rust Language to uncover the secret incantations for building high performance and memory safe Python extensions using Rust.

Rust has a lot to offer in terms of safety and performance for high-level programming languages such Python, Ruby, Js and more with its easy Foreign Function Interface capabilities which enables developers to easily develop bindings for foreign code.

https://us.pycon.org/2018/schedule/presentation/82/

Transcript

1. Pumping Python modules using Rust PyCon US 2017 Vigneshwer Dhinakaran | Mozilla TechSpeaker | @dvigneshwer
2. Need for native extension Performance Freedom in memory management Access to low-level APIs
3. Solutions to native extension Python community & it’s strong lib ecosystem to the rescue
4. Native extension aren’t new to Python devs
5. Native extension aren’t new to Python devs
6. Native extension aren’t new to Python devs
7. Native extension aren’t new to Python devs
8. Reality: Not many of us can write C/C++ extensions
9. Writing C/C++ extension is scary & painful Manual memory management Segfaults Data Races
10. Other options? • Cython • Numba • PyPy • ...
11. What’s the main ask here for us? C/C++ ‘s Performance, Portability & Embeddability With Guaranteed Safety
12. Rust is a system programming language that has great control like C/C++, delivers productivity like in Python, and is super safe.
13. Zen of Rust Memory Safety without Garbage Collection Rich Typesystem High level iterators Freedom from data race Welcoming Community
14. Facts & Figures
15. Speeding up crypto on Wire desktop apps https://medium.com/@wireapp/speeding-up-crypto-on-wire-desktop-apps-3ff37fc98c3f Up to 141× faster
16. Fixing Python Performance with Rust https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html
17. Snips Uses Rust to Build an Embedded Voice Assistant
18. Tokio MiniHTTP The library is a proof-of-concept implementation of a simple HTTP/1.1 server using Tokio. http://aturon.github.io/blog/2016/08/11/futures/
19. 2016 -18 For the third year in a row, Rust is stackoverflow's most loved language https://insights.stackoverflow.com/survey/2018/
20. Friends of Rust Organizations running Rust in production. (https://www.rust-lang.org/en-US/friends.html)
21. History: How did it all begin?
22. Browsers Compete for performance & need Safety at its core
26. Gradual adoption works.
27. Rust is easy to learn!
28. Variables
29. Functions
30. Flow control
31. Flow control
32. Looping
33. Types
34. Package Manager
35. Ownership The act, state, or right of possessing something
36. Garbage Collection Heap Memory Circle(5) Circle(5) Circle(5)
37. Referencing Now both bookingService and repairService hold a reference to vehicle
38. Same implementation in Rust Doesn't compile!
39. Ownership • Rust moves ownership by default • The owner has the right to destroy the thing it owns • The memory is freed as soon as the owned variable goes out of it's scope • Hence vehicle may already be destroyed at the point when it's passed to repair_service • Rust catches these errors at compile time
40. Borrowing The act, state, or right of possessing something
41. Making it work in Rust Lending things is the key!
42. Borrowing • A reference is passed without transferring ownership • One can borrow immutable (&) or mutable (&mut) but not both at the same time • Shared ownership can be achieved through special pointers with runtime checks
43. Mutability Rules • All variables are immutable by default • Only one mutable reference at a time ( But as many immutable &’s as you want ) • Mutable references block all other access (The &mut must go out of scope before using other &’s )
44. Yes !!! Safe memory management without GC
45. High-level iterators in Rust
46. High-level iterators Execution time : ~ 250 ms
47. High-level iterators
48. Rayon ~ 50 ms
49. custom data type in Rust
50. Struct • Structs are like lightweight classes • They can have fields • They can be instantiated
51. Struct But not partially !
52. Struct Rust has option enum for such cases. Every Option is either Some and contains a value, or None, and does not.
53. Struct can have methods
54. Traits Traits are Rust’s sole notion of interface Traits must be brought into scope with use It has to be looked up in a vtable at run time ~ DYNAMIC DISPATCH
55. Generics talk is now generic with a Talk trait bound The compiler generates specialized code ~ Static Dispatch Zero -Cost Abstraction !!
56. Developing & Shipping Rust ext. in Python modules
57. How does this work? • A Foreign Function Interface is a way for a programming language to define functions and enable a different (foreign) programming language to call those functions • Based on platform-dependant C application binary interface • Rust produces efficient C bindings
58. Rust is ideal for creating and using FFI
59. Rust FFI capabilities • Building the project • Create the extensions
60. Interfacing & building the python module • build.py • setup.py
61. Other libs • setuptools-rust https://pypi.org/project/setuptools-rust/ • milksnake https://github.com/getsentry/milksnake
62. Rust-Cpython
63. Rust-Cpython : Integration • Import the dependencies • Write the rust equivalent logic • Export the rust function
64. Rust-Cpython : Benchmarks
66. Tools • rustup: the Rust toolchain installer • rustfmt : the Rust code formatting tool • rust-clippy : Rust code linter • Cargo : package manager • ...
67. 67 Community Photo credit: David McSpadden https://www.flickr.com/photos/familyclan/15535822737/
68. 68 RFC Process
70. Summary
71. When should you use Rust? • Rewrite modules of your code that are computationally heavy (Complex math optimizations etc) • For accessing the hardware(ex :- IOT applications, device drivers, GPU’s etc) • Implementing advance concurrency paradigms • Python to Rust calls are non-trivial, so be wise!
72. What does Rust have to offer? • Modern C/C++ replacement • Memory Safety without Garbage Collection • No runtime needed • Rich Typesystem • Strong Functional Programming influence
73. Feel Awesome
74. Thanks! @dvigneshwer dvigneshwer@gmail.com https://tinyurl.com/PyConUS2018