Upgrade to Pro — share decks privately, control downloads, hide ads and more …

RustとPython

Hideo Hattori
September 17, 2018

 RustとPython

PyCon JP 2018 9/17 talk session
Python Extension Module written in Rust.

Hideo Hattori

September 17, 2018
Tweet

More Decks by Hideo Hattori

Other Decks in Programming

Transcript

  1. Rust Rust と と Python Python PyCon PyCon J JP

    P 2018 2018 2018.09.17 Hideo Hattori 2018.09.17 Hideo Hattori 1
  2. 6

  3. Hello World Hello World fn main() { let name =

    "Rust"; println!("Hello {}!", name); } 7
  4. rustup rustup Installer like venv , pyenv , ... stable

    and nightly rustup install nightly 10
  5. cargo cargo & & package manager & package repository like

    setup.py , pip & PyPI.org crates.io crates.io 11
  6. cargo cargo - Start Project - Start Project $ cargo

    new myproject $ tree myproject myproject ├── Cargo.toml └── src └── main.rs 12
  7. Ownership and Borrowing Ownership and Borrowing fn main() { let

    a = vec![1, 2, 3]; let b = a; println!("a={:?}", a); println!("b={:?}", b); } 15
  8. $ rustc ownership.rs error[E0382]: use of moved value: `a` -->

    ownership.rs:5:24 | 3 | let b = a; | - value moved here 4 | 5 | println!("a={:?}", a); | ^ value used here after move | = 16
  9. fn main() { let a = vec![1, 2, 3]; let

    b = &a; println!("a={:?}", a); println!("b={:?}", b); } 17
  10. Concurrency with Concurrency with std::thread std::thread use std::thread; fn main()

    { let mut threads = vec![]; for i in 0..4 { threads.push(thread::spawn(|| { println!("this is thread number {}", i); })); } for thread in threads { let _ = thread.join(); } } 18
  11. error[E0373]: closure may outlive the current function, but it borrows

    `i`, which is owned --> concurrency.rs:7:36 | 7 | threads.push(thread::spawn(|| { | ^^ may outlive borrowed value `i` 8 | println!("this is thread number {}", i); | - `i` is borrowed here help: to force the closure to take ownership of `i` (and any other referenced variables), u | 7 | threads.push(thread::spawn(move || { | ^^^^^^^ error: aborting due to previous error For more information about this error, try `rustc --explain E0373`. 19
  12. use std::thread; fn main() { let mut threads = vec![];

    for i in 0..4 { threads.push(thread::spawn(move || { println!("this is thread number {}", i); })); } for thread in threads { let _ = thread.join(); } } 20
  13. Blazingly Fast Blazingly Fast minimal runtime zero-cost abstractions std::thread (rayon,

    crossbeam, ...) The computer language Benchmarks Game - Rust vs C++ g++ 21
  14. Command-Line Tools Command-Line Tools - like grep tool - fuzzy

    finder in Rust - Replacement for ls written in Rust - blazingly fast CLOC program ripgrep skim exa tokei 24
  15. Motivation Motivation more fast ... C Extension more safety write

    Extension modules use Rust crates in Python world 29
  16. Start Python Extension Project Start Python Extension Project $ rustup

    default nightly $ cargo new --lib example $ cd example 33
  17. Cargo.toml Cargo.toml [package] name = "example" version = "0.1.0" authors

    = ["Hideo Hattori"] [lib] name = "example" crate-type = ["cdylib"] [dependencies] [dependencies.pyo3] version = "0.4" features = ["extension-module"] 35
  18. setup.py setup.py from setuptools import setup from setuptools_rust import Binding,

    RustExtension setup(name='example', version='0.1', rust_extensions=[ RustExtension('example', 'Cargo.toml', binding=Binding.PyO3)], zip_safe=False) 36
  19. src/lib.rs src/lib.rs #![feature(specialization)] extern crate pyo3; use pyo3::prelude::*; use pyo3::py::modinit

    as pymodinit; #[pymodinit(example)] fn init_example(py: Python, m: &PyModule) -> PyResult<()> { #[pyfn(m, "hello")] fn hello(name: &str) -> PyResult<()> { println!("Hello {}", name); Ok(()) } Ok(()) } 37
  20. Let s use!! Let s use!! $ python >>> import

    example >>> example.hello() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: Required argument ('name') (pos 1) not found >>> example.hello('Rust') Hello Rust 39
  21. Benchmark Benchmark ## Ranking real uap 0.0813 (100.0) ******************** fast-woothee

    1.0989 ( 7.4) * woothee 3.2219 ( 2.5) * uap(non-cache) 64.3114 ( 0.1) ## Matrix real [01] [02] [03] [04] [01] uap 0.0813 100.0 1352.1 3964.4 79132.1 [02] fast-woothee 1.0989 7.4 100.0 293.2 5852.5 [03] woothee 3.2219 2.5 34.1 100.0 1996.1 [04] uap(non-cache) 64.3114 0.1 1.7 5.0 100.0 https://gist.github.com/hhatto/c951a981e8a3ee4d1bbcf96cb93d5f5e 43
  22. Implement Python Class Implement Python Class use pyo3::prelude::*; #[pyclass] struct

    Writer { _wtr: csv::Writer<BufWriter<fs::File>>, token: PyToken, } #[pymodinit(_fcsv)] fn init_mod(_py: Python, m: &PyModule) -> PyResult<()> { m.add_class::<Writer>()?; Ok(()) } 45
  23. Implement Python Class Implement Python Class #[pymethods] impl Writer {

    #[new] fn __new__(obj: &PyRawObject, path: String, kwargs: Option<&PyDict>) -> PyResult<()> { : obj.init(|t| Writer { _wtr: wtr, token: t, }) } fn writerow(&mut self, py: Python, arg: PyObject) -> PyResult<()> { : Ok(()) } } 46
  24. Implement Python Iterator Implement Python Iterator #[pyproto] impl PyIterProtocol for

    Reader { fn __iter__(&mut self) -> PyResult<pyobject> { Ok(self.into()) } fn __next__(&mut self) -> PyResult<option<pyobject>> { let mut record = csv::StringRecord::new(); match self._rdr.read_record(&mut record) { Ok(true) => { let py = self.py(); Ok(Some(record.iter().collect::<vec<&str>>().to_object(py))) } _ => Err(exc::StopIteration::new("stop")), } } } 47
  25. use this use this # example.py import fcsv for row

    in fcsv.reader('hello.csv'): print(row) 48
  26. Benchmark of Benchmark of Writer Writer ## Ranking real fcsv.writerows

    0.0519 (100.0) ******************** fcsv.writerow 0.0677 ( 76.7) *************** unicodecsv.writerows 1.3566 ( 3.8) * std.writerows 1.4500 ( 3.6) * ## Matrix real [01] [02] [03] [04] [01] fcsv.writerows 0.0519 100.0 130.4 2614.4 2794.3 [02] fcsv.writerow 0.0677 76.7 100.0 2005.3 2143.4 [03] unicodecsv.writerows 1.3566 3.8 5.0 100.0 106.9 [04] std.writerows 1.4500 3.6 4.7 93.6 100.0 https://gist.github.com/hhatto/a101f904e516c0ea8519cc5a50fcf586 49
  27. Benchmark of Benchmark of Reader Reader ## Ranking real fcsv.reader

    0.5444 (100.0) ******************** unicodecsv.reader 0.6474 ( 84.1) ***************** std.reader 0.7892 ( 69.0) ************** ## Matrix real [01] [02] [03] [01] fcsv.reader 0.5444 100.0 118.9 145.0 [02] unicodecsv.reader 0.6474 84.1 100.0 121.9 [03] std.reader 0.7892 69.0 82.0 100.0 https://gist.github.com/hhatto/a101f904e516c0ea8519cc5a50fcf586 50
  28. fn _basename(path_str: &str, is_bytes: bool) -> PyObject { let i

    = match memchr::memrchr(MAIN_SEPARATOR as u8, path_str.as_bytes()) { Some(v) => v + 1, None => 0, }; : } 53
  29. Benchmark Benchmark methodname % real[p,r] user[p,r] sys[p,r] n abspath 45.53%

    10.15s, 5.53s 6.86s, 2.81s 3.25s, 2.69s 100000 basename 53.52% 0.71s, 0.33s 0.70s, 0.33s 0.00s, 0.00s 100000 dirname 57.43% 1.02s, 0.43s 1.01s, 0.43s 0.00s, 0.00s 100000 isabs 56.55% 0.59s, 0.25s 0.59s, 0.25s 0.00s, 0.00s 100000 islink 0.25% 3.78s, 3.77s 0.01s, 0.01s 0.01s, 0.00s 50 exists 0.25% 3.78s, 3.77s 0.01s, 0.01s 0.01s, 0.00s 50 lexists 0.70% 3.77s, 3.74s 0.01s, 0.01s 0.01s, 0.00s 50 54
  30. Benchmark (2) Benchmark (2) methodname % real[p,r] user[p,r] sys[p,r] n

    split 53.85% 1.17s, 0.54s 1.17s, 0.54s 0.00s, 0.00s 100000 splitext 62.02% 1.22s, 0.46s 1.21s, 0.46s 0.00s, 0.00s 100000 relpath 52.97% 0.02s, 0.01s 0.01s, 0.01s 0.01s, 0.00s 50 normpath 57.27% 2.02s, 0.86s 2.01s, 0.86s 0.00s, 0.00s 100000 realpath 1.08% 13.39s, 13.25s 0.05s, 0.02s 0.02s, 0.03s 50 join 23.01% 0.24s, 0.19s 0.24s, 0.18s 0.00s, 0.00s 100000 expanduser 67.45% 1.50s, 0.49s 1.49s, 0.48s 0.00s, 0.00s 100000 expandvars 61.37% 1.21s, 0.47s 1.19s, 0.47s 0.00s, 0.00s 100000 55