Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Wrapping C libraries into Python modules

Wrapping C libraries into Python modules

This presentation aims to show how libraries like Python PIL, ScyPy, NumPy do interfaces with C libraries. Python allows this communication natively. We’ll see the concepts, the Python compiler resources, code API and some code examples.

Gustavo Pantuza

April 02, 2021
Tweet

More Decks by Gustavo Pantuza

Other Decks in Programming

Transcript

  1. ❖ Why a C module ❖ How to create a

    module ❖ How to install a module ❖ Code API and built in properties ❖ How to use a module ❖ Experiments ❖ Analysis Agenda
  2. Why a C module? ❖ Heavy computation (numpy) ❖ Increase

    performance (Scipy) ❖ Don't want to recreate mature a stable C libraries (PIL) ❖ Get control over the memory allocation (embedded systems) ❖ Embed Python in some application (GoDot game engine)
  3. Python.h ❖ Objects ❖ Funcions ❖ Types ❖ Macros https://docs.python.org/3/c-api/intro.html

    ❖ Includes common headers {stdio,string,errno,limits,assert,stdlib}.h
  4. Python.h ❖ Functions and Variables have prefixes ➢ Py -

    Used by modules ➢ _Py - Internal Python interpreter use https://docs.python.org/3/c-api/intro.html ❖ Avoid using those prefixes ➢ Confuses the Interpreter ➢ Legibility ➢ Conflicts in future versions of Python
  5. How to create a module #include <Python.h> static PyObject *

    hello (PyObject *self) { return Py_BuildValue("s", "Hello Pythonist"); }
  6. Objects ❖ Retorn PyObject * ➢ Reference to an opaque

    object ➢ Every Python object pointer can best cast to a PyObject ❖ Most python objects are allocated in the Heap ➢ Pymalloc != malloc static PyObject * https://docs.python.org/3/c-api/module.html
  7. Defining Functions static PyObject* my_function_with_no_args (PyObject *self); static PyObject* my_function

    (PyObject *self, PyObject *args); static PyObject* my_function_with_keywords (PyObject *self, PyObject *args, PyObject *kwargs);
  8. Objects PyObject* Py_BuildValue (const char *format, ...) return Py_BuildValue("s", "Hello

    Pythonist"); format C type c char f float i int d double format C type u Py_UNICODE* O PyObject* [...] ... {...} ...
  9. How to create a module #include <Python.h> static PyObject *

    hello (PyObject *self) { return Py_BuildValue("s", "Hello Pythonist"); } static char docstring[] = "Hello world module for Python written in C";
  10. How to create a module #include <Python.h> static PyObject *

    hello (PyObject *self) { return Py_BuildValue("s", "Hello Pythonista"); } static char docstring[] = "Hello world module for Python written in C"; static PyMethodDef module_methods[] = { {"hello", (PyCFunction) hello, METH_NOARGS, docstring}, {NULL, NULL, 0, NULL} };
  11. Function entry type struct PyMethodDef { char *ml_name; /* Module

    name called by Python */ PyCFunction ml_meth; /* Function reference */ int ml_flags; /* Function parameters type */ char *ml_doc; /* Function description */ }; https://docs.python.org/3/c-api/structures.html#c.PyMethodDef
  12. How to create a module #include <Python.h> static PyObject *

    hello (PyObject *self) { return Py_BuildValue("s", "Hello Pythonist"); } static char module_docstring[] = "Hello world module for Python written in C"; static PyMethodDef module_methods[] = { {"hello", (PyCFunction) hello, METH_NOARGS, module_docstring}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initmodule(void) { Py_InitModule("module", module_methods); }
  13. Initializing a module PyObject* Py_InitModule(char *name, PyMethodDef *methods) PyObject* Py_InitModule3(

    char *name, PyMethodDef *methods, char *doc) PyObject* Py_InitModule4(char *name, PyMethodDef *methods, char *doc, PyObject *self, int apiver)
  14. Installation from distutils.core import setup from distutils.core import Extension setup(

    name='module', version='1.0', ext_modules=[Extension('module', ['hello.c'])] ) setup.py
  15. Extension Extension('module', ['hello.c']) ❖ Describes C/C++ extensions in setup.py ❖

    Only a Python class with attributes ❖ When Extensions object exist, calls build_ext function cpython/Lib/distutils/extension.py
  16. Extension class build_ext(Command): objects = self.compiler.compile( sources, output_dir=self.build_temp, macros=macros, include_dirs=ext.include_dirs,

    debug=self.debug, extra_postargs=extra_args, depends=ext.depends ) cpython/Lib/distutils/command/build_ext.py
  17. Installation $> python setup.py install running install running build running

    build_ext building 'module' extension creating build creating build/temp.linux-x86_64-2.7 {gcc compiles the module} running install_lib copying build/lib.linux-x86_64-2.7/module.so -> /path/site-packages running install_egg_info Removing path/module-1.0-py2.7.egg-info Writing /path/site-packages/module-1.0-py2.7.egg-info $> python setup.py --help
  18. Installation {gcc compiles the module} gcc -pthread -fno-strict-aliasing -fmessage-length=0 -grecord-gcc-switches

    -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -g -DNDEBUG -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -g -DOPENSSL_LOAD_CONF -fwrapv -fPIC -I/usr/include/python2.7 -c hello.c -o build/temp.linux-x86_64-2.7/hello.o creating build/lib.linux-x86_64-2.7 gcc -pthread -shared build/temp.linux-x86_64-2.7/hello.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/module.so $> man gcc
  19. Shared Object (.so) ❖ Linkage happens during program load ❖

    Changes in the module does not implies changes in the Python code
  20. How to use the module $> ipython In [1]: from

    module import hello In [2]: hello() Out[2]: 'Hello Pythonist' In [3]: help(hello)
  21. Dynamic load In [1]: from sys import modules In [2]:

    modules['module'] ----------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-2-f0b257567ce0> in <module>() ----> 1 modules['module'] KeyError: 'module' In [3]: from module import hello In [4]: modules['module'] Out[4]: <module 'module' from '/path/to/lib/python2.7/site-packages/module.so'>
  22. Dynamic linkage $> ldd /path/to/lib/python2.7/site-packages/module.so linux-vdso.so.1 (0x00007ffe0bd94000) libpython2.7.so.1.0 =>/usr/lib64/libpython2.7.so.1.0(0x00007f72ca71b000) libpthread.so.0

    => /lib64/libpthread.so.0 (0x00007f72ca4fe000) libc.so.6 => /lib64/libc.so.6 (0x00007f72ca15f000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f72c9f5b000) libutil.so.1 => /lib64/libutil.so.1 (0x00007f72c9d58000) libm.so.6 => /lib64/libm.so.6 (0x00007f72c9a52000) /lib64/ld-linux-x86-64.so.2 (0x000055df9d3ae000) $> man ldd
  23. Symbol table $> objdump -t /path/to/lib/python2.7/site-packages/module.so | grep text 0000000000000690

    l d .text 0000000000000000 .text 0000000000000690 l F .text 0000000000000000 deregister_tm_clones 00000000000006d0 l F .text 0000000000000000 register_tm_clones 0000000000000720 l F .text 0000000000000000 __do_global_dtors_aux 0000000000000760 l F .text 0000000000000000 frame_dummy 0000000000000790 l F .text 0000000000000015 hello 00000000000007b0 g F .text 000000000000001d initmodule $> man objdump
  24. Experiments ❖ 1000 lists with 1000 random integers ranging from

    1 to 1000 ❖ Compute times of execution of the module and of standard library ❖ Plot histograms with mean times
  25. Algorithm for(int i = 0; i < seq_size; i++) {

    ... for(int j = 0; j < seq_size; j++) { ... if(_PyLong_AsInt(item_i) == _PyLong_AsInt(item_j)) { count++; } } ... } Module
  26. Algorithm for(int i = 0; i < seq_size; i++) {

    ... for(int j = 0; j < seq_size; j++) { ... if(_PyLong_AsInt(item_i) == _PyLong_AsInt(item_j)) { count++; } } ... } O(n²) Module
  27. Algorithm Utiliza uma Heap Queue ou Priority Queue 0 1

    2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 https://github.com/python/cpython/blob/master/Lib/heapq.py#L33 Standard library
  28. Algorithm Utiliza uma Heap Queue ou Priority Queue 0 1

    2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 https://github.com/python/cpython/blob/master/Lib/heapq.py#L33 Standard library O(n log n)