Bridging Python and C & Introduction to ctypes

Some solutions to work with C code/library in Python, and introduction to ctypes, a pure-Python solution to make use of compiled C shared objects. Also some tips and tricks when working with ctypes (and C in general).

Notes:

Slide 6
========

So we’re not going to talk about SWIG, SIP, Shiboken, etc. Just Python and C.

Slide 9
========

First: You don’t have the code (yet).
Second: You have the code, but you need to compile it.
Third: You have a compiled binary, and you want to use it in Python.

Slide 12
========

Suitable if: You need to write some C anyway, and is knowledgable enough with CPython internals.

Slide 13
========

Suitable if: You’re dealing with very little or non existing C code. Writing Cython is a lot easier than C, so you get speed in the binding with much hassle.

Slide 17
========

You can see how similar ctypes code is to straight C.

Slide 21
========

Also there are some types from stdint.h and others, e.g. c_int32 (mapped to int32_t). The naming rules are pretty straightforward.

Slide 23
========

Another example. (Python equivalent in the next slide.)

Notice three topics:

void * and non-NULL-terminated char * printing.
int * “out” parameter, pass by pointer, and address-of (&) operator.
int declaration.

Slide 24
========

Rough Python equivalent of the previous C code. Look at how each set of syntaxes is translated.

Note: C type variables are always initialised in ctypes, even if you don’t pass a value! c_int() is equivalent to c_int(0).

Slide 30
========

This is “Hello World” with ctypes. The point here is, function annotations are just hints, and are optional.

They are recommended, but you should use them as a tool, not an obligation.

Slide 31 & 32
========

You might get into trouble if you follow the C declaration blindly. Remember, C is weakly-typed.

Slide 35
========

Remember, unions and structures are all Python classes, and can have normal Python attributes! Use them!

Slide 41
========

This last one helps me very often. I find it extremely helpful to write some simple C programs first, then translate them into Python with ctypes.

Transcript

1. Bridging Python and C Also: Introduction to ctypes 1
2. Me • Call me TP • Follow @uranusjr • https://uranusjr.com 2
4. 4 http://macdown.uranusjr.com
5. 5 www. .com
6. Notice: C++ is not C 6
7. 7 Python C
8. 8 Python C Binding
9. Why? • I want to write C code!! • I have some C code • I want to access a C library 9
10. 10 Python .so Binding
11. To compile, or not to compile. 11
12. Python C API 12 Python .so Binding C source compile compile C source
13. Cython 13 Python .so Binding C source Cython compile compile
14. No Compiling 14 Python .so Binding
15. https://en.wikipedia.org/wiki/Foreign_function_interface A foreign function interface (FFI) is a mechanism by which a program written in one programming language can call routines or make use of services written in another. 15
16. FFI • libffi: Use C functions in other languages • Bundled in CPython • ctypes: Expose libffi to Python • Available in many implementations 16
17. Show me the code! http://bit.ly/ejdb-example 17
18. Basics 18
19. Find the library 19 import  ctypes   from  ctypes.util  import  find_library   path  =  find_library('ejdb')   _  =  ctypes.cdll.LoadLibrary(path)
20. 20 ejdbversion  =  _.ejdbversion   ejdbversion.argtypes  =  []   ejdbversion.restype  =  ctypes.c_char_p const  char  *ejdbversion(); C Python
21. 21 ctypes C Python c_bool _Bool bool c_char char 1-char binary c_wchar wchar_t 1-char text c_int int int c_uint unsigned int int c_double double float c_char_p char * binary or None c_wchar_p wchar_t * text or None c_void_p void * int POINTER(type) type * (Not used directly)
22. 22 ejdbversion  =  _.ejdbversion   ejdbversion.argtypes  =  []   ejdbversion.restype  =  c_char_p   version  =  ejdbversion()   print(version)   #  b'1.2.6'   #  Notice  the  auto  type  conversion.   #  BTW  this  is  Python  3.
23. 23 void  *bson_data2(bson  *bs,  int  *len);   int  size  =  0;   void  *data  =  bson_data2(bs,  &size);   if  (data)          printf("%.*s\n",  size,  (char  *)data);
24. 24 bson_data2.argtypes  =  [BSONP,  POINTER(c_int)]   bson_data2.restype  =  c_void_p   size  =  c_int(0)   data  =  bson_data2(bs,  byref(size))   if  data:        #  0  for  NULL.          print(string_at(data,  size.value))
25. 25 typedef  struct  {          char  *cname;          int  cnamesz;          TCTDB  *tdb;          EJDB  *jb;          void  *mmtx;   }  EJCOLL; class  EJCOLL(Structure):          _fields_  =  [                  ('cname',  c_char_p),                  ('cnamesz',  c_int),                  ('tdb',  TCTDBREF),                  ('jb',  EJDBREF),                  ('mmtx',  c_void_p),          ] C Python
26. 26 typedef  union  {          char  bytes[12];          int  ints[3];   }  bson_oid_t; class  BSONOID(Union):        _fields_  =  [                  ('bytes',  c_char  *  12),                  ('ints',  c_int  *  3),          ] C Python
27. 27 enum  bson_binary_subtype_t  {          BSON_BIN_BINARY  =  0,          BSON_BIN_FUNC  =  1,          BSON_BIN_BINARY_OLD  =  2,          BSON_BIN_UUID  =  3,          BSON_BIN_MD5  =  5,          BSON_BIN_USER  =  128   }; BSON_BIN_BINARY  =  0   BSON_BIN_FUNC  =  1   BSON_BIN_BINARY_OLD  =  2   BSON_BIN_UUID  =  3   BSON_BIN_MD5  =  5   BSON_BIN_USER  =  128 C Python
28. 28 ejdbgetcoll.argtypes  =  [EJDBREF,  c_char_p]   ejdbgetcoll.restype  =  EJCOLLREF   coll_r  =  ejdbgetcoll(jb,  b'people')   coll  =  coll_r.contents        #  Dereferencing.   print(coll.cname)                  #  Field  access.
29. More Tricks 29
30. 30 import  ctypes   import  ctypes.util   #  POSIX.   path  =  ctypes.util.find_library('libc')   libc  =  ctypes.cdll.LoadLibrary(path)   #  Windows.   #  libc  =  ctypes.cdll.LoadLibrary('msvcrt')   _  =  lib.puts(b'Hello  World!')
31. 31 #  const  char  *bson_data2(   #          const  bson  *bs,  int  *size);   bson_data2  =  _.bson_data2   bson_data2.argtypes  =  [BSONREF,  POINTER(c_int)]   bson_data2.restype  =  c_char_p   bs  =  ...   size  =  c_int()   print(bson_data2(bs,  byref(size)))   #  b'\x0e'              <-­‐  Oops
32. 32 #  const  char  *bson_data2(   #          const  bson  *bs,  int  *size);   bson_data2  =  _.bson_data2   bson_data2.argtypes  =  [BSONREF,  POINTER(c_int)]   bson_data2.restype  =  c_void_p   bs  =  ...   size  =  c_int()   data_ptr  =  bson_data2(bs,  byref(size))   print(string_at(data_ptr,  size.value))   #  b'\x0e\x00\x00\x00\x10A\x00\x01\x00\x00\x00\x00'
33. When in Rome • Array and pointer manipulation • pointer and addressof • Type-casting is your weapon • cast • Opaque pointers • c_void_p 33
34. 34 data  =  b'...'   #  NO!   bson_append_binary(          bs,  b'photo',  BSON_BIN_BINARY,          data,  len(data))   #  YES.   buf  =  create_string_buffer(data,  len(data))   bson_append_binary(          bs,  b'photo',  BSON_BIN_BINARY,          buf,  len(data))
35. 35 class  BSONOID(Union):          _fields_  =  [...]          def  __str__(self):                  #  ...          @classmethod          def  from_string(cls,  s):                  oid  =  cls()                  bson_oid_from_string(byref(oid),  s)                  return  oid
36. 36 int  main(int,  const  char  *[])   {          EJDB  *jb  =  ejdbnew();          //  Do  things...          ejdbdel(jb);          return  0;   } def  main():          jb  =  EJDB()          #  Garbage  collection  FTW! C Python
37. 37 class  EJDB:          def  __init__(self):                  self._wrapped  =  ejdbnew()          #  When  do  we  call  `ejdbdel()`?  
38. 38 class  EJDB:          def  __init__(self):                  self._wrapped  =  ejdbnew()          def  __del__(self):                  ejdbdel(self._wrapped) NO!
39. bit.ly/python-ctypes-raii 39
40. Warnings • Using ctypes ≈ writing C • It will crash if things go wrong • I mean Segmentation Fault 40
41. Advices • Test your modules • Protect things with wrappers • Python 3 faulthandler • When in doubt, write some C 41
42. Anyway, should I use it? 42
43. Yes • Beyond CPython • No need of a C compiler • Don’t need the C source 43
44. Maybe Not • Need to compile dependencies anyway • C is fucking annoying • Don’t have a C API? 44
45. Choose the Right Tool 45
46. To be continued…? 46