A Python and a Gopher walk into a bar - Embedding Python in Go.

Success stories about rewriting Python applications in Go are not big news anymore. The pros and cons are well known, best practices are in place, and the standard library is there to help. But what if you want to keep some of your Python code? When we chose to port the Datadog Agent to Go, we needed to maintain support for our existing library of plugins written in Python. During the talk we will share lessons learned from our experiences with cgo, the GIL and the quest for performance as we bridge multiple languages in a single application.

Transcript

1. A Python and a Gopher walk into a bar - Embedding Python in Go. Massimiliano Pippi
2. • Software Engineer ◦ C++, Python and Go • OSS fan and contributor • 1+ year at Datadog, working on Agent and Integrations Hi, I’m Massi!
3. • SaaS based infrastructure, app and logs monitoring • Open Source Agent • Time series data (metrics and events) • Intelligent Alerting and Insightful Dashboards • Trillions of data points processed per day About Datadog
4. Monitor everything
5. Meet the Datadog Agent Agent check • Written in Python • Open Source, https://github.com/DataDog/dd-agent
6. The anatomy of a check import psutil from checks import AgentCheck class SystemSwap(AgentCheck): def check(self, instance): swap_mem = psutil.swap_memory() self.rate('system.swap.swapped_in', swap_mem.sin) self.rate('system.swap.swapped_out', swap_mem.sout) self.gauge(‘system.swap.total’, swap_mem.total) self.gauge(‘system.swap.used’, swap_mem.used)
7. The way to Go - Our Goals • Make it smaller faster stronger • Keep Python as an extension language ◦ ~75 Python checks currently part of the official package ◦ Python is the right tool to implement most of them ◦ Undetermined number of custom checks in the wild ◦ Update a check without recompiling the Agent
8. Embedding for the win • Python can be embedded: you keep an interpreter in memory and make it run Python code at will • CPython provides a C API to allow embedding • Cgo enables the creation of Go packages that call C code.
9. Demo time! Let’s run a Python module from a go application • https://github.com/masci/golab17/tree/master/01
10. The dreadful GIL • GIL stands for global interpreter lock • Prevents multiple threads from executing Python code at once • Yes, even if those threads are run on a multi-core processor
11. The dreadful GIL • Embedding CPython means embedding the GIL • The GIL knows about threads created from Python code... • ...but the GIL can’t do its job if we run Python from separate Go threads!
12. Demo time! Run Python in different goroutines and watch the world burn • https://github.com/masci/golab17/tree/master/02 • Rule of thumb: any time you use Python in some piece of code that could be executed in a separate thread, lock the GIL!
13. The revenge of the dreadful GIL • We lock and unlock the GIL in goroutines, not threads • The GIL protects a specific thread state, we cannot lock/unlock it from different threads! • But the Go scheduler might pause and resume goroutines in different threads
14. Demo time! See what happens when the Go scheduler relocates our Pythonic goroutines (spoiler alert: it’ll crash your software) • https://github.com/masci/golab17/tree/master/03
15. Beyond embedding: extending! • Once you have an embedded interpreter, you can extend Python capabilities with Go code • This involves a little bit of C so no demo here • Still very easy to achieve, Python scripts import a module that actually lives in memory and points to Go instructions
16. Extending Python: the Go code //export MyGoFunc func MyGoFunc() { fmt.Println(“Hello, World from Go!”) }
17. Extending Python: the C code static PyMethodDef MyMethods[] = { {"my_func", (PyCFunction)myGoFunc, METH_VARARGS, "YAY!"}, {NULL, NULL} // guards }; PyObject *m = Py_InitModule("my_module", MyMethods);
18. Extending Python: the Python code # WARNING! This only works on the embedded interpreter import my_module my_module.my_func() # prints “Hello, World from Go!”
19. Lessons learned: the good • Embedded Python plays nice with Go concurrency model • From/To Python overhead is negligible ◦ BenchmarkCallPyFunc 300000 3606 ns/op • Extending Python is a very powerful tool ◦ Expose functions and data to the Python world
20. Lessons learned: the bad • The GIL prevents Python parallel execution ◦ This was expected • The GIL feels the effects of the Go scheduler ◦ Honestly, didn’t see this coming • Using multiple interpreters doesn’t help ◦ They share a unique GIL
21. Lessons learned: the ugly • You must carry on some C code, how much depending on the use case • You will likely carry on some Python code too, to offer base classes and utilities to external modules running in embedded mode
22. What we have now • Datadog Agent 6.0.0-beta1 ◦ Embedded CPython 2.7.13 ◦ Linux, OSX and Windows ◦ We now run checks concurrently, knowing that many of them will wait for each other... ◦ ...even if some of them were ported to Go, so we also have some parallelism
23. Thanks for listening! • Try Datadog at https://www.datadoghq.com • Find our OSS on https://github.com/DataDog • Our tech blog is http://engineering.datadoghq.com Check out the new Agent on Github! https://github.com/DataDog/datadog-agent/
24. WE’RE HIRING! NYC, PARIS, Remote
25. Embedding: an example // #cgo pkg-config: python-2.7 // #include <Python.h> import "C" C.Py_Initialize() cmd := C.CString(“print ‘Hello, World!’”) // cmd must be freed! C.PyRun_SimpleString(cmd) C.Py_Finalize() • we still do go build and that’s it.
26. Embedding: a better example • Use go-python to eliminate boilerplate ◦ https://github.com/sbinet/go-python import "github.com/sbinet/go-python" python.Initialize() python.PyRun_SimpleString(“print ‘Hello, World!’”) python.Finalize()