Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Josh Triplett - Networking without an OS

Josh Triplett - Networking without an OS

Many Python modules, such as socket and select, wrap low-level functionality written in C and provided by the OS. But what if you don't have an OS, and don't want any C code? We implemented client and server networking in Python itself, for a bare-metal environment running without an OS.

Our socket and select implementations support Python HTTP server and client modules, which we'll demo live.

https://us.pycon.org/2016/schedule/presentation/1945/

PyCon 2016

May 29, 2016
Tweet

More Decks by PyCon 2016

Other Decks in Programming

Transcript

  1. PyCon 2015 BIOS Implementation Test Suite (BITS) Python in GRUB

    and EFI, without an OS Explore and test hardware and firmware
  2. BITS Capabilities Interactive Python interpreter (with line editing and tab

    completion) Direct access to hardware and physical memory
  3. BITS Capabilities Interactive Python interpreter (with line editing and tab

    completion) Direct access to hardware and physical memory Python can call EFI firmware protocols via ctypes
  4. BITS Capabilities Interactive Python interpreter (with line editing and tab

    completion) Direct access to hardware and physical memory Python can call EFI firmware protocols via ctypes Most of the Python standard library
  5. Why networking in firmware? Send scripts or test data into

    the machine Read test data or logs from the machine
  6. Why networking in firmware? Send scripts or test data into

    the machine Read test data or logs from the machine Avoid relying on a writable filesystem
  7. Why networking in firmware? Send scripts or test data into

    the machine Read test data or logs from the machine Avoid relying on a writable filesystem Speed up edit/compile/boot/run cycle
  8. Bridging EFI networking and Python sockets Could call EFI network

    protocols directly Want compatibility with existing Python networking code Python modules import socket and select socket (Python) imports _socket (C)
  9. Sockets overview “Berkeley” sockets Standard on UNIX/POSIX systems, and on

    Windows via WinSock Focusing exclusively on TCP/IP connections
  10. Creating a socket int s = socket(AF_INET, SOCK_STREAM, 0); AF_INET

    - IP SOCK_STREAM - TCP Can use s as either client or server socket
  11. select - waiting for activity Pass sets of file descriptors

    to monitor for reading and for writing
  12. select - waiting for activity Pass sets of file descriptors

    to monitor for reading and for writing And for exceptions, but ignoring that here
  13. select - waiting for activity Pass sets of file descriptors

    to monitor for reading and for writing And for exceptions, but ignoring that here Pass a timeout
  14. select - waiting for activity Pass sets of file descriptors

    to monitor for reading and for writing And for exceptions, but ignoring that here Pass a timeout Waits for a connected socket to have data to recv or send
  15. select - waiting for activity Pass sets of file descriptors

    to monitor for reading and for writing And for exceptions, but ignoring that here Pass a timeout Waits for a connected socket to have data to recv or send Waits for a listening socket to have a connection
  16. select - waiting for activity Pass sets of file descriptors

    to monitor for reading and for writing And for exceptions, but ignoring that here Pass a timeout Waits for a connected socket to have data to recv or send Waits for a listening socket to have a connection Core of the main loop in most network servers
  17. select - waiting for activity Pass sets of file descriptors

    to monitor for reading and for writing And for exceptions, but ignoring that here Pass a timeout Waits for a connected socket to have data to recv or send Waits for a listening socket to have a connection Core of the main loop in most network servers Many OS-specific replacements for scalability and performance
  18. Python bindings import socket, select s = socket.socket() - defaults

    to TCP/IP s.connect s.bind, s.listen, s.accept
  19. Python bindings import socket, select s = socket.socket() - defaults

    to TCP/IP s.connect s.bind, s.listen, s.accept s.sendall, s.recv
  20. Python bindings import socket, select s = socket.socket() - defaults

    to TCP/IP s.connect s.bind, s.listen, s.accept s.sendall, s.recv rl,wl,xl = select.select([s],[],[]) if s in rl:
  21. CPython implementation socketmodule.c and selectmodule.c Extensive dependencies on POSIX and

    on C sockets API Would have to implement those APIs in C Handle C arguments, addresses, buffer management
  22. CPython implementation socketmodule.c and selectmodule.c Extensive dependencies on POSIX and

    on C sockets API Would have to implement those APIs in C Handle C arguments, addresses, buffer management Would have to call EFI protocols from C
  23. CPython implementation socketmodule.c and selectmodule.c Extensive dependencies on POSIX and

    on C sockets API Would have to implement those APIs in C Handle C arguments, addresses, buffer management Would have to call EFI protocols from C Or, have many callbacks from C to Python
  24. BITS implementation C helper for safe asynchronous event handling Otherwise

    entirely Python Python makes all EFI protocol calls
  25. Behind the scenes efi.system_table is a data structure efi.system_table.ConOut is

    a pointer to a protocol .contents dereferences a ctypes pointer
  26. Behind the scenes efi.system_table is a data structure efi.system_table.ConOut is

    a pointer to a protocol .contents dereferences a ctypes pointer Most EFI calls expect a “this” pointer
  27. Behind the scenes efi.system_table is a data structure efi.system_table.ConOut is

    a pointer to a protocol .contents dereferences a ctypes pointer Most EFI calls expect a “this” pointer ctypes converts "Hello world\r\n" to a Unicode string
  28. Behind the scenes efi.system_table is a data structure efi.system_table.ConOut is

    a pointer to a protocol .contents dereferences a ctypes pointer Most EFI calls expect a “this” pointer ctypes converts "Hello world\r\n" to a Unicode string ctypes returns the error code from EFI
  29. Resource management EFI uses manual memory management Python uses garbage

    collection Python GC doesn’t know about references from EFI
  30. Resource management EFI uses manual memory management Python uses garbage

    collection Python GC doesn’t know about references from EFI Must keep Python object alive as long as EFI references it
  31. Resource management EFI uses manual memory management Python uses garbage

    collection Python GC doesn’t know about references from EFI Must keep Python object alive as long as EFI references it Must explicitly free EFI resources when no longer referenced from Python
  32. EFI networking protocols EFI_IP4_CONFIG2_PROTOCOL Read current IP configuration, or start

    IP configuration EFI_TCP4_SERVICE_BINDING_PROTOCOL Create a new EFI_TCP4_PROTOCOL, like socket()
  33. EFI networking protocols EFI_IP4_CONFIG2_PROTOCOL Read current IP configuration, or start

    IP configuration EFI_TCP4_SERVICE_BINDING_PROTOCOL Create a new EFI_TCP4_PROTOCOL, like socket() EFI_TCP4_PROTOCOL
  34. EFI networking protocols EFI_IP4_CONFIG2_PROTOCOL Read current IP configuration, or start

    IP configuration EFI_TCP4_SERVICE_BINDING_PROTOCOL Create a new EFI_TCP4_PROTOCOL, like socket() EFI_TCP4_PROTOCOL EFI socket API: Configure, Connect, Accept, Transmit, Receive, Close
  35. EFI networking protocols EFI_IP4_CONFIG2_PROTOCOL Read current IP configuration, or start

    IP configuration EFI_TCP4_SERVICE_BINDING_PROTOCOL Create a new EFI_TCP4_PROTOCOL, like socket() EFI_TCP4_PROTOCOL EFI socket API: Configure, Connect, Accept, Transmit, Receive, Close
  36. EFI networking protocols EFI_IP4_CONFIG2_PROTOCOL Read current IP configuration, or start

    IP configuration EFI_TCP4_SERVICE_BINDING_PROTOCOL Create a new EFI_TCP4_PROTOCOL, like socket() EFI_TCP4_PROTOCOL EFI socket API: Configure, Connect, Accept, Transmit, Receive, Close Glossing over quirks, bugs, error handling, workarounds, and compatibility with older versions
  37. Impedance mismatch: select versus Receive/Accept select checks for pending data

    or connections Does not read data or accept connection
  38. Impedance mismatch: select versus Receive/Accept select checks for pending data

    or connections Does not read data or accept connection EFI can only check for data by calling Receive with a valid buffer
  39. Impedance mismatch: select versus Receive/Accept select checks for pending data

    or connections Does not read data or accept connection EFI can only check for data by calling Receive with a valid buffer Solution: buffer received data, call Receive when buffer empty
  40. Impedance mismatch: select versus Receive/Accept select checks for pending data

    or connections Does not read data or accept connection EFI can only check for data by calling Receive with a valid buffer Solution: buffer received data, call Receive when buffer empty Likewise for Accept
  41. Impedance mismatch: select versus Receive/Accept select checks for pending data

    or connections Does not read data or accept connection EFI can only check for data by calling Receive with a valid buffer Solution: buffer received data, call Receive when buffer empty Likewise for Accept Similar to the implementation of sockets in an OS kernel
  42. Impedance mismatch: Poll EFI_TCP4_PROTOCOL not updated directly from low-level interrupts

    Data processed infrequently, even with asynchronous call running
  43. Impedance mismatch: Poll EFI_TCP4_PROTOCOL not updated directly from low-level interrupts

    Data processed infrequently, even with asynchronous call running Caller expected to call Poll periodically if waiting
  44. Impedance mismatch: Poll EFI_TCP4_PROTOCOL not updated directly from low-level interrupts

    Data processed infrequently, even with asynchronous call running Caller expected to call Poll periodically if waiting Improves performance by orders of magnitude
  45. Impedance mismatch: Poll EFI_TCP4_PROTOCOL not updated directly from low-level interrupts

    Data processed infrequently, even with asynchronous call running Caller expected to call Poll periodically if waiting Improves performance by orders of magnitude Solution: call Poll inside helpers for select
  46. Impedance mismatch: asynchronous callbacks All EFI socket calls asynchronous Calls

    take a “completion token” with EFI_EVENT to signal when done
  47. Impedance mismatch: asynchronous callbacks All EFI socket calls asynchronous Calls

    take a “completion token” with EFI_EVENT to signal when done For sockets, EFI_EVENT must have a callback function
  48. Impedance mismatch: asynchronous callbacks All EFI socket calls asynchronous Calls

    take a “completion token” with EFI_EVENT to signal when done For sockets, EFI_EVENT must have a callback function Need to handle callback safely from Python
  49. Python concurrency model Python expects bytecode ops and C calls

    to run to completion Global Interpreter Lock (GIL)
  50. Python concurrency model Python expects bytecode ops and C calls

    to run to completion Global Interpreter Lock (GIL) Data may have inconsistent state when callback occurs
  51. Python concurrency model Python expects bytecode ops and C calls

    to run to completion Global Interpreter Lock (GIL) Data may have inconsistent state when callback occurs Almost all CPython functions prohibited
  52. Python concurrency model Python expects bytecode ops and C calls

    to run to completion Global Interpreter Lock (GIL) Data may have inconsistent state when callback occurs Almost all CPython functions prohibited Same problem arises with Ctrl-C and signals
  53. Py_AddPendingCall Register a callback (with context) Python calls it at

    the next safe point Can call arbitrary CPython functions from the callback
  54. Handling events C module provides event callback function pointer Python

    code creates EFI_EVENT with C callback C callback invokes universal Python callback
  55. Handling events C module provides event callback function pointer Python

    code creates EFI_EVENT with C callback C callback invokes universal Python callback Python callback dispatches to event-specific callback via dict
  56. Handling events C module provides event callback function pointer Python

    code creates EFI_EVENT with C callback C callback invokes universal Python callback Python callback dispatches to event-specific callback via dict dict keeps Python objects live while EFI_EVENT references them
  57. Implementing select Takes read and write lists of sockets Or

    file descriptors; map to sockets via dict
  58. Implementing select Takes read and write lists of sockets Or

    file descriptors; map to sockets via dict Loop over sockets until timeout expires
  59. Implementing select Takes read and write lists of sockets Or

    file descriptors; map to sockets via dict Loop over sockets until timeout expires Call _read_ready or _write_ready
  60. Implementing select Takes read and write lists of sockets Or

    file descriptors; map to sockets via dict Loop over sockets until timeout expires Call _read_ready or _write_ready _read_ready checks queue, calls Receive or Accept
  61. Implementing select Takes read and write lists of sockets Or

    file descriptors; map to sockets via dict Loop over sockets until timeout expires Call _read_ready or _write_ready _read_ready checks queue, calls Receive or Accept If already running from previous call, call Poll
  62. Implementing select Takes read and write lists of sockets Or

    file descriptors; map to sockets via dict Loop over sockets until timeout expires Call _read_ready or _write_ready _read_ready checks queue, calls Receive or Accept If already running from previous call, call Poll Handle connection closure to provide EOF from recv
  63. Implementing select Takes read and write lists of sockets Or

    file descriptors; map to sockets via dict Loop over sockets until timeout expires Call _read_ready or _write_ready _read_ready checks queue, calls Receive or Accept If already running from previous call, call Poll Handle connection closure to provide EOF from recv Queues data or error when callback called
  64. Implementing select Takes read and write lists of sockets Or

    file descriptors; map to sockets via dict Loop over sockets until timeout expires Call _read_ready or _write_ready _read_ready checks queue, calls Receive or Accept If already running from previous call, call Poll Handle connection closure to provide EOF from recv Queues data or error when callback called _write_ready returns true if connected
  65. Implementing select Takes read and write lists of sockets Or

    file descriptors; map to sockets via dict Loop over sockets until timeout expires Call _read_ready or _write_ready _read_ready checks queue, calls Receive or Accept If already running from previous call, call Poll Handle connection closure to provide EOF from recv Queues data or error when callback called _write_ready returns true if connected Always calls Poll if connected
  66. class socket __init__: Create EFI_TCP4_PROTOCOL from binding protocol __del__/close: Cancel

    all outstanding callbacks connect: Call Configure and Connect
  67. class socket __init__: Create EFI_TCP4_PROTOCOL from binding protocol __del__/close: Cancel

    all outstanding callbacks connect: Call Configure and Connect recv: Return data from queue
  68. class socket __init__: Create EFI_TCP4_PROTOCOL from binding protocol __del__/close: Cancel

    all outstanding callbacks connect: Call Configure and Connect recv: Return data from queue Spin on _read_ready if queue empty; this starts a Receive
  69. class socket __init__: Create EFI_TCP4_PROTOCOL from binding protocol __del__/close: Cancel

    all outstanding callbacks connect: Call Configure and Connect recv: Return data from queue Spin on _read_ready if queue empty; this starts a Receive sendall: Call Transmit; save status for
  70. class socket __init__: Create EFI_TCP4_PROTOCOL from binding protocol __del__/close: Cancel

    all outstanding callbacks connect: Call Configure and Connect recv: Return data from queue Spin on _read_ready if queue empty; this starts a Receive sendall: Call Transmit; save status for bind: Save provided address and port for later calls
  71. class socket __init__: Create EFI_TCP4_PROTOCOL from binding protocol __del__/close: Cancel

    all outstanding callbacks connect: Call Configure and Connect recv: Return data from queue Spin on _read_ready if queue empty; this starts a Receive sendall: Call Transmit; save status for bind: Save provided address and port for later calls listen: Call Configure for listening socket
  72. class socket __init__: Create EFI_TCP4_PROTOCOL from binding protocol __del__/close: Cancel

    all outstanding callbacks connect: Call Configure and Connect recv: Return data from queue Spin on _read_ready if queue empty; this starts a Receive sendall: Call Transmit; save status for bind: Save provided address and port for later calls listen: Call Configure for listening socket accept: Return queued connection if any
  73. class socket __init__: Create EFI_TCP4_PROTOCOL from binding protocol __del__/close: Cancel

    all outstanding callbacks connect: Call Configure and Connect recv: Return data from queue Spin on _read_ready if queue empty; this starts a Receive sendall: Call Transmit; save status for bind: Save provided address and port for later calls listen: Call Configure for listening socket accept: Return queued connection if any Spin on _read_ready if queue empty; this starts an Accept
  74. Try it out yourself BITS: https://biosbits.org/ QEMU/KVM Client works automatically

    For server, use hostfwd option to forward ports inside
  75. Try it out yourself BITS: https://biosbits.org/ QEMU/KVM Client works automatically

    For server, use hostfwd option to forward ports inside OVMF: Open Virtual Machine Firmware, EFI for QEMU
  76. Try it out yourself BITS: https://biosbits.org/ QEMU/KVM Client works automatically

    For server, use hostfwd option to forward ports inside OVMF: Open Virtual Machine Firmware, EFI for QEMU Or try it on physical hardware
  77. Try it out yourself BITS: https://biosbits.org/ QEMU/KVM Client works automatically

    For server, use hostfwd option to forward ports inside OVMF: Open Virtual Machine Firmware, EFI for QEMU Or try it on physical hardware Check BIOS settings to enable EFI network stack
  78. Try it out yourself BITS: https://biosbits.org/ QEMU/KVM Client works automatically

    For server, use hostfwd option to forward ports inside OVMF: Open Virtual Machine Firmware, EFI for QEMU Or try it on physical hardware Check BIOS settings to enable EFI network stack Use wired Ethernet