MPTCP: when one path is not enough

Multipath TCP (MPTCP) started to appear in the Linux kernel in the v5.6 version, released in 2020. Before a rewrite to fit Netdev maintainers requirements, this complex protocol was supported in a fork, with millions of users. A lot of changes have been introduced since then.

This session will cover different aspects about MPTCP and its development. First, what is MPTCP, its use-cases, and he different components. Then how easy it is to use MPTCP today on Linux. There will be some words about the current status, what has been built around, what are the planned next steps, and eventually some discussions about what would be great to have later on.

Matthieu BAERTS

Transcript

1. Multipath TCP: When one path is not enough Matthieu Baerts (NGI0) • Kernel Recipes 26 • 23.09.2026
2. Plan • What is MPTCP? • How to use MPTCP on Linux today? • A long road to get MPTCP upstream • Current status & next steps 2
3. What is MPTCP? 3
4. Multipath TCP (MPTCP – RFC-8684) Exchange data for a single connection over different paths, simultaneously ⇒ Bring new capabilities Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 4
5. Multipath TCP (MPTCP – RFC-8684) Exchange data for a single connection over different paths, simultaneously ⇒ Best network selection Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 5
6. Multipath TCP (MPTCP – RFC-8684) Exchange data for a single connection over different paths, simultaneously ⇒ Seamless handover Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 6
7. Multipath TCP (MPTCP – RFC-8684) Exchange data for a single connection over different paths, simultaneously ⇒ Seamless handover Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 7
8. Multipath TCP (MPTCP – RFC-8684) Exchange data for a single connection over different paths, simultaneously ⇒ Seamless handover Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 8
9. Multipath TCP (MPTCP – RFC-8684) Exchange data for a single connection over different paths, simultaneously ⇒ Seamless handover Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 9
10. Multipath TCP (MPTCP – RFC-8684) Exchange data for a single connection over different paths, simultaneously ⇒ Seamless handover Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 10
11. Multipath TCP (MPTCP – RFC-8684) Exchange data for a single connection over different paths, simultaneously ⇒ Network aggregation Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 11
12. Multipath TCP (MPTCP – RFC-8684) Use cases: • Mobile devices: ◦ “walk-out” scenario • Home Gateways: ◦ combine networks, e.g. DSL + cellular or low Earth orbit satellites • Data centres: ◦ fast recoveries, select best paths, aggregation 12
13. Concept: Subflow and Fallback • Subflow: Each path of an MPTCP connection. A subflow is a regular TCP connection carrying extra options in the TCP header. • Fallback: If the other host does not support MPTCP, or in case of a middlebox intercepting TCP connections, there will be fallbacks to TCP. The MPTCP protocol is complex to cope with various middleboxes. 13
14. Concept: Path Manager Typically, different needs for the clients and servers: Which path to create / remove? Which addresses to announce? ? ? Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 14
15. Concept: Packet Scheduler On which available path packets will be sent? Reinject to another path? ? ? ? ? Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 15
16. How to use MPTCP on Linux today? 16
17. MPTCP on Linux • The complexity is handled by the kernel • Opt-in (with possibilities to force apps to use MPTCP): socket(AF_INET(6), SOCK_STREAM, IPPROTO_MPTCP); • Minimal behaviour changes for apps compared to TCP • Path-Manager configured via userspace, e.g. manually: ip mptcp endpoint add <IP address> dev <interface> <type> 17
18. MPTCP on Linux • Some tools can automatically set up the MPTCP endpoints, e.g. NetworkManager and mptcpd • Some apps natively support MPTCP, e.g. cURL, HAProxy, Apache Server, Lighttpd, NGINX, systemd sockets, Go apps (enabled by default on the server side), etc. Check mptcp.dev/apps.html • Possibilities to force using MPTCP, e.g. mptcpize (LD_PRELOAD), mptcpify (eBPF/BCC), GODEBUG=multipathtcp=1, SystemTAP, etc. Check mptcp.dev/setup.html 18
19. MPTCP on Linux • Most Linux distributions have MPTCP support enabled, and mptcpd packages, including specialised ones like OpenWrt, RPiOS, HAOS. • Debugging tools supports MPTCP, e.g ss -M, ip mptcp, nstat, tcpdump, ptcpdump, WireShark • Server: an MPTCP listen socket will create a TCP socket if the client didn’t request MPTCP: good to have MPTCP enabled by default! 19
20. Demo 20
21. A long road to get MPTCP upstream 21
22. A bit of history • MPTCP started to appear upstream in v5.6 (Jan. 2020) • But started as a prototype in 2008/2009 󰎐🍟🍻 • MPTCPv0 RFC published in January 2013 • Used in production on servers having millions of clients 22
23. Maintaining a fork • It is easy to fork … ◦ but you will pay for it! • The Linux kernel is big, complex, very active • The fork was quite invasive: ◦ 21k lines in total ◦ 2.5k in TCP / IP / … with many “if (mptcp)” ◦ With duplicated functions adapted for MPTCP case 23
24. The fork couldn’t be accepted upstream • Linux TCP is highly optimized • New implementation cannot affect existing TCP stack: ◦ No performance regressions ◦ Maintainable and possibility to disable it ◦ Can be extended via the userspace • Cannot take the initial fork: ◦ Built to support experiments and rapid changes but not generic enough ◦ Special purpose implementation of MPTCP 24
25. Upstreaming: requirements • Rewriting (almost) everything from scratch • A different concept: introduction of “MPTCP socket” • Minimal differences in TCP code thanks to TCP ULP (+ SKB ext) • Carefully review and detail modifications in TCP stack • APIs to extend the path-manager and the scheduler • And … A lot of work! 25
26. Current status & next steps 26
27. Current status: general • Supports most of the protocol features: multiple subflows, announce addresses and priority, fast close, reset reasons, etc. • Info from MIB counters (nstat), INET_DIAG interface (ss) and MPTCP_INFO / MPTCP_FULL_INFO socket options • CI validating patches sent on the ML (virtme-ng, GitHub Actions, etc.) Patches CI Results 27
28. Current & Future: socket options • Current: Supports most common socket options: SO, IP, IPV6, TCP: ◦ Imitating TCP’s behaviour ◦ But adapted to MPTCP case: ◦ ▪ Inherit the behaviour on all subflows, including future ones? e.g. KeepAlive ▪ Or only on the first one? e.g. TCP FastOpen Still possible to change the per-subflow behaviour with eBPF • Future: Support more uncommon ones, e.g. (MP)TCP_REPAIR 28
29. Current status: path managers • In-kernel: Global settings per network namespace: e.g. via ip mptcp ◦ Set endpoints: IP addresses, flags (client-server sides, backup, fullmesh) ◦ Set limits: max subflows to establish or accept ◦ Monitor connections: created, established, closed, announced, etc. • Userspace: Per connection: e.g. via mptcpd ◦ Reacting to “events” by sending “commands” ⇒ Communications: using Netlink 29
30. Current status: PM: Deployment behind a Load-Balancer • Initial path: with a random server behind a stateless load-balancer Anycast: each server is reusing the same public IP Load-Balancer Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 30
31. Current status: PM: Deployment behind a Load-Balancer • Additional paths: how a stateless load-balancer can pick the same server? Anycast: each server is reusing the same public IP Load-Balancer Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 31
32. Current status: PM: Deployment behind a Load-Balancer • Additional paths: how a stateless load-balancer can pick the same server? ⇒ Servers: tell client not to use initial address and announce a new one. Anycast: each server is reusing the same public IP Load-Balancer Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain 32
33. Current status: PM: Deployment behind a Load-Balancer • Additional paths: how a stateless load-balancer can pick the same server? ⇒ Servers: tell client not to use initial address and announce a new one. Load-Balancer Smartphone and WiFi icons by Blurred203 and Antü Plasma under CC-by-sa, others from Tango project, public domain Anycast: each MPTCPis server supported reusing the by CDNs? same public IP 🤞 33
34. Future: Path Manager & Packet Scheduler extension • BPF extension (struct_ops): • ◦ To adapt to specific use-cases, at a lower cost ◦ Including quite a bit of clean-up in the current code! Support more corner cases, e.g. paths from “too heterogeneous” environments • Better support Linux phones use-cases. 34
35. Future: in-kernel MPTCP usage • To be used by other subsystems from within the kernel: ◦ BPF sockmap ◦ kTLS ◦ nVME over MPTCP ◦ etc. ⇒ more and more features “outside MPTCP” (harder to review) 35
36. Questions Discussions • 🌐 mptcp.dev • ✉ mptcp@lists.linux.dev • 💬 IRC: #mptcp on Libera.chat • 🎙 Online Meetings • 📰 blog.mptcp.dev • @mptcp@social.kernel.org – @matttbe@fosstodon.org 36
37. Backup 37
38. Workflow CI Patches • ML / Patchwork • Patches in Git • Build + tests Results • Logs / Artefacts • Failures / Instabilities • Publications 38