K9P: Kubernetes as 9P Files

K9P, a virtual file system, exposes the state of a Kubernetes cluster as files. Our terminals have been optimized over the last 40 years towards working with files, kubectl not so much. K9P allows us to carry the mantra of "everything is a file" to the distributed computing extreme.

K9P allows you to integrate Kubernetes resources into an existing workflow, or create new ones. Scale a Deployment by writing to a file. Locate failing Pods with grep. Update configuration in ConfigMaps with sed.

Transcript

1. K9P: Kubernetes as 9P Files Terin Stock 20 November 2019
2. Terin Stock @terinjokes Kubernetes Engineer, Cloudflare
3. What if… • Orchestrate compute across multiple hosts • Orchestrate storage across multiple hosts • Compute separate from users • Applications running in a sandbox • Service discovery
4. What if… it's 1992? • Linux 1.0: +2 years • Process Containers (later, cgroups): +14 years • LMCTFY and Docker: +21 years • Kubernetes: +23 years
5. What if… it's 1992? Plan 9! Glenda, the Plan 9 Bunny Drawn by Renée French Дмитрий Царьков (CC-SA-4.0)
7. K9P
8. What is K9P? • Kubernetes exposed as a 9P filesystem • Goal to provide easy interaction with Kubernetes ◦ Use familiar tools with clusters ◦ Integrate into file-based workflows
9. 9P…? • Networked filesystem connecting Plan 9 systems and components ◦ windows ◦ network connections ◦ processes ◦ files
10. 9P RPC Messages • version • auth • attach • flush • walk • create • open • read • write • clunk • remove • stat • wstat • error
11. 9P in the modern era • DataKit and VPNKit by Docker • VirtFS in QEMU • WSL by Microsoft • Crostini and gVisor by Google
12. Linux Support • Linux has native support when compiled with CONFIG_NET_9P. # mount -t 9p -o trans=tcp,port=1564 198.51.100.1 /mnt/k8s
13. macOS and Windows • Both operating systems have native implementations, not (currently) easily exposed to users. ◦ macOS: 9P for VirtFS landed in 10.14.4 ◦ Windows: 9P is used for WSL • In the meantime, there are FUSE implementations
14. What can I use K9P for?
15. Everything is a file!
16. Everything is a file! $ ls -lp ./namespaces/kube-system/deployments/coredns -rw-rw-rw- 1 4294967294 4294967294 5297 Jul 3 17:03 data.yaml -rw-rw-rw- 1 4294967294 4294967294 0 Jul 3 17:03 scale -r--r--r-- 1 4294967294 4294967294 0 Jul 3 17:03 status
17. Pipelines and Integrations
18. Pipelines and Integrations Scale by writing to a file $ echo 5 > ./namespaces/kube-system/deployments/coredns/scale
19. Pipelines and Integrations Search for non-running Pods $ grep -L Running ./namespaces/kube-system/pods/*/status ./namespaces/kube-system/pods/coredns-5fc75767bb-7f9gx/status
20. Pipelines and Integrations Tail Pod logs $ tail ./namespaces/kube-system/pods/coredns-*/logs/* 2019-07-04T00:20:08.602Z [ERROR] plugin/errors: 2 www.example.com. A: read udp 198.51.100.34:58230->198.51.100.2:53: i/o timeout
21. Pipelines and Integrations Modify cluster resources $ sed -i 's/1.6.4/1.6.5/' ./namespaces/kube-system/deployments/coredns/data.yaml
22. Controllers and Operators
23. Controllers and Operators • No language libraries required ◦ Everything is a file. ◦ Most programming languages have great filesystem support.
24. Controllers and Operators • No networking code. • No network access. ◦ 9P client can run at a different level of network access
25. Controllers and Operators • Very testable ◦ Inputs and outputs are just files.
26. Demo
27. Implementing K9P
28. Implementing K9P • Server implemented in Go ◦ 9P server exposed over TCP ◦ Uses Kubernetes's client-go • Each mount is associated with a new connection to Kubernetes Drawn by Renée French
29. 9P and Go • Many available implantations: p9p, neinp, styx, p9 (gVisor) ◦ Each has a different level of implementation, abstraction, and paradigms ◦ K9P uses p9p from Docker.
30. Files • Kubernetes resources are mapped to directory layouts ◦ Cluster-scope resources are mapped to /cluster ◦ Namespace-scope resources are mapped inside of /namespaces • Each supported resource type has a handler implementation. ◦ Would like to derive implementations where possible.
31. Challenges
32. Polling • 9P has no subscription mechanism ◦ No way for the server to push new information to clients ◦ Required to open directories and files to observe changes
33. Managing State • This is my first virtual filesystem, continue to be unsure about the best way to implement and manage the filesystem state. ◦ Currently, fids are mapped to resources and a lot of state is generated on demand. Performance isn't often great.
34. Versions • The Kubernetes API uses resources versions to prevent modifications based on old versions of resources. • User's view of a file may fall out of sync with the server's resource version. ◦ Future improvement: associate a qid with the resource's version, know when a conflict might happen.
35. Conflicts and Errors • Inflexibility to report conflicts and issues ◦ Can only return errors to client requests ◦ Client often attempt to turn stringy errors into standard system errors.
36. Authentication • 9P defers authentication to filesystem implementations The 9P protocol does not prescribe an authentication method. Instead, client and server communicate by reading from and writing to a special file. • Future improvement to setup an authentication protocol to exchange user or session tokens.
37. Discussion
38. 9P and more • Only scratched the surface of what 9P can do. ◦ Go out and experiment ◦ Build your own filesystems ▪ KubeCon talk videos as a filesystem? It could be the next YouTube! • NBD implements network-backed block devices. • CUSE allows character devices to be implemented in userspace.
39. Wide Area SHell • Puppet's wash is a similar idea ◦ Shell to interact with remote and cloud-native infrastructure with unix-like tools ◦ Some support as a FUSE filesystem
40. 9P's Future? • Strings for UIDs and GIDs didn't catch on elsewhere. • Lacks support for some expected things: ◦ symlinks can be iffy ◦ xattrs ◦ sockets • Protocol can be versioned and extended, but who's coordinating?
41. Use K9P at your own risk! It might delete something important.
42. More K9P • Learn more + code: https://terin.ee/u/k9p • Find me in the hallway track for questions.
43. Back Matter
44. Colophon The main body text, including headers, were set in Cambo by Argentinian foundry Huerta Tipográfica based on the style of traditional Khmer type. Monospace text was set in Anonymous Pro by Minnesota font designer Mark Simonson. It was inspired by mid-90s freeware Macintosh font Anonymous 9. The one quote was set in Open Sans, a humanist font designed by Coloradan Steve Matteson at the Monotype foundry.
45. Colophon Glenda, the Plan 9 Bunny; the Go Gopher; and the unnamed dachshund were designed by Renée French. Glenda, the Plan 9 Bunny is copyright Alcatel-Lucent S.A., used with permission. The Go Gopher is licensed under the Creative Commons 3.0 Attributions license. The dachshund was shamelessly lifted from the cover of The Practice of Programming, copyright Alcatel-Lucent S.A.