etcd: Next Steps with the Cornerstone of Distributed Systems

Demos on the GitHub repo: https://github.com/philips/2016-LinuxCon-etcd-Next-Steps-with-the-Cornerstone-of-Distributed-Systems

Transcript

1. Next Steps with the Cornerstone of Distributed Systems Brandon Philips @brandonphilips | brandon.philips@coreos.com Demo Code: http://goo.gl/R6Og3Y Free Stickers @ Podium!
2. Why build etcd? MOTIVATION
3. Uncoordinated Upgrades
4. ... ... ... ... ... ... Unavailable Uncoordinated Upgrades
5. Motivation CoreOS cluster reboot lock - Decrement a semaphore key atomically - Reboot and wait... - After reboot increment the semaphore key
6. 3 CoreOS updates coordination
7. CoreOS updates coordination 3
8. ... CoreOS updates coordination 2
9. ... ... ... CoreOS updates coordination 0
10. ... ... ... CoreOS updates coordination 0
11. ... ... CoreOS updates coordination 0
12. ... ... CoreOS updates coordination 1
13. ... ... ... CoreOS updates coordination 0
14. CoreOS updates coordination
15. Store Application Configuration config
16. config Start / Restart Start / Restart Store Application Configuration
17. config Update Store Application Configuration
18. config Unavailable Store Application Configuration
19. Requirements Strong Consistency - mutual exclusive at any time for locking purpose Highly Available - resilient to single points of failure & network partitions Watchable - push configuration updates to application
20. Requirements CAP Theorem - We want CP - We want something like Paxos
21. Common Problem Google - “All” infrastructure relies on Paxos GFS Paxos Big Table Spanner CFS Chubby
22. Common Problem Amazon - Replicated log for ec2 Microsoft - Boxwood for storage infrastructure Hadoop - ZooKeeper is the heart of the ecosystem
23. #GIFEE and Cloud Native Solution COMMON PROBLEM
24. 10,000 Stars on Github 250 contributors CoreOS, Google, Red Hat, EMC, Cisco, Huawei, Baidu, Alibaba...
25. Kubernetes, Cloud Foundry Diego, Canal, many others THE HEART OF CLOUD NATIVE
26. Fully Replicated, Highly Available, Consistent ETCD KEY VALUE STORE
27. History of etcd ◦ 2013.8 Alpha release (v0.x) ◦ 2015.2 Stable release (v2.0+) ◦ stable replication engine (new Raft implementation) ◦ stable v2 API ◦ 2016.6 (v3.0+) ◦ efficient, powerful API ◦ highly scalable backend
28. Production Ready long running failure injection tests no known data loss issues no known inconsistency issues
29. How does etcd work? • Raft consensus algorithm ◦ Using a replicated log to model a state machine ◦ "In Search of an Understandable Consensus Algorithm" (Ongaro, 2014) • Three key concepts ◦ Leaders ◦ Elections ◦ Terms
30. How does etcd work? • The cluster elects a leader for every given term • All log appends (--> state machine changes) are decided by that leader and propagated to followers • Much much more at http://raft.github.io/
31. How does etcd work? • Written in Go, statically linked • /bin/etcd ◦ daemon ◦ 2379 (client requests/HTTP + JSON API) ◦ 2380 (peer-to-peer/HTTP + protobuf) • /bin/etcdctl ◦ command line client • net/http, encoding/json, golang/protobuf, ...
32. Clustering ETCD BASICS
40. etcd cluster failure tolerance
41. locksmith ETCD APPS
42. locksmith • cluster wide reboot lock ◦ "semaphore for reboots" • CoreOS updates happen automatically ◦ prevent all the machines restarting at once...
43. Cluster Wide Reboot Lock server1 server2 server3
44. Cluster Wide Reboot Lock server1 server2 server3 needs reboot
45. Cluster Wide Reboot Lock • Need to reboot? Decrement the semaphore key (atomically) with etcd • manager.Reboot() and wait... • After reboot, increment the semaphore key in etcd (atomically)
46. Cluster Wide Reboot Lock server1 server2 server3 needs reboot Sem=1
47. Cluster Wide Reboot Lock server1 server2 server3 needs reboot Sem=1 Lock()
48. Cluster Wide Reboot Lock server1 server2 server3 needs reboot Sem=0 Lock() acquired
49. Cluster Wide Reboot Lock server1 server2 server3 Sem=0 Reboot()
50. Cluster Wide Reboot Lock server1 server2 server3 Sem=0 Rebooting...
51. Cluster Wide Reboot Lock server1 server2 server3 Sem=0 Rebooting... needs reboot
52. Cluster Wide Reboot Lock server1 server2 server3 Sem=0 Rebooting... needs reboot Lock()
53. Cluster Wide Reboot Lock server1 server2 server3 Sem=0 Rebooting... needs reboot Lock() fail
54. Cluster Wide Reboot Lock server1 server2 server3 Sem=0 Rebooting... needs reboot wait...
55. Cluster Wide Reboot Lock server1 server2 server3 Rebooted! Sem=0 needs reboot wait...
56. needs reboot Cluster Wide Reboot Lock server1 server2 server3 Sem=0 Unlock() wait...
57. needs reboot Cluster Wide Reboot Lock server1 server2 server3 Sem=1 Unlock() wait...
58. needs reboot Cluster Wide Reboot Lock server1 server2 Sem=1 Lock() server3
59. needs reboot Cluster Wide Reboot Lock server1 server2 Sem=0 Lock() acquired server3
60. Reboot() Cluster Wide Reboot Lock server1 server2 Sem=0 server3
61. Canal (formerly flannel) ETCD APPS
62. 192.168.1.10 192.168.1.40
63. 192.168.1.10 192.168.1.40
64. 10.0.0.3 10.0.0.8 10.0.1.10 10.0.1.20 192.168.1.10 192.168.1.40
66. 192.168.1.10 192.168.1.40 10.0.0.0/24 10.0.1.0/24
67. routes to 192.168.1.40 192.168.1.10 192.168.1.40 10.0.0.0/24 10.0.1.0/24
68. 192.168.1.40 10.0.1.0/24 192.168.1.10 routes to 192.168.1.10
69. Canal Today • virtual (overlay) network for constrained envs • BGP for physical environments • Connection policies • Built for Kubernetes useful in other systems with CNI
70. skydns ETCD APPS
71. skydns • Service discovery and DNS server • backed by etcd for all configuration/records
72. vulcand ETCD APPS
73. vulcand • "programmatic, extendable proxy for microservices" • HTTP load balancer • etcd for storing all configuration
75. confd ETCD APPS
76. confd • simple configuration templating • for "dumb" applications • watch etcd for changes, render templates with new values, reload applications
77. Kubernetes ETCD APPS
78. Simple cluster operations Secure and Simple API Friendly operational tools
79. Guides & Tools coreos.com/kubernetes
80. etcd benchmarks SCALING
81. Performance: 1k keys
82. Performance: etcd2 - 600K keys Snapshot caused performance degradation
83. Performance: etcd2 - 600K keys Snapshot triggered elections
84. Performance: Zookeeper default
85. Performance: Zookeeper default Snapshot triggered election
86. Performance: Zookeeper snapshot disabled GC
87. Performance: etcd3 vs zk snapshot disabled
88. Performance: etcd3 vs zk snapshot disabled
89. Memory: 512MB data - 2M 256B keys
90. Reliability ETCD DESIGN
91. Reliability • 99% at small scale is easy ◦ Failure is infrequent and human manageable • 99% at large scale is not enough ◦ Not manageable by humans • 99.99% at large scale ◦ Reliable systems at bottom layer
92. WAL, Snapshots, Testing HOW DO WE ACHIEVE RELIABILITY
93. Write Ahead Log • Append only ◦ Simple is good • Rolling CRC protected ◦ Storage & OSes can be unreliable
94. Snapshots • Torturing DBs for Fun and Profit (OSDI2014) ◦ The simpler database is safer ◦ LMDB was the winner • Boltdb an append only B+Tree ◦ A simpler LMDB written in Go
95. Testing Clusters Failure • Inject failures into running clusters • White box runtime checking ◦ Hash state of the system ◦ Progress of the system
96. Testing Cluster Health with Failures Issue lock operations across cluster Ensure the correctness of client library
97. dash.etcd.io TESTING CLUSTER
98. etcd/raft Reliability • Designed for testability and flexibility • Used by large scale db systems and others ◦ Cockroachdb, TiKV, Dgraph
99. Do one thing ETCD VS OTHERS
100. Only do the One Thing ETCD VS OTHERS
101. Do it really well ETCD VS OTHERS
102. Proxy, Caching, Watch Coalescing, Secondary Index FUTURE WORK
103. github.com/coreos/etcd GET INVOLVED
105. Linux
107. Runs on any platform Consistency across clouds Trivial dev, test, & prod Faster time to market
108. Training San Francisco September 13 & 14 New York City September 27 & 28 San Francisco October 11 & 12 New York City October 25 & 26 Seattle November 10 & 11 https://coreos.com/training
109. Build, Store and Distribute your Containers quay.io
110. Thank you! Brandon Philips @brandonphilips | brandon@coreos.com | coreos.com We’re hiring in all departments! Email: careers@coreos.com Positions: coreos.com/ careers