Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Real-World Kubernetes Deployments @ OSCON 2016
Search
Brandon Philips
May 16, 2016
Programming
720
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Real-World Kubernetes Deployments @ OSCON 2016
Brandon Philips
May 16, 2016
Transcript
116 slides
Real World Kubernetes Deployments failure domains, upgrades, high-availability @coreoslinux @brandonphilips
Follow Along Instructions http://bit.ly/1XeUbMW Stickers Upfront Decorate your laptop, dog,
kid, phone.
Brandon Philips CTO, CoreOS github.com/philips
None
Build, Store and Distribute your Containers quay.io
Linux
Secure the Internet MISSION
Separate Apps from OS STRATEGY
Make Servers Consistent STRATEGY
Tolerate Machine Failures STRATEGY
Make Servers Easy to Upgrade STRATEGY
Simplify Application Upgrades STRATEGY
None
None
None
None
None
Application Packaging 1
Abstract away app from the OS OS App
None
None
Linux at Scale 2
Patches to the OS and kernel are hard Retest after
updates No automation SECURITY Dependency breakage Uptime risk APPLICATION
None
Auto-updating browsers fixed security We got HTML5 at the same
time
Clustering 3
Operations Paradise Easy scale out Painless app upgrades Tolerant of
machine failure
App Req/sec: 6,000 App Healthy: True
App Req/sec: 6,000 App Healthy: True
App Req/sec: 7,000 App Healthy: True
App Req/sec: 8,000 App Healthy: True
App Req/sec: 7,000 App Healthy: True
App Req/sec: 6,000 App Healthy: True
App Req/sec: 8,000 App Healthy: True
App Req/sec: 7,000 App Healthy: True
App Req/sec: 8,000 App Healthy: True
App Req/sec: 8,000 App Healthy: True
3 Application packaging Clustering Linux at scale
3 Application packaging Clustering Linux at scale
Follow Along Instructions https://github.com/philips/repositories 2016-OSCON-containers-at-scale
CoreOS+Kubernetes vagrant, aws, bare metal, etc coreos.com/kubernetes/docs/latest/
kubernetes architecture in practice
worker kubelet worker kubelet worker kubelet scheduler & API worker
kubelet w ku t worker kubelet
worker kubelet worker kubelet scheduler & API
worker & API works on 1 node too
kube-aws Initial Cluster Setup
worker kubelet worker kubelet controller scheduler, etcd & API
Demo Boot up a Cluster
Demo Run an App
Demo Understand the Network
Domains Let's Talk About Failure
Failure domains are regions or components of the infrastructure which
contain a potential for failure.
These regions can be physical or logical boundaries, and each
has its own risks and challenges to architect for.
Failure Feud - Machine Failure - Network/Disks/RAM/Processor/Power Supply - Rack
Failure - Network/Power - Data Center Failure - Network/Power/Fire/Semi-trucks - Internet Failure - Network/Political/Natural
Failure Analysis Kid Celebrating
None
Kid Hitting His Eye Failure Analysis - Failure is caused
by human error - Celebration continues; eye unnecessary - Kid has two eyes can continue seeing - Brain elects new eye automatically
Primary Datastore etcd operations
/etc distributed hence, the name...
a clustered key-value store GET and SET operations
a building block for higher order systems primitives for building
reliable distributed systems
Demo play.etcd.io
None
None
None
None
None
None
None
Failure Analysis etcd
worker kubelet worker kubelet scheduler & API
kube-aws high availability in cloud
scheduler & API EBS { ASG
etcd protects against - Machine Failure - Replication, automatic leader
election - Flakey Disk Failure - CRC checksums on WAL files - Network Failure - Timeouts and linearized state machine
etcd does not protect against - Denial of Service -
Future work on proxies - Lying etcd Peers - We do a ton of functional testing a hedge - Buggy or Broken Clients - Client deleting all keys requires restore from backup
Demo etcd restore backup
1 2 3 4 { Log
1 2 3 4 Entries
1 2 3 4 Indexes
Kubernetes Control API Service, Scheduler, Controller Manager
Failure Analysis Kubernetes
Demo etcd down for API server
worker kubelet worker kubelet scheduler & API
scheduler & API
Demo etcd restore for API server
scheduler & API
Demo node partition from API
worker kubelet worker kubelet scheduler & API
Demo node scaling up
worker kubelet worker kubelet scheduler & API worker kubelet
Demo node scheduled outage API
worker kubelet worker kubelet scheduler & API
Demo node unplanned outage
worker kubelet worker kubelet scheduler & API
Demo node downgrade/upgrade outage
worker kubelet worker kubelet scheduler & API
Future Work Upstream Kubernetes and Elsewhere
Upstream rktnetes Auth/OIDC Node self-signed TLS
Scaling 15x scheduler performance 30k pods on 1k nodes SIG-scale
Automatic Node Drain Locksmith Design Doc
None
Performance etcd3 /ZooKeeper snapshot disabled
Performance etcd3 /ZooKeeper snapshot disabled
Memory 10GB 2.4GB 0.8GB 512MB data - 2M 256B keys
Sounds good, but... Is anyone successful with all this in
prod?
Publically traded options exchange
Containers on CoreOS are powering ISE's high- throughput, low-latency financial
exchange Running in production Bare metal & AWS Billions of transactions a day 150 million req/sec
None
Secure the Internet MISSION
Separate Apps from OS STRATEGY
Make Servers Consistent STRATEGY
Tolerate Machine Failures STRATEGY
Make Servers Easy to Upgrade STRATEGY
Simplify Application Upgrades STRATEGY
None
Thank you! Brandon Philips @brandonphilips |
[email protected]
| coreos.com We’re
hiring in all departments! Email:
[email protected]
Positions: coreos.com/ careers
More Decks by Brandon Philips
See All by Brandon Philips
Node.js Workflow with Minikube and Skaffold
philips
0
310
Manage the App on Kubernetes
philips
0
390
Production Backbone Monitoring Containerized Apps
philips
0
260
KubeCon EU 2017: Dancing on the Edge of a Volcano
philips
1
900
rkt - KubeCon EU keynote - 2017
philips
1
330
FOSDEM_Keynote_2017-_.pdf
philips
0
190
Tectonic Summit Day 2 Keynote
philips
0
440
Kubernetes: Simple to Manage Anywhere (self-hosted, Tectonic upgrade demo)
philips
0
470
KubeCon Keynote 2016- Distributed Systems Simplified on Kubernetes
philips
2
610
Other Decks in Programming
See All in Programming
[DroidKnights 2026] Workflow AX: From Auto-Documented Policies to E2E Testing
raeminkang
0
580
自作Discord音声botをLambda MicroVMsに移行した話 / Migrating My Discord Voice Bot to Lambda MicroVMs
akihisaikeda
1
350
あなたはMCPの機能を、まだ知らない。
4geru
1
400
エラーチャンネルのボールを拾い続けて鍛えるエンジニアの地力 / phpcon-niigata2026
meihei3
1
230
Будущее локальных LLM – Алексей Голобурдин, PythoNN
sobolevn
0
120
将来の標準フォーマッター?ktfmtを深堀る
t45k
0
170
AI時代のIaC入門 〜なぜインフラをコードで管理したいのか?〜 / Introduction to IaC in the AI Era
akihisaikeda
6
1.5k
はじめてのPLATEAU
nokonoko1203
1
650
조합이폭발하는디자인시스템이걸눈으로도테스트하라고요
giljongyu
0
770
Coding with AI: How to trust the storyteller (AWS Community Day Adria, Ljubljana 2026)
slobodan
0
150
クラウドエージェントは、 何がそんなに便利なのか / what makes cloud agents so useful
rkaga
5
900
React Native에 현대적인 빌드 도구 더하기: Rollipop 구현기
leegeunhyeok
0
180
Featured
See All Featured
What Being in a Rock Band Can Teach Us About Real World SEO
427marketing
0
1.1k
Java REST API Framework Comparison - PWX 2021
mraible
34
9.8k
For a Future-Friendly Web
brad_frost
184
11k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
2k
Accessibility Awareness
sabderemane
1
300
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Max Prin - Stacking Signals: How International SEO Comes Together (And Falls Apart)
techseoconnect
PRO
0
530
Between Models and Reality
mayunak
4
580
エンジニアに許された特別な時間の終わり
watany
109
250k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
10
1.4k
Building AI with AI
inesmontani
PRO
1
1.4k
Music & Morning Musume
bryan
48
7.5k