KubeCon EU 2017: Dancing on the Edge of a Volcano

Google Slides: https://docs.google.com/presentation/d/11AjFqwB8QYBHuvmqetdeHOAaGBQDvnL875d3AcKs0CA/edit#slide=id.g1d26e1a83e_0_599

Transcript

1. Jess Frazelle (@jessfraz) Brandon Philips (@brandonphilips)
2. Dancing on the Edge of a Volcano Kubernetes Security Processes Jess Frazelle (@jessfraz) Brandon Philips (@brandonphilips)
3. Today's Major Parts • Context Setting • Open Source and Security Disclosure • Kubernetes Security Process • Helping Out
4. Today's Major Parts • Context Setting • Open Source and Security Disclosure • Kubernetes Security Process • Helping Out
5. Computer Security in 2017 BP
9. Why is Server Security Important?
10. 3,424,000,000 Internet Users Source: http://www.internetlivestats.com/internet-users/
11. 29,000,000 Software Developers and IT Practitioners Source: https://www.infoq.com/news/2014/01/IDC-software-developers
12. We are Outnumbered!
13. 238,975,082 New Internet Users in 2016 Source: http://www.internetlivestats.com/internet-users/
15. Documents Commerce Communications
16. ~100,000,000 Servers Worldwide
17. 3 Per Person In the Software and IT Industry
18. 100+ Per Person At the Internet Giants
19. Kubernetes.
20. Today's Major Parts • Context Setting • Open Source and Security Disclosure • Kubernetes Security Process • Helping Out
21. Challenges with Open Source and Security Disclosure JF
22. An open process Need to be open to vendors and community but also keep information from those who can do harm before bugs are fixed.
23. GPG: Do you even?
24. Can we make everyone happy? Researchers, community, vendors… All sides want something different.
25. Users Expectations - Want software without bugs - Want to be able to upgrade without breaking changes - Want to be let know when security fixes are made so they can act on them accordingly
26. Original Reporter Expectations - Want “a human to say acknowledged” - Want updates on status, transparency - Want bug to be fixed in a certain timeframe
27. Project Expectations - Want bugs to be fixed quickly and embargo to be kept secret until time of disclosure - Want upgrades to remain seamless - Want to continue to instill trust with users
28. How to get people to upgrade? - Challenges with making sure people know - Use mailing lists, establish a pattern for the project that allows people to follow along - How to handle if very severe… pray you don’t have someone make up a fun name
29. Challenge: name the vulnerability
30. How bugs get reported... I found a bug… should I be evil or let upstream know... Let me google “X security disclosure” or I’ll just try to email security@
31. How bugs get reported... I’ll lookup the project’s recommended method for disclosures I found a bug… I’ll file an issue on the repo...
32. I found a bug… I’ll file an issue on the repo... - Not everyone can recognize the severity of a bug at first glance. - Allow for mistakes and have a process for handling them in a way that it will not confuse the original reporter or cause unnecessary attention. Mistakes happen
33. Learning from Other Communities BP
34. The Common Stuff
35. General Recommendations We Got - Make documentation Googleable (X security disclosure) - Have a dedicated response team - Coordinate via a mailing list - Don't force the use of GPG - Have internal documentation on the process - Use a severity leveling system - Request CVEs when appropriate*
36. Docker: Document everything JF
37. Docker: Tooling for everything JF
38. Linux Kernel: Don't Negotiate Set timeline expectations; don't negotiate on embargos BP
39. Node.js: Help the Ecosystem BP Fix the core but also help track the ecosystem and notify
40. Go: Early warning systems JF Give people a heads up that a vulnerability is coming
41. Go: Small team, explicit contacts Mailing lists fail, people don't read email, list everyone. JF
42. Today's Major Parts • Context Setting • Open Source and Security Disclosure • Kubernetes Security Process • Helping Out
43. Where we are today BP
44. Kubernetes Normal Release Process BP
45. Kubernetes Release Process k8s 1.5.x k8s 1.6.0 k8s 1.6.x Bug fixes & security patches Features Features k8s 1.7.0 k8s 1.7.x Bug fixes & security patches Features
46. Kubernetes Release Process k8s 1.5.x k8s 1.6.0 k8s 1.6.x Bug fixes & security patches Features Features Bug fixes & security patches ~9 Months*
47. The Kubernetes v1.6 Release Schedule • … 4-6 weeks of coding … • Jan 30th (Monday) v1.6.0-alpha.1 • Feb 13th (Monday) v1.6.0-alpha.2 • Feb 20th (Monday) Cut release-1.6 branch and v1.6.0-beta.0 • Feb 27th (Monday) Start code freeze • Feb 28th (Tuesday) v1.6.0-beta.1 • March 7th (Tuesday) - v1.6.0-beta.2 • March 14th (Tuesday) Lift code freeze and v1.6.0-rc.1 • March 22nd (Wednesday) - v1.6.0
48. The Release Team • Release Management Team Lead • Release Branch Manager • Docs Lead • Features Lead • Bug Triage Lead • Test Infra Lead • Automated Upgrade Testing Lead • Manual Upgrade Testing Lead • Testing Lead • Product Lead
50. Lots of moving parts
51. So lots and lots of testing...
52. Kubernetes Security Release Process JF
53. Bug report Fix team summoned Bug is reported through the disclosure process. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions
54. Bug report Fix team summoned The “Fix Team” is organized and a “Lead” makes sure relevant engineers are included in the discussion. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions
55. Bug report Fix team summoned Patches are posted to a private repo. They are tested and reviewed there by the “fix team.” within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions
56. Bug report Fix team summoned “Fix Lead” emails kubernetes-announce and kubernetes-security-announce informing users a fix will be made available on a certain date. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions
57. Bug report Fix team summoned If the bug is deemed critical enough, downstream distributions will be given notice about the bug and patches ahead of time. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions
58. Bug report Fix team summoned Release is cut from the private repo and patches to the public repo are made after binaries are available. within 24 hours Fix Development Disclosure of “fix forthcoming” to users within 1-14 days within 1-7 days within 1-21 days Release day Patch disclosure to distributions
59. Scoring Issues w/ CVSS
60. Scoring Issues w/ CVSS
61. 5.0/10 Heartbleed
62. Why is Heartbleed only a 5.0? CVSS 2.0 scoring is failing to take into account quite a few factors: 1. It’s a target of opportunity for attackers: 30-70% of the Internet was effected 2. It’s being actively and successfully exploited on the Internet 3. It’s easy to exploit
63. 7.8/10 Dirty COW
64. Areas for Improvement.
65. Rotating the Responsibilities - Challenges - Responsible security process requires non-public work - Running the process is not glamorous - Incentives - Many organizations rely on Kubernetes; the work is important - Critical security announcement will be highly referenced - Next Steps - We need a way of making decisions! Kubernetes Governance Discussion concludes BP
66. Communication to Users - Send an email to kubernetes-announce and call it a day?! - How do you make announcements immediately actionable? - Should vendors be informed ahead of time with a fix? 1 JF
67. Today's Major Parts • Context Setting • Open Source and Security Disclosure • Kubernetes Security Process • Helping Out
68. Do Something. BP
69. Some Homework - Join Kubernetes Announce! - Review the Kubernetes Security Process - Governance discussion to figure out - Help improve https://github.com/kubernetes/test-infra
70. Thank You! Jess Frazelle (@jessfraz) Brandon Philips (@brandonphilips)