Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Heartbleed at Acquia

Avatar for Marc Seeger Marc Seeger
May 20, 2014
Technology
0
14k

Heartbleed at Acquia

A quick presentation on how we handled Heartbleed at Acquia. Held at a DevOps Boston meetup.
Avatar for Marc Seeger

Marc Seeger

May 20, 2014
Tweet

More Decks by Marc Seeger

See All by Marc Seeger
Security in DECT
Avatar for Marc Seeger rb2k
2
160
The DIRAC video codec
Avatar for Marc Seeger rb2k
1
76
Communitygetriebene Android Systemerweiterungen
Avatar for Marc Seeger rb2k
1
53
Alternative infrastructure
Avatar for Marc Seeger rb2k
1
170
NoSQL Lunch and Learn
Avatar for Marc Seeger rb2k
9
8.5k
Lunch and Learn: Cucumber and Capybara
Avatar for Marc Seeger rb2k
7
21k

Other Decks in Technology

See All in Technology
Azure Developer CLI と Azure Deployment Environment / Azure Developer CLI and Azure Deployment Environment
Avatar for ののし nnstt1
1
130
Introduction to Sansan, inc / Sansan Global Development Center, Inc.
Avatar for Sansan, Inc. sansan33
PRO
0
2.6k
うちの会社の評判は？SNSの投稿分析にAIを使ってみた
Avatar for Kiyotaka Doumae doumae
0
280
Cursor Meetup Tokyo
Avatar for Shunta Komatsu iamshunta
2
710
障害を回避するHttpClient再入門 / Avoiding Failures HttpClient Reintroduction
Avatar for Yusuke Uehara uskey512
1
160
toittaにOpenTelemetryを導入した話 / Mackerel APM リリースパーティ
Avatar for cohalz cohalz
1
490
大手企業のAIツール導入の壁を越えて：サイバーエージェントのCursor活用戦略
Avatar for Gunther Brunner gunta
19
5.5k
技術書典18結果報告
Avatar for Mutz mutsumix
2
180
Slackひと声でブログ校正！Claudeレビュー自動化編
Avatar for Yusuke Shimizu yusukeshimizu
3
180
OTel meets Wasm: プラグイン機構としてのWebAssemblyから見る次世代のObservability
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
1
300
GigaViewerにおけるMackerel APM導入の裏側
Avatar for koudenpa 7474
0
460
会社紹介資料 / Sansan Company Profile
Avatar for Sansan, Inc. sansan33
PRO
6
360k

Featured

See All Featured
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
46
14k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
63
7.8k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
52
2.8k
Side Projects
Avatar for Sacha Greif sachag
454
42k
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
178
9.7k
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
76
5.8k
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
233
140k
Dealing with People You Can't Stand - Big Design 2015
Avatar for Cassini Nazir cassininazir
367
26k
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
407
66k
KATA
Avatar for Megan Lloyd mclloyd
29
14k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
29
1.7k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
507
140k

Transcript

  1. Marc Seeger (@rb2k)
 Boston Devops Meetup
 May 20th 2014 at

  2. Act 1: Technology

  3. How it all started 7:24 PM

  4. How it all started 7:30 PM

  5. How it all started 7:26 PM

  6. How it all started 7:33 PM

  7. How it all started

  8. Quick risk assessment Lucid: [00:35:27] [email protected]:~# openssl version OpenSSL 0.9.8k

    25 Mar 2009 ! Precise: [00:34:37] [email protected]:~# openssl version OpenSSL 1.0.1 14 Mar 2012

  9. Where’s Waldo OpenSSL 8000 EC2 Machines: - 99.9% of them

    puppetized - Candidates: - Balancers - SVN Servers - Appliances - ELBs - 3rd party AMIs - Unique little snowflakes
 (Jira, Crucible,…)

  10. Let the patching begin

  11. Rollout Australia: ! Con: - Spiders - Snakes ! Pro:

    - Ops is awake

  12. Rollout

  13. Scan www

  14. Waiting on ELBs…

  15. Internal Certificates

  16. Suddenly: “reverse” Heartbleed

  17. Act 2: Communication

  18. Internal • Pre-determined chat rooms • Dial-in conference bridges •

    A communication plan Thanks SSAE-16, PCI and FedRAMP… I guess :)

  19. Statuspage + Twitter * Powered by StatusPage.io *

  20. Documentation https://docs.acquia.com/articles/heartbleed-acquia-cloud

  21. Proactive communication Phone calls by Acquia support, TAMs, …

  22. Since then: Post mortem

  23. Since then: Incident Commander (shamelessly stolen from Heroku) http://en.wikipedia.org/wiki/Incident_command_system

  24. Since then: Dedicated resource to vet security threats

  25. Since then: Clean up intranet docs

  26. Since then: Additional tooling

  27. We’re hiring (shameless self promotion) bit.ly/acquiajobs