Heartbleed at Acquia

A quick presentation on how we handled Heartbleed at Acquia. Held at a DevOps Boston meetup.

Transcript

1. Marc Seeger (@rb2k)
 Boston Devops Meetup
 May 20th 2014 at
2. Act 1: Technology
3. How it all started 7:24 PM
4. How it all started 7:30 PM
5. How it all started 7:26 PM
6. How it all started 7:33 PM
7. How it all started
8. Quick risk assessment Lucid: [00:35:27] root@bal-2.dev:~# openssl version OpenSSL 0.9.8k 25 Mar 2009 ! Precise: [00:34:37] root@master.dev:~# openssl version OpenSSL 1.0.1 14 Mar 2012
9. Where’s Waldo OpenSSL 8000 EC2 Machines: - 99.9% of them puppetized - Candidates: - Balancers - SVN Servers - Appliances - ELBs - 3rd party AMIs - Unique little snowflakes
 (Jira, Crucible,…)
10. Let the patching begin
11. Rollout Australia: ! Con: - Spiders - Snakes ! Pro: - Ops is awake
12. Rollout
13. Scan www
14. Waiting on ELBs…
15. Internal Certificates
16. Suddenly: “reverse” Heartbleed
17. Act 2: Communication
18. Internal • Pre-determined chat rooms • Dial-in conference bridges • A communication plan Thanks SSAE-16, PCI and FedRAMP… I guess :)
19. Statuspage + Twitter * Powered by StatusPage.io *
20. Documentation https://docs.acquia.com/articles/heartbleed-acquia-cloud
21. Proactive communication Phone calls by Acquia support, TAMs, …
22. Since then: Post mortem
23. Since then: Incident Commander (shamelessly stolen from Heroku) http://en.wikipedia.org/wiki/Incident_command_system
24. Since then: Dedicated resource to vet security threats
25. Since then: Clean up intranet docs
26. Since then: Additional tooling
27. We’re hiring (shameless self promotion) bit.ly/acquiajobs