Architecture and Devops 2015

Talk at Devops Madrid, June 2015

1. architecture && devops @ticketeaEng
2. ticketea Who we are? @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_
3. ticketea Designers @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ X X X X X X X X X X X X X
4. ticketea Development @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ X X X X
5. ticketea QA @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ X X X X X X X X X X X X X X X
6. ticketea Sysadmins @esanchezm @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @Ethervoid @imanolcg @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ X X X X X X X X X X X X X X X X
7. do your own deploys!
8. Our systems
10. ticketea Why use AWS? ✤ We don’t have sysadmins. ✤ Our traffic is spike-oriented. ✤ Auto-managed AWS services are really helpful. ✤ They do multi-tenancy easy. ✤ They improve their service continually. ✤ The uptime of some of their services is 99.999999999%
11. ticketea AWS services EC2 RDS S3 CloudFront EBS ELB ElasticCache DynamoDB SES Route53 SNS SQS VPC CloudWatch
12. I can quit whenever I want
13. The Good, the bad and the ugly with AWS
14. ticketea EC2 && ELB ✤ Keep an eye on the “Events” page of the admin console. ✤ Use always more than one AZ. ✤ We reached the maximum number of instances of EC2! ✤ Pre-heat the ELBs (open a ticket in the support area).
15. ticketea RDS ✤ Vertical auto-scaling and spikes are not friends. ✤ The point-in-time-snapshots system is awesome. ✤ The MultiAZ allows you tu vertically scale without downtime. ✤ Create new replicas is a one-click process.
16. ticketea SQS ✤ Everything can fail: also 3 AZ of one region. ✤ Automate fallbacks to other regions. ✤ Be prepared to have a 2nd plan to synchronise services. ✤ No messages lost, but receive them more than once (maybe).
17. ticketea SES ✤ We’ve been flagged as SPAM because of our neighbours. ✤ Be careful with the service limits. ✤ Automate fallbacks to other regions. ✤ Configure properly your domain (SPF and DKIM). ✤ Use SNS to log the bounces/complains rate.
18. ticketea Route53 ✤ Check the expiration date of your domains and SSL certs.
19. ticketea VPC ✤ It’s really difficult to migrate from EC2. ✤ We’re trying to migrate gradually. ✤ Check ClassicLink to facilitate the process. ✤ Create a VPN to get access to your network. ✤ Use a Route53 internal domain to name your services.
20. Don’t panic, we’re devops!
21. Graphing and logging
22. ticketea Error reporting ✤ sentry ✤ fabric (fka. crashlytics)
23. ticketea Graphing ✤ graphite ✤ grafana ✤ CloudWatch
24. ticketea Logging and alarming ✤ rsyslog ✤ cabot ✤ Slack connected through SNS
25. Deploying code
26. ticketea The Rat ‘devops’ race API Heracles Odin Frontend Chef Puppet Ansible Thor Caronte Ansible “The Reborn”
28. ticketea From puppet to ansible, through chef
29. ticketea Executing commands ✤ fab -f heracles.py stop_consuming_messages:queue ✤ fab -f frontend.py maintenance_on ✤ fab -f sg.py open_sg ✤ fab -f api.py fast_rollback:version
30. ticketea Development environment ✤ Use the same ansible roles ✤ Meta-roles: roles/ api/ tasks/main.yml tkt-api/ meta/main.yml api.yml development.yml
31. ticketea server
32. ticketea Our architecture
33. ticketea Our architecture
34. ticketea Our architecture
35. ticketea Our architecture
36. ticketea Our architecture
37. ticketea Our architecture
38. ticketea Our architecture
39. ticketea offices
40. ticketea Distributed systems ✤ Granular failing. ✤ Avoid single point of failure. ✤ Easy code maintainability ✤ Choose the best tech for each project. ✤ Scale the team easier.
41. ticketea Distributed systems ✤ Hard to deploy. ✤ Hard to see the whole picture. ✤ More complexity: easy to fail. ✤ When shit happens: you need a plan B.
42. Forseti
43. ticketea Why did we create forseti? ✤ It was 2013 and there was no UI for Autoscaling in AWS. ✤ Other tools were too overkill for us. ✤ We wanted something small, quick and for CLI. ✤ We wanted to play around with AWS API :)
44. ticketea What’s forseti? ✤ An AWS tool. ✤ It mainly manages auto-scaling groups. ✤ It’s the director of our deploys.
45. ticketea How to use it? forseti deploy app
46. ticketea deploy_and_snapshot deployer 1. Stops any autoscaling process. 2. It relies in other tools to deploy your app. 3. Choose a random instance to create an AMI. 4. Setup the autoscaling group, policies and alarms.
47. ticketea Any other thing? ✤ It has other commands: i. status ii. regenerate iii. list iv. clean
48. ticketea Where can you get it? https://github.com/ticketea/forseti
49. ticketea Questions?
50. @ticketeaEng Thanks!