Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lessons Learned Reviewing 150 Infrastructures

Lessons Learned Reviewing 150 Infrastructures

Since April 2018 we've had the opportunity to perform a structured review of the architectural and operational choices of 150 platform teams. In this talk (given at QCon London 2020) I explore some themes, talk about common mistakes, and give some advice on how to avoid these yourselves.

The Scale Factory

March 04, 2020
Tweet

More Decks by The Scale Factory

Other Decks in Technology

Transcript

  1. 0 45 90 135 180 Mar-2018 May-2018 Jul-2018 Sep-2018 Nov-2018

    Jan-2019 Mar-2019 May-2019 Jul-2019 Sep-2019 Nov-2019 Jan-2020 REVIEWS RUN_ @jtopper
  2. WELL ARCHITECTED ORIGINS_ Catalogue of emergent good practices Observed by

    AWS Field Solutions Architects Codified and shared Platform agnostic* @jtopper
  3.          

      !     !" #   $# #!  !! %! #  %  ! % "  $ White Papers Review Tool @jtopper
  4. Performance Efficiency Cost Optimisation Operational Excellence Reliability Security Well Architected

    Core Serverless Applications High Performance Computing IoT (Internet of Things) 9 11 9 8 9 2 3 2 1 1 4 3 3 4 2 4 11 6 10 4 @jtopper 46 9 16 35
  5. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate

    internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? @jtopper
  6. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate

    internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? NI WA CI WA WA NI NI @jtopper
  7. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate

    internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? NI WA CI WA WA NI NI High Risk @jtopper
  8. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate

    internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? NI WA CI WA WA NI NI Medium Risk @jtopper
  9. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate

    internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? NI WA CI WA WA NI NI Medium Risk @jtopper
  10. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate

    internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? NI WA CI WA WA NI NI Well Architected @jtopper
  11. QUESTION OPS 1_ • Evaluate external customer needs • Evaluate

    internal customer needs • Evaluate compliance requirements • Evaluate threat landscape • Evaluate tradeoffs • Manage benefits and risks • None of these How do you determine what your priorities are? NI WA CI Well Architected 77% WA WA NI NI 93% 87% 90% 85% 89% 89% 0% WA Rank: 1 @jtopper
  12. QUESTION PERF 3_ • Understand storage characteristics and requirements •

    Evaluate available configuration options • Make decisions based on access patterns and metrics • None of these How do you select your storage solution? WA CI Well Architected 70% NI NI 84% WA Rank: 2 78% 73% 5% @jtopper
  13. QUESTION REL 5_ • Deploy changes in a planned manner

    • Deploy changes with automation • None of these How do you implement change? WA CI Well Architected 63% NI 83% 67% 6% WA Rank: 3 @jtopper
  14. QUESTION REL 9_ • Define recovery objectives for downtime and

    data loss • Use defined recovery strategies to meet the recovery objectives • Test disaster recovery implementation to validate the implementation • Manage configuration drift on all changes • Automate recovery • None of these How do you plan for disaster recovery? WA CI High Risk 79% NI 33% HRI Rank: 1 WA WA NI 33% 25% 39% 16% 31% (87%) @jtopper
  15. QUESTION SEC 11_ • Identify key personnel and external resources

    • Identify tooling • Develop incident response plans • Automate containment capability • Identify forensic capabilities • Pre-provision access • Pre-deploy tools • Run game days • None of these How do you respond to a [security] incident? WA CI High Risk 75% NI 51% HRI Rank: 2 WA WA NI 27% 39% 0% 11% 27% 10% 3% 35% NI NI NI (93%) @jtopper
  16. QUESTION SEC 8_ • Define data classification requirements • Define

    data protection controls • Implement data identification • Automate identification and classification • Identify the types of data • None of these How do you classify your data? WA CI High Risk 75% HRI Rank: 3 WA WA 61% 39% 17% 4% 59% 23% NI NI (88%) @jtopper
  17. QUESTION COST 9_ • Establish a cost optimisation function •

    Develop a workload review process • Review and implement services in an unplanned way • Review and analyse this workload regularly • Keep up to date with new service releases • None of these How do you evaluate new services? WA CI High Risk 71% HRI Rank: 4 WA 34% 26% 84% NI NI (79%) NI 43% 63% 1% @jtopper
  18. QUESTION REL 8_ • Use playbooks for unanticipated failures •

    Conduct root cause analysis and share results • Inject failures to test resiliency • Conduct game days regularly • None of these How do you test resilience? WA CI High Risk 67% HRI Rank: 5 WA 25% NI (92%) NI 73% 6% 0% 16% @jtopper
  19. QUESTION OPS 3_ • Use version control • Test and

    validate changes • Use config management systems • Use build/deploy systems • Perform patch management • Share design standards • Implement practices to improve code quality • Use multiple environments • Make frequent, small, reversible changes • Fully automate integration and deployment • None of these How do you reduce defects, ease remediation, and improve flow into production? NI WA Well Architected 14% WA WA Rank: 23 @jtopper NI NI NI NI NI NI NI CI 90% 87% 78% 82% 37% 57% 83% 81% 63% 52% 3%
  20. QUESTION OPS 6_ • Identify key performance indicators • Define

    workload metrics • Collect and analyse workload metrics • Establish workload metric baselines • Learn expected patterns of activity for workload • Alert when workload outcomes are at risk • Alert when workload anomalies are detected • Validate the achievement of outcomes and the effectiveness of KPIs and metrics • None of these How do you understand the health of your workload? WA Well Architected 46% WA WA Rank: 21 @jtopper NI NI NI NI NI CI 53% 62% 72% 51% 54% 40% 34% 37% WA 14%
  21. QUESTION SEC 2_ • Define human access requirements • Grant

    least privileges • Allocate unique credentials per person • Manage credentials based on lifecycle • Automate credential management • Grant access through roles or federation • None of these How do you control human access? WA CI High Risk 47% HRI Rank: 20 WA WA 70% 58% 90% 70% 13% 62% 3% NI NI (88%) @jtopper NI
  22. QUESTION SEC 3_ • Define programmatic access requirements • Grant

    least privileges • Automate credential management • Allocate unique credentials per component • Grant access through roles or federation • Implement dynamic authentication • None of these How do you control programmatic access? WA CI High Risk 57% HRI Rank: 15 WA 40% 70% 24% 68% 58% 22% 13% NI NI (89%) @jtopper NI NI
  23. TEAMS ARE OK AT CHOOSING CORRECT SERVICES_ Database choices match

    workload Storage choices match workload Compute choices sometimes not right- sized. @jtopper
  24. TEAMS ARE OK AT MAKING SOFTWARE CHANGES_ Automation tools are

    being used Full CD remains out of reach Change batch sizes need to be smaller @jtopper
  25. TEAMS ARE BAD AT THINKING ABOUT FAILURE MODES_ Not considering

    business requirements No risk analysis of failure modes Poor documentation Almost no attempt to rehearse outages @jtopper
  26. TEAMS ARE BAD AT MONITORING FOR FAILURE MODES_ Monitoring happening

    Data not used for much Tracing almost non-existent @jtopper
  27. TEAMS NEED TO DO BETTER AT SECURITY_ Poor hygiene around

    patching Limited data classification Mediocre human access control Bad programmatic access control Low adoption of security monitoring tools @jtopper
  28. TOP BREACH CAUSES_ Using components with known vulnerabilities Security misconfiguration

    Injection Weak auth / session management Missing function access control https:/ /snyk.io/blog/owasp-top-10-breaches/ @jtopper
  29. WHAT NEXT?_ Read the white papers: https:/ /aws.amazon.com/architecture/well-architected/ Run your

    own review(s) https:/ /aws.amazon.com/well-architected-tool/ Consider engaging an AWS Well-Architected partner https:/ /scalefactory.com/services/well-architected/ (funding available) @jtopper