Share
Embed
Operating Microservices
Listen closely. Close your eyes. I know you hear it coming. It's the microservices buzz. Well, by now it is more of a quake. Catherine Rodriguez and Michael Winslow have been involved with 2 enterprise level microservices implementations. One was greenfield and the other was a migration from a FAT service. It turns out that the 2 easiest things to do when moving to microservices are to DECIDE to do it and to WRITE THE CODE. If you stick around after that, you better be ready for a ride!
Michael will address considerations like scaling, testing and releasing your microservices.
Catherine will cover logging and traceability.
Michael Scott Winslow
June 03, 2019
Transcript
58 slides
-
@catherinerod27 | @michaelswinslow Michael Winslow Core Applications - Comcast Spaces
> Tabs Operating Microservices @michaelswinslow /michaelswinslow Catherine Rodriguez Core Applications - Comcast I have a twin sister /catherinerodriguez1 @CatherineRod27 -
@catherinerod27 | @michaelswinslow Some background: 1. Michael was on a
DEV team that built web services using Spring MVC 2. Catherine was on the Splunk Platform team (logging, observability, etc.) 3. At some point, the team and leadership decided to pursue Microservices 4. Things worked out well, but it was not always sunshine Now you have a choice: • Hear about the lessons your team learned! (red) • Hear about how great your team is at making microservices! (blue) -
@catherinerod27 | @michaelswinslow Is There a Need To Move to
Microservices? Are you Netflix? No no Start -
@catherinerod27 | @michaelswinslow This is a MONOLITH Service Repository (DAO)
eligibility user device eligibility user device eligibility user device Controller eligibility user device -
@catherinerod27 | @michaelswinslow Service Repository This is a MICROSERVICE Service
Repository Service Repository Controller device Controller user Controller eligibility -
@catherinerod27 | @michaelswinslow Service Repository This is a MICROSERVICE Service
Repository Service Repository Controller device Controller user Controller eligibility Note: Large organizations may dedicate a small team to each microservice. A team that has a great deal of developers who can work independently may benefit from this model. -
@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service
Repository Controller eligibility Service Repository API Gateway -
@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service
Repository Controller eligibility Service Repository API Gateway “Oh no! For some reason requests to the ELIGIBILITY API have gone way up! At least 3x the traffic of any other API!” -
@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service
Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery -
@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service
Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery 1 2 3 -
@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service
Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery Note: Elasticity is one of the major reasons to move to microservices. If your traffic tends to increase and decrease dramatically, you may find value. Especially in the cloud. -
@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service
Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery -
@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service
Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery Retry 3/3 -
@catherinerod27 | @michaelswinslow HYSTRIX Controller device Service Repository Controller user
Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Discovery Retry 3/3 -
@catherinerod27 | @michaelswinslow Summary: 1. Monolith vs Microservice 2. API
Gateway - Zuul 3. Service Discovery - Eureka 4. Software Load Balancer – Ribbon 5. Circuit Breaker - Hystrix Now you have a choice: • That was amazing! Now tell us what your team learned! (red) • That was amazing! Now tell us what your team learned! (blue) -
@catherinerod27 | @michaelswinslow “Being able to scale and distribute your
application resources is certainly alluring. However, this doesn't come for free. Complexity begets complexity. Soon tools are needed to manage the complexity. Then tools are needed to manage the tools.” – Ryan Emerle, Principal Engineer -
@catherinerod27 | @michaelswinslow Sample: We need to call our API
to determine if a particular USER is ELIGIBLE to purchase a DEVICE -
@catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility
user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog -- -- -- -- process boundary syslog entry -
@catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility
user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- -- -- -
@catherinerod27 | @michaelswinslow MONOLITH MICROSERVICES syslog applog syslog applog 03
-- 08 -- Note: Increased number of process boundaries also creates more surface area for security attacks during runtime. -
@catherinerod27 | @michaelswinslow So What? Microservices inflate the System Logs
a little (and security risk). It’s totally worth it! -
@catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility
user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- -
@catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility
user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.fetchUser(EligibilityService.java:1012) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityService.java:5) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityController.java:27) STACK TRACE for Monolith -
@catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility
user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.fetchUser(EligibilityService.java:1012) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityService.java:5) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityController.java:27) STACK TRACE for Monolith -
@catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility
user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- Stack trace has visibility of everything in the Monolith -
@catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility
user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- -
@catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility
user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.loadUserByUserName(UserController.java:127) STACK TRACE for Microservice -
@catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility
user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- Stack trace only has visibility of the current microservice -
@catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility
user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- All context of where the request started is lost -
@catherinerod27 | @michaelswinslow Distributed Tracing and the logs that come
with it [Dapper] [Zipkin] [Money] [OpenTelemetry] @pauljamescleary -
@catherinerod27 | @michaelswinslow “Microservices did not introduce the need for
distributed tracing, but they certainly increased the need.” – Anthony Orapallo, Engineering Leader -
@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service
Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 07 { "correlationId": 1, "method":"EligibilityController.evaluateEligibility", "level":"INFO"…} { "correlationId": 1, "method":"EligibilityService.evaluateEligibility", "level":"INFO"…} { "correlationId": 1, "method":"EligibilityService.fetchUser", "level":"INFO"…} { "correlationId": 1, "method":“UserController.fetchUser", "level":"INFO"…} { "correlationId": 1, "method":“UserService.loadUserByUsername", "level":"INFO"…} { "correlationId": 1, "method":“UserRepository.getUser", "level":"INFO"…} … { "correlationId": 1, "method": “UserRepository.getUser" "level": “ERR", "message": "variable username is null", "timestamp": 1385053862.3072, "_user_id": 9001, } correlationId = 1 -
@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service
Repository Controller eligibility Service Repository API Gateway { "correlationId": 1, "method":"EligibilityController.evaluateEligibility", "level":”INFO"…} { "correlationId": 1, "method":"EligibilityService.evaluateEligibility", "level":”INFO"…} { "correlationId": 1, "method":"EligibilityService.fetchUser", "level":”INFO"…} { "correlationId": 1, "method":“UserController.fetchUser", "level":”INFO"…} { "correlationId": 1, "method":“UserService.loadUserByUsername", "level":”INFO"…} { "correlationId": 1, "method":“UserRepository.getUser", "level":”INFO"…} … { "correlationId": 1, "method": “UserRepository.getUser" "level": “ERR", “message": "variable username is null", "timestamp": 1385053862.3072, "_user_id": 9001, } MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 07 0 200 400 600 800 1000 1200 1400 1600 1 2 3 4 5 6 7 8 9 LOG ENTRIES # OF REQUESTS Log File Sizes Monolith Microservices 4x -
@catherinerod27 | @michaelswinslow What you did for ONE you now
have to do for MANY -
@catherinerod27 | @michaelswinslow code review + CI CD to all
environments firewalls + observability security scans -
@catherinerod27 | @michaelswinslow Suggestions: 1. Do not create all of
your microservices at once 2. Monitor the usage to find your most utilized endpoints and use that to prioritize your migration 3. Have a template for creating new microservices and USE IT! -
@catherinerod27 | @michaelswinslow Version Control and Compatibility can be a
nightmare @michaelswinslow -
@catherinerod27 | @michaelswinslow Release 1.5.7 A RELEASE version now consists
of a configuration of many microservices. Each microservice version increments independently from the RELEASE and other microservices. -
@catherinerod27 | @michaelswinslow “…when you publish a new version of
a mobile app, the old version doesn’t just magically go away.” – Mark Dappollone, Senior Android Developer Dappollone, M (2018). Future Proofing Mobile Applications from https://medium.com/datadriveninvestor/future-proofing-mobile-applications-56f37e80cd4c -
@catherinerod27 | @michaelswinslow Note that the World Health Organization now
classifies burn-out as an official medical disorder. World Health Organization (2019). QD85 Burn-Out from https://icd.who.int/browse11/l-m/en#/http://id.who.int/icd/entity/129180281 -
@catherinerod27 | @michaelswinslow Recap with DOs and DON’Ts: DO have
a reason to move to microservices (Resiliency, Cost-Optimization, etc.) DON'T practice RDD (Resume-Driven Development) DO communicate with all teams about monitoring, telemetry and distributed tracing DON'T count on traditional logging techniques (Stack Trace) DO peel off one microservice at a time (Strangler Application pattern) DON'T refactor everything at once DO understand that all teams are impacted (Operations, Release Mgmt, Security, etc.) DON'T say “we're doing microservices!” #YOLO DO staff appropriately and monitor employee health DON'T ignore the human impact of your decision -
@catherinerod27 | @michaelswinslow Recap with DOs and DON’Ts: DO encourage
DevOps teams to pursue microservices if there are benefits DON'T underestimate the time, effort and money involved -
@catherinerod27 | @michaelswinslow Michael Winslow Core Applications - Comcast Spaces
> Tabs @michaelswinslow /michaelswinslow Catherine Rodriguez Core Applications - Comcast I have a twin sister /catherinerodriguez1 @CatherineRod27 Thank You! -
@catherinerod27 | @michaelswinslow Helpful Links 1. Netflix Zuul - https://github.com/Netflix/zuul
2. Netflix Eureka - https://github.com/Netflix/eureka 3. Netflix Ribbon - https://github.com/Netflix/ribbon 4. Netflix Hystrix - https://github.com/Netflix/hystrix 5. Netflix Vizsceral - https://github.com/Netflix/vizceral 6. Google Dapper – https://ai.google/research/pubs/pub36356 7. Comcast Money - https://github.com/Comcast/money 8. Twitter Zipkin - https://zipkin.apache.org/ Tom Martin, Splunk – Zipkin and Splunk Mark Dappalone, Comcast – FutureProofing Your Mobile Applications