Monitoring OSA 2017 NYC

Transcript

1. Monitoring Reactive Microservices Henrik Engström (@h3nk3) Senior Software Engineer, Lightbend O’Reilly Software Architecture, NYC - April 2017
2. >whoami
4. • “TRADITIONAL” AND REACTIVE APPLICATIONS • MICROSERVICES • MONITORING (DIFFERENT TYPES OF) APPLICATIONS • CHALLENGES IN MONITORING AND MITIGATIONS • PEEKING INTO THE FUTURE • PRODUCTION MONITORING • LIGHTBEND MONITORING AGENDA
5. “TRADITIONAL” AND REACTIVE APPLICATIONS
6. IT IS 2017 AND WE STILL USE • Synchronous local and remote calls • Single machine apps - scaling is an afterthought • Non resilient approaches Result: brittle, non-scaling applications
7. WHAT CAN WE DO ABOUT IT? Use message driven/asynchronous programming!
8. PROBLEM SOLVED?
9. REACTIVE MANIFESTO http://www.reactivemanifesto.org/ • Created in September 2014, +18k signatures • Consists of four traits • Responsive • Resilient • Elastic • Message Driven
10. RESPONSIVE
11. RESILIENT
12. E L A S T I C
13. MESSAGE DRIVEN
14. MICROSERVICES
15. WHAT WE TRY TO GET AWAY FROM
16. WHERE WE COULD END UP…
17. RECIPES FOR MICROSERVICES • Isolate everything • Act autonomously • Do one thing and do it well • Own your state • Embrace asynchronous message passing
18. http://www.lightbend.com/resources/e-books
19. MONITORING TRADITIONAL APPLICATIONS
20. SYNCHRONOUS APPS • Metrics based on entry/exit points • Context packed stack traces are available • Logs are (more) descriptive • Thread locals can be used to transfer contexts
21. MONITORING ASYNCHRONOUS APPLICATIONS
22. ASYNCH STACK TRACE [info] at cinnamon.sample.failure.B$$anonfun$receive$2.applyOrElse(FailureDemo.scala:102) [info] at akka.actor.Actor$class.aroundReceive(Actor.scala:467) [info] at cinnamon.sample.failure.B.aroundReceive(FailureDemo.scala:86) [info] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [info] at akka.actor.ActorCell.invoke(ActorCell.scala) [info] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [info] at akka.dispatch.Mailbox.run$$original(Mailbox.scala:220) [info] at akka.dispatch.Mailbox.run(Mailbox.scala:29) [info] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [info] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [info] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [info] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [info] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
23. JUMP ASYNCH BOUNDARIES
24. HOW WE DO IT
25. EXAMPLE SPI abstract class ActorInstrumentation { def systemStarted(system: ActorSystem): Unit def systemShutdown(system: ActorSystem): Unit def actorStarted(actorRef: ActorRef): Unit def actorStopped(actorRef: ActorRef): Unit def actorTold(actorRef: ActorRef, message: Any, sender: ActorRef): AnyRef def actorReceived(actorRef: ActorRef, message: Any, sender: ActorRef, context: AnyRef): Unit def actorCompleted(actorRef: ActorRef, message: Any, sender: ActorRef, context: AnyRef): Unit // … }
26. INSIDE THE SAUSAGE FACTORY // ActorCell.scala final def invoke(messageHandle: Envelope): Unit = try { //… systemImpl.instrumentation.actorReceived( self, messageHandle.message, messageHandle.sender, context) messageHandle.message match { // … } systemImpl.instrumentation.actorCompleted( self, messageHandle.message, messageHandle.sender, context) } catch handleNonFatalOrInterruptedException { e 㱺 handleInvokeFailure(Nil, e) // …
27. MONITORING DISTRIBUTED APPLICATIONS
28. DISTRIBUTED TRACING In a nutshell: • Create event for each “occurrence” • Persist these events • Deduct information based on the events • Transfer contexts at remote boundaries
29. PAPER NOTE EXPERIMENT Henrik Received: 4.22.23 Sent: 4.22.29 Peter Received: 4.22.32 Sent: 4.22.40 Björn Received: 4.22.40 Sent: 4.22.50 Duncan Received: 4.22.51 Sent: 4.22.58
30. WHAT IS WRONG WITH THIS? TIME Henrik Received: 4.22.23 Sent: 4.22.29 Peter Received: 4.22.32 Sent: 4.22.40 Björn Received: 4.22.40 Sent: 4.22.50 Duncan Received: 4.22.51 Sent: 4.22.58
31. APPROACH TO ACHIEVING ORDER
32. PAPER NOTE EXPERIMENT Trace Id: 123 Parent Id: - Id: Peter Received: 4.22.32 Sent: 4.22.40 Trace Id: 123 Parent Id: Peter Id: Henrik Received: 4.22.23 Sent: 4.22.29 Trace Id: 123 Parent Id: Henrik Id: Björn Received: 4.22.40 Sent: 4.22.50 Trace Id: 123 Parent Id: Björn Id: Duncan Received: 4.22.51 Sent: 4.22.58
33. CORRECT ORDER TRUE TIME Trace Id: 123 Parent Id: - Id: Peter Received: 4.22.32 Sent: 4.22.40 Trace Id: 123 Parent Id: Peter Id: Henrik Received: 4.22.23 Sent: 4.22.29 Trace Id: 123 Parent Id: Henrik Id: Björn Received: 4.22.40 Sent: 4.22.50 Trace Id: 123 Parent Id: Björn Id: Duncan Received: 4.22.51 Sent: 4.22.58
34. VISIBILITY
35. CHALLENGES IN MONITORING AND MITIGATIONS
36. HANDLING SCALE
37. HANDLING SCALE
38. EPHEMERALITY
39. STAYING COST EFFECTIVE
40. CORRELATION
41. VARIETY
42. HIGHER AVAILABILITY 9 9 9 9 9 9 9 9 9
43. PEEKING INTO FUTURE<THE>
44. EXAMINING DEPLOYMENT TRENDS • 1970s: Mainframes • 1980s: Minicomputers • 1990s: Unix servers • 2000s: Windows on x86, Linux on x86 • 2010s: Cloud computing, Serverless/FaaS
45. FaaS/Serverless/NoOps AWS Lambda: • Short lived functions • Triggered by events • Stateless • Auto scaling • Pay per 100ms of invocation
46. AWS LAMBDA EXAMPLE exports.handler = (event, context, callback) => { callback(null, "Hello from Lambda"); };
47. ACTORS object ActorA { def props: Props = Props[ActorA] } class ActorA extends Actor { def receive: Receive = { case x => println(s"Got: $x") } } // … in main class … val system: ActorSystem = ActorSystem("MicroserviceA") system.actorOf(ActorA.props) ! "Hey there!"
48. FUTURES import scala.util.{Success, Failure} val f: Future[List[String]] = Future { session.getRecentPosts } f onComplete { case Success(posts) => for (post <- posts) println(post) case Failure(t) => println("An error has occured: " + t.getMessage) }
49. SO, IT’S NOT REALLY NEW! Actors are location transparent Futures are anonymous blocks of code executed some time Serverless/FaaS just highlights a monitoring need that already exists!
50. PRODUCTION MONITORING
51. http://www.lightbend.com/resources/e-books
53. INTERESTING FRAMEWORKS
55. LIGHTBEND MONITORING
56. FEATURE LIST (2017-04) • Akka Actors • Akka remoting and clustering • Dispatchers/Thread Pools • MDC Propagation • Various backend integration (ES, StatsD, …) • Sandbox environment (EKG) for easy exploration
57. EXAMPLE INTEGRATION: DATADOG
58. UPCOMING FEATURES - Q2 2017 • More Akka cluster stuff! • Akka HTTP metrics • Expanded distributed tracing capabilities - OpenTracing • Integration enhancements
59. HOW TO GET IT • Free to use during development • Requires subscription to use in production Create, free, account to get started: https://www.lightbend.com/account/register Demo: https://demo.lightbend.com
60. THANKS FOR LISTENING! Q & A @h3nk3 henrik.engstrom@<company name>.com