Contexts in Context

The context type allows deadlines, cancelation signals, and other request-scoped values, to propagate through request chains and across API boundaries, but in practice how can these features be used? In this talk Matt will talk about how Monzo use the humble context to power a variety of features in their Go based microservice platform.

Transcript

1. Contexts in Context Matt Heath
2. ➡ ➡ ❓ ➡
3. Hi, I’m Matt
4. @mattheath
11. Nov 2015 Apr 2019
12. Nov 2015 Apr 2019
13. Nov 2015 Apr 2019
14. Nov 2015 Apr 2019
15. Nov 2015 Fastest growing UK Bank 1.8M Customers Apr 2019
21. package context type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} }
22. package context type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} }
23. package context type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} }
24. package context type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} }
25. package context type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} }
26. package context type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} }
27. ctx := context.Background()
28. ctx := context.Background() ctx = context.WithValue(ctx, "TraceID", "boop")
29. ctx := context.Background() ctx = context.WithValue(ctx, "TraceID", "boop") ctx.Value("TraceID") => "boop"
30. ctx := context.Background() ctx = context.WithValue(ctx, "TraceID", "boop") ctx.Value("TraceID") => "boop" Contexts are immutable
31. func(ctx, ...)
32. func(ctx, ...) func(ctx, ...)
33. func(ctx, ...) func(ctx, ...)
34. func(ctx, ...) func(ctx, ...) func(ctx, ...) func(ctx, ...) func(ctx, ...)
35. func(ctx, ...) func(ctx, ...) func(ctx, ...) func(ctx, ...) func(ctx, ...)
36. func(ctx, ...) func(ctx, ...) func(ctx, ...) func(ctx, ...) func(ctx, ...)
38. func(ctx, ...) req.WithContext(ctx)
39. func(ctx, ...) req.WithContext(ctx) Remote API
40. func(ctx, ...) req.WithContext(ctx) Remote API ⏱
41. func(ctx, ...) req.WithContext(ctx) Remote API ⏱ Deadline Exceeded err
42. package context type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} }
47. LICENCE WITH RESTRICTIONS WE ARE HERE A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! FEB 2015 JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY JUN JUL MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC PRE APPLICATION APPLICATION MOBILISATION LAUNCH LICENCE WITH RESTRICTIONS 50K MAX DEPOSIT WE ARE HERE A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY JUN JUL MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC E APPLICATION APPLICATION MOBILISATION LAUNCH LICENCE WITH RESTRICTIONS WE ARE HERE AUG
 2016 JAN
 2016 Feb
 2015 APR
 2017 A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! FEB 2015 JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC PRE APPLICATION APPLICATION MOBILISATION LICENCE WITH RESTRICTIONS WE ARE HERE
48. LICENCE WITH RESTRICTIONS WE ARE HERE A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! FEB 2015 JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY JUN JUL MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC PRE APPLICATION APPLICATION MOBILISATION LAUNCH LICENCE WITH RESTRICTIONS 50K MAX DEPOSIT WE ARE HERE A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY JUN JUL MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC E APPLICATION APPLICATION MOBILISATION LAUNCH LICENCE WITH RESTRICTIONS WE ARE HERE AUG
 2016 JAN
 2016 Feb
 2015 APR
 2017 A UK banking licence is authorised by the PRA and regulated by the PRA and FCA1, allowing deposit- taking and balance sheet lending. Once granted, it allows firms to passport across Europe, accessing This is followed by a “mobilisation” phase during which final capital is raised and IT systems are completed, before launching to the public. We received a UK banking licence in August 2016! FEB 2015 JAN 2016 JAN 2017 MAR FEB FEB APR MAR MAR APR MAY APR JUN MAY JUL JUN AUG JUL SEP AUG NOV SEP DEC NOV DEC PRE APPLICATION APPLICATION MOBILISATION LICENCE WITH RESTRICTIONS WE ARE HERE 2 YEARS AGO!!
50. ?
51. ☁
52. ⛈
53. ⛅
54. Application
55. Application
56. APPLICATION Application
59. Feb 2015 Apr 2019 1000 services
68. package slog
 
 func Error(ctx context.Context, msg string, params ...interface{}) { if l := DefaultLogger(); l != nil { l.Log(Eventf(DebugSeverity, ctx, msg, params...)) } }
69. package slog
 
 func Error(ctx context.Context, msg string, params ...interface{}) { if l := DefaultLogger(); l != nil { l.Log(Eventf(DebugSeverity, ctx, msg, params...)) } }
70. func thing(ctx context.Context, ...) {
 // do things... 
 slog.Error(ctx, “Oh no!”, err)
 
 }
71. func thing(ctx context.Context, ...) {
 // do things... 
 slog.Error(ctx, “Oh no!”, err)
 
 }
72. 2019-04-17 15:38:09.134539276 [DBG] Slog -> Auth request: 
 {"auth_type": "Auth", "merchant": {"merchant_description": "Deliveroo! LONDON UK", "merchant_id": "XXXX"}, "network": “Mastercard" ...}
 {opentracing_id=2855967a9de91afb, trace_id=19ce030f-f652-4b4d-5f35-5d75f472f482, service=service.blah, service_version=d7b742fe5c03998748348fe9d7d81a6b8c7358d, hostname=s- blah-79f9b54887-j62d2}
 
 (handle_auth_message.go:25 handleAuthMessage)
73. 2019-04-17 15:38:09.134539276 [DBG] Slog -> Auth request: 
 {"auth_type": "Auth", "merchant": {"merchant_description": “Deliveroo! LONDON UK", "merchant_id": "XXXX"}, "network": “Mastercard" ...}
 {opentracing_id=2855967a9de91afb, trace_id=19ce030f-f652-4b4d-5f35-5d75f472f482, service=service.blah, service_version=d7b742fe5c03998748348fe9d7d81a6b8c7358d, hostname=s- blah-79f9b54887-j62d2}
 
 (handle_auth_message.go:25 handleAuthMessage)
74. API
75. API Service API Service API Service API Service API Service API Service API Service API Service Edge Gateway
76. Accounts Cards Pots Feed Payments … Edge Gateway
77. Cards Pots Payments … Accounts New
 Exciting
 API!! Feed Edge Gateway
78. Cards Pots Payments … Accounts New
 Exciting
 API!! Feed Edge Gateway
79. Edge Gateway Accounts Cards Pots Payments … New
 Exciting
 API!! Feed
80. Load Balancer Edge Gateway
81. Load Balancer Edge Gateway Request ID: 19ce030f-f652-4b4d-5f35-5d75f472f482
82. // Generate new ID // Store value in context tid, err := NewTraceID() // v4 UUID if err == nil { req.Context = context.WithValue(req.Context, "Trace-Id", tid) }
83. // Generate new ID // Store value in context tid, err := NewTraceID() // v4 UUID if err == nil { req.Context = context.WithValue(req.Context, "Trace-Id", tid) }
84. // Generate new ID // Store value in context tid, err := NewTraceID() // v4 UUID if err == nil { req.Context = context.WithValue(req.Context, "Trace-Id", tid) }
85. Load Balancer Edge Gateway Request ID: 19ce030f-f652-4b4d-5f35-5d75f472f482 Request ID: 19ce030f-f652-4b4d-5f35-5d75f472f482 API Service
86. Service A Service B Load Balancer Edge Gateway Request ID: 19ce030f-f652-4b4d-5f35-5d75f472f482 Request ID: 19ce030f-f652-4b4d-5f35-5d75f472f482 Request ID: 19ce030f-f652-4b4d-5f35-5d75f472f482 API Service
87. Service Service
88. Service Service Context to Headers Headers to Context HTTP
89. // Add headers previously stashed in the context if stash, ok := req.Context.Value(ctxKey).(http.Header); ok && stash != nil { for k, v := range stash { if req.Header[k] == nil { req.Header[k] = v } } } // Stash headers in the context stash := http.Header{} for k, v := range req.Header { for _, r := range passedHeaders { if r.MatchString(k) { stash[k] = v } } } if len(stash) > 0 { req.Context = context.WithValue(req.Context, ctxKey, stash) }
90. // Add headers previously stashed in the context if stash, ok := req.Context.Value(ctxKey).(http.Header); ok && stash != nil { for k, v := range stash { if req.Header[k] == nil { req.Header[k] = v } } } // Stash headers in the context stash := http.Header{} for k, v := range req.Header { for _, r := range passedHeaders { if r.MatchString(k) { stash[k] = v } } } if len(stash) > 0 { req.Context = context.WithValue(req.Context, ctxKey, stash) }
91. // Add headers previously stashed in the context if stash, ok := req.Context.Value(ctxKey).(http.Header); ok && stash != nil { for k, v := range stash { if req.Header[k] == nil { req.Header[k] = v } } } // Stash headers in the context stash := http.Header{} for k, v := range req.Header { for _, r := range passedHeaders { if r.MatchString(k) { stash[k] = v } } } if len(stash) > 0 { req.Context = context.WithValue(req.Context, ctxKey, stash) }
92. // Add headers previously stashed in the context if stash, ok := req.Context.Value(ctxKey).(http.Header); ok && stash != nil { for k, v := range stash { if req.Header[k] == nil { req.Header[k] = v } } } // Stash headers in the context stash := http.Header{} for k, v := range req.Header { for _, r := range passedHeaders { if r.MatchString(k) { stash[k] = v } } } if len(stash) > 0 { req.Context = context.WithValue(req.Context, ctxKey, stash) }
93. Service Service Context to Headers Headers to Context HTTP
94. Service Service Context to Headers Headers to Context
95. Service Service ✨ ✨ ✨ HTTP HTTP HTTP
99. Correlate logs Observe database queries Queue publishing Distributed locks
100. Correlate logs Observe database queries Queue publishing Distributed locks Understand request flow What’s slow?! What’s broken?!
103. Service Service
104. Service Service cancel() or ctx.Done()
105. Service Service x cancel() or ctx.Done() Terminate connection (http1)
 Cancel request (http2)
106. Service Service cancel() or ctx.Done() context cancelled Terminate connection (http1)
 Cancel request (http2) x
107. Cancel downstream requests
108. Cancel outstanding work! Decrease tail latencies! Reduce load on your database!
109. Cancel outstanding work! Decrease tail latencies! Reduce load on your database!
110. rspCh := make(chan Response, 1) go dispatchRequest(ctx, rspCh)
111. rspCh := make(chan Response, 1) go dispatchRequest(ctx, rspCh) select { case rsp := <-rspCh: return rsp case <-ctx.Done(): return ctx.Error() }
112. rspCh := make(chan Response, 1) go dispatchRequest(ctx, rspCh) select { case rsp := <-rspCh: return rsp case <-ctx.Done(): return ctx.Error() }
113. rspCh := make(chan Response, 1) go dispatchRequest(ctx, rspCh) select { case rsp := <-rspCh: return rsp case <-ctx.Done(): return ctx.Error() }
114. // check if the context is cancelled! 
 select { case <-ctx.Done(): return ctx.Error() default: } wooooooo… let’s go!
115. Cancel outstanding work! Decrease tail latencies! Reduce load on your database!
116. Cancel outstanding work! Decrease tail latencies! Reduce load on your database!
 
 Unless operations should complete…
117. Service A Service B Load Balancer Edge Gateway API Service
118. Service A Service B Load Balancer Edge Gateway API Service Client disconnects
119. Service A Service B Load Balancer Edge Gateway API Service Client disconnects Cascading failure!! Errors everywhere!! Intense panic!
120. func(ctx, ...) func(ctx, ...) func(ctx, ...) func(ctx, ...) func(ctx, ...)
121. func(ctx, ...) func(ctx, ...) func(context.Background(), ...) func(ctx, ...) func(ctx, ...)
122. func(ctx, ...) func(ctx, ...) func(context.Background(), ...) func(ctx, ...) func(ctx, ...) DANGER ZONE
123. Request scoped variables Dependency Injection Understand your systems.
125. monzo.com/careers