Instrumenting CI Pipelines

Transcript

1. Instrumenting CI Pipelines @smithclay Testing in Production Meetup, San Francisco 2/20/2018
2. A BRIEF HISTORY OF OBSERVABILITY THOUGHT- LEADERSHIP CENTERED ON THE URBAN COAST OF THE WESTERN UNITED STATES intro. FOR CURIOUS DEVELOPMENT AND OPERATIONAL PROFESSIONALS
3. THAT TWITTER POST REGARDING OBSERVABILITY (credit: @gphat, https://blog.twitter.com/engineering/en_us/a/2013/observability-at-twitter.html fig. 1
4. THE THREE PILLARS OF OBSERVABILITY Metrics Logs Traces (credit: @copyconstruct, https://medium.com/@copyconstruct/logs-and-metrics-6d34d3026e38) fig. 3
5. THE bourgonIAN VENN DIAGRAM (credit: @peterbourgon, https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html) fig. 2
6. ? HOW SHOULD WE DEFINE 'OBSERVABILITY' FOR CI TOOLS? q. 1
7. AN EXPIERMENT IN INSTRUMENTING A CI PIPELINE IN THE CLOUD
8. CI PIPELINE: TYPICAL MONITORING BATTERIES INCLUDED https://github.com/capitalone/Hygieia (as seen at AWS re:Invent) (diagram: @peterbourgon)
9. CI PIPELINE: WHAT DOES TRACING LOOK LIKE? Waterfall/Gantt Chart 'CI run scoped (?)' (diagram: @peterbourgon)
10. CASE STUDY: AWS CodeBuild (no servers required!) Converting steps to segments in a trace
11. HOW TO INSTRUMENT THE PIPELINE? (I.E. NO OPERATING SYSTEM ACCESS) Log Events converted to a trace. Needs: timestamp and relationship to other events Optional: other useful attributes (i.e. container name) Traces are just related events! We can convert! (diagram: @peterbourgon)
12. COLLECTING TRACE EVENTS THE REALLY LAZY WAY tail -f $temp_file | ./traceformer -app-name DevOpsTest &> output.log & Just read events in from STDIN and convert them to a tracing format (in this case New Relic APM) with some hacked together golang code.
13. CI TRACES, VISUALIZED (APM STYLE) Latency breakdown of different build steps. Metrics created from traces.
14. CONNECTING CI TRACES TO PROD PERFORMANCE Attributes add additional context to traces. ... this can then be correlated with production performance.
15. LESSONs LEARNED INSTRUMENTING THE CI PIPELINE 1) If you have a complex CI pipeline, consider traces. 2) We should be connecting production performance to the processes (i.e. pipeline runs) that deployed and tested them. (diagram: @peterbourgon)
16. THANK YOU. @smithclay—New Relic Testing in Production Meetup, San Francisco 2/20/2018