FaaS Measurement Fundamentals

"Don't Worry About Servers, Still Worry About Metrics: FaaS Measurement Fundamentals"

Talk at Gluecon 2017.

Transcript

1. Don't Worry About Servers Still Worry About Metrics FaaS Measurement Fundamentals @smithclay New Relic Gluecon 5/24/17
2. Metrics are what we measure *hopefully useful things
3. λ This Thing Appeared The Magic Hat of Werner Vogels How do we understand it?
4. Hyped tech wish list Metrics Trends Alerting CTA Logging Detail Tracing Cause Analytics All of the above
5. MALTA Observability Index for FaaS Metrics Alerting Logging Tracing Analytics Maturity Level
6. Built-in FaaS Metrics* Error Count Function Invocation Count Function Duration (ms) * not comprehensive, but the important ones
7. Why does function invocation time vary so much?
8. Event Trigger 1. Invoke λ 2. Run 3. End Result Error Timeout or or
9. Cold Start vs Warm Start Event Trigger Handler Code Warm Function Invocation Time Create Initialize Handler Code Cold
10. What's inside AWS Lambda? "It's containers" — Person waving their hands
11. λ: Running Commands for Discovery const exec = require('child_process').exec; exports.handler = (trigger, cb) => { exec('whoami', (err, stdout) => { console.log(stdout); return cb(null); }); } [LOG TIME] sbxuser_1066
12. λ is a UNIX system?! I know this!
13. Let's run SSH in λ λ ssh process SSH Tunnel Firewall: no inbound ports
14. SSH in Lambda Architecture λ node.js wrapper go sshd binary (x64) Go SSH Crypto Libs process.exec() https://github.com/smithclay/faassh
15. Max Session Length: 5 minutes (custom prompt optional) https://github.com/smithclay/faassh
16. Info from /proc 2x Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz cat /proc/cpuinfo 3857664 kB cat /proc/meminfo ixgbevf (EC2 10Gbps Network Driver) cat /proc/modules c4.large EC2 Compute-Optimized Instance (?)
17. c4.large instance λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λs in theory on a VM 10 Gbps λ = Running 128 MB Function λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ = Not Running 128 MB Function
18. Frozen functions help avoid cold starts https://www.kernel.org/doc/Documentation/cgroup-v1/freezer- subsystem.txt cgroup freezer subsystem λ λ λ
19. Internals Recap It's just containers on a VM Functions frozen when not running No magic unikernels involved
20. How do we measure and prevent cold starts? "Use Kubernetes" — Troll
21. Cold Start Discovery var SO_SO_COLD = true; exports.handler = function(trigger, cb) { console.log('Cold? %s', SO_SO_COLD); SO_SO_COLD = false; return callback(cb) } https://github.com/smithclay/lambda-proc-info
22. Warming automation Scheduled Event λ Only effective for non- concurrent execution! 4 minute interval http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html
23. Sending cold start events to analytics λ console.log(coldStart) Logs λ POST to Event DB (Insights) CloudWatch Log Filter Trigger
24. Cold Starts Visualized ~7 hrs ~8 hrs
25. λ Host Uptime Cold starts happen when hosts change! ~8hrs
26. λ Host Subnet Hopping 10.13 10.12 10.11 10.13 10.12 10.12 10.12 10. 11 # of AZs in us-west-2: 3
27. What's the maximum concurrency of your function? one > 1 A scheduled event will warm it until host retires. More advanced strategy needed*
28. "Advanced" Strategy for i in `seq 1 $NUM_EXECUTIONS`; do echo "[$i] Executing $AWS_LAMBDA_FUNCTION_NAME..." aws lambda invoke ... done https://gist.github.com/smithclay/e89dfe35fe2a4938db56bb12df76777c
29. Multiple containers running on a single host to serve parallel requests Tracking /proc/sys/kernel/random/boot_id and hostname Confirmed: cold start happens on container init.
30. So is this just a container PaaS? High-availability/multiple zones Elastic fleet of compute-optimized VMs A very good scheduling algorithm Design (freezing, limits, etc) for very fast invocation Only if your PaaS has...
31. FaaS in Production Reality λ λ Dev Prod Orchestration (!?) Version/Deploy Monitoring Security Cold Start Mgmt The "learning cliff" Great tweet from @mfdii
32. FaaS Isn't a Silver Bullet "I've got a fast, computationally-intensive task that I need to perform occasionally in response to a well-defined event that isn't that sensitive to latency." —The Ideal FaaS Developer // TO DO: measure && share results
33. Thanks. @smithclay New Relic Gluecon 5/24/17