Just Enough Ops for Developers

Originally presented at DjangoCon US 2022

Most developers don’t want to think about operations (aka Ops, DevOps, etc.) PaaS providers (Heroku, Fly, Render, etc.) do an awesome job of getting your app live on the internet, but there’s a lot to ops beyond just deployment.

This talk will answer the following questions:

* How much CPU and memory should I give my services?
* How do I know if the app is overallocated (costing too much money) or underallocated (slow/overloaded)?
* How can I make my app faster?
* Will autoscaling help me save costs?
* What about serverless?

If you’re a developer and want to run your applications successfully without deep DevOps knowledge, this talk is for you. It will help if you have some basic Django developer experience, but other than that, no specific knowledge is necessary!

Transcript

1. Peter Baumgartner Just Enough Ops for Developers DjangoCon US 2022 @ipmb lincolnloop.com
2. About Me • Founder at Lincoln Loop — lincolnloop.com • Co-author of High Performance Django — highperformancedjango.com • Building AppPack — apppack.io
3. Prepping your project for production Watch my talk at DjangoCon 2019
4. Why just enough Ops?
5. PaaS & Managed Services are really good
6. Forget about 😴 • System security/hardening • Routing/networking • Secrets management • Deployments • Scaling how to, not when to • Process Management (systemd, docker, etc.) • Hardware failures
7. But! You need to understand the basics • CPU • RAM • I/O
10. CompSci 101
12. Disk (HDD, SSD) Persistent File Storage
13. • Disk access is slow • Usually ephemeral Disk (HDD, SSD) Persistent File Storage
14. CPU (processor) Code execution
15. How many CPUs do I need?
16. How many CPUs do I need? It depends
17. 1 request/1 process/1 CPU
18. https://fastapi.tiangolo.com/async/ Cook = CPU Cashier = Python Process Customer = Request
19. How many CPUs do I need? It depends
20. How many cooks do I need? It depends
21. More Users → More CPU Rule of 👍
22. 2 requests/1 process/1 CPU
23. 2 requests/2 process/2 CPU
24. 💸
25. How do I minimize cost?
26. How do I maximize CPU usage?
27. Multiple Processes per CPU How do I maximize CPU usage?
28. 2 requests/2 processes/1 CPU
29. 2 requests/2 processes/1 CPU
30. 2 requests/2 processes/1 CPU
31. gunicorn \ --workers=3 \ …
32. Tuning Worker Count Each app is different • Start at double your CPU count • You can experiment with adding workers until • Not enough memory • Response times plateau or degrade
33. Improve Application Performance How do I maximize CPU usage?
34. 1 CPU / 1 minute 100ms response time = 6000 requests 1s response time = 60 requests
35. ⏳Slow responses ⏱More CPU time 💰Higher cost
36. I/O CompSci 101
37. I/O (Input/Output) Examples • Reading/writing files • Database queries • Object storage (S3) • Search index queries • Third-party APIs
38. Everything waits for input/output
40. Synchronous I/O is blocking 🤓
41. 🤓 Asynchronous I/O is non-blocking
42. Async Django Views since Django 3.1 / ORM since Django 4.1
43. Asynchronous programming is hard 🤯
44. Jeff Atwood “Hardware is cheap, Programmers are expensive
45. gevent How do I maximize CPU usage?
46. 🪄 magic How do I maximize CPU usage?
47. 2 workers with gevent
48. gunicorn \ --worker-class=gevent \ --worker-connections=50 \ …
49. Not Enough CPU
51. Backlog
52. Error Codes • 502 Bad Gateway • 503 Service Unavailable • 504 Gateway Timeout
53. Memory (RAM) Ephemeral cache
54. • “Warm” code cache • Python objects • File processing/manipulation • Network sockets • File descriptors Memory (RAM) Ephemeral cache
55. • Fast • Limited Memory (RAM) Ephemeral cache
56. Memory usage for a typical Django app
57. 128 - 512+ MB Memory usage for a typical Django app
58. Memory usage for a typical Django app 128 - 512+ MB per process
59. 👷 4 gunicorn workers 📊 512 MB per process 🟰 2 GB of memory
60. Over 100% memory → ☠
61. Not enough memory? • ⬆ Increase allowed memory • ⬇ Reduce processes/workers
62. Memory usage should be stable
63. Stable Memory Usage
64. Variable (but Stable) Memory Usage
65. Django memory tips • Don’t read huge files into a string/byte object • Don’t process a huge queryset • Use Model.objects.iterator() • Use .values() to avoid creating a model instance • Use .only() to avoid loading large text fields
66. Memory Leak
67. Memory Leak Causes • Bug in C extension (no garbage collection) • Leaving file descriptors or network sockets open (use context managers) • Global objects (sometimes accidental)
68. gunicorn \ --max-requests=10000 \ --max-requests-jitter=500 \ …
69. Databases
70. Same rules apply
71. Remember: disk is slow 🌋 Database should fit in RAM Rule of 👍
72. Scaling
74. Horizontal Vertical
75. Autoscaling is great 🎉
76. Autoscaling is great 🎉
77. …but it’s not magic 🪄
78. Serverless
79. 1 request/1 process/1 CPU Serverless
80. Pricing Per request + Allocated resources per millisecond of response time Serverless
81. • Scaling • CPU Allocation • Application server (workers, gevent, etc.) Forget about 😴 Serverless
82. Worry about 😥 • Budgeting/variable costs • Cold starts • Database connections • Limitations (upload size, max duration) • Remote shell access Serverless
83. Final Thoughts 🧐 • Get to know your application — “observability” • CPU usage • Memory usage • Response times • Error rates/uptime
84. Thanks! 👋 @ipmb lincolnloop.com apppack.io