The frugal developer

Transcript

1. The frugal developer
2. Developers Guide to Cost Optimisation _
3. A tale in three acts: Plan, Build, and Run
4. Plan_
5. Cost Optimisation > Minimsation _
6. Plan_ |- Architecture
8. It’s crucial to find the right balance between your technical and business needs – to find the sweet spot that aligns with your risk tolerance and budget. Remember, frugality is about maximizing value, not just minimizing spend. And to do that, you need to determine what you’re willing to pay for.
9. > Switch to Graviton processor architecture
11. > Switch to Graviton processor architecture > Design Architecture
12. > Switch to Graviton processor architecture > Design Architecture > Use managed services
13. “ DynamoDB has reduced prices for on-demand throughput by 50% and global tables by up to 67% ”
14. cloud Managed Open Source Self Manage •Open Source technology skill and expertise •Installation and configuration of OSS •Scaling and performance tuning •Security configuration •Patching – updates and security fixes •Upstream contributions and fixes •Support of open source technology •Infrastructure patching – security and functional •Compute orchestration, provisioning •Cluster scaling •Physical hardware, host OS/kernel, networking, and facilities •Physical data centre capabilities Self host •Use open source technology Customer Manages
15. Plan_ |- Architecture |- Pushing back on requirements
16. > Make cost a non functional requirement
18. > Make cost a non functional requirement > Systems that last align cost to business
20. Build_
21. Build_ |- Run local
22. Localstack SAM Finch TestContainers mwaa-local-runner aws-lambda-runtime-interface-emulator CDK
23. Build_ |- Run local |- Shut down dev servers
24. Creating ephemeral dev environments
26. Build_ |- Run local |- Shut down dev servers |- Developing for spot
27. Build_ |- Run local |- Shut down dev servers |- Developing for spot |- Code optimisation
28. > Cumulative API calls
29. import boto3 import time s3 = boto3.client('s3') bucket_name = "your-bucket-name" # List the bucket contents 1000 times (excessive API calls) for _ in range(1000): objects = s3.list_objects_v2(Bucket=bucket_name) # Download the same files over and over instead of caching for obj in objects.get("Contents", []): key = obj["Key"] s3.download_file(bucket_name, key, f"/tmp/{key}") # Re-downloading every time # Uploading large file in small chunks without using multipart upload with open("large_file.dat", "rb") as f: while chunk := f.read(1024): # Reading in tiny 1KB chunks s3.put_object(Bucket=bucket_name, Key="uploads/large_file.dat", Body=chunk)
30. > Cumulative API calls > Database optimisations
31. import boto3 dynamodb = boto3.client("dynamodb") table_name = "your-dynamodb-table" # Scanning entire table instead of querying specific records response = dynamodb.scan(TableName=table_name) # Process items (even if we only need a few) for item in response.get("Items", []): print(item)
32. > Cumulative API calls > Database optimisations > Container image size
33. > Cumulative API calls > Database optimisations > Container image size > Refactoring and upgrading
35. "We have deployed ACCP to thousands of cloud instances hosting our source of truth Apache Cassandra databases as well as our performance sensitive Memcached tiers. ACCP is successfully reducing crypto overhead by up to 90% in various workloads primarily TLS and message digests. For example, ACCP accelerates our internode communication, quorum data reads, anti-entropy repair, and continuous S3 backups. We were pleased with how easy it was to integrate ACCP with both off-the-shelf and in-house Java software.” Joey Lynch Netflix Cloud Data Engineering Team Up to 90% crypto overhead reduction
36. Build_ |- Run local |- Shut down dev servers |- Developing for spot |- Code optimisation |- Automate everything
37. > Use Amazon Q Developer to generate automate code
38. > Use Amazon Q Developer to generate automate code > Optimise infrastructure as code
39. Run_
40. Run_ |- Debugging costs
41. Run_ |- Debugging costs |- Optimise your serverless workloads
43. > Cold start optimisation
44. > Cold start optimisation > Optimise your dependencies
46. > Cold start optimisation > Optimise your dependencies > Set the right logging levels
47. > Cold start optimisation > Optimise your dependencies > Set the right logging levels > Review Well Architected best practices
49. Tools_
50. > P: Amazon Q Developer
51. > P: Amazon Q Developer > B: OSS Tools
52. > P: Amazon Q Developer > B: OSS Tools > B: Infracost (Terraform)
53. https://github.com/infracost/infracost
54. > P: Amazon Q Developer > B: OSS Tools > B: Infracost (Terraform) > B: AWS Lambda Power Tuning
55. > P: Amazon Q Developer > B: OSS Tools > B: Infracost (Terraform) > B: AWS Lambda Power Tuning > R: Autospotting and Saving Expert (OSS)
57. > P: Amazon Q Developer > B: OSS Tools > B: Infracost (Terraform) > B: AWS Lambda Power Tuning > R: Autospotting and Saving Expert (OSS) > R: Steampipe with Thrifty mod
59. Takeaway_
60. Takeaway_ |- You have options, but starting early is key |- There are some great tools can use |- Amazon Q Developer can help you optimize and reduce your costs
61. https://s12d.com/builder-id Get started with Amazon Q Developer for free today
62. https://www.thefrugalarchitect.com/
63. Thank you_ @094459@hachyderm.io https://www.linkedin.com/in/ricardosueiras 094459.bsky.social https://github.com/094459