How to select the righ EBS?

How to select the right EBS? by Ahmed Shendi during AWS MENA community days 2020.

Transcript

1. COMMUNITY DAY MENA How to select the right EBS
2. Ahmed Shendy Senior DevOps Engineer
3. COMMUNITY DAY MENA Agenda What is Amazon EBS Amazon EBS Volume Types Choosing an Amazon EBS volume type Performance Best Practices for EBS Saving Cost with EBS
4. COMMUNITY DAY MENA What is Amazon EBS? o Block storage as service o Create, attach volumes through an API o Service accessed over network o Volumes persist independent of EC2 o Detach and attach between instances within the same availability zone AWS Cloud Availability Zone NAT gateway Instance Instance Amazon EC2 Auto Scaling Amazon EC2 Amazon Elastic Block Store
5. COMMUNITY DAY MENA Amazon EBS Volume Types
6. COMMUNITY DAY MENA Amazon EBS Volume Types SSD HDD gp2 General Purpose SSD io1 Provisioned IOPS SSD st1 Throughput Optimized HDD sc1 Cold HDD
7. COMMUNITY DAY MENA Amazon EBS volume: General Purpose SSD Baseline: 100 to 16,000 IOPS @ 3 IOPS/GB Burst: 3,000 IOPS (< 1000 GB) Throughput: Up to 256 MiB/s Latency: Single-digit ms Capacity: 1 GiB to 16 TiB I/O Size: Up to 256 KiB (logical merge) gp2 General Purpose SSD Great for boot volumes, low-latency applications, and bursty databases
8. COMMUNITY DAY MENA Burst bucket: gp2 gp2 General Purpose SSD
9. COMMUNITY DAY MENA Burst and baseline: gp2
10. COMMUNITY DAY MENA How long can I burst on gp2? 0 0 150 300 450 600 750 Minutes of Burst 200 400 600 800 Volume size in GiB
11. COMMUNITY DAY MENA Amazon EBS volume: Provisioned IOPS Baseline: 100 – 64,000 IOPS Throughput: Up to 1000 MiB/s Latency: Single-digit ms Capacity: 4GiB – 16 TiB I/O Size: Up to 256 KiB (logical merge) io1 Provisioned IOPS Ideal for critical applications and databases with sustained IOPS
12. COMMUNITY DAY MENA Scaling Provisioned IOPS SSD : io1 0 0 40,000 20,000 30,000 10,000 50,000 64,000 IOPS 1.28 2 6 9 16
13. COMMUNITY DAY MENA Amazon EBS volume: Throughput Optimized Baseline: 40 MiB/s per TiB, up to 500 MiB/s Burst: 250 MiB/s per TiB, up to 500 MiB/s Capacity: 500 GiB to 16 TiB I/O Size: Up to 1 MiB (logical merge) st1 Throughput Optimized HDD Ideal for large-block, high-throughput sequential workloads
14. COMMUNITY DAY MENA Amazon EBS volume: Cold HDD Baseline: 12 MiB/s per TiB, up to 192 MiB/s Burst: 80 MiB/s per TiB, up to 250 MiB/s Capacity: 500 GiB to 16 TiB I/O Size: Up to 1 MiB (logical merge) sc1 Cold HDD Ideal for sequential throughput workloads, such as logging and backup
15. COMMUNITY DAY MENA Choosing an Amazon EBS volume type
16. COMMUNITY DAY MENA Understand your mission Databases PostgreSQL, MySQL Cassandra, MongoDB Data & Analytics Kafka, Splunk, Hadoop, Data Warehousing Media Transcoding, Encoding, Render Farms File CIFS, NFS Archive
17. COMMUNITY DAY MENA Select the right volume • Typically high performance requirements • Mostly random I/O • Journal is sequential • Highly workload dependent Recommendation: SSD Databases PostgreSQL, MySQL Cassandra, MongoDB
18. COMMUNITY DAY MENA Select the right volume • Typically high throughput requirements • Mostly sequential I/O • Sustained I/O Recommendation: Throughput Optimized HDD (st1) Media Transcoding, Encoding, Render Farms
19. COMMUNITY DAY MENA Select the right volume • Typically high throughput requirements • Mostly sequential I/O • Daily periodicity Recommendation: HDD Data & Analytics Kafka, Splunk, Hadoop, Data Warehousing
20. COMMUNITY DAY MENA Select the right volume • Typically low throughput requirements • Bursty workloads • Cost sensitive Recommendation: Cold HDD (sc1) File CIFS, NFS Archive
21. COMMUNITY DAY MENA How do AWS count I/O When possible, AWS logically merge sequential I/O, to minimize I/O charges on io1, and maximize burst on gp2, sc1, and st1 io1 and gp2 : up to 256 KiB st1 and sc1: up to 1 MiB
22. COMMUNITY DAY MENA How do AWS count I/O Example 1: Random I/Os: • 4 random I/Os (non-sequential) • Each I/O 64 KiB
23. COMMUNITY DAY MENA How do AWS count I/O Example 2: Sequential I/Os: • 4 sequential I/Os • Each I/O 64 KiB
24. COMMUNITY DAY MENA How do AWS count I/O Example 2: Sequential I/Os: • 4 sequential I/Os • Each I/O 64 KiB
25. COMMUNITY DAY MENA How do AWS count I/O Example 2: Sequential I/Os: • 4 sequential I/Os • Each I/O 64 KiB
26. COMMUNITY DAY MENA How do AWS count I/O Example 3: Large I/O: • 1 I/O • 1024 KiB
27. COMMUNITY DAY MENA How do AWS count I/O Example 4: Mixed I/Os: • 2 * 512 KB sequential I/O s • 2 * 64 KB random I/O s • 2 * 128 KB sequential I/O s
28. COMMUNITY DAY MENA Need more details
29. COMMUNITY DAY MENA Performance Best Practices for EBS
30. COMMUNITY DAY MENA EBS Optimized • Dedicated EBS bandwidth • Up to 19 Gbps (2,375 MiB/s) per instance • Burst available for smaller instances Tip: Ensure EBS optimized bandwidth can support your volumes
31. COMMUNITY DAY MENA EBS Optimized • Dedicated EBS bandwidth • Up to 19 Gbps (2,375 MiB/s) per instance • Burst available for smaller instances Tip: Ensure EBS optimized bandwidth can support your volumes
32. COMMUNITY DAY MENA Performance: EBS Optimized
33. COMMUNITY DAY MENA Best practices: RAID When to RAID? - Storage requirements > 16 TiB - Throughput requirement > 1,000 MiB /s - IOPS requirement > 64,000
34. COMMUNITY DAY MENA Best practices: RAID Avoid RAID for redundancy • EBS data is already replicated • RAID 1 halves available EBS bandwidth • RAID 5 / 6 loses 20 % to 30% of usable I/O to parity
35. COMMUNITY DAY MENA Monitor I/O characteristics using CloudWatch You can monitor the I/O characteristics with each volume's CloudWatch volume metrics. Important metrics to consider include the following: • BurstBalance • VolumeReadBytes • VolumeWriteBytes • VolumeReadOps • VolumeWriteOps • VolumeQueueLength BurstBalance: displays the burst bucket balance for gp2, st1, and sc1 volumes as a percentage of the remaining balance To determine your volume's average I/O size, divide VolumeWriteBytes by VolumeWriteOps. The same calculation applies to read operations. If your I/O latency is higher than you require, check VolumeQueueLength to make sure your application is not trying to drive more IOPS than you have provisioned.
36. COMMUNITY DAY MENA Saving Cost with EBS
37. COMMUNITY DAY MENA EBS Cost gp2 $0.10 per GiB io1 $0.125 per GiB $0.065 per PIOPS st1 $0.045 per GiB sc1 $0.025 per GiB Snapshot storage for all volume types is $0.05 per GiB per month * All prices are per month, prorated to the second, and from the us-west-2 Region as of Nov 2019
38. COMMUNITY DAY MENA Elastic Volumes: Size your volumes correctly!
39. COMMUNITY DAY MENA Tag Volumes and Snapshots on Create
40. COMMUNITY DAY MENA Set DeleteOnTermination = TRUE if storage lifecycle = instance lifecycle DeleteOnTermination = FALSE DeleteOnTermination = TRUE USE WITH CARE – You might delete volumes you need
41. COMMUNITY DAY MENA Tier to lower cost volume type if volume is not in use
42. COMMUNITY DAY MENA Don't use io1 unless you are sure that you need it
43. COMMUNITY DAY MENA Thank You