用恐龍書剖析 AWS Legacy Infra 的效能瓶頸

用恐龍書剖析 AWS Legacy Infra 效能瓶頸 3ric@DevOpsDays 2025/06/05 https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSAjPMB4-bmf2OJZZ6NLlFThjTzrQTvnK8D-Q&s

Senior Reboot Engineer

3 | ©2023 Trend Micro Inc. 趨勢科技 DevOps 人才招募中！ ▍
(Sr.) DevOps Engineer Help build and maintain the cloud infrastructure for Trend Vision One! • 1+ years of experience as a DevOps Engineer • Familiarity with cloud and containerization technologies such as AWS, Azure, and Kubernetes would be a big plus ▍ (Sr.) Staff DevSecOps Engineer Drive security and efficiency across Trend Micro’s entire software development lifecycle! • 7+ years of experience in software development (including 3+ years in DevOps) • Familiarity with cloud and containerization technologies such as AWS, Azure, and Kubernetes \ 更多資訊請見 104 /

Generated by chatGPT

Legacy Infra Long Polling 120s Generated by AWS Diagram MCP
Server + Claude 3.7 Sonnet

https://memeprod.ap-south-1.linodeobjects.com/user-template/bc9dab6e864a081e134b6b502041db74.png Generated by Claude 3.7 Sonnet

https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-cloudwatch-metrics.html

https://memeprod.sgp1.digitaloceanspaces.com/meme/de361a6f1a65f343c525c40f21f89e52.png

https://sre.google/sre-book/effective-troubleshooting/

Generated by AWS Diagram MCP Server + Claude 3.7 Sonnet

UnHealthy HostCount CLB 5XX Backend 5XX SurgeQueue Length Spillover Count
Route53 X X X X X CLB O O X O O EC2 O O O O O Elasticache X X O X X

Application / NodeJS CPU % MEM % Disk % Disk
I/O Network I/O Network Connections Open Files

https://down-tw.img.susercontent.com/file/8059c4cc16fd85bde4680472519616e5

https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html

https://gd-hbimg.huaban.com/007904a643021ad3c8a8c228d4fa21c55726d74125284-XCYoCi_fw658 https://down-tw.img.susercontent.com/file/8059c4cc16fd85bde4680472519616e5 https://im1.book.com.tw/image/getImage?i=https://www.books.com.tw/img/001/082/45/0010824592.jpg&v=5cf64854k&w=375&h=375

理論上一台Server可以接收多少TCP連線? C(lient)10k/100k問題假設Server提供服務的IP和Port只有一個 192.168.1.1:80

理論上一台Client可以發起多少TCP連線? 假設Client的IP只有一個

Source / Client Destination / Server Protocol Source IP Client
IP Source Port Client Port Destination IP Server IP Destination Port Server Port TCP 1 2 3 4 TCP Connection 4 Tuple

理論上一台Server可以接收多少TCP連線? 假設Server提供服務的IP和Port只有一個 192.168.1.1:80 Source / Client Destination / Server Protocol
Source IP Client IP Source Port Client Port Destination IP Server IP Destination Port Server Port TCP 2^32 42.9億 2^16 65536 192.168.1.1 固定 80 固定 TCP 固定

Source / Client Source Port Destination / Server Destination Port
Protocol 1 1.2.3.4 2^16 65536 10.20.30.40(固定) 443(固定) TCP(固定) 2 5.6.7.8 2^16 65536 10.20.30.40(固定) 443(固定) TCP(固定) 3 9.10.11.12 2^16 65536 10.20.30.40(固定) 443(固定) TCP(固定) 4 13.14.15.16 2^16 65536 10.20.30.40(固定) 443(固定) TCP(固定) 5 17.18.19.20 2^16 65536 10.20.30.40(固定) 443(固定) TCP(固定) N IPv4 2^32 42.9億 2^16 65536 10.20.30.40(固定) 443(固定) TCP(固定)

理論上一台Client可以發起多少TCP連線? 假設Client的IP只有一個 Source / Client Destination / Server Protocol Source
IP Client IP Source Port Client Port Destination IP Server IP Destination Port Server Port TCP 1.2.3.4 固定 2^16 65536 2^32 42.9億 2^16 65536 TCP 固定

Source / Client Source Port Destination / Server Destination Port
Protocol 1.2.3.4(固定) 2^16 10.20.30.40 80(固定) TCP(固定) 1.2.3.4(固定) 2^16 10.20.30.40 443(固定) TCP(固定) 1.2.3.4(固定) 2^16 10.20.30.60 443(固定) TCP(固定) 1.2.3.4(固定) 2^16 10.20.30.70 443(固定) TCP(固定) 1.2.3.4(固定) 2^16 10.20.30.80 443(固定) TCP(固定) 1.2.3.4(固定) 2^16 10.20.30.90 443(固定) TCP(固定)

實際上呢? •可用記憶體 •作業系統可開啟的最大檔案數量 •檔案系統的inode數量 •net.ipv4.ip_local_port_range (32768 60999) •etc

CLB ENI可以接收/發起多少連線? Generated by AWS Diagram MCP Server + Claude
3.7 Sonnet Agent->CLB(接收) CLB -> EC2 (發起)

Agents -> CLB ENI Public Source / Agent IP Destination
/ CLB Public Protocol Source IP Client IP Source Port Client Port Destination IP CLB IP Destination Port CLB Port TCP 看有多少 agent 預設 32768~60999 約28232個看有幾張 CLB ENI 443 TCP

CLB ENI Private -> EC2 Private Source / Client Destination
/ Server Protocol Source IP Client IP Source Port Client Port Destination IP Server IP Destination Port Server Port TCP 看有幾張 CLB ENI 預設 32768~60999 約28232個看有幾張 EC2 ENI 8888 TCP

111k Connections x2 ??? Connections 1M Connections 111k Connections 150k Connections

https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/ts-elb-error-message.html#ts-elb-errorcodes-http503

猜測是不是這個問題 •不管CLB ENI背後原理，依照連線組成定義角度 •假設問題是CLB來不及擴展ENI •或是 ENI可接收連線、可發起連線到上限， •導致CLB出現5XX •HealthyCheck也是透過TCP，造成UnHealthy等連鎖問題 •AWS沒有公開CLB擴展的邏輯

觀察到的有趣現象 •Agent數量相同 •A Region CLB ENI 數量到 N 就上不去了 •B
Region CLB ENI 數量會超過 N •為什麼?

https://memeprod.ap-south-1.linodeobjects.com/user-template/fca06d622e0014ada9701a7fe171fafc.png

https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html

https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html TCP 4-Tuple

NLB

AWS文件並沒有說明NLB架構和背後技術 https://media.istockphoto.com/id/1407160246/zh/%E5%90%91%E9%87%8F/danger-triangle-icon.jpg?s=612x612&w=0&k=20&c=XhxcETr5nz171Jv_vIM9JSFBtkD7JwS6egp7GHIr58w=

https://research.google/pubs/maglev-a-fast-and-reliable-software-network-load-balancer/ 不過Google有公開他們的NLB論文

Google NLB AWS NLB OSI Layer L4 L4 靜態IP Virtual
IP (VIP) AWS Elastic IP (EIP) 延遲 Anycast 效能 Kernel bypass (類似 Zero Copy) 黑魔法 Y Y

CLB -> NLB https://media.istockphoto.com/id/1327689563/zh/%E5%90%91%E9%87%8F/icon-emoji-party-confetti-in-clubhouse-social-network-happy-birthday-cracker-isolated- vector.jpg?s=612x612&w=0&k=20&c=U0An07nn_Q5ZaCaG9MkAJefLCgo3SzvqnJXXRz4uBNY=

Key Takeaways

Thank You! https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSAjPMB4-bmf2OJZZ6NLlFThjTzrQTvnK8D-Q&s

用恐龍書剖析 AWS Legacy Infra 的效能瓶頸

用恐龍書剖析 AWS Legacy Infra 的效能瓶頸

More Decks by Eric

Other Decks in Technology

Featured

Transcript