Grafana Alloy Best Practice

Speaker: Eric Huang
Event: COSCUP 2024

Transcript

1. Grafana Alloy Best Practice
2. Eric Huang LINE Taiwan / SRE 2021 : E-SUN bank 2022 : LINE Taiwan Kubernetes, Rust, eBPF 2 titaneric chen-yi-huang
3. 01 Introduction
4. • Collect metrics from client side & browser • Monitor web application performance • Discover error • Track user behavior (session) Real User Monitoring (RUM) Source: Web Vitals, User-centric performance metrics, Grafana Faro OSS 4
5. (Distributed) Tracing Source: Observability primer | OpenTelemetry 5 • Represent the full journey of request though distributed environment • Improve the visibility of the app • Diagnose the source of error
6. (Distributed) Tracing 6 Source: COSCUP 2024
7. “Alloy is a flexible, high performance, vendor- neutral distribution of the OpenTelemetry Collector” Key features: • Custom components • Chained components • Debugging utilities Adopt faro.receiver component with Faro SDK Grafana Alloy Source: Grafana Alloy | Grafana Alloy documentation 7
9. “Grafana Faro includes a highly configurable web SDK for real user monitoring that instruments browser frontend applications to capture observability signals.” Key features: • Monitoring applications performance • Captures errors, logs, user activity • Instrument performance and observe full stack Grafana Faro Web SDK Source: Grafana Faro OSS | Web SDK for real user monitoring (RUM) 9
11. 02 Results
12. End-to-End Tracing Spans include: • frontend app (nextjs) • ingress controller (traefik) • web framework (flask) • http client library (requests) 12
13. RUM dashboards official RUM dashboard: Loki datasource 13
14. RUM dashboards improved RUM dashboard: Prometheus datasource 14
15. Session/Trace Explore 15
16. Session Detail 16
17. 03 Present Architecture
18. Technology stack 18 Infra managed by JP, KR Serve managed by TW SRE
19. Architecture (User) 19
20. Architecture (SRE) 20
21. How to design? 04
22. Requirements Must have: • Adopt present observability platform • Easy to deploy alloy service automatically • Control traffic load sent from real user Nice to have: • Easy application for new tenant • Slack workflow • Sample code for SSR and CSR app • Nextjs based sample app 22
23. Alloy Architecture (User) 23
24. Alloy Architecture (SRE) 24
25. Why? • Adopt gateway instead of individual ingress for each cluster? • Unified traffic control by SRE • Decouple the business logic and telemetry traffic • Easy deployment for alloy • Cut down the Security Review procedure 25
26. Why? • Choose Contour instead of Traefik, or other ingress controller? • Contour is more performant and less memory consumption • Envoy Gateway is considered, but k8s version is not compatible 26
27. • Handle incoming large amount of traffic • Load test and tuning for Contour and Alloy • 3 levels of protections 1. Client side sampling 2. Contour rate limit 3. Grafana Alloy rate limit • Increasing load from Loki and Tempo • Continuously tuning for Loki and Tempo • Individual rate limit for each tenant Challenges Load Test Report: Alloy: 1500 RPS (1 core, 1Gi) Envoy: 10000 connection (3 core, 1Gi) 27
28. • Web vitals is stored in Loki instead of Prometheus • Adopt Loki Rulers to ingest Loki query result into Prometheus • Faster loading for real user monitoring dashboard • Constrained trace propagation in present architecture • Upgrade or update the trace propagation in the intermediate • Block trace propagation header from API gateway • Add allowed list for trace context header (e.g., TraceParent, Uber-Trace-Id) Challenges 28
29. Challenges 29 Source: DevOpsDays Taipei 2024 Source: DevOpsDays Taipei 2023
30. Future work 05
31. • Upgrade Traefik to v3.0 to adopt OpenTelemetry • Resolve the issue of unbalanced requests to OTEL collector • Zero-code instrumentation by eBPF (e.g., Grafana Beyla) • Continuously tuning for Tempo, Loki, and Alloy 31
32. Q&A 06