Ask an OpenShift Admin | Ep 132 | Multi-Cluster Observability

Christian Stark and Roger Floren are guests on this weeks Ask an OpenShift Admin Livestream - check it out here: https://www.youtube.com/watch?v=wUT1F0Vvlkk

They're talking about mutlicluster observability!

Transcript

1. Ask an OpenShift Admin MultiCluster Observability Date: • July 31th, 2024 1
2. 2 Red Hat Advanced Cluster Management What’s new in 2.11 - Highlights ▸ Expanded Kubernetes support matrix ▸ Automated ROSA import - DP ▸ Global Hub Search - TP ▸ Enhanced OLM operator integration - GA ▸ Policy violation debug (diff) in the UI ▸ OpenShift Virtualization enhancements ▸ Right-sizing - Enhanced DP ▸ Fine-grained RBAC for RHACM Observability - TP
3. OpenShift Virtualization enhancements Observe containerized and virtualized workloads across the hybrid cloud 3 Red Hat Advanced Cluster Management for Kubernetes 2.11 Use enhanced Search to quickly see all virtual machines on your fleet. Gain deeper insights and visibility into your OpenShift Virtualization inventory with a ready-to-use dashboard. Generally Available (GA)
4. RHACM Observability v2.11 4
5. Components/Architecture 5 Components: ▸ Hub side ･ MultiCluster observability operator ･ Thanos components ･ Grafana ･ AlertManager ▸ Managed cluster side ･ Endpoint operator ･ Metrics collector ･ Prometheus and exporters (Architecture diagram) Export metrics to 3rd party tools Kafka Victoria Metrics
6. Console Navigation 6 Click link “Grafana” in “Clusters” page, navigate to Grafana console
7. What’s New in this Release 7 ▸ ACM-8879 - [Dev Preview] T-Shirt Sizing configuration for ACM Observability instances ▸ ACM-9865 - Observability Support for Microshift ▸ ACM-10865 - [Tech-preview] support fine-grain RBAC
8. ACM-8879 - [Dev Preview] T-Shirt Sizing configuration for ACM Observability instances 8 Control the scale of ACM Observability without AdvancedConfig! With Instance Sizes, users can now configure a set of resource requests across all their Observability components, sized proportionally, using a single field in their MCO CR, InstanceSize (provided Hub cluster has enough resources). Supported sizes: default, minimal, small, medium, large, 2xlarge, 4xlarge More details in: https://github.com/stolostron/stolostron/tree/main/dev-preview -
9. ACM-9865 - Observability Support for Microshift 9 Feature Description/Why Microshift spokes were not supported due to various issues. They are now partially supported: • All spoke components are running like any *KS spoke, including prometheus. • Some metrics like ETCD ones are not scraped yet. How to Enable/Use Import the cluster as any other cluster. -
10. ACM-10865 - [Tech-preview] support fine-grain RBAC 10 Feature Description/Why - By default, RBAC is enforced per managed cluster (all or none) - Large managed clusters shared by multiple teams / applications in the organization want metrics access restricted to just their application's metrics Fine-grain RBAC provides Namespace level granularity by limiting access to specific namespaces. How to Enable/Use 1. Define Cluster roles for granular metrics access, rules define - apiGroup: cluster.open-cluster-management.io # always - resources: managedclusters # always - resourceNames: <managed cluster names> - verbs: metrics/<namespaces> - metrics/* indicates all namespaces # special case - (no other wild cards allowed) 2. Define Cluster role bindings to bind users to groups Documentation link here kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: ocm-metrics-access rules: - verbs: - metrics/open-cluster-management-agent - metrics/open-cluster-management-agent-addon apiGroups: - cluster.open-cluster-management.io resources: - managedclusters resourceNames: - devcluster1 - devcluster2 kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: ocm-metrics-access-binding subjects: - kind: User apiGroup: rbac.authorization.k8s.io name: user1 roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ocm-metrics-access
11. Known Issues 11 ▸ cluster-monitoring-config config map has some options overwritten by ACM Observability (see https://issues.redhat.com/browse/ACM-11724 or https://github.com/stolostron/rhacm-docs/pull/6518 for more details). ･ Planned to be fixed in 2.12.
12. RHACM Observability Architecture 12
13. ▸ Metrics and alerting only ▸ All-in-one solution provisioned by ACM ▸ User experience unified in Grafana ▸ One-click fleetwide monitoring stack configuration ▸ Integration with custom-tailored storage/forwarding systems What is OpenShift Observability today? 13 An All-In-One Solution with an Opinionated Design ACM Multi-Cluster Observability as of today
14. ▸ MCOA packaged as an additional addon w/ MCO ▸ Standalone community Thanos Operator ▸ Native CRD driven configuration, distributed and managed via addon framework 14 “Observe a fleet like a single cluster” Future ACM Observability
15. Multi-Cluster Observability Configuration Learning From Our Own OCP/ACM Ecosystem Key Objectives • Customer flexibility ◦ Selecting only the most relevant signals ◦ Adapting to different infrastructure • Cost efficiency ◦ Maximizing single-cluster solutions • Single-pane control ◦ Providing a single pane-of-glass across the fleet
16. Thanos-Operator 16 Why standalone Thanos Operator? ▸ Kubebuilder architecture with full lifecycle control of Thanos components ▸ Closely built with focus on compatibility with widely used Prometheus Operator (OCP CMO is based on this) ▸ Community-friendly, expecting more and more contributions from upstream users/adopters of Thanos ▸ No upstream widely-adopted operator for Thanos exists yet ▸ In ACM, benefits of opinionated Thanos Operator CRDs and Red Hat guided customisation capabilities