Upgrade to Pro — share decks privately, control downloads, hide ads and more …

kube-prometheusを気軽にKustomize

 kube-prometheusを気軽にKustomize

Prometheus Operator を気軽にインストールして使うために、kube-prometheusをKustomize して使います。

O. Yuanying

May 28, 2020
Tweet

More Decks by O. Yuanying

Other Decks in Technology

Transcript

  1. Who am I ? • OTSUKA, Motohiro / Yuanying ◦

    Z Lab Corporation ◦ 技術 ▪ Ruby (2007 - ) ▪ Cloud Foundry (2012 -) ▪ Docker (2013 - ) ▪ Python (2014 - ) ▪ Kubernetes / OpenStack (2014 - ) ▪ Go (2017 - ) ◦ 趣味 ▪ ロードバイク (2009 - ) ▪ トライアスロン (2010 - )
  2. Currently I’m interested in... 自宅: (本番環境) 1. Control Plane: Raspberry

    Pi 4 4GB x3 2. Worker Node: ECS Liva Z N4200 x3 3. Storage Node: ASRock J3160-ITX 実家: (テスト環境) 1. VM 4GB x7 on NUC x2
  3. Master Master Master Worker Worker Worker kubelet kubelet kubelet kubelet

    kubelet kubelet etcd etcd etcd kube-apiserver kube-apiserver kube-apiserver kube-proxy kube-proxy kube-proxy kube-cm kube-cm kube-scheduler kube-scheduler kube-proxy kube-proxy kube-proxy 凡例 Systemd Unit Static Pod DaemonSet Deployment Flannel Flannel Flannel Flannel Flannel Flannel metallb metallb metallb pod-checkpointer pod-checkpointer pod-checkpointer coredns metallb metallb metallb
  4. Kubernetes クラスタ kubelet kubelet kubelet apiserver apiserver kube-apiserver keepalived クラスタ

    haproxy haproxy haproxy controller manager kube-controller manager scheduler kube-scheduler VIP DaemonSets DaemonSets Deployments Deployments Kubernetes Service Service etcd クラスタ etcd node etcd node etcd node
  5. Kubernetes クラスタ etcd クラスタ etcd node etcd node etcd node

    kubelet kubelet kubelet apiserver apiserver kube-apiserver controller manager kube-controller manager scheduler kube-scheduler systemd sevice Deployments Deployments Kubernetes Service Service (MetalLB) Static Pod DaemonSets
  6. Prometheus で k8s および k8s 上のアプリケーションを監視 するための k8s Manifest を生成するライブラリ

    1. Prometheus Operator 2. HA Prometheus 3. HA Alertmanager 4. Prometheus node-exporter 5. Prometheus Adapter for Kubernetes Metrics APIs 6. kube-state-metrics 7. Grafana + Dashboard https://github.com/coreos/kube-prometheus
  7. local kp = (import 'kube-prometheus/kube-prometheus.libsonnet' ) + // Uncomment the

    following imports to enable its patches // (import 'kube-prometheus/kube-prometheus-anti-affinity.libsonnet') + // (import 'kube-prometheus/kube-prometheus-managed-cluster.libsonnet') + // (import 'kube-prometheus/kube-prometheus-node-ports.libsonnet') + // (import 'kube-prometheus/kube-prometheus-static-etcd.libsonnet') + // (import 'kube-prometheus/kube-prometheus-thanos-sidecar.libsonnet') + // (import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet') + { _config+:: { namespace: 'monitoring', }, }; { ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields (kp.kubePrometheus) } + { ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.filter((function(name) name != 'serviceMonitor' ), std.objectFields (kp.prometheusOperator)) } + // serviceMonitor is separated so that it can be created after the CRDs are ready { 'prometheus-operator-serviceMonitor' : kp.prometheusOperator.serviceMonitor } + { ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields (kp.nodeExporter) } + { ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields (kp.kubeStateMetrics) } + { ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields (kp.alertmanager) } + { ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields (kp.prometheus) } + { ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields (kp.prometheusAdapter) } + { ['grafana-' + name]: kp.grafana[name] for name in std.objectFields (kp.grafana) }
  8. 公式リポジトリに コンパイル済みのマニフェストが Quickstart 用に置いてある # Create the namespace and CRDs,

    and then wait for them to be availble before creating the remaining resources kubectl create -f manifests/setup until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done kubectl create -f manifests/
  9. Quickstart のマニフェストで痒いところ 1. kube-rbac-proxy が arm64 に対応していない 2. Alertmanager の通知先を指定したい

    3. Ingress がいない 4. Prometheus/Alertmanager の externalUrl がおかしい 5. 永続化されていない 6. 自作の textfile collector を使いたい
  10. global: hipchat_api_url: 'https://hipchat.foobar.org/' … - name: 'team-X-hipchat' hipchat_configs: - auth_token:

    <auth_token> room_id: 85 message_format: html notify: true alertmanager.yaml
  11. secretGenerator: - name: alertmanager-main namespace: monitoring behavior: merge files: -

    secrets/alertmanager.yaml kustomization.yaml 既存のSecretもsecretGeneratorで マージできる!
  12. apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: grafana namespace: monitoring labels:

    app: grafana spec: rules: - host: grafana.$(DOMAIN) http: paths: - backend: serviceName: grafana servicePort: 3000 tls: - hosts: - grafana.$(DOMAIN) secretName: grafana-tls ingress.yaml
  13. spec: rules: - host: grafana.$(DOMAIN) http: ... tls: - hosts:

    - grafana.$(DOMAIN) secretName: grafana-tls ingress.yaml var/varReference を使うことで、 ingress のホスト部分のカスタマイズ が簡単に!
  14. apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: k8s namespace: monitoring spec:

    externalUrl: https://prometheus.$(DOMAIN)/ storage: volumeClaimTemplate: spec: storageClassName: "iscsi-targetd-vg-targetd" resources: requests: storage: 10Gi prometheus.yaml CRD にパッチするだけ。
  15. 永続化されていない (grafana) 1. pvc を追加。 2. grafana deployment にパッチ a.

    volumes に追加した pvc を設定 b. emptyDir の設定を削除
  16. apiVersion: apps/v1 kind: Deployment metadata: name: grafana namespace: monitoring spec:

    template: spec: securityContext: fsGroup: 3111 volumes: - name: grafana-storage emptyDir: null persistentVolumeClaim: claimName: grafana-storage grafana.yaml $patch: delete できないので null を 指定。
  17. spec: containers: - name: grafana volumeMounts: - mountPath: /grafana-dashboard-definitions/0/k8s-cluster-overview name:

    grafana-dashboard-k8s-cluster-overview readOnly: false volumes: - name: grafana-dashboard-k8s-cluster-overview configMap: name: grafana-dashboard-k8s-cluster-overview grafana.yaml ちなみに、ダッシュボードの追加はこ んな感じ。
  18. textfile collector を使いたい 1. textfile collector のディレクトリをマウント。 2. textfile collector

    のディレクトリを指定するフラグを追加。 1 は strategic merge patch が使えるが、2 には JSON 6902 patch が必要。
  19. apiVersion: apps/v1 kind: DaemonSet metadata: name: node-exporter namespace: monitoring spec:

    template: spec: containers: - name: node-exporter volumeMounts: - mountPath: /srv/txt_collector name: textfile-dir readOnly: false volumes: - name: textfile-dir hostPath: path: /var/lib/node-exporter node-exporter.yaml