Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
OpenShift.Run-2024-Medik8s
Search
orimanabu
March 28, 2024
Technology
2
1.6k
OpenShift.Run-2024-Medik8s
OpenShift.Run 2024でのMedik8sセッションの資料です。
orimanabu
March 28, 2024
Tweet
Share
More Decks by orimanabu
See All by orimanabu
Whats_new_in_Podman_and_CRI-O_2025-06
orimanabu
3
220
podman_update_2024-12
orimanabu
2
500
KubeVirt Networking ONIC 2024
orimanabu
7
1.8k
Podman_CRI-O_update_2024-02
orimanabu
5
1.6k
Podman_update_2023-11
orimanabu
2
860
OVN-Kubernetes-Introduction-ja-2023-01-27.pdf
orimanabu
3
3.4k
Podman_CRI-O_update_2022-08
orimanabu
2
1.1k
skupper-introduction
orimanabu
0
880
OpenShift.Run-2022-makaizo
orimanabu
2
1.3k
Other Decks in Technology
See All in Technology
20250913_JAWS_sysad_kobe
takuyay0ne
2
260
まずはマネコンでちゃちゃっと作ってから、それをCDKにしてみよか。
yamada_r
2
120
メルカリIBISの紹介
0gm
0
810
使いやすいプラットフォームの作り方 ー LINEヤフーのKubernetes基盤に学ぶ理論と実践
lycorptech_jp
PRO
1
180
Modern Linux
oracle4engineer
PRO
0
170
COVESA VSSによる車両データモデルの標準化とAWS IoT FleetWiseの活用
osawa
1
400
Rustから学ぶ 非同期処理の仕組み
skanehira
1
160
Oracle Cloud Infrastructure IaaS 新機能アップデート 2025/06 - 2025/08
oracle4engineer
PRO
0
120
Autonomous Database - Dedicated 技術詳細 / adb-d_technical_detail_jp
oracle4engineer
PRO
4
10k
スマートファクトリーの第一歩 〜AWSマネージドサービスで 実現する予知保全と生成AI活用まで
ganota
2
320
AIエージェントで90秒の広告動画を制作!台本・音声・映像・編集をつなぐAWS最新アーキテクチャの実践
nasuvitz
3
390
Apache Spark もくもく会
taka_aki
0
150
Featured
See All Featured
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
6k
How STYLIGHT went responsive
nonsquared
100
5.8k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
Building Flexible Design Systems
yeseniaperezcruz
329
39k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
[RailsConf 2023] Rails as a piece of cake
palkan
57
5.8k
Building an army of robots
kneath
306
46k
The World Runs on Bad Software
bkeepers
PRO
70
11k
How to Think Like a Performance Engineer
csswizardry
26
1.9k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Practical Orchestrator
shlominoach
190
11k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Transcript
- Workload Availability for OpenShift - ノード障害時にStatefulSetも 自動回復したい! そんなときは medik8s.io
Manabu Ori Red Hat 1 OpenShift.Run 2024 2024-03-28
2 ▸ worker-2で動いていたPodにEvictionが発動 ▸ k8sの自動復旧の仕組みにより、他のノードで自動的に ...起動してない子がいる?! ▸ worker-2で動いていたPodは Terminating のまま
こういう状況見たことありませんか? ▸ worker-2でノード障害が発生!すると...
3 こういう状況見たことありませんか? ▸ ...といった状況を自動で復旧させる方法をご紹介します 勝手に復旧!
4 ▸ デフォルト設定では、ノード障害発生からおよそ 5分40秒でPodのEvictionが始まる ノード障害時の動き ノード障害発生 Taint付与 Eviction発動 • node.kubernetes.io/not-ready
• node.kubernetes.io/unreachable tolerationSecondsのデフォルトは 300秒 node-monitor-grace-period (40秒) default-not-ready-toleration-seconds default-unreachable-toleration-seconds (300秒)
5 ▸ ノード障害時にPodが自動回復しないパターン: at-most-one semantics ・ StatefulSet ・ RWOなPVを持つPod ▸
KubeVirt/OpenShift Virtualizationでノード障害時にVMが別ノードで自動回復しない Kubernetesのありがちな悩み ▸ 本資料では、これを自動でなんとかするための仕組みをご紹介します ・ キーワード: ・ Workload Availability for OpenShift ・ Node Health Check Operator, Self Node Remediation Operator, ... ・ Medik8s https://www.medik8s.io/ ・ そういえば、最近(k8s v1.28で)Non Graceful Node Shutdownの機能がGAしましたね...
6 ▸ Failure Detectionの仕組みでノード障害を検知 ・ Machine Health Check (OpenShift標準機能、IPI前提) ・
Node Health Check Operator (UPIでも使用可) ・ SIG ClusterのExternal Remediation APIを利用 ▸ 検知の結果を元に、Remediationの仕組みを使って自動復旧 ・ Self Node Remediation (IPMI/Redfish必要なし) ・ Fence Agents (IPMI/Redfish前提) ・ Machine Deletion (APIでノード削除) 概要 Failure Detection Remediation Machine Health Check Node Health Check Operator Self Node Remediation Operator Fence Agents Remediation Operator Machine Deletion Remediation Operator OpenShift標準機能 追加Operator Legend: 障害検知 自動復旧
7 概要 Node Health Check Operator NodeHealthCheck SelfNode Remediation Self
Node Remediation Operator SelfNode Remediation Config SelfNode Remediation Template Soft Watchdogに よる回復処理 FenceAgent Remediation Fence Agent Remediation Operator SelfNode Remediation Template FenceAgent Remediation Template IPMIによる フェンシング ノード障害を検知し、 Remediationが必要と判断する と、Templateから XXRemediation CRを生成 NHC Operatorがノード障害を検 知し、Remediationが必要と判断 すると、Templateから XXRemediation CRを生成 Self Node Remediation Fence Agent Remediation
8 ▸ 処理の概要: ・ 各ノードのNodeConditionsを見て、NodeHealthCheckで定義した判断基準/しきい値にひっ かかるかを確認 ・ 障害発生と判断でき、かつRemediation処理を実施するべき状況であれば、 Remediation Request
カスタムリソースを作成 ・ このCRを見てRemediation処理が走る ▸ ノードのラベルに応じて異なる対応が可能 ・ NodeHealthCheckでnodeSelectorを指定 ▸ もしクラスターのバージョンアップ中であれば、それが完了するまで remediation処理をスキップする ・ https://github.com/medik8s/node-healthcheck-operator/blob/2edb6fd6d1b0294f19 d62c2e91815e44b96b74f4/controllers/nodehealthcheck_controller.go#L249-L255 ・ 今のところOpenShiftのみ Node Healthcheck Operator
9 ▸ NodeHealthCheck CRの例 Node Healthcheck Operator apiVersion: remediation.medik8s.io/v1alpha1 kind:
NodeHealthCheck metadata: name: nodehealthcheck-sample spec: minHealthy: 51% pauseRequests: - <pause-test-cluster> remediationTemplate: apiVersion: self-node-remediation.medik8s.io/v1alpha1 name: self-node-remediation-resource-deletion-template namespace: openshift-workload-availability kind: SelfNodeRemediationTemplate escalatingRemediations: - remediationTemplate: apiVersion: self-node-remediation.medik8s.io/v1alpha1 name: self-node-remediation-resource-deletion-template namespace: openshift-workload-availability kind: SelfNodeRemediationTemplate order: 1 timeout: 300s selector: matchExpressions: - key: node-role.kubernetes.io/worker operator: Exists unhealthyConditions: - type: Ready status: "False" duration: 300s - type: Ready status: Unknown duration: 300s 最低限残す必要がある Healthyノード数 一時的にRemediationを止めたいときに使用 メインのRemediation処理のテンプレート Remediation処理を多重化したいときはここにリ ストする 適用するノードの指定 ノード障害と判断する条件
10 ▸ MachineHealthCheck/NodeHealthCheckで障害を検知した際、自動復旧を行う仕 組 みのひとつ ・ MHC/NHCコントローラがSelfNodeRemediation CRを作成 ・ SelfNodeRemediation
CRを見てSNR OperatorがRemediation処理を実行 ▸ IPMI/Redfishやノード操作のAPIを必要としない ・ Software Watchdogを使用 Self Node Remediation Operator apiVersion: self-node-remediation.medik8s.io/v1alpha1 kind: SelfNodeRemediation metadata: name: selfnoderemediation-sample namespace: openshift-operators spec: remediationStrategy: <remediation_strategy> status: lastError: <last_error_message> apiVersion: self-node-remediation.medik8s.io/v1alpha1 kind: SelfNodeRemediationConfig metadata: name: self-node-remediation-config namespace: openshift-operators spec: safeTimeToAssumeNodeRebootedSeconds: 180 watchdogFilePath: /dev/watchdog isSoftwareRebootEnabled: true apiServerTimeout: 15s apiCheckInterval: 5s maxApiErrorThreshold: 3 peerApiServerTimeout: 5s peerDialTimeout: 5s peerRequestTimeout: 5s peerUpdateInterval: 15m (次ページ参照)
11 ▸ ResourceDeletion ・ ノード上のPodおよびPVCを削除する ▸ OutOfServiceTaint ・ KEP-2268: Non
graceful node shutdownの機能を使用 ・ PVを持つPodが動いているノードに障害が発生した場合でも、 PVのデ タッチおよびPodの削除ができるようになる ・ ノードがNotReady状態かつ「node.kubernetes.io/out-of-service」 Taintが付与されている場合に、kube-controller-managerがPVのデ タッチ/Pod削除処理を行う ▸ Automatic ・ OutofServiceTaintが利用できる場合は使用し、そうでなければ ResourceDeletionを使用する (デフォルト設定) Remediation Strategy
12 ▸ fence agentを使って電源オン/オフを行う ・ よく使われるのはfence_ipmilan Fence Agent Remediation Operator
apiVersion: fence-agents-remediation.medik8s.io/v1alpha1 kind: FenceAgentsRemediationTemplate metadata: name: fence-agents-remediation-template-fence-ipmilan namespace: openshift-workload-availability spec: template: spec: agent: fence_ipmilan nodeparameters: --ipport: master-0-0: '6230' master-0-1: '6231' master-0-2: '6232' worker-0-0: '6233' worker-0-1: '6234' worker-0-2: '6235' sharedparameters: '--action': reboot '--ip': 192.168.123.1 '--lanplus': '' '--password': password '--username': admin retryCount: '5' retryInterval: '5' timeout: '60'
13 ▸ 03:56:00 worker-2電源断 実際の動き (1) controllers.NodeHealthCheck Node is going
to match unhealthy condition {"node": "worker-2", "condition type": "Ready", "condition status": "Unknown", "duration left": "4m59.920799888s"} controllers.NodeHealthCheck.resource manager Creating a remediation CR {"CR name": "worker-2", "CR kind": "SelfNodeRemediation", "namespace": "openshift-workload-availability"} ▸ 03:56:35 [node-healthcheck-controller-manager] Unhealhtyノードの検出 ▸ 04:01:35 [node-healthcheck-controller-manager] SelfNodeRemediation作成 node-monitor-grace-period default-not-ready-toleration-seconds default-unreachable-toleration-seconds
14 ▸ 04:04:37 [node-healthcheck-controller-manager] SelfNodeRemediation更新 (phase: Reboot-Completed) ▸ 04:04:38 [self-node-remediation-controller]
out-of-service taint add 実際の動き(2) kube-controller-manager-master-2 kube-controller-manager I0328 04:04:47.598293 1 reconciler.go:277] "attacherDetacher.DetachVolume started: node has out-of-service taint, force detaching" node="worker-2" volumeName="kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com^0001-0011-openshift-storage-0000000000000002-67bda59f-cf8f-4ad3-9ad9-df681d058b9c" kube-controller-manager-master-2 kube-controller-manager I0328 04:04:47.634199 1 reconciler.go:277] "attacherDetacher.DetachVolume started: node has out-of-service taint, force detaching" node="worker-2" volumeName="kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com^0001-0011-openshift-storage-0000000000000002-25426b29-4dfa-48bf-8099-e7827b7cdca1" controllers.SelfNodeRemediation out-of-service taint added {"selfnoderemediation": "openshift-workload-availability/worker-2", "new taints": [{"key":"node.kubernetes.io/unreachable","effect":"NoSchedule","timeAdded":"2024-03-28T03:56:35Z"},{"key":"node.kubernetes.io/unreachable","effect":"NoExecute","time Added":"2024-03-28T03:56:40Z"},{"key":"medik8s.io/remediation","value":"self-node-remediation","effect":"NoExecute","timeAdded":"2024-03-28T04:01:36Z"},{"key":"node. kubernetes.io/unschedulable","effect":"NoSchedule","timeAdded":"2024-03-28T04:01:36Z"},{"key":"node.kubernetes.io/out-of-service","value":"nodeshutdown","effect":"No Execute","timeAdded":"2024-03-28T04:04:38Z"}]} ▸ 04:04:47 [kube-controller-manager] force detaching ▸ 04:01:37 [node-healthcheck-controller-manager] SelfNodeRemediation更新 (phase: Pre-Reboot-Completed) safeTimeToAssumeNodeRebootedSeconds
15 ▸ 04:04:48 [node-healthcheck-controller-manager] SelfNodeRemediation更新 (phase: Fencing-Completed) ▸ 04:04:48 [self-node-remediation-controller]
out-of-service taint remove 実際の動き (3) controllers.SelfNodeRemediation out-of-service taint removed {"selfnoderemediation": "openshift-workload-availability/worker-2", "new taints": [{"key":"node.kubernetes.io/unreachable","effect":"NoSchedule","timeAdded":"2024-03-28T03:56:35Z"},{"key":"node.kubernetes.io/unreachable","effect":"NoEx ecute","timeAdded":"2024-03-28T03:56:40Z"},{"key":"medik8s.io/remediation","value":"self-node-remediation","effect":"NoExecute","timeAdded":"2024-03-28T0 4:01:36Z"},{"key":"node.kubernetes.io/unschedulable","effect":"NoSchedule","timeAdded":"2024-03-28T04:01:36Z"}]}
16 ▸ Workload Availability for Red Hat OpenShift ・ https://access.redhat.com/documentation/en-us/workload_availability_for_red_hat_openshift
▸ Node Health Check Operator ・ https://www.redhat.com/en/blog/node-health-check-operator ▸ Medik8s ・ https://www.medik8s.io/ ▸ Ø Kubernetes Failover Improvement: Non-Graceful Node Shutdown - Yuiko Mori, NEC Solution Innovators ・ https://www.youtube.com/watch?v=28dbE24j_zc&list=PLbzoR-pLrL6q7ytV2im0AoNcJcADAeBtm ・ https://jpn.nec.com/oss/community/blog/kubernetes-failover-improvement-non-graceful-node-sh utdown.html ・ https://qiita.com/y-mo/items/ecb9207175543392ffbe 参考文献
17 ▸ KEP-2268: Non graceful node shutdown ・ https://github.com/kubernetes/enhancements/tree/master/keps/sig -storage/2268-non-graceful-shutdown
・ https://github.com/kubernetes/kubernetes/pull/108486 ▸ External Remediation API ・ https://github.com/kubernetes-sigs/cluster-api/pull/3190 参考文献
linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat 18 Red Hat is the world’s
leading provider of enterprise open source software solutions. Award-winning support, training, and consulting services make Red Hat a trusted adviser to the Fortune 500. Thank you