Recap: KubeCon+CloudNativeCon 2021 NA / Kubernetes Meetup Tokyo #47

Kubernetes Meetup Tokyo #47のRecapです
YouTube: https://youtu.be/fvxp8bUOgtI?t=1893
Event: https://k8sjp.connpass.com/event/229659/

Transcript

1. Recap: Automated, Distributed Systems Testing for Kubernetes Controllers Aya Igarashi (@Ladicle), SWE at Z Lab Kubernetes Meetup Tokyo #47
2. https://kccncna2021.sched.com/event/lV0S
3. • Kuberenetesͷ(Custom)ControllerΛҟৗܥͰਖ਼ ͘͠ಈ͘͜ͱΛอূ͢Δͷ͸೉͍͠ • Fault Injectionͯ͠ControllerͷόάΛൃݟ͢Δͨ Ίͷsieveͱ͍͏OSSΛ։ൃͨ͠ • ಛ௃͸ιʔείʔυͷมߋ͕ෆཁͰɺൃݟͨ͠ό άͷ࠶ݱ͕Մೳͳ͜ͱ • cassandra-operator΍percona-xtradb-cluster- operatorͳͲͷOSSͰόάΛ࣮ࡍʹൃݟͨ͠ Recap: distributed-system pitfall \ /
4. Interesting Points Event-oriented Common Bug Patterns • ͸͡Ίͯίϯτϩʔϥ࣮૷ͯ͠ ͍ͨ࣌ʹ೰ΜͩΑ͏ͳɺϋϚΓ ͕ͪͳϙΠϯτ͕ύλʔϯͱ͠ ͯݴޠԽ͞Ε͍ͯΔ • πʔϧΛ࢖Θͳͯ͘΋͓֮͑ͯ ͘ͱ࣮૷ͷ࣌ʹࢀߟʹͳΔ͸ͣ (Chaos tool) Inject Fault = Bug!! Find Diff
5. Bug Pattern 1 ίϯτϩʔϥ͕Πϕϯτͷड͚औ Γʹࣦഊ͢ΔͱɺϦιʔε͕ਖ਼ৗ ʹॲཧ͞Εͳ͍ύλʔϯ Observability-gaps Bug https://github.com/instaclustr/cassandra-operator/issues/398 ACTION౳ʹґଘͤͣಛఆͷΠϕϯ τΛऔΓ͜΅ͯ͠΋࣍ͷΠϕϯτ ౳ͰॲཧͰ͖ΔΑ͏ʹ࣮૷͢Δ όάͷྫ: Orphan PVC 1. Scale Down 2. Delete Pod
6. Bug Pattern 2 Atomicity Violation https://github.com/rabbitmq/cluster-operator/issues/782 όάͷྫ: Reconcileͷ్தͰ༧ظͤ͵Τϥʔ ΍ఀࢭ͕ىͬͯ͜΋࠶։Ͱ͖ΔΑ ͏ʹ࣮૷͢Δ Reconcile͕׬ྃ͢Δલʹίϯτ ϩʔϥ͕ఀࢭͯ͠ɺϦιʔε͕த ్൒୺ͳঢ়ଶʹͳΔύλʔϯ 1. Extend PVC volume to 2Gi 2. Delete STS 3. Crash controller 4. restart controller 5. Create New STS reference 1Gi PVC
7. Bug Pattern 3 Time Traveling https://jira.percona.com/browse/K8SPXC-716 HAߏ੒ͷΫϥελʹ͓͍ͯɺίϯ τϩʔϥͷ࠶ىಈ࣌ʹݹ͍Ϧιʔ εΛऔಘͯ͠όάʹͳΔύλʔϯ Ϧιʔεͷόʔδϣϯ΍ґଘؔ܎ (OwnerReference)͕ਖ਼͍͔͠ͳͲ ॲཧલͷόϦσʔγϣϯʹ஫ҙ όάͷྫ: 1. Create Cluster w/ same ID 2. Receive old event Reference invalid STS...
8. References Event Schedule: https://kccncna2021.sched.com/event/lV0S YouTube: https://youtu.be/6JnhjgOaZVk GitHub: https://github.com/sieve-project/sieve Research Paper: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-sun.pdf
9. Thank you! Aya Igarashi, Z Lab Twitter & GitHub: @Ladicle