Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What is Patroni, really?

What is Patroni, really?

Avatar for Polina Bungina

Polina Bungina

May 09, 2025
Tweet

Other Decks in Programming

Transcript

  1. • What is it, really? • Automatic failover done wrong

    • Patroni overview ◦ how it works? ◦ notable features Agenda 2
  2. What is it, really? • Originated from Governor project by

    Compose, in 2015 • Main functions: ◦ Automatic failover ◦ Cluster creation and initial setup ◦ Cluster management ◦ ~ Monitoring 4
  3. Running two nodes only primary Automatic failover done wrong standby

    WAL stream health-check Should I… promote? 7
  4. Avoiding split-brain • STONITH (shoot the other node in the

    head) • Must use a secondary network • Almost impossible to get it right Automatic failover done wrong 10
  5. Single witness node primary Automatic failover done wrong standby WAL

    stream witness node (arbiter) health-check health-check 11
  6. primary Automatic failover done wrong standby WAL stream witness node

    health-check health-check 12 Single witness node
  7. primary Automatic failover done wrong standby WAL stream witness node

    (arbiter) health-check health-check Promote standby! 13 Single witness node STONITH
  8. Things to consider • Think about network partition • Prevent

    split-brain → fencing ◦ STONITH • Shut down • Kill old connections, re-configure proxy ◦ Self-fencing (locally) • Watchdog Automatic failover done wrong 15
  9. DC3 Local agents primary Automatic failover done wrong 17 agent

    witness node (arbiter) DC1 standby agent standby agent DC2
  10. primary agent witness node (arbiter) DC1 (isolated) DC3 Local agents

    Automatic failover done wrong 18 standby agent standby agent DC2
  11. But how to do it right? primary standby WAL stream

    19 agent agent witness node (arbiter) Quorum
  12. General idea • State stored in Distributed Configuration Store (DCS)

    ◦ Etcd, ZooKeeper, Consul, Kubernetes control-plane • Built-in distributed consensus (RAFT, Zab) • Key-value store • Atomic CAS (compare-and-swap) operations • Lease/Session/TTL to expire data (/leader, /members/*) • Watches for keys How it works? 23
  13. Patroni overview UPDATES /leader, /status, … primary How it works?

    node B standby UPDATES /members/B WATCHES /leader node A /leader: “A”, ttl: 30 24 UPDATE /leader, value=A, prev=A, ttl=30
  14. Self-fencing node A primary How it works? node B standby

    UPDATES /members/B WATCHES /leader UPDATES /leader read-only instance /leader: “A”, ttl: 19 25
  15. Self-fencing standby How it works? node B standby UPDATES /members/B

    WATCHES /leader UPDATES /leader node A demote /leader: “A”, ttl: 9 26 read-only instance
  16. Self-fencing standby How it works? node B standby CREATES /leader

    NOTIFIES /leader expired node A /leader: “A”, ttl: 0 promote 2. 1. 3. 27 read-only instance
  17. Leader race standby How it works? node C standby CREATE

    /leader, prevExists=false value=C node B 28 CREATE /leader, prevExists=false value=B FAILED SUCCESS promote
  18. Communication with DCS – leader How it works? get /

    (whole cluster) update /leader update /status write /failover update /sync … HA loop sleep for loop_wait 29 retriable
  19. ttl, loop_wait, retry_timeout How it works? loop_wait + 2 *

    retry_timeout <= ttl get / (whole cluster) update /leader (10) (10) (30) 30
  20. Data stored in DCS $ etcdctl get --keys-only --prefix /service/demo

    /service/demo/config /* global (dynamic) configuration */ /service/demo/initialize /* cluster identifier */ /service/demo/leader /* who is the primary? */ /service/demo/members/patroni1 /service/demo/members/patroni2 /* discovery */ /service/demo/members/patroni3 /service/demo/status /service/demo/history /* failover history */ /service/demo/failover /* manual failover/switchover */ /service/demo/sync /* synchronous mode */ How it works? 31
  21. Data stored in DCS $ patronictl list + Cluster: demo

    (7497665970948870167) --------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +----------+------------+---------+-----------+----+-----------+ | patroni1 | 172.18.0.2 | Leader | running | 1 | | | patroni2 | 172.18.0.7 | Replica | streaming | 1 | 0 | | patroni3 | 172.18.0.3 | Replica | streaming | 1 | 0 | +----------+------------+---------+-----------+----+-----------+ How it works? data retrieved from DCS 32
  22. Data stored in DCS $ etcdctl get --print-value-only --prefix /service/demo/leader

    patroni1 $ etcdctl get --print-value-only --prefix /service/demo/initialize 7497665970948870167 How it works? 33
  23. Data stored in DCS $ etcdctl get --keys-only --prefix /service/demo/members/patroni2

    { "conn_url": "postgres://172.18.0.7:5432/postgres", "api_url": "http://172.18.0.7:8008/patroni", "state": "running", "role": "replica", "version": "4.0.5", "xlog_location": 67425896, /* max(receive_lsn or 0, replay_lsn or 0) */ "replication_state": "streaming", "timeline": 1 } How it works? 34
  24. Data stored in DCS $ etcdctl get --print-value-only --prefix /service/demo/status

    { "optime": 67425896, /* pg_current_wal_flush_lsn() */ "slots": { "patroni2": 67425896, "patroni3": 67425896, /* members slots */ "patroni1": 67425896, “my_logical_slot: 67425700 /* permanent slots */ }, "retain_slots": [ "patroni1", "patroni2", /* member_slots_ttl */ "patroni3" ] } How it works? 35
  25. Data stored in DCS $ etcdctl get --print-value-only --prefix /service/demo/config

    { "loop_wait": 10, "ttl": 30, "retry_timeout": 10, "maximum_lag_on_failover": 1048576, "postgresql": { "parameters": { "max_connections": 100 }, /* applied to all members (global) */ "use_pg_rewind": true }, "synchronous_mode": "quorum" } How it works? 36
  26. pg_controldata hack • max_connections max_worker_processes max_wal_senders max_prepared_transactions max_locks_per_transaction PG restriction:

    value on primary ≤ value on standbys • Patroni only allows it to be set globally How it works? 37
  27. pg_controldata hack • New cluster from a backup/standby cluster, max_connections

    = 80 • $ pg_controldata $PGDATA ... max_connections setting: 100 ... • start fails WARNING: hot standby is not possible because of insufficient parameter settings DETAIL: max_connections = 80 is a lower setting than on the primary server, where its value was 100. How it works? 38
  28. pg_controldata hack • => start Postgres with the value from

    pg_crontroldata (100) and inform users: INFO: max_connections value in pg_controldata: 100, in the global configuration: 80. pg_controldata value will be used. Setting 'Pending restart' flag • $ patronictl list + Cluster: my-standby (7387342692208361967) -+---------------------+----+-----------+-----------------+---------------------------+ | Member | Host | Role | State | TL | Lag in MB | Pending restart | Pending restart reason | +--------------+------------+----------------+---------------------+----+-----------+-----------------+---------------------------+ | my-standby-0 | 10.2.26.68 | Standby Leader | in archive recovery | 46 | | * | max_connections: 300->100 | +--------------+------------+----------------+---------------------+----+-----------+-----------------+---------------------------+ +-----------------+---------------------------+ | Pending restart | Pending restart reason | +-----------------+---------------------------+ | * | max_connections: 300->100 | +-----------------+---------------------------+ How it works? 39
  29. Notable features • Standby cluster – running cascading replication to

    a remote datacenter (region) [docs] • Synchronous mode – manage “synchronous_standby_names” to enable synchronous replication whenever there are healthy standbys available [docs] • Quorum-based failover – reduce latencies, compensating higher latency of replicating to one synchronous standby by other standbys [docs] • DCS failsafe mode – survive temporary DCS outages without primary demotion [docs] [slides] • Citus support [docs] [article] What else? 41
  30. More links • Patroni – Postgres.FM podcast • Patroni tutorial

    (A bit outdated but still good) • Step-by-step Patroni cooking guide talk slides • Official documentation (Read the docs! No, seriously…) • Changelog (new features and bugfixes) • Patroni channel in the PostgreSQL Slack What else? 42