Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Patroni 3.0: What's New and Future Plans (Berli...

Alexander Kukushkin
September 19, 2023
120

Patroni 3.0: What's New and Future Plans (Berlin meetup)

Patroni is one of the most popular and advanced solutions for PostgreSQL high-availability. It is integrated with a variety of distributed configuration stores allowing Patroni to work with PostgreSQL cluster information in a consistent way ensuring that there is only one leader at a time. Unlike the majority of existing solutions for automatic failover, Patroni requires a minimal effort to configure the HA cluster and supports auto-discovery of new nodes.

This talk will consist of a short introduction to the ideas behind Patroni, overview of the major features that we have released lately, cover some interesting failure scenarios which were fixed, and share some plans for the future development. Of course, the talk will not be complete without showing a live demo of some brand-new features.

Alexander Kukushkin

September 19, 2023
Tweet

Transcript

  1. Patroni 3.0: What's New and Future Plans PostgreSQL Meetup, Berlin

    Alexander Kukushkin, Polina Bungina 2023-09-18 1
  2. About us 2 Polina Bungina • Senior Software Engineer @ZalandoTech

    [email protected] Alexander Kukushkin • Principal Software Engineer @Microsoft • PostgreSQL Contributor, The Patroni guy • [email protected] • Twitter: @cyberdemn
  3. Agenda • Brief introduction to automatic failover and Patroni •

    New features • Future plans • Live Demo 3
  4. Distributed Configuration (Key-Value) Store • Consul, Etcd (v2/v3), Zookeeper, Kubernetes

    API • Service Discovery ◦ Every Postgres node maintains a key with its state ◦ Leader key points to the primary • Lease/Session/TTL to expire data (i.e. leader key) • Atomic CAS operations • Watches for important keys (i.e. leader key) 5
  5. Patroni: Normal operation 6 Node A Primary Node B Standby

    Node C Standby /leader: A, ttl: 30 UPDATE /leader, A, ttl=30, prev=A WATCH /leader WATCH /leader SUCCESS
  6. Patroni: primary dies, leader key holds 7 Node A Primary

    Node B Standby Node C Standby /leader: A, ttl: 7 WATCH /leader WATCH /leader BANG!
  7. Patroni: leader key expires 8 Node A Node B Standby

    Node C Standby /leader: A, ttl: 0 NOTIFY /leader, expired=true NOTIFY /leader, expired=true
  8. Patroni: leader race 9 Node A Node B Standby Node

    C Standby Node B: GET http://A:8008/patroni -> failed/timeout GET http://C:8008/patroni -> wal_lsn: 100 Node C: GET http://A:8008/patroni -> failed/timeout GET http://B:8008/patroni -> wal_lsn: 100
  9. Patroni: leader race 10 Node A Node B Standby Node

    C Standby CREATE /leader, B, ttl=30, prevExists=false CREATE /leader, C, ttl=30, prevExists=false FAIL SUCCESS
  10. Patroni: promote and continue replication 11 Node A Node B

    Standby Node C Primary WATCH /leader promote
  11. DCS Failsafe Mode • Case: Postgres is running as primary

    only when Patroni can maintain leader lock in DCS • Before: primary is demoted when lock can’t be updated • Now: Patroni will keep primary if all members of the cluster agree with it 13 $ patronictl edit-config --- +++ @@ -4,3 +4,4 @@ use_pg_rewind: true retry_timeout: 10 ttl: 30 +failsafe_mode: on Apply these changes? [y/N]: y Configuration changed Documentation: DCS Failsafe Mode
  12. Citus integration 14 Coordinator Primary Worker 1 Primary Worker 2

    Primary Worker 3 Primary citus=# SELECT nodeid, groupid, nodename FROM pg_dist_node order by groupid; nodeid | groupid | nodename --------+---------+------------- 1 | 0 | 172.19.0.9 3 | 1 | 172.19.0.7 2 | 2 | 172.19.0.2 4 | 3 | 172.19.0.13 (4 rows) Documentation: Citus support
  13. Logical Slots Failover • Case: logical replication slots are lost

    after failover. • Before: don’t allow connections before logical slots are recreated • Now: copy slots from the primary and use pg_replication_slot_advance() to keep logical slot ready. 15 $ patronictl edit-config --- +++ @@ -1,6 +1,12 @@ loop_wait: 10 retry_timeout: 10 ttl: 30 +permanent_slots: + my_slot: + database: testdb + plugin: test_decoding Apply these changes? Configuration changed
  14. pg_rewind improvements 16 • Postgres v13+ supports pg_rewind --restore-target-wal ◦

    But, opt out --restore-target-wal on v13 and v14 if postgresql.conf if outside of $PGDATA (Debian/Ubuntu) @Gunnar "Nick" Bluth • For older versions Patroni tries to fetch missing WALs when pg_rewind fails • Archive WALs before calling pg_rewind on the old primary ◦ pg_rewind might remove them even if they are needed for Postgres
  15. Switchover with Debezium • Case: Postgres on stop waits until

    all WALs are streamed ◦ Debezium doesn’t properly handle keepalive messages • Before: Patroni keeps updating the leader key while Postgres is being stopped (indefinitely) • Now: the leader key is removed when pg_controldata starts reporting “shut down” and there are nodes ready to fail over 17
  16. General improvements • Removed support of Python < 3.6 ◦

    Introduced type hints! ◦ Psycopg 3! • Compatibility with PostgreSQL 16 • Set permissions for files and directories created in $PGDATA 18
  17. General improvements • pre_promote - run a script before pg_ctl

    promote ◦ Abort if the exit code != 0 • before_stop - run a script before pg_ctl stop @Le Duane ◦ pgbouncer PAUSE, terminate Debezium connections 20
  18. Quorum based failover (aka Quorum Commit) • PostgreSQL v10+: synchronous_standby_names="ANY

    k (*)" ◦ Examples: 1. "ANY 2 (node1,node2)", 2. "ANY 2 (node1,node2,node3)" • Challenge: figure out during failover whether the node was synchronous ◦ Was the node2 synchronous in the example 2? 22
  19. Quorum based failover: math • synchronous_standby_names="ANY 2 (m2,m3,m4)" • /sync:

    {leader: m1, sync: [m2,m3,m4], quorum: 1} • synchronous_standby_names="ANY 1 (m2,m3,m4)" • /sync: {leader: m1, sync: [m2,m3,m4], quorum: 2} 23
  20. Quorum based failover: challenges 24 • How to change synchronous_standby_names

    and /sync that we can always identify sync node? • Example: ◦ synchronous_standby_names="ANY 1(m2,m3)" ◦ /sync: {leader: m1, sync: [m2,m3], quorum: 1} ◦ Node m4 joins the cluster: 1. change /sync to {leader: m1, sync: [m2,m3,m4], quorum: 2} 2. change synchronous_standby_names="ANY 1(m2,m3,m4)"
  21. Integrate Patroni with pg_failover_slots • https://github.com/EnterpriseDB/pg_failover_slots • But Patroni already

    solved logical failover slots problem! Why? ◦ Extension has mechanisms to wait for physical standbys before sending data to logical subscriber ▪ pg_failover_slots.standby_slot_names, pg_failover_slots.standby_slots_min_confirmed ◦ Works similar to synchronous_standby_names="ANY k (s1, s2, s3)" 25
  22. Improve Citus support • Register replica nodes in pg_dist_node ◦

    for read scaling (easy) ◦ to use them as failover targets (hard) 26
  23. Get rid of non-inclusive terminology • role: master -> primary

    ◦ Most of preparations are done in 3.0 ▪ If running something older, better to upgrade to 3.1.x first ✓ Kubernetes pod labels is a challenge ◦ Migration will require temporary labels and 3 rolling upgrades 27