The relational model in the modern development age

Transcript

1. The relational model in the modern
   development age
   Shlomi Noach
   PlanetScale
   FOSDEM 2022
   Towards an improved operational paradigm,
   illustrated by Vitess
   

   View Slide

2. Agenda
   - Relational databases as production systems
   - Modern expectations
   - Schema deployment friction
   - Deployment sheduling
   - Deployment conflicts
   - Undeploying
   - Redeploying, idempotency & version control
   

   View Slide

3. About me
   Engineer at PlanetScale
   Author of open source projects orchestrator, gh-ost,
   freno and others
   Maintainer for Vitess
   Blog at http://openark.org
   github.com/shlomi-noach
   @ShlomiNoach
   

   View Slide

4. Founded Feb. 2018 by co-creators of Vitess, ~85
   employees, HQ San Francisco, remote team
   MySQL-compatible serverless database platform, built
   for developers
   Built on top of Vitess
   
   

   View Slide

5. Vitess
   A database clustering system for horizontal scaling of
   MySQL
   ● CNCF graduated project
   ● Open source, Apache 2.0 licence
   ● Contributors from around the community
   

   View Slide

6. The Relational Model
   Data changes vs. schema changes, and where relational
   databases fell short
   

   View Slide

7. Deploying: size and time
   Deployments can be measured in minutes, hours or days
   

   View Slide

8. Deploying: submission & runtime
   Scheduling migrations
   

   View Slide

9. Scheduling: offered paradigm
   Asynchronous migrations
   Database scheduler
   

   View Slide

10. Vitess: migration scheduler
    Migrations run asynchronous, decoupled from caller
    Flexible scheduling:
    Single pending migration at a time
    Single running migration at a time
    Concurrent migrations
    

    View Slide

11. Deployment conflicts
    Choose between a required deployment or a required
    operation
    ● Server restart/reboot
    ● Promotion/failover
    

    View Slide

12. Deployment conflicts
    Interrupt due to conflicting operations
    

    View Slide

13. Deployment conflicts
    Withhold due to conflicting operations
    

    View Slide

14. Deployment conflicts
    Pursue and withhold operation
    

    View Slide

15. Deployment conflicts
    Deployment is coupled with the running database server
    process
    

    View Slide

16. Deployment conflicts: offered
    paradigm
    Process, server & failure agnostic deployments
    

    View Slide

17. Vitess: failure agnostic migrations
    mysql> SET @@ddl_strategy=’online’;
    mysql> ALTER TABLE my_table
    ADD INDEX name_idx(name(24)), DROP COLUMN c;
    Auto-resumes after restart
    Auto resumes on promoted replica
    

    View Slide

18. Undeploying
    Rolling back a deployment
    

    View Slide

19. Undeploying
    Blast radius
    

    View Slide

20. Undeploying
    Feasibility
    

    View Slide

21. Undeploying
    Feasibility: constructing the reverse ALTER statement
    mysql> ALTER TABLE my_table DROP INDEX state_idx;
    mysql> ALTER TABLE my_table ADD INDEX
    state_idx(state, token(32));
    

    View Slide

22. Undeploying
    Feasibility: dropped data
    mysql> ALTER TABLE my_table DROP COLUMN my_col;
    mysql> DROP TABLE my_table;
    

    View Slide

23. Undeploying
    Time
    

    View Slide

24. Undeploying: offered paradigm
    Revert as first class citizen
    

    View Slide

25. Vitess: REVERT
    mysql> SET @@ddl_strategy=’online’;
    mysql> REVERT VITESS_MIGRATION
    ‘a1dac193_4b86_11ec_a827_0a43f95f28a3’;
    Restore dropped tables, populated with data
    Restore dropped columns, populated with data
    Quick operation, proportional to time since migration
    completion, not to table size.
    

    View Slide

26. Vitess: REVERT
    Near instant REVERT made possible by:
    mysql> SET @@ddl_strategy=’online
    -allow-concurrent -postpone-completion’;
    mysql> REVERT VITESS_MIGRATION
    ‘A1dac193_4b86_11ec_a827_0a43f95f28a3’;
    +--------------------------------------+
    | uuid |
    +--------------------------------------+
    | bf4598ab_8d55_11eb_815f_f875a4d24e90 |
    +--------------------------------------+
    Runs, but does not complete, a reverting migration
    

    View Slide

27. Vitess: REVERT
    Either choose to actually revert in case of trouble, or
    abort the revert if all goes well:
    mysql> ALTER VITESS_MIGRATION
    'bf4598ab_8d55_11eb_815f_f875a4d24e90' COMPLETE;
    mysql> ALTER VITESS_MIGRATION
    'bf4598ab_8d55_11eb_815f_f875a4d24e90' CANCEL;
    

    View Slide

28. Redeploying
    Painfully untrivial
    

    View Slide

29. Redeploying
    Relational databases and imperative changes
    

    View Slide

30. Redeploying
    Analyzing errors
    mysql> ALTER TABLE my_table ADD COLUMN my_column
    INT NOT NULL DEFAULT 0;
    

    View Slide

31. Redeploying
    Lack of errors, undesired operation
    mysql> ALTER TABLE my_table ADD INDEX (status);
    mysql> ALTER TABLE my_table
    PARTITION BY HASH(id) PARTITIONS 4;
    

    View Slide

32. Redeploying
    What is your current schema?
    

    View Slide

33. Redeploying
    What is your current schema?
    db_changes/0001_initial
    db_changes/0002_add_products
    db_changes/0003_key_on_user
    db_changes/0004_refactor_address_to_another_table
    

    View Slide

34. Redeploying
    So what is your current schema?
    

    View Slide

35. Redeploying
    Effective source of truth?
    

    View Slide

36. Redeploying
    Would you agree to read code by series of patches?
    

    View Slide

37. Redeploying
    Captive in decades old flow
    

    View Slide

38. Redeploying: offered paradigm
    Declarative schema changes
    

    View Slide

39. Vitess: declarative changes
    mysql> SET @@ddl_strategy='online -declarative';
    -- The following migration creates a new table, as
    the table does not exist:
    mysql> CREATE TABLE decl_table(
    id INT PRIMARY KEY
    );
    +--------------------------------------+
    | uuid |
    +--------------------------------------+
    | b06475e5_8a74_11eb_badd_f875a4d24e90 |
    +--------------------------------------+
    

    View Slide

40. Vitess: declarative changes
    mysql> SET @@ddl_strategy='online -declarative';
    -- The next migration will implicitly ALTER the
    table decl_table into desired state:
    mysql> CREATE TABLE decl_table(
    id INT PRIMARY KEY,
    ts TIMESTAMP NOT NULL
    );
    +--------------------------------------+
    | uuid |
    +--------------------------------------+
    | b7d6e6fb_8a74_11eb_badd_f875a4d24e90 |
    +--------------------------------------+
    

    View Slide

41. Vitess: declarative changes
    mysql> SET @@ddl_strategy='online -declarative';
    -- Next migration does not change table structure,
    hence is a noop and implicitly successful:
    mysql> CREATE TABLE decl_table(
    id INT PRIMARY KEY,
    ts TIMESTAMP NOT NULL
    );
    +--------------------------------------+
    | uuid |
    +--------------------------------------+
    | 110574b1_8a75_11eb_badd_f875a4d24e90 |
    +--------------------------------------+
    

    View Slide

42. Vitess: declarative changes
    A hybrid approach: per-table declarative
    Exploration continues
    

    View Slide

43. Fear of deployment
    “Risky” schema deployments
    Human-handled large deployments
    Avoid dealing with the database
    

    View Slide

44. Existing paradigm: conclusion
    We are caught in a paradigm that has evolved an
    elaborate ecosystem, yet one that does not meet modern
    expectations
    Outdated paradigms go beyond schema deployments
    

    View Slide

45. The relational developer paradigm
    Asynchronous migrations
    Scheduled migrations
    Migrations decoupled from server, survive reboot and
    failover
    Revertible migrations as first class citizens
    Retriable migrations
    Support declarative, idempotent migrations
    

    View Slide

46. Links
    Vitess project home page and docs: https://vitess.io/
    Vitess repo: https://github.com/vitessio/vitess
    Vitess Slack workspace: https://vitess.io/slack
    PlanetScale docs: https://docs.planetscale.com
    

    View Slide

47. Thank you!
    Questions?
    github.com/shlomi-noach
    @ShlomiNoach
    

    View Slide