Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data as Code

Data as Code

Sean Scott

August 28, 2023
Tweet

More Decks by Sean Scott

Other Decks in Technology

Transcript

  1. Data as Code: Exploring Data and Database Portability Latin America

    Oracle User Group Community Tour '23 July 29 - August 16, 2023 @ViscosityNA www.viscosityna.com Sean Scott Oracle ACE Director DoK Community Ambassador Managing Principal Consultant @oraclesean linkedin.com/in/soscott/ [email protected]
  2. The Intersectionality of Cloud Native and "Groundhog Day" @ViscosityNA www.viscosityna.com

    Latin America Community Tour '23 - Uruguay August 14, 2023 Sean Scott Oracle ACE Director DoK Community Ambassador Managing Principal Consultant @oraclesean linkedin.com/in/soscott/ [email protected]
  3. DATABASE RELIABILITY ENGINEERING ⁘ DEVOPS & AUTOMATION HIGH AVAILABILITY ⁘

    BUSINESS CONTINUITY ⁘ DISASTER RECOVERY MODERNIZATION ⁘ OBSERVABILITY ⁘ ENGINEERED SYSTEMS AutoUpgrade ⁘ Zero Downtime Migrations ⁘ Patching Real Application Clusters ⁘ Data Guard ⁘ Sharding Docker/Containers ⁘ Terraform ⁘ Ansible Exadata ⁘ Oracle Database Appliance AHF ⁘ TFA ⁘ GIMR ⁘ CHA @ViscosityNA www.viscosityna.com Sean Scott Oracle ACE Director DoK Community Ambassador Managing Principal Consultant @oraclesean linkedin.com/in/soscott/ [email protected]
  4. @ViscosityNA www.viscosityna.com Oracle on Docker Running Oracle Databases in Linux

    Containers Download a free sample chapter: https:/ /oraclesean.com 20% Discount Code: OracleDocker https:/ /link.springer.com
  5. @ViscosityNA www.viscosityna.com As creators, buyers, and influencers, developers have become

    instrumental in deciding what products pass muster. Ultimately, the C-suite may call the shot, but the developer is often guiding its aim. Boston Consulting Group "The Growing Influence of Developers in Enterprise Tech Sales" https://www.bcg.com/publications/2022/developers-influence-in-enterprise-tech-sales
  6. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA 6:00 AM Phil w akes up

    Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM
  7. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main 6:00 AM Phil

    w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM
  8. @ViscosityNA www.viscosityna.com Feature Request Customers are complaining that the Phil

    microservice is narcissistic and misogynistic. Redesign Phil to be a decent human being.
  9. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 6:00

    AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM
  10. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─

    bank-robber 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM
  11. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─

    bank-robber └─ piano-student 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM
  12. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─

    bank-robber └─ piano-student └─ redemption 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM
  13. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─

    bank-robber └─ piano-student └─ redemption 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM Punxsutawney_TEST Feb 2 FUNNY
  14. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA 6:00 AM Phil w akes up

    Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM Punxsutawney_TEST Feb 3 NOT FUNNY Phil Connors: main └─ day-2 └─ bank-robber └─ piano-student └─ redemption
  15. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─

    bank-robber └─ piano-student └─ redemption 6:00 AM Alarm doesn't go off! N ed is PO TU S! 7:00 AM M ultiple groundhogs? 7:20 AM M iam i Beach? 9:00 AM Punxsutawney_TEST Polluted Feb 2 BIZARRE
  16. @ViscosityNA www.viscosityna.com Problems • Every test further pollutes data. •

    Restoring databases is operationally difficult, requiring: • Downtime • Specialized skills/DBA intervention • Every database is a production database • Making databases idempotent is technically difficult
  17. @ViscosityNA www.viscosityna.com Common Solutions • Build exceptions/elasticity into CI/CD pipelines

    • Technically difficult • Requires upkeep • Not absolute • Assume certain errors can be ignored • Technology solution ($$$$$)
  18. @ViscosityNA www.viscosityna.com Data is Code Every record in every database

    is created in one of three ways: • INSERT • UPDATE • DELETE
  19. @ViscosityNA www.viscosityna.com "[F]or a long time, DBAs were in the

    business of crafting silos and snowflakes. Their tools were different, their hardware was different, and their languages were different." Laine Campbell, Charity Majors Database Reliability Engineering O'Reilly
  20. @ViscosityNA www.viscosityna.com "The days in which this model can prove

    itself to be effective and sustainable are numbered." Laine Campbell, Charity Majors Database Reliability Engineering O'Reilly
  21. @ViscosityNA www.viscosityna.com "Our systems are no more or less important

    than any other components serving the needs of the business. [T]he components of database clusters are not sacred." Laine Campbell, Charity Majors Database Reliability Engineering O'Reilly
  22. @ViscosityNA www.viscosityna.com An Image is the game board and rules;

    Containers host and persist games; The superimposed Layer saves moves; Deleting a Container removes its Layer.
  23. @ViscosityNA www.viscosityna.com CONTAINER_NAME=punxsutawney_test mkdir -p /data/${CONTAINER_NAME}/data docker volume create \

    --opt type=none \ --opt o=bind \ --opt device=/data/${CONTAINER_NAME}/data \ ${CONTAINER_NAME}_data
  24. @ViscosityNA www.viscosityna.com Database Image Rule #1: If there is a

    database, start it. If not, create a new database.
  25. @ViscosityNA www.viscosityna.com Database Container Startup docker run -d \ --name

    ${CONTAINER_NAME} \ -e ORACLE_SID=ORCL \ -v /data/${CONTAINER_NAME}:/opt/oracle/oradata \ oracle/database:19.3.0-ee
  26. @ViscosityNA www.viscosityna.com Clone the Data Directory cp -rp /data/punxsutawney/ \

    /data/${CONTAINER_NAME} cp -rp /NFS/gold_data/punxsutawney/ \ /data/${CONTAINER_NAME}
  27. @ViscosityNA www.viscosityna.com Clone the Data Directory cp -rp /data/punxsutawney/ \

    /data/${CONTAINER_NAME} cp -rp /NFS/gold_data/punxsutawney/ \ /data/${CONTAINER_NAME} cp -rp /NFS/gold_data/punxsutawney/2023-02-02.0600/ \ /data/${CONTAINER_NAME}
  28. @ViscosityNA www.viscosityna.com Versioned Data 2023-02-02 ARMORED_TRUCKS: NULL LAKES_AND_RIVERS: NULL IMPRESSIVE_PIANO:

    NULL ICE_SCULPTING: NULL select * from knowledge where fname = 'Phil' and lname = 'Connors';
  29. @ViscosityNA www.viscosityna.com Entrypoints • Generally adapted but not an OCI

    standard • Implementation varies by image, vendor/author • Setup only • Setup & startup
  30. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─

    bank-robber └─ piano-student └─ redemption 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tape segm ent 7:20 AM Blizzard! 9:00 AM Punxsutawney_TEST Feb 2 FUNNY
  31. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main 6:00 AM Phil

    w akes up Punxsutawney_TEST Feb 3 PHIL & RITA LIVE HAPPILY EVER AFTER
  32. @ViscosityNA www.viscosityna.com Data as Code: Standardizes data Decouples DBAs from

    the development lifecycle Accelerates & streamlines development Assures test reliability and quality