Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data corrupting architectures we know and love ...

Data corrupting architectures we know and love - VelocityConf San Jose 2019

There’s an old distributed systems adage, “You can have a second computer when you learn how to use the first one.” When it comes to data-access patterns, most of our favorite patterns are unsafe on a single computer. Most of our applications assume a concurrency model where all access is serialized. What happens when that mental model meets distributed data?

Sean Allen reviews data race and corruption problems that exist on single-machine systems and shows how we’ve transferred many of those patterns over to distributed systems and distributed state. You’ll learn the basics of data races, deadlocks, and problems in even the simplest of our concurrent data access patterns and how our existing scaling patterns replicate the same issues across multiple machines, increasing the potential problems, as well as designs that can alleviate them.

Sean T Allen

June 12, 2019
Tweet

More Decks by Sean T Allen

Other Decks in Technology

Transcript

  1. VP OF ENGINEERING @ WALLAROO LABS MEMBER OF THE PONY

    CORE TEAM AUTHOR OF “STORM APPLIED” SPOILED BOSTON SPORTS FAN @SEANTALLEN @WALLAROOLABS @PONYLANG @NHLBRUINS SEAN T. ALLEN
  2. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE TODAY’S TOPICS INCLUDE…

    ▸ Data corruption ▸ Concurrency ▸ Data races ▸ Shared values ▸ Atomic operations
  3. OUR MOST POPULAR ARCHITECTURES CAN BE TRACED TO WEB APPPLICATIONS.

    Sean T. Allen DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
  4. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE A BIT MORE

    DETAILED… REQUEST to the APPLICATION on to the DATABASE back to the APPLICATION and out as a RESPONSE
  5. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE WE COULD GET

    MORE DETAILED STILL… REQUEST to the APPLICATION on to the DATABASE back to the APPLICATION and out as a RESPONSE (WEB BROWSER) to the (WEB SERVER + DEVELOPER CODE) on to the DATABASE back to the (WEB SERVER + DEVELOPER CODE) and out as a (WEB BROWSER)
  6. DISTRIBUTED SYSTEMS AREN’T HARD. THEY’RE EASY*. Sean T. Allen DATA

    CORRUPTING ARCHITECTURES WE KNOW AND LOVE * sometimes
  7. DISTRIBUTED SYSTEMS AREN’T HARD. THEY’RE EASY*. CONCURRENCY IS THE HARD

    PART. Sean T. Allen DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE * sometimes
  8. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED

    COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return new value to the user
  9. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED

    COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return original value to the user SPOT THE DATA CORRUPTION?
  10. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED

    COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return new value to the user
  11. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED

    COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return original value to the user LET’S ADD SOME CONCURRENCY
  12. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED

    COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return new value to the user ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return new value to the user
  13. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED

    COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return original value to the user ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return original value to the user WHAT’S THE FINAL VALUE? 1? 2?
  14. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE THE PAGES SERVED

    COUNTER ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return new value to the user ‣ Read current value from the database ‣ Add one to update it ‣ Write new value to the database ‣ Return new value to the user
  15. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current

    value from the database 0 ‣ Read current value from the database 0 ‣ Add one to update it 1 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Write new value to the database 1 ‣ Return new value to the user 1 ‣ Return new value to the user 1 Request A Request B
  16. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current

    value from the database 0 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Return new value to the user 1 ‣ Read current value from the database 1 ‣ Add one to update it 2 ‣ Write new value to the database 2 ‣ Return new value to the user 2 Request A Request B
  17. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current

    value from the database 0 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Read current value from the database 1 ‣ Return new value to the user 1 ‣ Add one to update it 2 ‣ Write new value to the database 2 ‣ Return new value to the user 2 Request A Request B
  18. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current

    value from the database 0 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Read current value from the database 1 ‣ Return new value to the user 1 ‣ Add one to update it 2 ‣ Write new value to the database 2 ‣ Return new value to the user 2 Request A Request B
  19. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current

    value from the database 0 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Read current value from the database 1 ‣ Return new value to the user 1 ‣ Add one to update it 2 ‣ Write new value to the database 2 ‣ Return new value to the user 2 Request A Request B
  20. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE RECIPE FOR A

    DATA RACE ▸ A shared value ▸ Concurrency ▸ At least one writer changing the shared value ▸ Non-atomic operations
  21. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE AN OPERATION IS

    ATOMIC IF IT’S ISOLATED FROM OTHER OPERATIONS HAPPENING AT THE SAME TIME.
  22. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current

    value from the database 0 ‣ Read current value from the database 0 ‣ Add one to update it 1 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Write new value to the database 1 ‣ Return new value to the user 1 ‣ Return new value to the user 1 Request A Request B
  23. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Read current

    value from the database 0 ‣ Read current value from the database 0 ‣ Add one to update it 1 ‣ Add one to update it 1 ‣ Write new value to the database 1 ‣ Write new value to the database 1 ‣ Return new value to the user 1 ‣ Return new value to the user 1 Request A Request B
  24. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Increment current

    value 1 ‣ Increment current value 2 ‣ Read new value 2 ‣ Read new value 2 ‣ Return new value to the user 2 ‣ Return new value to the user 2 Request A Request B
  25. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ‣ Increment current

    value 1 ‣ Increment current value 2 ‣ Read new value 2 ‣ Read new value 2 ‣ Return new value to the user 2 ‣ Return new value to the user 2 Request A Request B
  26. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1

    1 1 2 Request A Request B Request C
  27. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1

    1 1 2 Request A Request B Request C
  28. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1

    1 1 2 Request A Request B Request C
  29. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1

    1 1 2 Request A Request B Request C
  30. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1

    1 1 2 Request A Request B Request C
  31. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1

    1 1 2 Request A Request B Request C
  32. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1

    1 1 2 Request A Request B Request C
  33. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1

    1 1 2 Request A Request B Request C
  34. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE 0 0 1

    1 1 2 Request A Request B Request C
  35. LAST WRITE WINS MEANS “SOME WRITES LOSE”. Sean Cribbs &

    Russell Brown DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
  36. LAST WRITE WINS IS A FANCY NAME FOR “LET’S CORRUPT

    SOME DATA”. Sean T. Allen DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE
  37. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE ALL HAVE DATA

    CORRUPTION BAKED INTO THE CORE ARCHITECTURE
  38. DATA CORRUPTING ARCHITECTURES WE KNOW AND LOVE RECIPE FOR A

    DATA RACE: CHANGE AT LEAST ONE ▸ A shared value ▸ Concurrency ▸ At least one writer changing the shared value ▸ Non-atomic operations