Life Beyond The Illusion Of Present

The idea of the present is an illusion. Everything we see, hear and feel is just an echo from the past. But this illusion has influenced us and the way we view the world in so many ways; from Newton’s physics with a linearly progressing timeline accruing absolute knowledge along the way to the von Neumann machine with its total ordering of instructions updating mutable state with full control of the “present”. But unfortunately this is not how the world works. There is no present, all we have is facts derived from the merging of multiple pasts. The truth is closer to Einstein’s physics where everything is relative to one’s perspective.

As developers we need to wake up and break free from the perceived reality of living in a single globally consistent present. The advent of multicore and cloud computing architectures meant that most applications today are distributed systems—multiple cores separated by the memory bus or multiple nodes separated by the network—which puts a harsh end to this illusion. Facts travel at the speed of light (at best), which makes the distinction between past and perceived present even more apparent in a distributed system where latency is higher and where facts (messages) can get lost. 

Transcript

1. Life BEYOND the ILLUSION of PRESENT Jonas Bonér CTO Typesafe @jboner
2. "Time is what prevents everything from happening at once." - John Archibald Wheeler
3. Newton's PHYSICS
4. THIS SIMPLIFIED MODEL IS VERY APPEALING TO US
5. von Neumann ARCHITECTURE
6. BACK THEN LIFE WAS GOOD
7. THEN, ALONG CAME CONCURRENCY MADE LIFE MISERABLE
8. Jim Gray's TRANSACTIONS SAVE THE DAY
10. WELL, ALONG CAME DISTRIBUTION MADE LIFE MISERABLE, again...
11. But don't be surprised UNFORTUNATELY, THIS IS NOT HOW THE WORLD WORKS
12. "The future is a function of the past." - A. P. Robertson
13. "The (local) present is a merge function of multiple concurrent pasts." — Me
14. val newLocalPresent = observedPasts. foldLeft(oldLocalPresent) { _ merge _ }
15. WE NEED TO EXPLICITLY MODEL THE LOCAL PRESENT AS FACTS DERIVED FROM THE MERGING OF MULTIPLE CONCURRENT PASTS
16. INFORMATION IS ALWAYS FROM THE PAST
17. THE TRUTH IS CLOSER TO EINSTEIN'S PHYSICS
18. INFORMATION HAS LATENCY
19. THE COST OF MAINTAINING THIS ILLUSION IS INCREASED CONTENTION & COHERENCY
20. AS LATENCY GETS HIGHER, THE ILLUSION CRACKS EVEN MORE
21. DISTRIBUTED SYSTEMS EVERYWHERE
22. THE NETWORK IS RELIABLE...NAT
23. "If a tree falls in a forest and no one is around to hear it, does it make a sound?" — Charles Riborg Mann
24. Information CAN (and will) GET LOST
25. HOW DO WE DEAL WITH INFORMATION LOSS IN REAL LIFE?
26. WE USE A SIMPLE PROTOCOL OF Confirm, Wait & Repeat
27. We fill in THE BLANKS
28. ...AND IF WE ARE WRONG, WE TAKE COMPENSATING ACTION APOLOGY-ORIENTED PROGRAMMING - PAT HELLAND (IN MEMORIES, GUESSES, AND APOLOGIES)
29. The bottom line: WE CAN'T FORCE THE WORLD INTO A SINGLE GLOBALLY CONSISTENT PRESENT
30. Should we just GIVE UP?
31. I BELIEVE THAT THERE IS A PATH FORWARD
32. WE NEED TO TREAT TIME AS A FIRST CLASS CONSTRUCT
33. WHAT IS TIME, really?
34. TIME IS THE SUCCESSION OF CAUSALLY RELATED EVENTS
35. How can we MANAGE TIME?
36. Think in FACTS
37. What is a FACT?
38. "A Fact is something that truly exists or happens: something that has actual existence, a true piece of information." — Merriam Webster
39. IMMUTABILITY is a requirement
40. So, do variables HAVE A PURPOSE IN LIFE?
41. "The assignment statement is the von Neumann bottleneck of programming languages and keeps us thinking in word-at-a-time terms in much the same way the computer's bottleneck does." — John Backus (Turing Award lecture 1977)
42. MUTABLE STATE NEEDS TO BE CONTAINED
43. Ok, but how should we MANAGE FACTS?
44. Functional PROGRAMMING
45. Logic PROGRAMMING
46. Dataflow PROGRAMMING
47. NEVER DELETE FACTS
48. "When bookkeeping was done with clay tablets or paper and ink, accountants developed some clear rules about good accounting practices. One never alters the books; if an error is made, it is annotated and a new compensating entry is made in the books. The books are thus a complete history of the transactions of the business. Update-in-place strikes many systems designers as a cardinal sin: it violates traditional accounting practices that have been observed for hundreds of years." — Jim Gray (1981)
49. CRUD
50. "Database is a cache of a subset of the log." — Pat Helland (2007)
51. Store facts in an EVENT LOG
52. The log allows TIME TRAVEL
53. Can we REWRITE THE PAST?
54. Allows us to shift our focus from DATA AT REST, to DATA IN MOTION
55. Stream Processing
56. CONSTRUCTING A SUFFICIENTLY CONSISTENT LOCAL PRESENT MEANS EMPLOYING CONSISTENCY MECHANISMS
57. Consistency WHAT? WHY? WHEN?
58. WE NEED TO DECOMPOSE THE SYSTEM USING CONSISTENCY BOUNDARIES
59. INSIDE DATA: OUR CURRENT PRESENT OUTSIDE DATA: BLAST FROM THE PAST BETWEEN SERVICES: HOPE FOR THE FUTURE — PAT HELLAND (DATA ON THE INSIDE VS DATA ON THE OUTSIDE)
60. MicroSERVICE
61. AGGREGATE Root
62. WITHIN THE CONSISTENCY BOUNDARY
63. EVENT Sourcing
64. BETWEEN THE Consistency Boundaries IT'S A ZOO
65. Decoupling in TIME / SPACE
66. STRONG CONSISTENCY The wrong default
67. Here, we are living in the LOOMING SHADOW OF IMPOSSIBILITY THEOREMS
68. FLP CONSENSUS IS IMPOSSIBLE
69. PROTOCOLS CLIMB THE LADDER OF KNOWLEDGE Cɸ: Common Knowledge (infinite number of i) Eiɸ: (Everyone knows * i) ɸ E3ɸ: (Everyone knows * 3) ɸ E2ɸ: Everyone knows Everyone knows ɸ E1ɸ: Everyone knows ɸ Sɸ: Someone knows ɸ
70. COMMON KNOWLEDGE IS NOT ATTAINABLE VIA PROTOCOL - JOSEPH HALPERN
71. CAP CONSISTENCY IS IMPOSSIBLE
72. Dissecting CAP
73. "The first principle of successful scalability is to batter the consistency mechanisms down to a minimum." – James Hamilton
74. EVENTUAL CONSISTENCY What does it really mean?
75. Tracking TIME is tracking CAUSALITY
76. RELYING ON TIMESTAMPS IS A BAD IDEA
77. Instead, rely on LOGICAL TIME
78. Lamport CLOCKS GLOBAL CAUSAL ORDERING BETWEEN
79. Vector CLOCKS PARTIAL CAUSAL ORDERING BETWEEN EVENTS
80. Causal CONSISTENCY
81. What CONSISTENCY DO YOU REALLY NEED AND when?
82. ACID 2.0 ASSOCIATIVE COMMUTATIVE IDEMPOTENT DISTRIBUTED
83. CONFLICT-FREE REPLICATED DATA TYPES
84. DISORDERLY PROGRAMMING CALM THEOREM
85. WE ARE JUST GETTING STARTED
86. WE HAVE A LONG ROAD AHEAD OF US...
88. Thanks FOR LISTENING
89. Life BEYOND the ILLUSION of PRESENT Jonas Bonér CTO Typesafe @jboner