Building Bulletproof Real-Time Apps with Phoenix LiveView + Stream Data

Elixir School's Day 2 workshop at ElixirConf 2019

Transcript

1. Building Bulletproof Real-Time Applications with Phoenix, LiveView, and StreamData
2. Introductions 2
3. Hello! I’m Sophie I’m an engineer and sometimes teacher at The Flatiron School where I build education and business tools in Elixir and Ruby. @sm_debenedetto 3 *and this is my dog
4. And... I’m Michael I’m a lead engineer at RentPath, where we build websites to help people find a place to live. @michaelstalker 4 *and this is my family
5. Elixir School A free, open-source Elixir curriculum Through the hard work of many volunteers around the world, we’ve developed and translated content covering everything from intro topics to deep dives. 5
6. Get Involved! Help us grow the Elixir School community Write a lesson, contribute a TIL (or longer!) blog post or provide a translation. Open an issue here describing your idea or simply open a PR with your work to get started. 6
7. Goals 7
8. What We’ll Learn ◦ Phoenix LiveView ◦ Phoenix PubSub ◦ Phoenix Presence ◦ Property-based Testing with StreamData 8
9. What We’ll Build A real-time ticket estimation tool ◦ Users can see other users in the “estimation room” in real-time ◦ Users can vote on tickets ◦ Users can see the winning vote tallied in real-time and move on to the next ticket 9
10. Pointing Party A collaborative ticket estimation tool for your team 10
12. User Starts Estimation Round When I click “start pointing!” Then the first card for estimation appears for all users in the room Features Real-Time User Presence When a new user joins/leaves Then I see the list of present users update in the UI 12
13. Ticket is Estimated When another user submits a vote Then I see the UI update to indicate they have voted Features Winner is Calculated When the last user submits a vote Then all users see the winning vote And a user can click “next card” 13
14. Application Starting State Log In with a username (We built some dummy authorization flows) See the static ticket estimation page Click the start button and... Oh no you can’t estimate a ticket! 14
15. We Need Real-Time! In the current state of our app, users cannot collaborate on ticket estimation. 15
16. What is the Real-Time Web? Users receive new information from the server as soon as it is available—no request required This represents a significant departure from traditional HTTP communication Made possible by WebSockets 16
17. 17 WebSockets
18. What are WebSockets? A Protocol built on top of TCP that allows for bi-directional, "full-duplex" communication between the client and the server by creating a persistent connection between the two. 18
19. 19 Server Client Request GET https://elixirschool.com Response <h1>Weclome to Elixir School!</h1> HTTP Protocol
20. 20 Server Client Step 1. Request to initiate WS connection Client Client GET / Host: elixirschool.com Upgrade: websocket Connection: upgrade
21. 21 Server Client Step 2. Open and maintain WS connections Client Client
22. 22 We’ll use WebSockets to maintain a persistent, bi-directional and stateful connection between our ticket estimation clients and the server with the help of Phoenix LiveView
23. 23 Phoenix LiveView
24. What is LiveView? Phoenix LiveView leverages server-rendered HTML and Phoenix’s native WebSocket tooling so you can build fancy real-time features without all that complicated JavaScript. 24
25. 25 So...I don’t need to write JavaScript ever again?
26. 26 Not exactly.
27. 27 You just don’t need it as often.
28. 28 How It Works
29. Connecting • Client connects to Phoenix server over WebSockets, at the LiveView socket endpoint • Phoenix server starts a LiveView process for that user’s socket 29
30. Rendering the Template • The LiveView leex (Live EEX) template is rendered, using the data from the LiveView process’ state 30
31. Handling Changes • Certain user interactions on the page will send a message over the socket to the LiveView process • The LiveView process will update its state and push a message back down the socket • The leex template will re-render with the new state 31
32. 32 A Closer Look
33. Roadmap 3. Handling Changes 2. Connecting to LiveView 33 1. Configuring LiveVIew
34. 34 1. Configuring LiveView
35. 35 Install the dependency
36. 36 # mix.exs def deps do [ {:phoenix_live_view, github: "phoenixframework/phoenix_live_view"} ] end
37. 37 Update our app’s config with the signing salt for our LiveView socket connection
38. 38 config :my_app, PointingParty.Endpoint, ... live_view: [ signing_salt: "YOUR_SECRET" ]
39. 39 Configure our app to support “Live EEX” templates
40. 40 config :phoenix, template_engines: [ leex: Phoenix.LiveView.Engine ]
41. 41 Import the LiveView render functions
42. 42 # lib/pointing_party_web.ex def view do quote do ... import Phoenix.LiveView, only: [live_render: 2, live_render: 3] end end
43. 43 Add the LiveView JS library to our NPM dependencies
44. 44 # assets/package.json { "dependencies": { ... "phoenix_live_view": "file:../deps/phoenix_live_view" } }
45. 45 Add the LiveView path to our dev live reload config
46. 46 # config/dev.exs config :pointing_party, PointingPartyWeb.Endpoint, live_reload: [ patterns: [ ..., ~r{lib/my_app_web/live/.*(ex)$} ] ]
47. Roadmap 3. Handling Changes 2. Connecting to LiveView 47 1. Configuring LiveVIew
48. 48 2. Connecting to LiveView
49. Connection Process • User visits ”/cards” in the browser 49
50. Connection Process • User visits ”/cards” in the browser • CardController mounts the LiveView 50
51. Connection Process • User visits ”/cards” in the browser • CardController mounts the LiveView • The LiveView renders the LEEX template as static HTML 51
52. Connection Process • User visits ”/cards” in the browser • CardController mounts the LiveView • The LiveView renders the LEEX template as static HTML • A JS snippet on that static HTML page sends the “WebSocket connect” request 52
53. Connection Process • User visits ”/cards” in the browser • CardController mounts the LiveView • The LiveView renders the LEEX template as static HTML • A JS snippet on that static HTML page sends the “WebSocket connect” request • The LiveView process re-renders the template over WebSockets, is now listening for events from the client 53
54. 54 LiveView Client Request GET /cards Render LEEX Template <h1>Start pointing!</h1> Step 1. Mount the LiveView + render static HTML CardController Mount LiveView
55. 55 Client Connect to LiveView socket Step 2. Establish WS connection LiveView LV Socket
56. 56 Client Connect to LiveView socket Step 3. Re-render the template LiveView Re-render template over WS connection LV Socket
57. We Will • Mount the LiveView socket endpoint • Define a LiveView and LEEX template • Teach our controller to render the LiveView 57
58. 58 Mount the LiveView socket at the “/live” endpoint
59. 59 defmodule PointingPartyWeb.Endpoint do use Phoenix.Endpoint, otp_app: :pointing_party socket "/live", Phoenix.LiveView.Socket end
60. 60 Define The LiveView
61. 61 # app/pointing_party_web/live/card_live.ex defmodule PointingPartyWeb.CardLive do use Phoenix.LiveView alias PointingPartyWeb.CardView def render(assigns) do Phoenix.View.render(CardView, "index.html", assigns) end def mount(_session, socket) do {:ok, assign(socket, is_pointing: false)} end end
62. 62 Define The LEEX Template
63. 63 <div> <%= if !@is_pointing do %> <div> <button> Start the Party! </button> </div> <% else %> <!-- card display here --> <% end %> </div>
64. 64 Include the LiveView socket connect JS
65. 65 // assets/js/app.js import LiveSocket from "phoenix_live_view" let liveSocket = new LiveSocket("/live") liveSocket.connect()
66. 66 Render the LiveView from the controller
67. 67 defmodule PointingPartyWeb.CardController do use PointingPartyWeb, :controller import Phoenix.LiveView.Controller def index(conn, _params) do %{assigns: %{username: username}} = conn live_render(conn, PointingPartyWeb.CardLive, session: %{username: username}) end end
68. 68 Now, when the user visits “/cards”, the LiveView process will start
69. 69 And render the static LEEX template
70. 70 <div id="phx-20gvOvqvFMA=" data-phx-view="PointingParty.CardsLiveView" data-phx-session="SFMyNTY.g3QACZAAEZGFY..."> ... </div>
71. 71 This will send the LiveView socket connect request
72. 72 Opening a bi-directional, stateful connection
73. 73 Over which the LEEX template will re-render
74. 74 3. Handling Changes
75. Roadmap 3. Handling Changes 2. Connecting to LiveView 75 1. Configuring LiveVIew
76. 76 <div> <%= if !@is_pointing do %> <div> <button> Start the Party! </button> </div> <% else %> <!-- card display here --> <% end %> </div>
77. We Will • Add a DOM element binding that will send an event from the client to LiveView 77
78. We Will • Add a DOM element binding that will send an event from the client to LiveView • Teach our LiveView how to handle this event 78
79. 79 Add the DOM element binding
80. 80 <button phx-click="start_party"> Start the Party! </button>
81. 81 The phx-click event will be sent over the WS to our LiveView
82. 82 Which will invoke the matching handle_event/3 function
83. 83 def handle_event("start_party", _value, socket) do # coming soon! end
84. 84 def handle_event("start_party", _value, socket) do [current_card | remaining_cards] = Card.cards() socket = assign(socket, is_pointing: true, current_card: current_card, remaining_cards: remaining_cards) {:noreply, socket} end
85. 85 Re-rendering the template
86. 86 <div> <%= if !@is_pointing do %> <div> <button phx-click="start_party"> Start the Party! </button> </div> <% else %> <!-- card display here --> <% end %> </div>
87. 87 <div> <h2><%= @current_card.title %></h2> <p><%= @current_card.description %></p> <form phx-submit="vote"> <label for="story-points">Story Points</label> <select name="points"> <option value="0">0</option> <option value="1">1</option> <option value="3">3</option> <option value="5">5</option> </select> </form> </div>
88. Your Turn!
89. Fork this repo
90. Feature Roadmap 2. Users see list of present users 3. All votes are tallied, displayed 1. User initiates pointing round 90
92. User initiates pointing round 92 Define the LiveView • Define a LiveView and LEEX template - We did this for you :) • Render the LiveView from the controller - We did this for you :) Handle Events • Add a phx-click event to the “start pointing” button • Define a handle_event/3 function to match this event and update socket.assigns accordingly Configure LiveView • We did this for you :)
93. Building Bulletproof Real-Time Applications Part 2: Phoenix PubSub
94. 94 We Have a Problem
95. 95 The pointing round is only started for the single user who clicked “start pointing”
96. 96 How can we broadcast shared changes to all of our users?
97. 97 We need PubSub!
98. 98 Phoenix PubSub
99. What is PubSub? PubSub (“publish/subscribe”) is a pattern in which we publish messages to a “topic”, such that those messages can be consumed by any number of subscribers. 99
100. What is Phoenix PubSub? The Phoenix PubSub library allows us subscribe Elixir processes to a shared topic and publish messages to those processes. Phoenix Channels abstract away the interactions with Phoenix PubSub, but you can also use the PubSub library directly, which is exactly what we’ll do from within our LiveView. 100
101. Phoenix PubSub and Distributed Elixir Phoenix PubSub is configured by default to be distribution-friendly with Phoenix.PubSub.PG2. With this adapter, messages are broadcast to processes sharing a topic across nodes. 101
102. 102 How It Works
103. 103 Subscribe users to a shared topic
104. 104 A user “joins” a pointing room when they mount the LiveView
105. 105 This is when we will subscribe that LiveView process to a shared PubSub topic
106. 106 Broadcasting Messages
107. 107 When a user sends an event to LiveView, we will broadcast it over PubSub to all the other users’ LiveViews
108. 108 Our LiveView code will implement a handle_info/3 function that receives the broadcast, updates state, and causes the template to re-render
109. 109 Client Submit “Start_pointing” event LiveView Client Client LiveView LiveView PubSub Broadcast to subscribers handle_info and update socket Re-render template
110. Building Bulletproof Real-Time Applications Part 3: Phoenix Presence
111. 111 We Have A Problem
112. 112 We don’t know which users are participating, who has voted or what their vote is!
113. 113 We don’t know which users are participating, who has voted or what their vote is!
114. 114 How can we sync and share stateful info, like who is present, across LiveView processes?
115. 115 Phoenix Presence
116. What is Phoenix Presence? The Phoenix.Presence module allows us to: • Register a process under a given topic • Store that info in a decentralized and resilient data store. • Broadcast presence-related events and sync presence data with ease. 116
117. 117 Phoenix Presence + Distributed Elixir =
118. Phoenix Presence Uses a CRDT A CRDT (Conflict-free Replicated Data Type) backs Phoenix Presence. What’s so special about a CFRDT? Unlike centralized data stores like Redis, Phoenix Presence… 118
119. “ …gives you high availability and performance because we don't have to send all data through a central server. It also gives you resilience to failure because the system automatically recovers from failures. - Chris McCord 119
120. 120 How It Works
121. Configure Presence • Define a module that uses Phoenix.Presence and shares a PubSub server with the rest of our application. 121
122. Register a Process • Use the Presence.Track behaviour to register a given user’s LiveView process under a shared topic 122
123. Broadcast Presence to Existing Users • Teach LiveView to handle the presence broadcast by fetching the list of present users from the presence data store 123
124. Building Bulletproof Real-Time Applications Part 4: Property-based testing with Stream Data
125. Testing Achieving 100% confidence there are no bugs (Just kidding.) 125
126. Pascal’s Triangle
127. 127 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
128. 128 1
129. 129 1 1 1
130. 130 1 1 1 1 2 1
131. 131 1 1 1 1 2 1 1 3 3 1
132. 132 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1
133. 133 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
134. “ 134 Given an integer n >= 0, write a function that returns the nth row of Pascal’s Triangle.
135. Possible Code Implementation defmodule Pascal do def row(n) do # Amazing calculations end end 135
136. Example-Based Unit Testing Stuff you’re probably used to. 136
137. Row 0 describe "row/1" do test "returns [1] for first row" do assert Pascal.row(0) == [1] end end 137
138. Row 1 describe "row/1" do test "returns [1, 1] for second row" do assert Pascal.row(1) == [1, 1] end end 138
139. Row 4 describe "row/1" do test "returns value for 5th row" do assert Pascal.row(4) == [1, 4, 6, 4, 1] end end 139
140. Advantages
141. Advantages ◦ Easy to reason about ◦ Target specific test cases
142. Disadvantages
143. Disadvantages ◦ Requires you to get edge cases right ◦ Doesn’t cover large indexes ◦ Doesn’t require a deep understanding of your domain
144. Stuff that’s going to feel hard. 144 Property-based Testing with StreamData
145. Streams of Random Data Using Generators 145
146. Booleans StreamData.boolean() |> Enum.take(5) # [false, true, false, false, false] 146
147. Integers StreamData.integer() |> Enum.take(5) # [0, -2, -2, 4, 3] 147
148. Strings StreamData.string(:alphanumeric) |> Enum.take(5) # ["", "", "vL2", "AbgJ", "XuH7"] 148
149. You Can Combine 149
150. Lists list = StreamData.list_of( StreamData.integer() ) Enum.take(list, 5) # [[], [-2, 1], [-1], [-2, -2, -1], []] 150
151. Maps map = StreamData.map_of( StreamData.atom(:alphanumeric), StreamData.integer() ) Enum.take(map, 5) # [%{}, %{}, %{iB3: -3, inF: -3, k: -1}, %{BSy: 2, z: -1}, %{}] 151
152. Lists of Maps StreamData.list_of( StreamData.map_of( StreamData.atom(:alphanumeric), StreamData.integer() ) ) 152
153. Lists of Maps of Lists of Integers StreamData.list_of( StreamData.map_of( StreamData.atom(:alphanumeric), StreamData.list_of(StreamData.integer()) ) ) 153
154. 154 [ [%{gc: []}], [%{O: [-1]}, %{f: [1, -2]}], [%{}], [ %{_: [], kTy: [4, -2, 0], u2h: [4, -2], w: [0, -3, 2, -4]}, %{_CK: [-2, -3, 3]} ], [ %{r: [-3, 2, -1, 2]}, %{_0: [1], _I: [-5, 0, 4, -3], _O: [-3, 3, 0], kz: [1, 4, 5, -1, -5]}, %{fr: [0, -2, 1, 3, 4]}, ] ]
155. Data Starts Small And gets larger 155
156. Integers, Revisited StreamData.integer() |> Enum.take(25) # [0, -2, 1, -3, -1, 1, 4, 7, -1, 9, 0, 8, 9, -12, 12, -5, 0, -7, -16, 8, 21, 1, -16, -7, -19] 156
157. 157 Lots of Tests, Data Shrinking
158. 158 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
159. Possible Test Implementation property "number of row elements is n+1" do check all number <- StreamData.integer(), row_index = abs(number) do element_count = length(Pascal.row(row_index)) assert element_count == row_index + 1 end end 159
160. assert length(Pascal.row(0)) == 1 160
161. assert length(Pascal.row(0)) == 1 assert length(Pascal.row(1)) == 2 161
162. assert length(Pascal.row(0)) == 1 assert length(Pascal.row(1)) == 2 assert length(Pascal.row(0)) == 1 162
163. assert length(Pascal.row(0)) == 1 assert length(Pascal.row(1)) == 2 assert length(Pascal.row(0)) == 1 assert length(Pascal.row(2)) == 3 163
164. assert length(Pascal.row(0)) == 1 assert length(Pascal.row(1)) == 2 assert length(Pascal.row(0)) == 1 assert length(Pascal.row(2)) == 3 # … assert length(Pascal.row(47)) == 48 164
165. assert length(Pascal.row(0)) == 1 assert length(Pascal.row(1)) == 2 assert length(Pascal.row(0)) == 1 assert length(Pascal.row(2)) == 3 # … assert length(Pascal.row(47)) == 48 assert length(Pascal.row(58)) == 117 165
166. assert length(Pascal.row(0)) == 1 assert length(Pascal.row(1)) == 2 assert length(Pascal.row(0)) == 1 assert length(Pascal.row(2)) == 3 # … assert length(Pascal.row(47)) == 48 assert length(Pascal.row(58)) == 117 # ?!?! 166
167. Stuff that’s true of any solution 167 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
168. Stuff that’s true of any solution ◦ Each row is symmetrical 168 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
169. Stuff that’s true of any solution ◦ Each row is symmetrical ◦ The first and last elements of each row are 1 169 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
170. Stuff that’s true of any solution ◦ Each row is symmetrical ◦ The first and last elements of each row are 1 ◦ After row 0, the second element in each list is the row index 170 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
171. Stuff that’s true of any solution ◦ Each row is symmetrical ◦ The first and last elements of each row are 1 ◦ After row 0, the second element in each list is the row index ◦ So is the second-to-last element 171 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
172. Stuff that’s true of any solution ◦ We can represent any row as a list 172 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
173. Stuff that’s true of any solution ◦ We can represent any row as a list ◦ Row 0 just has one element: 1 173 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
174. Stuff that’s true of any solution ◦ We can represent any row as a list ◦ Row 0 just has one element: 1 ◦ For all rows after 0, 1 appears twice 174 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
175. Stuff that’s true of any solution ◦ We can represent any row as a list ◦ Row 0 just has one element: 1 ◦ For all rows after 0, 1 appears twice ◦ Each row n has n+1 elements in it 175 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
176. Stuff that’s true of any solution ◦ We can represent any row as a list ◦ Row 0 just has one element: 1 ◦ For all rows after 0, 1 appears twice ◦ Each row n has n+1 elements in it ◦ In each row, numbers go up, then down 176 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 …
177. Potential Properties ◦ Data type ◦ Data shape ◦ Count of specific values ◦ Presence/emptiness ◦ Size ◦ Membership ◦ Range ◦ Symmetry ◦ Relationships
178. Testing Votes
179. Voting Results def calculate_votes(users) do case winning_vote(users) do top_two when is_list(top_two) -> {"tie", top_two} winner -> {"winner", winner} end end
180. A Property Test property "calculated vote is a list or an integer" do check all users <- user_generator, {_event, winner} = calculate_votes(users) # We'll assert something here later end end
181. A Missing Generator setup do # Create generator here [user_generator: user_generator] end
182. Your Turn
183. Properties of Our Code
184. What’s true of any solution? def calculate_votes(users) do case winning_vote(users) do top_two when is_list(top_two) -> {"tie", top_two} winner -> {"winner", winner} end end
185. Your Turn Writing our properties and assertions
186. I :heart: StreamData And you should, too.
187. Resources ◦ “An introduction to property-based testing” by Scott Wlaschin ◦ “Choosing properties for property-based testing” by Scott Wlaschin ◦ Property-Based Testing with PropEr, Erlang, and Elixir and PropEr Testing by Fred Hebert ◦ SteamData documentation ◦ “Testing the Hard Stuff and Staying Sane” by John Hughes ◦ “Picking Properties to Test in Property Based Testing” by Michael Stalker 187
188. Recap