The Big Ball of Nouns

A talk about modular software design in Elixir given at CodeBEAM Lite Amsterdam 2018

Transcript

1. mkaszubowski94 Maciej Kaszubowski THE BIG BALL OF NOUNS
2. mkaszubowski94 Hi, I’m Maciej
3. mkaszubowski94 2+ years of Elixir
4. mkaszubowski94 Functional Programming
5. mkaszubowski94 immutable data no side effects explicit code
6. mkaszubowski94 immutable data no side effects explicit code simpler software
7. mkaszubowski94 and yet…
8. mkaszubowski94 time complexity
9. mkaszubowski94 time development progress complexity
10. mkaszubowski94 why is that?
11. mkaszubowski94 f(x) explicit input explicit output what we want
12. mkaszubowski94 f(x) input output what we want g(x) output
13. mkaszubowski94 input output what we have
14. mkaszubowski94
15. mkaszubowski94
16. mkaszubowski94
17. mkaszubowski94
18. mkaszubowski94
19. mkaszubowski94
20. mkaszubowski94
21. mkaszubowski94
22. mkaszubowski94 on a higher level, our systems are not very functional
23. mkaszubowski94 Database mutable state shared dependency complexity
24. mkaszubowski94 Object Object Object Object Object Object SHARED STATE SHARED STATE
25. mkaszubowski94 Maciej Kaszubowski What can we do?
26. mkaszubowski94 https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/ Event Sourcing / CQRS
27. mkaszubowski94 Domain Driven Design
28. mkaszubowski94 Actor Model https://www.brianstorti.com/the-actor-model/
29. mkaszubowski94 Microservices HTTP HTTP HTTP
30. mkaszubowski94 these are hard and risky
31. mkaszubowski94 a monolith, stateless app with a relational DB is still your safest option
32. mkaszubowski94 Maciej Kaszubowski Modular Software Design
33. mkaszubowski94 1972
34. mkaszubowski94 one big problem
35. mkaszubowski94
36. mkaszubowski94 Modular design smaller, simpler modules strong boundaries low coupling
37. mkaszubowski94 Each module can be: understood in isolation modified in isolation tested in isolation replaced or removed
38. mkaszubowski94 ok, but how?
39. mkaszubowski94 Maciej Kaszubowski Example Project
40. mkaszubowski94 create a new job name image url price publish John Alchemist create new job manage jobs find a job logout description date
41. mkaszubowski94 your jobs John Alchemist create new job manage jobs find a job logout my awesome job accept my other job cancel accept cancel yet another one publish and one more accepted this didn’t go well canceled
42. mkaszubowski94 browse jobs John Alchemist create new job manage jobs find a job logout my awesome job my other job yet another one and one more filters 100$ 50$ 3500$ 99$ xxxx yyy zzz search
43. mkaszubowski94 start this job John Alchemist create new job manage jobs find a job logout Job Title I need something done start a job https://example.com/jobs/1234 100$
44. mkaszubowski94 defmodule ModularElixir.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
45. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish()
46. mkaszubowski94 new feature: slugs
47. mkaszubowski94 start this job John Alchemist create new job manage jobs find a job logout my awesome job Don’t ask, you’ll get paid start a job https://example.com/jobs/my-awesome-job
48. mkaszubowski94 defmodule ModularElixir.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :price, :integer field :slug, :string field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
49. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug()
50. mkaszubowski94 new feature: pay before publishing
51. mkaszubowski94 create a new job name image url price continue John Alchemist create new job manage jobs find a job logout description date
52. mkaszubowski94 pay for job pay 110$ John Alchemist create new job manage jobs find a job logout my awesome job job price………….………..100$ fee (10%)…………………….10$ total……………………………..110$ Don’t ask, you’ll get paid Your job will be published after the payment
53. mkaszubowski94 defmodule ModularElixir.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
54. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() — accept() - cancel() - publish() - create()
55. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create()
56. mkaszubowski94 new feature: job reminders
57. mkaszubowski94 browse jobs John Alchemist create new job manage jobs find a job logout my awesome job my other job yet another one and one more filters 100$ 50$ 3500$ 99$ xxxx yyy zzz search Upcoming job Job starting in 54 minutes ! x
58. mkaszubowski94 defmodule ModularElixir.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
59. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
60. mkaszubowski94 jobs JobContext Payments preload jobs
61. mkaszubowski94 jobs JobContext User Context users join (jobs_count) Payments preload jobs
62. mkaszubowski94 jobs JobContext User Context users join (jobs_count) Payments preload jobs Chat Context
63. mkaszubowski94 jobs JobContext User Context users join (jobs_count) Payments preload jobs Chat Context JobSubtask Context
64. mkaszubowski94 jobs JobContext User Context users join (jobs_count) Payments JobSubtask Context preload jobs Chat Context JOB USER CONVERSATION MESSAGE PAYMENT SUBTASK
65. mkaszubowski94 In most projects: • few nouns / entities • a lot of interactions between them • logic is organised around the nouns • a lot of coupling as a result
66. mkaszubowski94 We’re used to the nouns • Object Oriented Programming • Databases • REST • CRUD
67. mkaszubowski94 nouns are not universal
68. mkaszubowski94 defmodule ModularElixir.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
69. mkaszubowski94 “job” means many different things
70. mkaszubowski94 Maciej Kaszubowski Extracting Behaviour
71. mkaszubowski94 the most important question: what is the desired outcome?
72. mkaszubowski94 1. what is the desired outcome?
73. mkaszubowski94 1. what is the desired outcome? 2. what data do I need to do this?
74. mkaszubowski94 1. what is the desired outcome? 2. what data do I need to do this? 3. how can I get this data?
75. mkaszubowski94 1. what is the desired outcome? 2. what data do I need to do this? 3. how can I get this data? 4. what should be exposed in the interface?
76. mkaszubowski94 Maciej Kaszubowski Extracting Slugs
77. mkaszubowski94 start this job John Alchemist create new job manage jobs find a job logout my awesome job Don’t ask, you’ll get paid start a job https://example.com/jobs/my-awesome-job
78. mkaszubowski94 desired outcome: jobs should be accessible via human-friendly slugs
79. mkaszubowski94 required data: id-slug mapping for jobs
80. mkaszubowski94 SEO Context
81. mkaszubowski94 SEO Context slugs
82. mkaszubowski94 SEO Context slugs - register_slug() - get_id_by_slug() - get_current_slug() Interface:
83. mkaszubowski94 def show(conn, %{"job_id"  slug}) do job = slug  SeoContext.fetch_id_by_slug("job")  JobContext.fetch_by_id() #  end
84. mkaszubowski94 def show(conn, %{"job_id"  slug}) do job = slug  SeoContext.fetch_id_by_slug("job")  JobContext.fetch_by_id() #  end Plug?
85. mkaszubowski94 defp put_slugs(jobs) do ids = Enun.map(jobs, & &1.id) slugs = SEOContext.fetch_slugs_by_ids(id) Enum.map(jobs, fn job  Map.put(job, :slug, slugs[job.id]) end) end
86. mkaszubowski94 defmodule ModularElixir.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
87. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
88. mkaszubowski94 SEO Context • slugs for other resources - easy to do
89. mkaszubowski94 SEO Context • slugs for other resources - easy to do • multiple slugs - easy to do
90. mkaszubowski94 SEO Context • slugs for other resources - easy to do • multiple slugs - easy to do • slugs are resolved at the controller level • no slugs in domain code
91. mkaszubowski94 no slugs in domain logic /api/jobs/my-awesome-job/subtasks from( s in Subtask, join: j in Job, on: s.job_id  j.id where: j.id  ^job_id or j.slug  ^job_id, # … ) /api/jobs/1/subtasks
92. mkaszubowski94 Maciej Kaszubowski Extracting Notifications
93. mkaszubowski94 browse jobs John Alchemist create new job manage jobs find a job logout my awesome job my other job yet another one and one more filters 100$ 50$ 3500$ 99$ xxxx yyy zzz search Upcoming job Job starting in 54 minutes ! x
94. mkaszubowski94 desired outcome: notification is sent when a job is starting soon
95. mkaszubowski94 required data: job information device information
96. mkaszubowski94 Job Reminders upcoming_jobs devices
97. mkaszubowski94 Job Reminders upcoming_jobs - put_job() - remove_job() - register_device() devices Interface:
98. mkaszubowski94 Job Reminders upcoming_jobs - put_job() - remove_job() - register_device() devices Interface: It’s hard to use (you have to remember to remove the job after starting)
99. mkaszubowski94 - register_device() Interface: Job Context search Job Reminders sent_reminders devices
100. mkaszubowski94 defmodule Notifications.Worker do use GenServer def handle_info(:work, state) do Notifications.send_notifications(now()) Process.send_after(self(), :work, @interval) {:noreply, state) end end
101. mkaszubowski94 defmodule Notifications do def send_notifications(datetime) do jobs = JobContext.search(…) already_sent = fetch_already_sent() sender = FirebaseAdapter :ok = Logic.send( jobs, datetime, already_sent, sender ) end end
102. mkaszubowski94 defmodule Notifications.Logic do def send(jobs, datetime, already_sent, sender) do Enum.each(jobs, fn job -> if starting_soon?(job) && not_sent_yet?(job, already_sent) do sender.send_notification(job) end end) end end
103. mkaszubowski94 logic is isolated from the data source
104. mkaszubowski94 Logic desired outcome Notifications Job Context required data data source
105. mkaszubowski94 Logic desired outcome Notifications required data events stream
106. mkaszubowski94 Logic desired outcome Notifications required data
107. mkaszubowski94 Logic desired outcome Notifications required data TESTS
108. mkaszubowski94 defmodule ModularElixir.Job do use Ecto.Schema schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
109. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
110. mkaszubowski94 Maciej Kaszubowski Extracting Publishing
111. mkaszubowski94 desired outcome: job should be published only after the payment is made
112. mkaszubowski94 desired outcome: job should be published only after the payment is made
113. mkaszubowski94 problem: …published where?
114. mkaszubowski94 problem: the same model is used for multiple behaviours
115. mkaszubowski94 JobContext • adding / publishing jobs • searching for available jobs • managing ongoing jobs • archive / history • …
116. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
117. mkaszubowski94 Job Publishing jobs Job Board jobs publish
118. mkaszubowski94 Job Publishing jobs Job Board jobs publish enforces job structure in params
119. mkaszubowski94 defmodule JobBoard do defmodule Job do @keys [:name, :description, ] @enforce_keys @keys defstruct @keys end def publish(%Job{} = job) do #  end end
120. mkaszubowski94 Job Publishing jobs Job Board jobs publish enforces job structure in params controls the rules for publishing
121. mkaszubowski94 Job Publishing - create() - get_drafts() - publish()
122. mkaszubowski94 defmodule ModularElixir.JobBoard.Job do use Ecto.Schema @schema_prefix “job_board” schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
123. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
124. mkaszubowski94 JobContext - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
125. mkaszubowski94 JobBoard - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
126. mkaszubowski94 JobBoard • can be tested in isolation (without mocking payments) • linear logic (without conditionals) • no risk of showing unpublished jobs
127. mkaszubowski94 JobPublishing • rules for publishing can be changed in isolation • can be used alongside other modules
128. mkaszubowski94 Job Board JobBoard params are enforced by the interface Job Publishing
129. mkaszubowski94 Job Board JobBoard params are enforced by the interface Admin
130. mkaszubowski94 Job Board JobBoard params are enforced by the interface TESTS
131. mkaszubowski94 JobBoard - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
132. mkaszubowski94 JobBoard jobs Ongoing Jobs jobs
133. mkaszubowski94 defmodule ModularElixir.JobBoard.Job do use Ecto.Schema @schema_prefix “job_board” schema "jobs" do field :name, :string field :description, :string field :image_url, :string field :paid, :boolean field :slug, :string field :started, :boolean field :notification_sent, :boolean field :price, :integer field :datetime, :utc_datetime belongs_to :creator, ModularElixir.User belongs_to :contractor, ModularElixir.User end end
134. mkaszubowski94 JobBoard - fetch_by_id() - search() - find_by_user_id() - start() - accept() - cancel() - publish() - create() - find_by_slug() - send_reminder()
135. mkaszubowski94 Maciej Kaszubowski Outcomes
136. mkaszubowski94 JobContext jobs
137. mkaszubowski94 JobContext • no clear responsibility • hard to change and understand • hard to test in isolation • grows with each new feature • hard to delete old features
138. mkaszubowski94 Job Publishing jobs Job Board jobs Reminders devices SEO slugs Ongoing Jobs jobs Job Archive jobs
139. mkaszubowski94 Smaller modules • clear responsibilities for each module • easy to change, refactor, understand • easy to test in isolation • tend to stay small • trivial to delete if no longer necessary
140. mkaszubowski94 focused on behaviour, not nouns
141. mkaszubowski94 the flow of the app is visible in the module structure and dependencies
142. mkaszubowski94 Job Publishing Job Board Ongoing Jobs Job Archive Reminders Payments
143. mkaszubowski94 the logic is much simpler
144. mkaszubowski94 UserContext users
145. mkaszubowski94 Auth credentials Profiles jobs Notifications devices
146. mkaszubowski94 Maciej Kaszubowski Isolate important decisions
147. mkaszubowski94 start with a list of important decisions
148. mkaszubowski94 Important decisions • what are the rules for publishing jobs? • when/what notifications are sent? • which jobs are visible on job board? • how is a contractor chosen for each job? • what happens after a job is started? when the job ends? • how payments work? • …
149. mkaszubowski94 then, create a module for each decision
150. mkaszubowski94 Maciej Kaszubowski Summary
151. mkaszubowski94 this is not a silver bullet
152. mkaszubowski94 it’s just a mindset
153. mkaszubowski94 focus on behaviour, not data
154. mkaszubowski94 put behaviour into modules
155. mkaszubowski94 Make each module • independent • easy to test • easy to understand • with a strong interface
156. mkaszubowski94 Maciej Kaszubowski THANKS!