ElixirConfUS 2017 - Phoenix After 100,000 lines

Transcript

1. Phoenix after 100,000 lines Renan Ranelli
2. whoami Renan Ranelli (also known as Milhouse) Senior Software Engineer @ (previously worked with Ruby & C# mainly)
3. whoami Renan Ranelli (also known as Milhouse) Senior Software Engineer @
4. Objectives ➔ Share my last 2 years experiences Working with Elixir in a "big" app ➔ Present & address concerns People have when considering Elixir for more "critical" projects ➔ Convince you it's great (& safe!) to use Elixir & Phoenix for big projects
5. First, a little context: What are we talking about?
6. Our context - Early stage startup (money was short) - B2B SaaS, "Enterprise application". We're not a "scalability" case. Sry. - Early stage ecosystem -- june 2015 - Elixir: 1.0.5 (1.5.1 today) - Phoenix: 0.14 (1.3.0 today) - Ecto: 0.13.1 (2.2.1 today) - No previous experience with Elixir in the "founding team"
7. Why we chose Elixir?!
8. Why we chose Elixir?! - Because it looks like Ruby? - No - Because it is super fast? - No - Because we needed to scale to a billion users overnight? - No… if we scaled to a million we would be rich - Because we were spending a lot of money on machines? - No. We didn't even have a product! - Because we understood the Erlang platform, weighted its pros and cons and found out that it was the best fit for our needs? - No. Definitely No.
9. Why we chose Elixir?! **Hireability**
10. Hireability We wanted a small but very effective & invested team. We then decided to use Elixir as a lever: "We are going to give you a great opportunity to work full time with Elixir & incredible people"
11. Some (common) concerns What are the risks of that choice?
12. Our fears concerns - Will we find libraries & other ecosystem related stuff? Will we need to build the world? - Will this whole thing turn into abandon-ware? - How productive is this framework? Should we be settling for something like Rails instead? - Will this be good for "complex" business logic? (which we do have lots) - Will this be stable in production? - (more particular to our case:) Will our hiring strategy work?
13. Our conclusion then: We won't know unless we try. Maybe in 2 years we'll be able to answer that
14. As expected, we had problems along the way.
15. Interacting with other databases - Interacting with "non-conventional" databases: In the beginning, we couldn't find good libraries to interact with Cassandra and ElasticSearch. - We ended up writing our own... - Today, we've got Xandra for Cassandra, but I don't know of any good alternative for ElasticSearch. Ping me if you do.
16. Authentication, Accounts, Users ... - At the time, we couldn't find anything ready for use… - We ended up writing our own... - Today, we've got a whole section in awesome elixir...
17. Massive Recompilations - With more than a thousand modules, we discovered that it is quite easy to get in a situation where a big chunk of the project is recompiled after changing just one file. - We suffered a lot, but got to understand the causes... - See http://milhouseonsoftware.com/2016/08/11/understanding-elixir-recompilation/ or my (unreleased) empex.co talk on this theme ~ demo ~
18. Timex many major updates - We started with Timex 0.15, and we're on 3.1.24; - We started before `Date`, `NaiveDateTime` and `DateTime` even existed. - That hurt. And we had **lots** of complex logic depending on time… and we also kind-of misused timezones... - Thank you core team for `{Naive,}DateTime.diff/3`
19. Dependency clash - Because Erlang lacked a hex.pm alternative for quite some time, some different libraries declare the same dependency name :/ - Unfortunately… there is no solution to this.
20. Deployment a lá "12 factor" - Deployment was a big unknown in the early days. - We had *lots* of trouble trying to mold releases into a "12 factor compliant" setup. Weird behaviours with environment variables... - We ended up running mix in production and we're doing exactly that until today. No problems since then :thumbs_up:
21. (and now some problems that were exclusively our fault)
22. Joining the "umbrella party" late - Our app started as a single phoenix app. - As the app grew large, we felt the need to split it into an umbrella application. - As you guessed, we weren't very well modularized. - Lots of pain to get this done. (It was our fault)
23. Fat Controllers - Is a terrible Idea. Don't do it. - See Chris' talk on Phoenix 1.3 ! https://www.youtube.com/watch?v=tMO28ar0lW8
24. Coarse testing - We started breaking down the fat controllers. - We got lazy and did not write much of unit tests as we were factoring out stuff from "fat controllers". Partially because things were too coupled, partially because of deadlines. - And that burned us. Bad. *Don't* ignore the testing pyramid. - In the beginning, everything was fine because the code in the controllers was concise and short. But that didn't last long.
25. Not using Ecto.Type - We reimplemented most of Ecto.Type functionality (very poorly) because we did not read the documentation thoroughly. - ~ code ~
26. Migration to a full SPA + GraphQL - We decided to reimplement the frontend as an SPA. Consuming a graphql api. - Rewriting was *not* a good idea. (it rarely is…) - **LOTS** of problems with React, ClojureScript, GraphQL, reinventing HTTP, yadda yadda. - IMHO that was our biggest mistake problem. It was not Phoenix or Elixir related at all. I will come back to this latter.
27. Incidental problems :s
28. But we also had awesome surprises!
29. Platform & operability! See https://www.youtube.com/watch?v=5SbWapbXhKo
30. Performance! (not only in production) See http://blog.plataformatec.com.br/2017/07/the-fallacies-of-web-application-performance/
31. Evolution! - Evolution of libraries, ecosystem, events and people interested. We started prior to phoenix & ecto 1.0; Both have come a *long* way. - Stability! Phoenix & Elixir we're beyond stable. We even forgot to add `Restart=on-failure` on systemd! - Even though the app got *really* big, it was much simpler to work with than {Rails,Spring} apps of similar size we worked with before
32. Contribution! - We landed a *lot* of patches in many popular libraries! (ecto, phoenix, erlcloud, plug, ja_serializer, timex, etc, etc) - We found a bug in the Elixir compiler! - We wrote some cool libraries! (like a stream-able zip encoder) - (and we also got some other stuff that we should be able to extract & open-source too)
33. Our hiring strategy... - It Worked! It really did! - We've got an awesome team of people (and growing!)
34. So, after 2 years, what about our initial concerns?
35. Our fears concerns - Will we find libraries & other ecosystem related stuff? Will we need to build the world? A lot before, a lot less today (always try not to!) - Will this become abandon-ware? Hell No! - How productive is this framework? Should I be settling for something like Rails instead? Phoenix is very productive! - Will this be good for "complex" business logic? Yep. No big issues - Will this be stable in production? As "stable" as "stable" can get. - (more particular to our case:) Will our hiring strategy work? It did.
36. Conclusion
37. Conclusion - Our biggest problems had nothing to do with Phoenix itself. - In 2015/2016 phoenix was enough to handle all our needs. Today it is even more so. Still lots of work to do, lots of opportunities to contribute. - Although Phoenix is new, it is *very well founded*. - The community *is* growing, the ecosystem *is* growing. Things are more and more *production ready & stable*.
38. If I were to build it all again, today, would I use Elixir & Phoenix? Hell YES!
39. Takeaway Phoenix *is* good for big "enterprise apps". The following estatement used to be very true: "you're trading maturity & productivity for performance". The amount of "maturity & productivity" we're trading is getting smaller and smaller. The future is bright. (and faster)
40. Obrigado!
41. Where to find me: Twitter: @renanranelli Github: /rranelli Blog: milhouseonsoftware.com