To Hell and Back: Why choosing Elixir to build a startup from scratch was both a terrible and an awesome decision

Building a chatbot and an e-commerce startup (www.fredbots.com) with 30k+ lines of Elixir and Phoenix code and almost redoing from scratch a few times.

Microservices and distribution done wrong, getting saved by a small bunny, reaching 178 million messages/day in a 8GB machine.

Slides from my Elixir User Group SP presentation on 18/Mar/17 - http://meetup.com/pt-BR/elug_sp/.

Transcript

1. To Hell and Back Why choosing Elixir to build a startup from scratch was both a terrible and an awesome decision
2. Who Co-founder and CEO at Fred (www.fredbots.com). Self-taught developer since 8. Used longest: PHP and Unity3D w/ C#. Almost a Computer Engineer, a Lawyer, a Veterinarian and a Computer Engineer again, but dropped out. Ex-snake handler and fishing tackle sales clerk. Founded and closed 2 other startups (and many others still archived). Alfred Reinold Baudisch @AlfredBaudisch / alfred@fredbots.com
3. How a WhatsApp bot got famous and evolved as the Brazilian WeChat and conversational commerce platform (goo.gl/v0qlJM) Food Delivery WhatsApp bot with unofficial API Exploded on the news Banned by WhatsApp FredApp, “WhatsApp for Business”, standalone chat + bot platform Angel investment Giving Birth
4. “Ecosystem with multiple platforms that enable the creation and deployment of automated transactional flows through chat. Sell and provide customer service automatically through chatbots in messaging apps.” TL;DR; chatbots for everyone, for everything.
5. The road to Elixir 1. Needed realtime communication for chat app. 2. socket.io. 3. The WhatsApp Architecture Facebook Bought For $19 Billion (Erlang and ejabberd). 4. Dive into Erlang: . Back to socket.io. 5. Elixir: . 6. Phoenix Channels: . 7. OTP: . 8. Now: 30k+ Elixir lines of code.
6. (18 microservices)
7. Fred Router Microservice transparent communication and discovery through custom Protocol and Dispatchers BEAM <> BEAM BEAM <> HTTP HTTP <> BEAM HTTP <> HTTP Context: BEAM – Elixir/Erlang apps communication. HTTP: External services (any language or 3rd party).
8. Deployment and Hot Reload • TL; DR; . • Every project a set of problems and headaches. • Almost every new deployment a new set of problems and headaches, making CI and CD a . • Dealing with Umbrellas was even worse. Context: July/2016.
9. Unified common demands Transformed Umbrellas into single projects Fred Router dropped (may open source soon)
10. Message Processing Pipeline Message comes from external messaging services (Facebook Messenger, etc). (fire and forget, async) |> Converted to Fred (i.e. FB to Fred). |> Added local sequence according to external sequence. |> Processed by the Engine (bot) only one time in order, if out of order reschedule. Processing may take a long time, per message. |> Publish responses to services in order, if out of order reschedule. Since some bots deal with sensitive data (insurance, health, payments) no message can EVER BE LOST and ORDER MUST BE RESPECT. AT ALL COSTS. NO EXCEPTIONS.
11. 1. GenStage + Flow 2. Riak Core 3. External Solutions * Riak Core ☑ * Mantra: “BEAM gives you everything, avoid external solutions and tools”
12. Riak Core
13. Complexity Context: ~September/2016. Development was getting overwhelming. Young ecosystem with few to zero programmers for hire in Brazil. Costly and chaotic infrastructure and CI/CD pipeline (even after microservice reduction).
14. Bunny Salvation
15. Persistence EVERYWHERE Acknowledgments Requeuing, Rejection, Dead Lettering Once you learn the possibilities with AMQP and RabbitMQ there is no looking back.
16. Our RabbitMQ Implementation 1. Deal with connections: GenServer + Registry. 2. Crashes when receiving messages: out of the box RabbitMQ handles this with ACKs. 3. Crashes when publishing messages: every message is a supervised task + GenServer watcher + cold start from persistent store. 4. High pressure: pool of GenServer workers, with many connections and channels (thousands).
17. Still needed a fast, but persistent data store ——————> We also use ETS in lots of places, but unfortunately it is not persistent. Mnesia is , so a big no.
18. 178 million messages/day in a 8GB server (+ RabbitMQ, Redis and PgSQL cluster)
19. Final Elixir Thoughts OTP everywhere (many GenServers, Agents, Tasks). It seems dauting, but OTP is just a few concepts, so don’t be afraid. Don’t worry about writing code the “Elixir/functional way” in the beginning. You’ll learn and improve quickly while working with it. Don’t try to reinvent the wheel and do something in a certain way just because someone said it is the “BEAM way”. They aren’t building your product or doing your job. (Even if “someone” is an authority). No need to learn Erlang in the beginning (later you will end up learning it anyway and you will also end up reading the docs a lot).
20. We are looking for Alchemists! www.fredbots.com THANK YOU!