Share
Embed
Microservices in Clojure
Lessons learned after 1.5+ years of developing/evolving microservices at Attendify. Ideas, problems, pitfalls, plans.
Oleksii Kachaiev
April 17, 2015
Transcript
37 slides
-
About Me ‣ Alexey Kachayev, @kachayev ‣ CTO at Attendify.com
‣ Clojure, Scala, Erlang engineer ‣ Active open source contributor ‣ Author of Fn.py library (Python) ‣ Hobbies: Haskell, Rust, CRDTs, compilers -
Agenda ‣ Product overview ‣ The big idea behind Microservices
‣ What we built and why ‣ Problems and pitfalls ‣ The Road Not Taken -
Attendify ‣ Mobile applications builder ‣ Thousands of mobile apps
‣ Private social networks in each application ‣ Real-time analytic ‣ Sponsored Posts (ads) ‣ EventWall for screen projection -
Microservices ‣ Your Server As a Function [1] ‣ Scaling,
multiple languages and bla-bla-bla… ‣ Hyped as well as NoSQL, BigData etc ‣ Just google it to find more information ‣ We use it because it’s convenient ‣ As well as split your code into small functions ‣ We moved from Django project almost 2 years ago -
Applicability ‣ If you don’t know how to split your
system into small services: -it’s too small to be split -you don’t know your system well enough -how are you going to scale your engineering team? -
Current State ‣ 7 services in Clojure (from a total
of 23) ‣ 82 RPC endpoints in Clojure (from a total of 290+) ‣ 17k+ LOC of Clojure code, 2850+ commits ‣ 4-6M requests handled each day ‣ 3 engineers work with Clojure on a regular basis ‣ Not only Clojure company (also Erlang, Scala, Go) -
Brief History ‣ Started 1.5 year ago ‣ With 2
services in Clojure (sophisticated data processing modules) ‣ Didn’t choose any of existing microservices framework or platform -
Ready-to-use Solutions ‣ All systems are targeted to fit in
predefined requirements (as any framework) ‣ We didn’t know all requirements in advance ‣ Requirements are subject to change (continuously) ‣ There is no “right way” ‣ Non-technical requirements (i.e. organization structure) are rarely portable -
Started From… ‣ JSON-RPC 2.0 protocol over HTTP transport ‣
Server: jetty & ring ‣ Service: implicit, ad-hoc definition, code copy & paste ‣ Deploy: JAR (uber), upstart, fab ‣ Discovery: URI with environment variables ‣ Security: HMAC request signature -
Next steps (1) ‣ Better JSON-RPC: -meta information -another multiplexing
procedure -named params -
Next steps (2) ‣ Deployment procedure: -move all fab commands
to shared library -save uberjars (each version) on S3 -ping and http-based health checker -report all activity to Slack -run:as (to connect local service to QA or Prod clusters) -
Next steps (3) ‣ Switched to httpkit (http server &
client) -better benchmarks but not really applicable for our case -wanted to use core.async for service definitions, but still using futures (it’s ok for us) -
More Services, New Problems ‣ logs: unification, collect/process ‣ errors
tracking: new type of errors (inter-service communication) ‣ auth: different levels and procedures ‣ metrics: collect, view, analyze ‣ protocol: dynamic typing is hard to scale -
Solutions So Far ‣ logs: used Loggly, not really a
problem now ‣ errors tracking: Rollbar for failure reports, either abstraction, timeouts handling as first class citizen ‣ metrics: used Graphite, now using InfluxDB ‣ protocol: schema library to params definition and validation -
Augustine ‣ shared library with s3-wagon ‣ defservice macro that
uses multimethod ‣ protocol definition/validation with schema ‣ auth level specification and control ‣ errors, exceptions and timeouts handling ‣ meta information, req/resp ID with flake algorithm -
Java ‣ You don’t need to know Java to write
Clojure ‣ Your http server/framework is written in Java ‣ “Java in Clojure” is easier than “Java in Java” ‣ Most probably you will deal with Java code somehow ‣ GC is your “good but very unpredictable” friend -
Java We Use ‣ io operations, streaming & buffering ‣
XLS reader ‣ base64 ‣ timers ‣ java.text.SimpleDateFormat -
Data Communication ‣ Databases: Riak, Redis, CouchDB, PostgreSQL ‣ Our
Clojure services are “data-centric” (mostly about data manipulations) ‣ “single data responsibility” sounds good, but doesn’t work in our case ‣ Databases are used a lot for cross-service communications to decrease inter-services coupling -
Actual Problems ‣ scaling is a hard problem even with
best instruments ‣ there is no “critical” problem that we can’t solve ‣ there is a big room for enhancements ‣ there is even bigger room for experiments -
Investigations (1) ‣ active investigations ‣ errors processing (even with
either, trying monads) ‣ distributed tracing (partially solved with req IDs) ‣ service discovery & (smart) load balancing ‣ binary protocol & TCP for inter-server communication -
Investigations (2) ‣ not really active investigations ‣ auto-generated SDKs
‣ back pressure control (looking at Hystrix) ‣ core.async (long story) ‣ tasks cancellation -
core.async (1) ‣ Your Server as a Transducer ‣ augustine
library accepts channel as a return type ‣ httpkit provides async interface ‣ but… futures work fine for us (still?) ‣ still experimenting… -
core.async (2) ‣ better timeouts ‣ better multiplexing ‣ easier
to deal with back-pressure control ‣ async abstractions are very leaky ‣ should reimplement most parts of the code ‣ hard to debug (just like futures) -
Finagle-Clojure ‣ github.com/finagle/finagle-clojure ‣ good interface to work with Thrift
‣ easy to start with basic template and docs ‣ inconvenient Scala runtime ‣ not-really-idiomatic Clojure ‣ no more comments for now (not using in production) -
Thoughts ‣ No regrets about our technical decision(s) ‣ We
have time to solve problems & concerns ‣ Clojure is ok for product development ‣ Clojure is ok when supporting old code ‣ Clojure development is hard when # engineers > 1** ‣ ** it’s hard to work with people in any case