A talk for Craftsman Guild on my team's use of Clojure at Netflix. Describes good, bad, and ugly lessons learned from going from a pure-Java codebase to Clojure in production.
hard problems and get out of their way. We strive to increase the freedom of our employees as we grow, enabling them to move quickly as the industry evolves. With that freedom comes increased responsibility. High performers thrive in that environment and make great choices for Netflix.” http://jobs.netflix.com/who-we-are.html
stateless web services ◦ Support Social APIs used by many Netflix UIs/devices • 3 engineers • 1 supportive manager • 1 medium-size-ish existing Java codebase
A symbol, used to name things :a-keyword ; Used for enumerations and map keys {:key1 "value1" :key2 "value2"} ; HashMap [:a :vector 1 2 3] ; Like java.util.ArrayList '(this is :a :list 1 2 3) • All data objects are immutable by default • Data is composable.
3.14) ; Use let to bind values to local ; names (let [r 3.0 c (* 2.0 Math/PI r) a (* Math/PI (* r r))] {:radius r :circumference c :area a}) ;=> {:radius 3.0, :circumference 18.84955592153876, :area 28.274333882308138}
(fn [x] (* x x)) ; Use defn to define a named function (defn square [x] (* x x)) ; Call a function (square 3) ;=> 9 ; Pass a function as an argument (map square [1 2 3]) ;=> [1 4 9]
and expression (and (even? a) (odd? b)) -> (if (even? a) (if (odd? b) true false) false) • Clojure code is data • A macro is a function that takes code and returns new code • Invoked by the compiler ; A macro can also reduce boilerplate (rx/fn [a] (* 2 a)) -> (reify rx.Func1 (call [this a] (* 2 a)))
(subscriber/subscriber+ 12345)) ;=> { ... user data from subscriber service ...} ; Check if they're in A/B test 4567 (ab/allocation+ s 4567) ;=> nil ; Forcibly allocate them to test 4567 cell 2 (ab/allocate+ s 4567 2) ; Get social info (social/profile+ s :netflix) ;=> { ... social connection status ...} Separate tool, free from constraints of Netflix platform
One-off maintenance jobs like database migrations • Dip toes into Netflix platform from Clojure • Dip toes into Netflix build infrastructure (Ant, Ivy, and Jenkins, oh my!)
Take it easy • No need to throw out working code • Clojure has good Java interop • When you need to write new Java, write Clojure instead • Tastefully add abstractions as you go
can! • Collect args, call a single entry point Be careful with caching. Breaks interactive model. ; Some Clojure code (ns com.netflix.mine) (defn func-to-call [x] (* 2 x)) // Invoke it from Java final Var require = RT.var("clojure.core", "require"); require.invoke(Symbol.intern("com.netflix.mine")); final Var funcToCall = RT.var("com.netflix.mine", "func-to-call"); assertEquals(198L, funcToCall.invoke(99L));
◦ Still a lot of Java so we have some helper macros for Mockito • Custom JUnit4 test runner that finds and runs Clojure tests ◦ Run tests from Eclipse or wherever ◦ Tests magically appear in jenkins • There are many options here. Our approach is pragmatic, bowing to playing nicely within Netflix build infrastructure
Pojos already! • Again, we’ve been pragmatic • For existing code, for the most part, stick with existing Java objects • For new code, use plain maps and simple Clojure types • Occasional conversion functions where “old meets new”
doesn’t magically go away in a dynamic language ◦ I still need to get the Cassandra Keyspace object to functions that use it ◦ It’s still DI even if the dependency is a simple function instead of an object implementing and interface • We currently take the “big context map” approach. Have other ideas we’d like to try
good at, but still isn’t perfect • … but I have broken prod by moving a Java class ref'd by property files • 1/10th the code that does 10x as much. Maybe it's not so bad? • Still a pain point, especially if test coverage is low...
else’s clojure code • Given an undocumented function with arbitrary args, what does it accept/produce? • Need more discipline about documentation, pre/post-conditions, schemas
clojure.lang.Compiler.eval(Compiler.java:6619) at clojure.lang.Compiler.eval(Compiler.java:6582) at clojure.core$eval.invoke(core.clj:2852) at clojure.main$repl$read_eval_print__6588$fn__6591.invoke(main.clj:259) at clojure.main$repl$read_eval_print__6588.invoke(main.clj:259) at clojure.main$repl$fn__6597.invoke(main.clj:277) at clojure.main$repl.doInvoke(main.clj:277) at clojure.lang.RestFn.invoke(RestFn.java:1096) at clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__1610.invoke (interruptible_eval.clj:56) at clojure.lang.AFn.applyToHelper(AFn.java:159) at clojure.lang.AFn.applyTo(AFn.java:151) at clojure.core$apply.invoke(core.clj:617) at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1788) at clojure.lang.RestFn.invoke(RestFn.java:425) at clojure.tools.nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eva 41) at clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn__1651$fn__1 invoke(interruptible_eval.clj:171) at clojure.core$comp$fn__4154.invoke(core.clj:2330) at clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__1644.invoke (interruptible_eval.clj:138) at clojure.lang.AFn.run(AFn.java:24) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:680) OMG, the Stacktraces!?!?!?!?
; annotate a constant (def pi "Pi, more or less" 3.14) ; annotate a function (ann area [Double -> Double]) (defn area [r] (* pi (* r r))) https://github.com/clojure/core.typed A la carte static type checking for Clojure Powerful, but invasive
Brittle? ◦ False negatives due to state ◦ False negatives due to normal failures • Occasional Hacks ◦ Services tuned for production use. Differs from integration test patterns ◦ Caches!
linking fb id to customer leaves user in not_connected state" (fixture/ensure-test-user-disconnected) (sabot/inject (at hystrix.linkVisitorToFacebookId eval (throw (RejectedExecutionException. "test-link-visitor-to-facebook-id-failure"))) => (try+ (fixture/ensure-test-user-facebook-connected) (throw (Exception. "Request unexpectedly succeeded.")) (catch (comp #{503} :status) e (let [s (subscriber/subscriber+ (:customer-id fixture/netflix-user))] ; make sure status is restored in subscriber and fb id is removed (is (= "not_connected" (get-in s [:social :connection-status]))))))))) Sabot is a Clojure library for injecting specific, fine-grained failures into a request Magic here!
intent • Take it easy ◦ Programmers love to wrap, especially Java • You'll get it wrong at least once • Wrapping or abstracting is language design, i.e. hard
breakers, fallbacks // Define a command in Java public class GetUserCommand extends HystrixCommand<User> { private final HttpClient client; private final long id; public GetUserCommand(HttpClient client, long id) { this.client = client; this.id = id; } @Override protected User run() { return client.get("/user/" + id, User.class); } @Override protected User getFallback() { return User.missing(id); } } // ... and use it .... new GetUserCommand(client, id).execute();
(require '[com.netflix.hystrix.core :as hystrix]) (hystrix/defcommand get-user {:hystrix/fallback-fn (fn [client id] (User/missing id))} [client id] (.get client id User)) ; ... and use it. OMG, just a fn call!! (get-user client 12345) This is what we’re doing, just defining a function So why does the Hystrix command have to look so different? No boilerplate here...
is an asynchronous sequence ; Raw RxJava (Observable/zip (reify rx.util.functions.Func3 (call [this a b c] (+ a b c))) stream-1 stream-2 stream-3) https://github.com/Netflix/RxJava Initially, we use raw Java interop to implement the “Func3” interface. This is tedious.
is an asynchronous sequence ; With a macro (Observable/zip (rx/fn [a b c] (+ a b c))) stream-1 stream-2 stream-3) https://github.com/Netflix/RxJava I know! Macros are great for eliminating boilerplate! Enter rx/fn macro
is an asynchronous sequence ; With a function (Observable/zip (rx/fn* +) stream-1 stream-2 stream-3) https://github.com/Netflix/RxJava But a function can do better, allowing composition with existing Clojure functions
.withRow(1234) .withColumn(“name”) .withValue(“dave”) .withTtl(90) .execute() // Consider a typical method call putColumn(keyspace, 1234, “name”, “dave”, 90); • No representation of the operation • Closer, but without a lot of work still isn’t manipulable, introspectable, reusable etc
that emulate existing idioms in the host language (Clojure) will ◦ Look better ◦ Be easier to understand without research ◦ Play better with existing features • In Hystrix, defcommand is structurally identical to defn ◦ Easy to switch. Easy to understand. • Pigpen (mostly) has semantics identical to Clojure data pipelines
data transforms easy • Not so in Rx, especially for expressions with “forks” • Enter rx/let-o to take care of the details (rx/let-o [?user (get-user-o 123) ?friends (rx/mapcat (fn [u] (map get-friends-o (:friends u))) ?user) ?ab (rx/mapcat get-ab ?user)] (rx/merge ?user ?friends ?ab))
gains in productivity and satisfaction • Yes, it's scary • Mental shift required ◦ Clojure examples to Clojure “in the large” ◦ People are still figuring this out