Babashka and GraalVM: taking Clojure to new places @ Clojure NYC meetup

Online presentation given at the Clojure NYC meetup in July 2020.

Transcript

1. and GraalVM; taking Clojure to new places Michiel Borkent @borkdude 2020-07-11
2. • CLI tools with instant startup! (< 10ms) • clj-kondo: a linter for Clojure that sparks joy • jet: convert between JSON, EDN and Transit • No eval: dynamic classloader not supported! +
3. DSL -> scripting • Added a query DSL to jet, a GraalVM CLI:
 
 $ jet --query '(map :id)' <<< '[{:id 1} {:id 2}]'
 [1 2] • Extend this DSL to significant subset of Clojure?

4. • Native Clojure scripting tool, single binary, no JVM • Can be used to replace “the grey areas” of bash • Installable via script, brew (macOS, linux), aur (linux), scoop (Windows) $ time bb '(+ 1 2 3)'
 6
 0.00s user 0.00s system 67% cpu 0.013 total
5. CLJ scripting Runtime Impl Startup* Interop Windows Execution Threads clojure JVM Java ~1048ms + + Compiled + planck JSCore CLJS / JS ~ 728ms + - Compiled - joker Native Go ~7ms - + Interpreted - babashka Native GraalVM ~10ms + + Interpreted + *) measured with multitime -n10 <cmd> -e '(+ 1 2 3)' on Ubuntu Bionic with Intel i7-3770K CPU @ 3.50GHz
6. Babashka goals • Fast starting Clojure scripting alternative for JVM Clojure • Easy installation: README ⟶ grab binary ⟶ run within seconds • Familiar: targeted at JVM clojure users • Cross-platform: #{linux, macOS, Windows} • Interop with commonly used classes (System, File, java.time.*, java.nio.*) • Multi-threading support (pmap, future, ...) • Batteries included (tools.cli, cheshire, ...) + external libraries + pods
7. Babashka non-goals • Performance • Long running performance intensive processes: use the JVM • Provide a mixed Clojure/Bash DSL (rather: be compatible with JVM Clojure) • Replace existing shells like Bash (rather: play well with them)
8. Shell interaction $ ls | bb -i '(filter #(-> % io/file .isDirectory) *input*)' ("doc" "examples" "logo" ...)
9. Predefine functions $ export BABASHKA_PRELOADS="(defn is-dir? [f] (-> f io/file .isDirectory))"
 
 $ ls | bb -i '(filter is-dir? *input*)' ("doc" "examples" "logo" ...)
10. Scripts $ pst.clj
 04:58 #!/usr/bin/env bb (def now (java.time.ZonedDateTime/now)) (def LA-timezone (java.time.ZoneId/of "America/Los_Angeles")) (def LA-time (.withZoneSameInstant now LA-timezone)) (def pattern (java.time.format.DateTimeFormatter/ofPattern "HH:mm")) (println (.format LA-time pattern))
11. Cross platform!
12. Included libs / namespaces • clojure.{core, edn, java.shell, java.io, pprint, set, string, test, walk, zip} • clojure.tools.cli • clojure.core.async • clojure.data.csv • cheshire.core (JSON) • clojure.xml • cognitect.transit • clj-yaml • babashka.curl
14. Compatibility with JVM • No deftype, definterface, reify (defrecord, defprotocol added recently) • Code that does not use these constructs often works • Possible to load library code using existing mechanisms: classpath
15. Classpath ;; spec.clj
 (require '[spartan.spec :as s])
 (s/explain (s/cat :x int? :y keyword?) [1 #{:foo}]) $ export BABASHKA_CLASSPATH=$(clojure -Spath ...) $ bb spec.clj
 #{:foo} - failed: keyword? in: [1] at: [:y]
16. Selection of compatible libs • spartan.spec: clojure.spec.alpha (1) for bb • clj-http-lite: lighter fork of cli-http-lite • medley: "missing" clojure utility functions • regal: create regular expressions from EDN/hiccup • camel-snake-kebab: word case conversions • aero: library for explicit, intentful configuration. • nubank/docopt: docopt in Clojure
17. $ clojure
18. $ deps.clj $ deps.clj -Sdeps ... C:\>deps.clj -Sdeps ...
19. Decomplecting babashka Libraries coming out of babashka: • edamame: EDN/Clojure parser • sci: a Small Clojure Interpreter
20. edamame (def parsed (edamame/parse-string "#(+ 1 2 %)" {:fn true})) ;;=> (fn* [%1] (+ 1 2 %1)) (meta parsed) ;;=> {:row 1, :col 1, :end-row 1, :end-col 11} - EDN/code parser - GraalVM compatible (no eval!) - location metadata - opt-in code-like features
21. Small Clojure Interpreter (def f (sci/eval-string "#(+ 1 2 %)")) (f 1) ;;=> 4 - Clojure interpreter - Works on JVM / GraalVM / JS - Sandboxing
 - Works in CLJS advanced compiled apps
 - Also available on NPM
22. Sci from JavaScript https://observablehq.com/@jeroenvandijk/untitled/5 $ npm install @borkdude/sci
 $ node
 > const { evalString, toJS } = require('@borkdude/sci');
 > x = evalString("(assoc {:a 1} :b 2)")
 > toJS(x) { a: 1, b: 2 }
23. Malli: serializable schemas (def my-schema [:and [:map [:x int?] [:y int?]] [:fn '(fn [{:keys [x y]}] (> x y))]]) (m/validate my-schema {:x 1, :y 0}) ; => true (m/validate my-schema {:x 1, :y 2}) ; => false
24. Function CLI args $ jet --from json --keywordize '(comp keyword str/upper-case)' \
 <<< '{"a": 1}' {:A 1}
25. clj-kondo hooks • The problem: • Clj-kondo doesn't recognize syntax of user-defined macros • Existing solutions: • Add built-in support • :lint-as - only adequate for identical syntax • :unresolved-symbol - works for suppressing, but also suppresses useful information
26. clj-kondo hooks (ns mylib) (defmacro with-bound [binding-vector & body] ,,,)
27. clj-kondo hooks
28. clj-kondo hooks (hooks.with-bound/with-bound
 '(my-lib/with-bound [a 1 ...] ...))) 
 => (let [a 1] {:with-bound/setting true} (inc 1))
29. Sci: adding libs (require '[cheshire.core :as json])
 
 (def sci-opts
 {:namespaces
 {'cheshire.core
 {'generate-string json/generate-string}}}) 
 (sci/eval-string "(require '[cheshire.core :as json]) (json/generate-string {:a 1})" sci-opts)
 ;;=> "{\"a\":1}"
30. Projects using sci • borkdude/babashka • borkdude/clj-kondo (Clojure linter) • borkdude/jet (JSON, EDN, Transit conversion) • epiccastle/spire (Ansible in Clojure) • metosin/malli (serializable schemas) • retrogradeorbit/bootleg (static HTML generation) • chlorine (Atom Clojure plugin) • dundalek/closh (Clojure shell replacement, mixed Clojure/Bash DSL) • liquidz/dad (config management, e.g. complex install scripts) • alekcz/pcp (Clojure PHP replacement) • theiceshelf/firn (Org-mode static site generator)
31. Bootleg: static site CLI
32. nREPL
33. Babashka pods Problems: • Including more libs into bb will cause GraalVM to take more RAM, can't build on free CI anymore • Including more libs makes binary grow, while not all libraries are useful to everyone • How to access functionality from other platforms useful for scripting (Rust, Go, Python)?
34. • Initial solutions: • Feature flags (PostgreSQL, HyperSQL, ...). Still supported. • Access native libraries (hard, brittle, too static) Babashka pods
35. • Standalone CLIs that can act as libraries to babashka • Can be built in Clojure + GraalVM or any other language (Rust, Python) • Architecture similar to nREPL • Uses bencode and (JSON or EDN) for RPC Babashka pods
36. • babashka-sql-pods (PostgreSQL, HyperSQL) • pod-babashka-etaoin (browser automation) • bootleg (HTML generation) • pod-babashka-filewatcher (implemented in Rust) • pod-babashka-parcera (whitespace preserving Clojure parser) • pod-lispyclouds-docker Babashka pods
37. Filewatcher pod (Rust)
38. PostgreSQL pod (clojure)
39. Future • Babashka • book.babashka.org • bb.deps.edn / .babashka/config.edn • clojure.spec integration • Datomic client (optional via feature flag)
 • Sci • clojure.datafy • Smaller JS bundle by configuration • More control over duration / interrupts
40. Clojure + GraalVM • CLJ-1472: issue with GraalVM and locking macro • Solved in 1.10.2-alpha1 (thanks Rich and Alex!) • https://github.com/lread/clj-graal-docs • https://github.com/BrunoBonacci/graalvm-clojure/
41. Companies using babashka / sci Github #254 Add your company to the list:
43. Conclusion Clojure might not be the best language for everything, like scripting Clojure is the best language for scripting.
44. Thanks to sponsors • Sponsors: • Github sponsors • OpenCollective • Ko-fi • Patreon
45. Thank you! https://github.com/borkdude/babashka
 https://github.com/borkdude/sci And the Small Clojure Interpreter