Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Haskell >>= DevOps

Haskell >>= DevOps

How Haskell fits into the DevOps ecosystem.
TL;DR: it glues DSLs.

Volodymyr Kyrylov

October 08, 2015
Tweet

More Decks by Volodymyr Kyrylov

Other Decks in Programming

Transcript

  1. • Pla%orm So+ware / SRE at Zalora, CS undergrad at

    NTUU KPI • We're hiring to our distributed team!
  2. • configura*on management • running systems • monitoring • deployment

    / con*nuous delivery • immutable infrastructure • supervising failures • ... • system integra*on
  3. Microgram • Applica(on realms (infra + configs) • Applica(on defini(ons

    (how to run / scale / monitor) • User APIs • Run(me converts defini(ons to real things (like infra) • Run(me handles opera(ons & automates labor • Fail
  4. /etc/passwd nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false root:*:0:0:System Administrator:/var/root:/bin/sh daemon:*:1:1:System Services:/var/root:/usr/bin/false _uucp:*:4:4:Unix to Unix

    Copy Protocol:/var/spool/uucp:/usr/sbin/uucico _taskgated:*:13:13:Task Gate Daemon:/var/empty:/usr/bin/false _networkd:*:24:24:Network Services:/var/networkd:/usr/bin/false _installassistant:*:25:25:Install Assistant:/var/empty:/usr/bin/false _lp:*:26:26:Printing Services:/var/spool/cups:/usr/bin/false _postfix:*:27:27:Postfix Mail Server:/var/spool/postfix:/usr/bin/false _scsd:*:31:31:Service Configuration Service:/var/empty:/usr/bin/false
  5. /etc/pf.conf block return out on em0 inet all set queue

    std pass out on em0 inet proto tcp \ from $developerhosts to any port 80 \ set queue developers pass out on em0 inet proto tcp \ from $employeehosts to any port 80 \ set queue employees pass out on em0 inet proto tcp \ from any to any port 22 \ set queue(ssh_bulk, ssh_interactive) pass out on em0 inet proto tcp \ from any to any port 25 \ set queue mail
  6. sendmail.mc define(`confTO_CONNECT', `1m')dnl define(`confTO_IDENT', `0')dnl define(`confTO_COMMAND', `2m')dnl LOCAL_NET_CONFIG # This

    rule ensures that all local mail is delivered using the # smtp transport, everything else will go via the smart host. R$* < @ $* .$m. > $* $#smtp $@ $2.$m. $: $1 < @ $2.$m. > $3
  7. YAML tasks: - name: take out of load balancer pool

    command: /usr/bin/take_out_of_pool {{ inventory_hostname }} ## ^ string template delegate_to: 127.0.0.1
  8. "Programming" with JSON (CloudForma8on) "Outputs" : { "MyOutput" : {

    "Value" : { "Fn::Join" : [ "%", [ "A-string", {"Ref" : "AWS::StackName" } ] ] } } }
  9. Puppet Although Puppet’s language is built around describing resources (and

    the rela6onships between them) in a declara6ve way, several parts of the language do depend on evalua6on order case $operatingsystem { centos, redhat: { $service_name = 'ntpd' } debian, ubuntu: { $service_name = 'ntp' } } package { 'ntp': ensure => installed, }
  10. Bash • somewhat ubiquitous and mostly unportable • hard to

    scale • strings • more strings • unexpected EOF while looking for matching `"' • also strings
  11. bash = "${pkgs.bash}/bin/bash"; base64 = "${pkgs.coreutils}/bin/base64"; jq = "${pkgs.jq}/bin/jq"; curl

    = "${pkgs.curl}/bin/curl -s --retry --fail"; awk = "${pkgs.gawk}/bin/awk"; openssl = "${pkgs.openssl}/bin/openssl";
  12. Nix • func&onal language • dynamically typed • lightweight "schema

    valida&on" (hard to say typing) • one side-effect (derivation, used to build a package manager framework)
  13. Nix mk-scope = paths: let self = { __nixPath =

    let to = k: v: { prefix = k; path = v; }; base = filterAttrs (p: _: p != "") scope-to-as; path = mapAttrsToList to (base // paths); in path; import = fn: self.scopedImport self fn; scopedImport = attrs: scopedImport (self // attrs); builtins = builtins // self; }; in self;
  14. • built a microgram implementa1on effec1vely compiled to "bash on

    steroids" • used a package manager toolchain for this • EDSL experiments in a dynamic language • hard to make private APIs • tried to add more side effects (like IO) • somewhat boring to work on
  15. Some OSS infra tools • zalora/upcast - declara1ve infra provisioning

    (like nixops/ terraform/fugue) • zalora/replicator - automated MySQL replica1on • zalora/sproxy - proxy that handles OAuth2 + ACL interface • zalora/aws-ec2 - EC2 extensions for aris1db/aws • unicron, a single-user cron • a lot more on Zalora's GitHub
  16. Lessons learned so far • a lot of effort spent

    on was/ng /me integra/ng with stringy world (e.g. to enable bash scrip/ng or SQL) • OTOH many of those tools integrate beAer with the UNIX world • ghci shell is a lot more fun to use • having to wait for builds to finish is a /me sink
  17. Deployment with nixpkgs • quite powerful • nixpkgs haskell infrastructure

    is hard to get to work • curated dependency sets before it was cool • lots of dependencies (disk space bloat) • especially bad with "distributed" builds (h<ps:/ /ghc.haskell.org/ trac/ghc/@cket/4012) • s@ll a top contender
  18. Deployment with anything else • people tend to build programs

    right on the target host • copy-paste of binaries between iden8cal systems (halcyon) • docker makes it easy to make distribu8on bloated (and generally sad) • high hopes for stack
  19. Deployment with anything else • hard to get builds fully

    sta2c (like golang does) • GHC with Musl + integer-simple to get rid of LGPL • btw GPL applies only for soAware redistribu2on to public :) • seriously consider GHCJS + node?
  20. hard to deploy haskell at scale - don't deploy it!

    :) don't write CLI tools! haskell is ideal for wri.ng glue to integrate various systems (control-plane services)
  21. The glue toolbox • parser combinators + quasi-quoters let you

    integrate with a lot of outside world • building and interpre:ng an AST is also nice • turn exis:ng CLI tools into libraries/DSLs (CLI integra:on to service integra:on) • wrap exis:ng interfaces with types
  22. Migra&ng from dynamically typed Nix • take all schemas and

    convert them to haskell • use Nix as a "query" language inside to access exis9ng code/data • Rebindable syntax is cool! Though you can't rebind let. types = { nullOr = fake-type-of "Maybe"; string = fake-type "Text"; int = fake-type "Integer"; bool = fake-type "Bool"; attrsOf = fake-type-of "Map"; listOf = fake-type-of "List"; unspecified = fake-type "Value"; };
  23. Integra(ng with 3rd-party APIs • servant-client is awesome • not

    enough implementa1ons for things like swagger (remember WSDL?) • amazonka has a great tailor-made code generator for AWS • perhaps something like F# type providers one day?
  24. Interpre'ng effects • free / opera*onal / prompt / extensible-effects

    • Freer monads, More extensible effects, ICFP 2015
  25. Integra(ng with bash • can't fully replace bash with turtle

    • use a bash DSL! data Expr :: * -> * where E :: Executable -> e -> Expr e Pipe :: Expr e -> Expr e -> Expr e Seq :: Expr e -> Expr e -> Expr e Or :: Expr e -> Expr e -> Expr e Redir :: Expr e -> FilePath -> Expr e Env :: [Pair] -> Expr e -> Expr e Sudo :: Expr e -> Expr e SSH :: Hostname -> e -> Expr e -> Expr e
  26. Integra(ng with the rest of the stringy world • UNIX

    is a minefield for experimen5ng with parser combinators! • a lot of perf analysis or systems explora5on is done by analysing streams of text • use haskell if lost in awk+perl+sed • see proger/lxkit and zalora/gctuner
  27. Takeaways • define your domain • isolate your effects •

    apply transforms • stay away from boring 8me sinks • cheat • don't forget to bring added value from the start because nobody cares about rewrites
  28. There are infinitely many DSLs to do cool things There

    are not enough languages that are powerful enough to map those domains PS. Please don't use bash