Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lambda the Ultimate Devops v2

Lambda the Ultimate Devops v2

Lambda the Ultimate Devops from Build Stuff Ukraine 2015

Volodymyr Kyrylov

November 23, 2015
Tweet

More Decks by Volodymyr Kyrylov

Other Decks in Programming

Transcript

  1. Demonstrate: • core func*onal programming values applied to pla5orm engineering:

    • transforma*on-driven programing • ADTs • types 1. increase development agility and flexibility 2. prevent from focusing on boring/obvious bugs
  2. Microgram • everything is an API • framework for composing

    (and running) alien so=ware ┌───────────┐ ┌───────────┬─────────┐ ┌───> app │ │ │ kernel │ ┌───────┐ │ └───────────┘ │ ├─────────┤ │infra? ├────────┐ │ ┌───────────┐ │ sdk │ openssl │ └────┬──┘realm ├──┴┬──> app │ │(nix-based)├─────────┤ └───────────┘ │ └───────────┘ │ │ nginx │ │ ┌───────────┐ │ ├─────────┤ └──> app │ │ │ myprog │ └───────────┘ └───────────┴─────────┘
  3. /etc/passwd nobody:*:-2:-2:Unprivileged User:/var/empty:/bin/false root:*:0:0:System Administrator:/var/root:/bin/sh daemon:*:1:1:System Services:/var/root:/bin/false _taskgated:*:13:13:Task Gate Daemon:/var/empty:/bin/false

    _networkd:*:24:24:Network Services:/var/networkd:/bin/false _lp:*:26:26:Printing Services:/var/spool/cups:/bin/false • nothing special, just TSV data
  4. /etc/pf.conf block quick from <bad_hosts> pass in on $ext_if proto

    tcp to $webserver port www \ (max-src-conn-rate 100/10, \ overload <bad_hosts> flush global) pass out on em0 inet proto tcp \ from $developerhosts to any port 80 \ set queue developers pass out on em0 inet proto tcp \ from any to any port 25 • macros, quick implies flow control
  5. sendmail.mc define(`confTO_CONNECT', `1m')dnl define(`confTO_IDENT', `0')dnl define(`confTO_COMMAND', `2m')dnl LOCAL_NET_CONFIG # This

    rule ensures that all local mail is delivered using the # smtp transport, everything else will go via the smart host. R$* < @ $* .$m. > $* $#smtp $@ $2.$m. $: $1 < @ $2.$m. > $3 • no comments
  6. "Programming" with JSON (CloudForma8on) { "Type": "AWS::ElasticLoadBalancing::LoadBalancer", "Properties": { "Listeners":

    [{ "SSLCertificateId": {"Fn::FindInMap": ["SSLMAP", "au", {"Ref": "EnvironmentType"}]}, }], "LoadBalancerName": {"Fn::Join": ["", ["app-web-", {"Ref": "EnvironmentName"}, ""]]} } } • people are actually expected to write that • lisp users can weep now
  7. YAML / Ansible tasks: - name: take out of load

    balancer pool command: /usr/bin/take_out_of_pool {{ inventory_hostname }} delegate_to: 127.0.0.1 • string interpola.on, apparently
  8. Puppet: resource-oriented DSL, isolated effects Although Puppet’s language is built

    around describing resources (and the rela6onships between them) in a declara6ve way, several parts of the language do depend on evalua6on order case $operatingsystem { centos, redhat: { $service_name = 'ntpd' } debian, ubuntu: { $service_name = 'ntp' } } package { $service_name: ensure => installed, }
  9. Puppet + Hiera Apparently trea+ng code and data separately is

    cool again! --- mysql::server::root_password: 'strongpassword' databases: gotcms: user: 'got' password: 'super_secret_db_password' host: 'localhost' grant: 'ALL'
  10. Building good Dockerfiles is mostly folklore FROM debian:jessie RUN apt-get

    update && \ apt-get install -y --no-install-recommends openjdk-7-jre-headless ADD http://someplace/zookeeper-3.4.6.tar.gz /opt/ RUN cd /opt && \ tar -zxvf zookeeper-3.4.6.tar.gz && \ mv zookeeper-3.4.6 zookeeper && \ rm -rf ./zookeeper-*tar.gz && \ mkdir -p /var/lib/zookeeper ADD entry.sh /entry.sh WORKDIR /opt/zookeeper EXPOSE 2181 2888 3888 VOLUME ["/var/lib/zookeeper", "/opt/zookeeper/conf", "/tmp/zookeeper"] ENTRYPOINT ["/entry.sh"]
  11. <?xml version='1.0' encoding='UTF-8'?> <project> <actions/> <description></description> <logRotator> <daysToKeep>7</daysToKeep> <numToKeep>-1</numToKeep> <artifactDaysToKeep>-1</artifactDaysToKeep>

    <artifactNumToKeep>-1</artifactNumToKeep> </logRotator> <keepDependencies>false</keepDependencies> <properties/> <scm class="hudson.scm.NullSCM"/> <canRoam>true</canRoam> <disabled>false</disabled> <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding> <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding> <triggers class="vector"/>
  12. <tr> <xsl:attribute name="style"> <xsl:choose> <xsl:when test="CONDITION"> <xsl:value-of select="'visibility: visible'"> </xsl:when>

    <xsl:otherwise> <xsl:value-of select="'visibility: collapse'"> </xsl:otherwise> </xsl:choose> </xsl:attribute> </tr>
  13. % cat databass.yaml sg: db-alpha.infra.zalora.io my: db-psi.infra.zalora.io id: db-psi.infra.zalora.io hk:

    db-psi.infra.zalora.io th: db-theta.infra.zalora.io ph: db-beta.infra.zalora.io vn: db-beta.infra.zalora.io
  14. • MariaDB channels are server-centric • our model is country-centric

    • countries can share servers have to aggregate!
  15. % cat databass.yaml | \ awk -F': ' '{print $2,

    $1}' | \ awk '{a[$1] = a[$1] (a[$1] ? "," : "") $2; } END { for(k in a) print k, a[k]; }' db-beta.infra.zalora.io ph,vn db-psi.infra.zalora.io my,id,hk db-alpha.infra.zalora.io sg db-theta.infra.zalora.io th
  16. Bash • a pi&all trap that looks easy • somewhat

    ubiquitous and mostly unportable • hard to scale • strings • more strings • unexpected EOF while looking for matching `"' • also strings
  17. Devops Success Stories 101 • when in doubt, "do it

    manually just this once" • when repeated 5 9mes, wrap with a string interpola9on library • if not enough, try to add flow control as a "declara9ve" construct • when failed, embed a random PL with unclear boundaries • of course, error handling is ad-hoc trial-and-error • weep
  18. db-slave-channels = let mapper = _: { db-name, masterhost, ...

    }: nameValuePair (to-key master-host) { inherit master-host; databases = [ db-name ]; }; reducer = { name, value }: all: all // { ${name} = value // { databases = all.${name}.databases or [] ++ value.databases; }; }; in fold reducer {} (mapAttrsToList mapper conf);
  19. bash = "${pkgs.bash}/bin/bash"; base64 = "${pkgs.coreutils}/bin/base64"; jq = "${pkgs.jq}/bin/jq"; curl

    = "${pkgs.curl}/bin/curl -s --retry --fail"; awk = "${pkgs.gawk}/bin/awk"; openssl = "${pkgs.openssl}/bin/openssl"; • automa'c dependency tracking!
  20. Nix • func&onal language • dynamically typed • lightweight "schema

    valida&on" (hard to say typing) • one side-effect (derivation, used to build a package manager framework)
  21. Case study: configuring Jenkins • string templa.ng again? • click

    through all the forms? • can we build a DSL?
  22. eris-sdk = erisJob { branch = "master"; shell = ''

    export SLACK_CHANNEL='#eris-facepalm' export SLACK_TIMEOUT=5m bin/slack make sdk ''; ssh-keys = [ credentials.hydrabot ]; triggers = [ (ghprb-trigger "eris-sdk") (github-push-trigger) ]; };
  23. mapAttrs' (realm-name: spec: deployJob "deploy-${realm-name}" { inherit realm-name; scm =

    eris_master; permissions = with spec; { build = humans-can-build ++ others-can-build; }; ssh-keys = attrValues credentials; })) (filterAttrs is-deployable realms);
  24. choice-parameter = { name , description ? "" , choices

    ? [ "this is a list" ] }: (term "hudson.model.ChoiceParameterDefinition" null [ (term "name" null name) (term "description" null description) (term "choices" { class = "java.util.Arrays$ArrayList"; } [ (term "a" { class = "string-array"; } (map (term "string" null) choices)) ]) ]);
  25. Run$mes • implementa*on of Apps APIs to linux services •

    same APIs - mul*ple backends • allows rapid experimenta*on: mesos/docker/diego/ ares
  26. apps runtimes targets ┌─────────┐ ┌──────────┐┌%%%%%%%%%%%%%%%%%%%┐ ┌───> ASG │ ┌───> mysql

    ││ nixos/systemd │──┼┐ └─────────┘ │ └──────────┘└───────────────────┘ ││ ┌───────────┐ │ ┌──────────┐┌###################┐ ││ ┌─────────┐ │ realm ├──┴┬──>strongswan││ adhoc │──┼┼───>baremetal│ └───────────┘ │ └──────────┘└───────────────────┘ ││ └─────────┘ │ ┌──────────┐┌&&&&&&&&&&&&&&&&&&&┐ ││ └──> someweb ││ ami ├──┘│ ┌─────────┐ └──────────┘└───────────────────┘ └──> ec2 │ └─────────┘
  27. Case study: containers • adhoc run+me deployed to a fleet

    of old RedHat 6 servers • adhoc apps were exposed to global state • adop+ng runc • +me to PoC - 3 hours • +me to produc+on - 12 hours (8 wasted tracing kernel code) • running on 2.6.32.602 with minor libcontainer patches
  28. infra = { ec2-instance = { web1 = m3large; web2

    = m3large; db-master = { infra, ... }: r3xlarge' { blockDeviceMapping."/dev/xvdm".disk = infra.ebs.database; }; cron = instance "m3.large"; }; elb = elb.defaults; ebs.database = { inherit (realm.ec2-args) region zone; size = 200; volumeType.gp2 = true; }; };
  29. Infrastructure Specs • nixops-inspired -> reimplemented in upcast • terraform

    implementa5on as an evening hack • autoscaling PoC was wri=en bash :O
  30. writeBashScript = name: script: let prelude = '' #!${pkgs.bash}/bin/bash set

    -e -o pipefail ''; in pkgs.runCommand name { inherit prelude script; } '' echo "$prelude" >> "$out" echo "$script" >> "$out" chmod +x "$out" ${ShellCheck}/bin/shellcheck \ "$out" '';
  31. • zalora/upcast - declara1ve infra provisioning (like nixops/terraform/fugue) • zalora/replicator

    - automated MySQL replica1on • zalora/sproxy - proxy that handles OAuth2 + ACL interface • zalora/aws-ec2 - EC2 extensions for aris1db/aws • unicron, a single-user cron • a lot more on Zalora's GitHub
  32. Correct by construc-on: safe Bash • harden your interoperability with

    type-safe APIs data Expr :: * -> * where E :: Executable -> e -> Expr e Pipe :: Expr e -> Expr e -> Expr e Seq :: Expr e -> Expr e -> Expr e Or :: Expr e -> Expr e -> Expr e Redir :: Expr e -> FilePath -> Expr e Env :: [Pair] -> Expr e -> Expr e Sudo :: Expr e -> Expr e SSH :: Hostname -> e -> Expr e -> Expr e
  33. Correct by construc-on: safe Nix data Nix :: * ->

    * where Null :: Nix () Lit :: Literal -> Nix a -- ... Var :: Index -> Nix a Lam :: Index -> Nix a -> Nix (Nix input -> Nix a) Ap :: Nix (Nix input -> Nix a) -> Nix input -> Nix a -- ... List1 :: Nix a -> Nix [Nix a]
  34. Typing the standard library listToAttrs :: Nix [Nix (Nix String,

    Nix a)] -> Nix (Attrset a) listToAttrs = Ap (Lit "builtins.listToAttrs") concatLists :: Nix [Nix [a]] -> Nix [a] concatLists = Ap (Lit "builtins.concatLists") map' :: Nix (a -> b) -> Nix [a] -> Nix [b] map' f xs = Ap (Ap (Lit "map") f) xs attrNames :: Nix (Attrset a) -> Nix [Nix String] attrNames = Ap (Lit "builtins.attrNames")
  35. Induc&on Composi&on! λ> let concatMapToAttrs = (\f -> listToAttrs .

    concatLists . map' f) λ> :t concatMapToAttrs concatMapToAttrs :: Nix (a1 -> Nix [Nix (Nix String, Nix a)]) -> Nix [a1] -> Nix (Attrset a)
  36. λ> :t null null :: Nix () λ> :t concatMapToAttrs

    null <interactive>:1:18: Couldn't match type ‘()’ with ‘a1 -> Nix [Nix (Nix String, Nix a)]’ Expected type: Nix (a1 -> Nix [Nix (Nix String, Nix a)]) Actual type: Nix ()
  37. Correct by construc-on: web APIs data DeploymentParams = -- ...

    data DeploymentStatus = -- ... type Deployments = "deployments.xml" :> Header "x-api-key" ApiKey :> ReqBody '[FormUrlEncoded] DeploymentParams :> Post '[XML] DeploymentStatus h"ps:/ /haskell-servant.github.io Type-safe generic programming!
  38. Interpre'ng effects wtf :: (Member Spec r, Member Upcast r)

    => Eff r (Map Machine ExitCode) wtf = infra >>= machines >>= traverse (`ssh` cmd) where cmd :: Commandline cmd = exec "wtf" [] • notable implementa-on: Eff-lang
  39. -- | Idempotent effects based on evaluating the spec (ok

    to cache). data Spec v where NixQuery :: FromJSON json => Query json -> Spec json Infra :: Spec Infras Stub :: Show a => a -> Spec () -- | Effects that require networking. data Upcast v where Machines :: Infras -> Upcast (Attrs Machine) NixBuild :: Query a -> Upcast [StorePath] NixInstantiate :: Query a -> Upcast [StorePath] MachineExec :: (Machine, Commandline) -> Upcast ExitCode SystemInstalls :: (Machine, StorePath) -> Upcast ExitCode NRNotification :: ApiKey -> [DNS] -> Upcast ()
  40. Integra(ng with the rest of the stringy world • UNIX

    is a minefield for experimen5ng with parser combinators! • a lot of perf analysis or systems explora5on is done by analysing streams of text • use haskell if lost in awk+perl+sed • see proger/lxkit and zalora/gctuner
  41. finally { • infrastructure tools come and go • they're

    filled with adhoc interfaces with poor composablity and/or expressiveness • "tradi=onal" composi=on prac=ces are too error- prone and divert your precious a?en=on to fixing irrelevant bugs • basic PLT tools like transforma=onal programming and type systems keep you sane