Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Boost your JQ IQ [JSON Parsing]

Boost your JQ IQ [JSON Parsing]

Presented at All Day DevOps (ADDO) 2020

Shows tips and tricks to using JQ effectively for automation and scripting.

Rob Hirschfeld

November 12, 2020
Tweet

More Decks by Rob Hirschfeld

Other Decks in Technology

Transcript

  1. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Why JQ?

    Not because it’s elegant, fun or easy. JK! While APIs are JSON native, people aren’t. • Pretty Print • Validate JSON • Parse JSON • Search JSON • Build JSON
  2. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) A LOT!

    It’s so critical, we embedded it into our golang CLI so we didn’t have to install it everywhere. If you cp & suffix “drpcli” with “jq” then it becomes a JQ parser! How much does RackN JQ?
  3. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Distributed Infrastructure

    as Code with API-driven Provisioning All our API models and events are JSON All our states are JSON All our IaC templates are JSON (ok, we source them in YAML because readability) What does RackN do?
  4. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Two Reference

    JSON files one.json machine object { “Name”: “x”, “Uuid”: “123-XYZ”, “Params: { “foo”: “bar” } } all.json list of machine objects [ { “Name”: “test1” … }, { “Name”: “test2” … }, { “Name”: “test3” … } ]
  5. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) cat one.json

    | jq . cat one.json | jq .Name cat all.json | jq .[].Name cat all.json | jq .[0].Name cat all.json | jq .[2:4] JSON dot navigation
  6. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) cat one.json

    | jq .Params.foo cat one.json | jq .Params.gohai-inventory nope!… be careful cat one.json | jq .Params.[gohai-inventory] nope!… be more careful cat one.json | jq ‘.Params.[“gohai-inventory”]’ cat one.json | jq .Params.\“gohai-inventory\” JSON dot navigation
  7. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) JSON math

    cat all.json | jq length cat all.json | jq keys cat one.json | jq keys cat all.json | jq .[0] cat all.json | jq first cat all.json | jq last.Name
  8. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) JSON pipes

    cat one.json | jq “.Params | length” cat one.json | jq “.Params | keys” cat one.json | jq “.Params | keys | length” cat one.json | jq “.Params[‘gohai-inventory’] | keys” nope… yikes quoting! cat one.json | jq '.Params["gohai-inventory"] | keys' cat one.json | jq “.Params[\"gohai-inventory\"] | keys”
  9. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Why do

    I care about quotes? export gohai=$(cat one.json | jq -r ‘.Params | keys | last’) echo $gohai cat one.json | jq '.Params["$gohai"]' nope! cat one.json | jq “.Params[\"$gohai\"] | keys” yes!
  10. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) JSON finding

    cat all.json | jq ‘.[] | select(.Name==”test1”)’ Can use lots of logic! Like has, in, >, etc but, be careful of unexpected output! cat all.json | \ jq '.[] | select(.Name | startswith("test"))'
  11. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Building JSON

    maps (arrays) this is set operations…. be careful cat all.json | \ jq 'map(.Name | startswith("test"))' not what you think! cat all.json | \ jq 'map(select(.Name | startswith("test")))'
  12. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) JSON flags

    cat one.json | jq .Endpoint fix that with -n cat one.json | jq -n .Endpoint cat one.json | jq -r .Name handy for bash loops: cat all.json | jq -r '.[].Uuid'
  13. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Bash Loop

    Examples all=$(cat all.json | jq -r '.[].Uuid') for uuid in $all; do echo “ID: $uuid” cat all.json \ | jq ".[] | select (.Uuid == \"$uuid\") | .Name" done
  14. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Bleh, Quoting!

    Help! --arg can be handy to remove \” quotes export name=”test1” cat all.json | \ jq --arg n "$name" \ ‘.[] | select(.Name == $n) | .Uuid’
  15. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Pipe from

    variables export all=$(cat all.json) jq .[].Name <<< "$all" We mix in Golang Templates export params=”{{ .Machine.Params }}” jq -r .[“gohai-inventory”] <<< "$params"
  16. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Building JSON

    (better than by hand!!) export base='{"foo": "bar"}' adds jq ‘. += {“rob”:”zehicle”}’ <<< $base removes jq 'del(."foo")' <<< "$base"
  17. TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Thank you!

    Resources: Online Dev Tooling: https://jqplay.org Reference: https://stedolan.github.com/jq This is stable stuff! Last major release was 2018