Share
Embed
Boost your JQ IQ [JSON Parsing]
Presented at All Day DevOps (ADDO) 2020
Shows tips and tricks to using JQ effectively for automation and scripting.
Rob Hirschfeld
November 12, 2020
Transcript
23 slides
-
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) NOVEMBER 12,
2020 Rob Hirschfeld, RackN Boost your JQ IQ -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Why JQ?
Not because it’s elegant, fun or easy. JK! While APIs are JSON native, people aren’t. • Pretty Print • Validate JSON • Parse JSON • Search JSON • Build JSON -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) A LOT!
It’s so critical, we embedded it into our golang CLI so we didn’t have to install it everywhere. If you cp & suffix “drpcli” with “jq” then it becomes a JQ parser! How much does RackN JQ? -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Distributed Infrastructure
as Code with API-driven Provisioning All our API models and events are JSON All our states are JSON All our IaC templates are JSON (ok, we source them in YAML because readability) What does RackN do? -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Two Reference
JSON files one.json machine object { “Name”: “x”, “Uuid”: “123-XYZ”, “Params: { “foo”: “bar” } } all.json list of machine objects [ { “Name”: “test1” … }, { “Name”: “test2” … }, { “Name”: “test3” … } ] -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) cat one.json
| jq . cat one.json | jq .Name cat all.json | jq .[].Name cat all.json | jq .[0].Name cat all.json | jq .[2:4] JSON dot navigation -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) cat one.json
| jq .Params.foo cat one.json | jq .Params.gohai-inventory nope!… be careful cat one.json | jq .Params.[gohai-inventory] nope!… be more careful cat one.json | jq ‘.Params.[“gohai-inventory”]’ cat one.json | jq .Params.\“gohai-inventory\” JSON dot navigation -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) JSON math
cat all.json | jq length cat all.json | jq keys cat one.json | jq keys cat all.json | jq .[0] cat all.json | jq first cat all.json | jq last.Name -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) JSON pipes
cat one.json | jq “.Params | length” cat one.json | jq “.Params | keys” cat one.json | jq “.Params | keys | length” cat one.json | jq “.Params[‘gohai-inventory’] | keys” nope… yikes quoting! cat one.json | jq '.Params["gohai-inventory"] | keys' cat one.json | jq “.Params[\"gohai-inventory\"] | keys” -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Why do
I care about quotes? export gohai=$(cat one.json | jq -r ‘.Params | keys | last’) echo $gohai cat one.json | jq '.Params["$gohai"]' nope! cat one.json | jq “.Params[\"$gohai\"] | keys” yes! -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) JSON finding
cat all.json | jq ‘.[] | select(.Name==”test1”)’ Can use lots of logic! Like has, in, >, etc but, be careful of unexpected output! cat all.json | \ jq '.[] | select(.Name | startswith("test"))' -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Building JSON
maps (arrays) this is set operations…. be careful cat all.json | \ jq 'map(.Name | startswith("test"))' not what you think! cat all.json | \ jq 'map(select(.Name | startswith("test")))' -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) JSON flags
cat one.json | jq .Endpoint fix that with -n cat one.json | jq -n .Endpoint cat one.json | jq -r .Name handy for bash loops: cat all.json | jq -r '.[].Uuid' -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Bash Loop
Examples all=$(cat all.json | jq -r '.[].Uuid') for uuid in $all; do echo “ID: $uuid” cat all.json \ | jq ".[] | select (.Uuid == \"$uuid\") | .Name" done -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Bleh, Quoting!
Help! --arg can be handy to remove \” quotes export name=”test1” cat all.json | \ jq --arg n "$name" \ ‘.[] | select(.Name == $n) | .Uuid’ -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Pipe from
variables export all=$(cat all.json) jq .[].Name <<< "$all" We mix in Golang Templates export params=”{{ .Machine.Params }}” jq -r .[“gohai-inventory”] <<< "$params" -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Building JSON
(better than by hand!!) export base='{"foo": "bar"}' adds jq ‘. += {“rob”:”zehicle”}’ <<< $base removes jq 'del(."foo")' <<< "$base" -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) ...And Don’t
Go Crazy In many cases, THE BEST JQ IS NO JQ -
TRACK: SITE RELIABILITY ENGINEERING SPEAKER: Rob Hirschfeld (@zehicle) Thank you!
Resources: Online Dev Tooling: https://jqplay.org Reference: https://stedolan.github.com/jq This is stable stuff! Last major release was 2018