Jinja and data manipulation

A first draft of my Jinja on Ansible presentation. How can we improve the user experience and make Jinja easier for typical Ansible use-cases.

Transcript

1. Jinja and data manipulation Dag Wieers @dagwieers LOADays 2018 – Antwerp
2. Who am I ? • Born as Dag Wieërs in Flanders, Belgium – Freelance Linux consultant • Doing Linux and Open Source since 1994 • Working for various companies (IT, Finance, Telco, Gov) – IBM, HP, EMC, Cisco, Punch; Euroclear, AXA, BNP Paribas, ING, KBC – Proximus, Telenet; Belgian Federal Police, Belgian Constitutional Court • Mostly as engineer/architect, but prefers hands-on too – The past 6 years this involved Ansible in various ways • Started developing Ansible from the very start (early 2012) • Wrote core functionality and basic modules: debug, fail, set_fact, mail, ... – Maintainer of: unarchive, xml, filetree, dense, hpilo, vmware_guest, Cisco ACI, IMC, … – Involved in: Core, Windows, VMware and Network/ACI communities
3. Jinja ? • One of many Python template engines (most popular ?) • Based on Django’s template engine • Features include: – Sandboxed execution – Compiles to optimal python code just in time – Easy to debug – Configurable syntax • Designed for Web-based templating in mind !
4. 3 types of delimiters (defaults) • Statements {% for item in items %} … {% endfor %} • Expressions {{ items|first }} • Comments {# If only I had more time, these examples would rock #} • Line statements (not worth mentioning)
5. Jinja and Ansible • Ansible builds on existing technology – SSH, YAML/JSON and Jinja • Ansible is not Web-related* ! – We actually care about data types, apart from strings – NativeEnvironment accepted upstream ! • http://jinja.pocoo.org/docs/2.10/nativetypes/ • https://github.com/ansible/ansible/pull/32738 • YAML and Jinja – A subtle-layered love/hate relationship – Let’s introduce a YAML ‘jinja’ type for Jinja expressions • https://github.com/ansible/proposals/issues/101
6. Jinja and Ansible... • Used for various things – Templating files (in-line or using template module) – Decision-making (e.g. when: or until:) – Data-manipulation (modeling facts or inventory data)
7. How to make playbooks readable • Move complexity out of playbooks 1. Start with well-designed inventory (possibly dynamic)** 2. Use templates to reduce playbook “spaghetti” 3. Logically group into self-suficient roles 4. Modify data and simplify expressions using Jinja2 filters** 5. Use (custom) lookup_plugins to iterate over collections 6. Push complex logic into (custom) modules (→ locality)
8. How to organize inventories and facts • Think about how you will be managing the data – Who’s going to maintain it – Where is it coming from – How should it be structured • Do not model inventory data for the task at hand ! • Do not duplicate inventory data for convenience ! • Use Jinja for manipulating data into structures you require for the job
9. Example #1: Past mistakes "ansible_date_time": { "date": "2018-04-21", "day": "21", "epoch": "1524299715", "hour": "10", "iso8601": "2018-04-21T08:35:15Z", "iso8601_basic": "20180421T103515915337", "iso8601_basic_short": "20180421T103515", "iso8601_micro": "2018-04-21T08:35:15.915448Z", "minute": "35", "month": "04", "second": "15", "time": "10:35:15", "tz": "CEST", "tz_ofset": "+0200", "weekday": "zaterdag", "weekday_number": "6", "weeknumber": "16", "year": "2018" }, hw_eth0: macaddress: 00:11:22:33:44:55 macaddress_dash: 00-11-22-33-44-55 "ansible_facts": { "ansible_all_ipv4_addresses": [ "192.168.122.1" ], … "ansible_default_ipv4": {}, … "ansible_enp0s25": { "active": false, "device": "enp0s25", "features": { "esp_hw_ofload": "of [fixed]", …
10. Example #2: Lists vs dictionaries • Lists - name: eth0 ipaddress: 1.2.3.4 netmask: 255.255.255.0 gateway: 1.2.3.1 mac: 00:ca:fe:ba:be:01 - name: eth1 ipaddress: 2.3.4.5 netmask: 255.255.255.0 mac: 00:de:ad:be:ef:01 • Dictionaries eth0: ipaddress: 1.2.3.4 netmask: 255.255.255.0 gateway: 1.2.3.1 mac: 00:ca:fe:ba:be:01 eth1: ipaddress: 2.3.4.5 netmask: 255.255.255.0 mac: 00:de:ad:be:ef:01
11. Data manipulation filters • Sorting lists or dicts: sort • Turning a dict into a list (and sort): dictsort • Modify/search lists of dicts: map • Filter lists of dicts: select / selectattr / reject / rejectattr • Group dicts by attribute: groupby
12. Problems with selecting data • Complex data-structures (nested dicts/lists) are beautiful – Until you start using them... – Iterating over nested dicts to selected data is hard – Raising unrecoverable errors on • missing attributes • empty lists • anything unsupported really • For the “aci-model” role we wrote aci_listify – But we plan to contribute something more generic
13. Examples 1) Only run when the target is listed on the output of the SCVVM server’s hyperv servers when: hostname is defined and hostname in hostvars[groups['winscvmm']|first].hyperv.stdout_lines 2) Get the IP addresses of all remaining cluster nodes (i.e. exclude the current target) candid: hosts: bdsol-aci51-candid-1: ansible_host: 10.48.16.161 bdsol-aci51-candid-2: ansible_host: 10.48.16.162 bdsol-aci51-candid-3: ansible_host: 10.48.16.163 cluster_addresses: "{{ hostvars|dictsort|selectattr('1.group_names', 'issuperset', ['candid'])| map(attribute='1.ansible_host')|reject('match', '^'~ansible_host~'$')|list }}"
14. Extending Jinja (in Ansible) is super easy • Jinja filters (aka. filter plugins) when: this|success == true • Jinja Tests (aka. test plugins) when: this is successful • Very easy, just python functions • Accepts arguments (first argument is piped in) • Jinja filters return any data type • Jinja tests return strictly booleans
15. Jinja and taste • Types and tests: inconsistent – https://github.com/ansible/ansible/pull/37517 • So 0 == false but 0 is not sameas false • Also true == 1.0 and true is number • And ’yes’ is sequence but also {} is sequence (iterable vs sequence ?) – Upstreamed: https://github.com/pallets/jinja/pull/824 • Functionally: incomplete – Standard filters are a weird bunch (if you’re not a web developer) – Be careful with input types • Syntactically: unlikable – Why on earth would we reuse the pipe-paradigm again ? – Terminology: Objects vs mappings vs dicts (Why ?) / Sequence vs list vs iterable – Templates with nested expressions on nested data, never happy indentation
16. Typical debugging problems • Debugging Jinja statements (online) – http://jinja.quantprogramming.com/ • Debugging playbooks – Use “debug” module and tags (like print statements) – Use “debug” callback plugin (more output) – Use “debug” strategy plugin • Interactive debug session for testing Jinja in real time
17. ERROR! A worker was found in a dead state • Out-of-memory due to missing brackets – Problem: memory: '{{ ansible_memtotal_mb*1024*1024|filesizeformat }}' – Solution: memory: '{{ (ansible_memtotal_mb*1024*1024)|filesizeformat }}' • This can happen to *everyone* – Imagine you expect an integer, but you get a string instead – https://github.com/ansible/ansible/issues/35344
18. Tell me your stories (or talk to me afterwards)