JQ and YQ for data science - Quick introduction

Quick introduction to jq and yq for data science first-year students. In the presentation, we review the key capabilities of the tools and do a quick demo for using them with real data from the ETH Zurich library registry.

Transcript

1. for data science newcomers Oleg Nenashev January 23, 2023
2. Agenda • Quick self-intro • jq introduction • Small demo
3. > whoami --community @oleg_nenashev oleg-nenashev
4. > whoami --academia “I have a PhD in Design… Hardware Design” Oleg @ Open Source Design Meetup Automation and Industrial Control Automation and Computer Systems Components of computing and control systems Thesis - Reengineering of digital hardware, and embedding test modules and interfaces into devices described by multilevel models BSc - 2009 MSc - 2011 PhD - 2015 Applied Computer Science in Economics Secondary MSc - 2011
5. > whoami --industry Since 2016, Neuchatel 2008 - 2016, Russia
6. Getting started! Source: commitstrip.com/en/2018/08/24/http- headers-ftw/
7. • One of the most popular data formats • Service REST APIs • Metadata • [Un]structured data
8. What is jq? “jq is a lightweight and flexible command-line JSON processor” “jq is like sed for JSON data” stedolan.github.io/jq
9. “jq is like sed for JSON data” Input JSON Resulting JSON ❖ Filtering ❖ Search ❖ Sorting
10. Data Access with jq • Index slices • Field slices • Optional fields support
11. Search with jq • Selector queries • Search by value • Functional processing
12. Advanced jq features Syntax • Math • Comparisons • Variable binding • Iterators • Micro-scripting • Regular expressions Tool • Stream processing • Modules / Plugins • IO operations
13. FAQ: How about YAML? … and other formats
14. YAML Engineering is HOT
15. Working with YAML • YAML is [almost] like JSON • Many tools • yq - a wrapper for YAML • github.com/mikefarah/yq • Other formats are supported
16. Getting Started with jq
17. jq - Where to get it? • Linux - [Any?] package manager ◦ sudo apt install jq • Windows - Chocolatey ◦ choco install jq • Docker - official image ◦ docker pull stedolan/jq • Python - jq binding ◦ pip install jq • VSCode plugins
18. VSCode Plugins Live editing Queries with preview Linting and previews
19. Try out jq online! https://jqplay.org
20. Demo time! Data Analysis with jq
21. Let’s do Big Data! ETH Library library.ethz.ch/en/archiving-and-digitising/o pen-data/interfaces-and-downloads.html Source: https://www.servicemax.com/uk/fsd/2017/11/03/comic-brake-tho ught-big-data-difficult-manage-meet-ginormous-data/
22. Our data • Let’s look up for Renku publications • https://daas.library.ethz.ch/rib/v3/search?q=any,contains,%22Renku%22 Source: en.wikipedia.org/wiki/Japanese_poetry#/media/File:' Kokinshu'-_Anthology_of_Classic_Japanese_Poetry _LACMA_M.91.250.332.jpg
23. Query examples cat eth-library-renku.json | jq … • Selector - '.docs[1]' • Selector fields - '.docs[].pnx.display.title' • Entries about data: '.docs[].pnx.display | select(.title[0] | contains("data"))'
24. Learn More • jq documentation ◦ https://stedolan.github.io/jq/ • jq tutorials ◦ Szymon Stepniak: youtube.com/playlist?list=PLKaiHc24qCTSOGkkEpeIMupEmnInqHbbV ◦ https://www.baeldung.com/linux/jq-command-json • yq documentation ◦ https://mikefarah.gitbook.io/yq/
25. Contacts: E-mail: o.v.nenashev@gmail.com GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS?
26. Credits • All developers of JSON APIs • Stephen Dolan, jq author ◦ https://github.com/stedolan • Mike Farah, yq author ◦ https://github.com/mikefarah • OpenMoji project ◦ https://openmoji.org