Write ETL or ELT data processing jobs with bricolage.

Introduce bricolage that's a batch job framework used in Cookpad Inc. Also explain ETL and ETL data processing briefly.

Transcript

1. Write ETL or ELT data processing jobs with bricolage. @inohiro at Cookpad Inc.
 RubyKaigi 2019 LT
2. self.introduce •@inohiro on Twitter and GitHub •Struggling over data business in Cookpad Inc.
3. ETL and ELT https://www.quora.com/Why-are-most-companies-moving-from-ETL-to-ELT SFG
4. bricolage •A batch job framework ‣ Designed to work with AWS services but works with PostgreSQL also ‣ For both ETL and ELT, especially ELT ‣ Written in Ruby https://github.com/bricolages/bricolage
5. Why bricolage •Simple and flexible ‣ Especially if you write SQL mainly •But we also can use Ruby
6. Job •Is written in SQL, Ruby, or even execs other scripts •Declined a job class ‣Help to write frequent patterns jobs ‣load, unload, insert, insert-delta, createview, exec, rebuild-drop, rebuild-rename, adhoc, … •You can define dependencies of jobs as jobnet(s)
7. Let’s see a rebuild-rename job
8. Rebuild a summary table with backup
9. What bricolage does Drop old and temporary tables
 that created previously Create new table and 
 insert summaries ✨ Swap new and old tables ♻
10. Good points for rubyists •Write transformation scripts in Ruby ‣ Use Bricolage::CommandLineApplication class •Define your own useful methods and use them with ERB
11. Thank you •Try to use bricolage •Cookpad is looking for data engineers. Let’s talk $