Productionizing Big Data - stories from the trenches

Presented at ScalaDays 2023 (Madrid, Spain)

1. Productionizing big data - stories from the trenches
2. Roksolana Diachuk •Engineering manager at Captify •Women Who Code Kyiv Data Engineering Lead •Speaker
3. AdTech methodologies deliver the right content at the right time to the right consumer AdTech
5. You have your pipelines in production What’s next?
6. Types of issues • Low performance • Human errors • Data source errors
7. Story #1. Unlucky query
8. Problem Drop 13 months of user profiles
9. Reporting
10. Problem 13 months hour=22042001
11. Loading mechanism loader.ImpalaLoaderConfig.periodToLoad: “P5D” loader.ImpalaLoaderConfig.periodToLoad: “P13M” val minTime = currentDay.minus(config.feedPeriod) 
 listFiles.filter(file => file.eventDateTime isAfter minTime)
12. Solution loader.ImpalaLoaderConfig.periodToLoad: “P5D” loader.ImpalaLoaderConfig.periodToLoad: “P1M” loader.ImpalaLoaderConfig.periodToLoad: “P13M” …
13. Story #2. Missing data
14. Data ingestion Data from Partner X Data costs attribution Extractor
15. Problem XX Advertiser ID, Language, XX Device Type, …, XX Media Cost (USD) X Advertiser ID, Language, X Device Type, …, X Media Cost (USD)
16. Solution • Rename old columns • Reload data for the week
17. Solution val colRegex: Regex = “””X (.+)“””.r val oldNewColumnsMapping = df.schema.collect { case oldColdName@colRegex(pattern) => (oldColName.name, (“XX “ + pattern)) } oldNewColumnsMapping.foldLeft(df) { case (data, (oldName, newName)) => data.withColumnRenamed(oldName, newName) }
18. XX Advertiser ID, Language, XX Device Type, …, XX Media Cost (USD) Solution
19. Story #3. Divide and conquer
20. Problem processing_time part-*.parquet filtering aggregations created part-*.parquet
21. • Slow processing • Large parquet files • Failing job that consumes lots of resources Problem
22. • Write new partitioned state • Run downstream jobs with smaller states • Generate seed partition column - xxhash64(fullUrl, domain) Solution
23. processing_time part-*.parquet created bucket=0 part-*.parquet part-*.parquet … bucket=9 part-*.parquet part-*.parquet processing_time part-*.parquet Solution
24. Story #4. Catch the evolution train
25. Data organisation evolution
26. Problem • Missing columns from the source • Impala to Databricks migration speed • Dependency with another team • Unhappy users
27. Log-level data Mapper Ingestor Transformer Data costs calculator Data costs attribution
28. Data costs attribution Data costs attribution Data extractor Impala loader
29. Data costs attribution Data extractor Impala loader Data costs attribution
30. Solution XX Advertiser ID, Language, XX Device Type, …, XX Partner Currency, XX CPM Fee (USD) XX Advertiser ID, Language, XX Device Type, …, XX Media Cost (USD) 26 columns 82 columns
31. Solution Data extractor New ingestion job
32. //final step is writing the data df.write .partitionBy(“event_date”, “event_hour”) .mode(SaveMode.Overwrite) .parquet(dstPath) Solution
33. Why this solution doesn’t work data_feed clicks.csv.gz views.csv.gz activity.csv.gz event_date clicks1.parquet clicks2.parquet
34. Impressions Clicks Conversions Attribution data source
35. Solution impressions clicks conversions clicks.csv.gz views.csv.gz activity.csv.gz
36. Story #5. Cleanup time
37. Corrupted data Data from Partner X Ingestor
38. Corrupted data Data from Partner X Ingestor IllegalArgumentException: Can't convert value to BinaryType data type
39. Solution • Adjust pipeline • Reload data for 3 days on S3 • Relaunch Databricks autoloader
40. Current solution impressions videoevents conversions impressions conversions Clicks clicks videoevents
41. Current solution impressions conversions clicks videoevents
42. Better solution impressions videoevents conversions impressions conversions clicks clicks videoevents
43. Conclusions
44. 2. Observability is the key 4. Plan major changes carefully 1. Set up clear expectations with stakeholders Prevention mechanisms 3. Distribute data transformation load
45. 2. Errors can be prevented 4. Data evolution is hard 1. Data setup is always changing Conclusions 3. There are multiple approaches with different tools
47. dead_ fl owers22 roksolana-d roksolanadiachuk roksolanad My contact info