Snowflake のアーキテクチャが どう「筋がいい」のかを解説する / Data Engineering Study #5

Data Engineering Study #5「噂のSnowflake Deep Dive」発表資料: https://forkwell.connpass.com/event/194269/
Youtube Live アーカイブ: https://www.youtube.com/watch?v=_kYW7EneUu4

Transcript

1. © 2020 Snowflake Inc. All Rights Reserved Snowflake Data Engineering Study #5 Yoshi Matsuzaki, Principal Cloud Support Engineer of Snowflake | Dec 2, 2020
2. © 2020 Snowflake Inc. All Rights Reserved ABOUT ME 2 Yoshi Matsuzaki Principal Cloud Support Engineer - Snowflake Work: Technical Support in English/Japanese (7:3) Launching Japanese language support team SQL functionality expert Escalation review Supportability readiness for new features Background: Infrastructure Engineer (MSP) Support Engineer (Cloud Vendor) Database Engineer of MySQL engines (Cloud Vendor)
3. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 3
4. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 4 DATA SOURCES OLTP DATABASES ENTERPRISE APPLICATIONS THIRD-PARTY WEB/LOG DATA IoT DATA CONSUMERS DATA MONETIZATION OPERATIONAL REPORTING AD HOC ANALYSIS REAL-TIME ANALYTICS → Data Platform → Data Cloud
5. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE ARCHITECTURE 5
6. © 2020 Snowflake Inc. All Rights Reserved Snowflake 6
7. © 2020 Snowflake Inc. All Rights Reserved 7 Snowflake 50-500 MB ( : S3) … Partition Pruning / NDV ( )
8. © 2020 Snowflake Inc. All Rights Reserved 8
9. © 2020 Snowflake Inc. All Rights Reserved 9 ( ) “ ( : immutable) ” - Wikipedia
10. © 2020 Snowflake Inc. All Rights Reserved 10 : (UPDATE) : → I/O … : →
11. © 2020 Snowflake Inc. All Rights Reserved 11 'a' 'b' 'c' Table Partitions UPDATE tab SET col = 'd' WHERE col = 'a'; 'a' 'b' 'c' Table Partitions 'd' v1 v2
12. © 2020 Snowflake Inc. All Rights Reserved 12 I/O … OLAP OLTP OLAP →
13. © 2020 Snowflake Inc. All Rights Reserved 13 / / : UNDO (MySQL, Oracle) UNDO (PostgreSQL) :
14. © 2020 Snowflake Inc. All Rights Reserved 14 → / → Time Travel 90 / SELECT * FROM tab AT (TIMESTAMP => ...); SELECT * FROM tab BEFORE (STATEMENT => <query-id>); CREATE TABLE tab CLONE src AT (TIMESTAMP => ...); → Fail Safe 7 GC ...
15. © 2020 Snowflake Inc. All Rights Reserved 15 Table SELECT col FROM tab; 'a' 'b' 'c' Partitions 'd' v1 v2 SELECT col FROM tab BEFORE (STATEMENT = ...); a b c b c d
16. © 2020 Snowflake Inc. All Rights Reserved 16 ( ) : M:N →
17. © 2020 Snowflake Inc. All Rights Reserved 17 ... / Zero Copy Cloning Data Sharing, Data Exchange ...
18. © 2020 Snowflake Inc. All Rights Reserved 18 Zero Copy Cloning → Data Sharing ( ) → Data Exchange (1:1 ) → Snowflake Data Marketplace ( ) → Snowflake Data Service Marketplace ( ) → ...
19. © 2020 Snowflake Inc. All Rights Reserved 19 → → →… (Extensibility)
20. © 2020 Snowflake Inc. All Rights Reserved 20 Extensibility ( ) Snowflake Snowflake extensible → (MP → S/C ) → (S/C → Zero Copy Cloning) → (Cloning → Data Sharing → ...) Snowflake
21. © 2020 Snowflake Inc. All Rights Reserved 21 ... but ( ) ... but ...
22. © 2020 Snowflake Inc. All Rights Reserved 22 → → Snowflake →
23. © 2020 Snowflake Inc. All Rights Reserved Snowflake
24. 2020 12 10 11:00 12:00am Live Snowflake Snowflake trocco Snowflake primeNumber
25. © 2020 Snowflake Inc. All Rights Reserved