Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Snowflake のアーキテクチャが どう「筋がいい」のかを解説する / Data Engi...

Snowflake のアーキテクチャが どう「筋がいい」のかを解説する / Data Engineering Study #5

Data Engineering Study #5「噂のSnowflake Deep Dive」発表資料: https://forkwell.connpass.com/event/194269/
Youtube Live アーカイブ: https://www.youtube.com/watch?v=_kYW7EneUu4

Yoshi Matsuzaki

December 03, 2020
Tweet

More Decks by Yoshi Matsuzaki

Other Decks in Technology

Transcript

  1. © 2020 Snowflake Inc. All Rights Reserved Snowflake Data Engineering

    Study #5 Yoshi Matsuzaki, Principal Cloud Support Engineer of Snowflake | Dec 2, 2020
  2. © 2020 Snowflake Inc. All Rights Reserved ABOUT ME 2

    Yoshi Matsuzaki Principal Cloud Support Engineer - Snowflake Work: Technical Support in English/Japanese (7:3) Launching Japanese language support team SQL functionality expert Escalation review Supportability readiness for new features Background: Infrastructure Engineer (MSP) Support Engineer (Cloud Vendor) Database Engineer of MySQL engines (Cloud Vendor)
  3. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 4 DATA

    SOURCES OLTP DATABASES ENTERPRISE APPLICATIONS THIRD-PARTY WEB/LOG DATA IoT DATA CONSUMERS DATA MONETIZATION OPERATIONAL REPORTING AD HOC ANALYSIS REAL-TIME ANALYTICS → Data Platform → Data Cloud
  4. © 2020 Snowflake Inc. All Rights Reserved 7 Snowflake 50-500

    MB ( : S3) … Partition Pruning / NDV ( )
  5. © 2020 Snowflake Inc. All Rights Reserved 9 ( )

    “ ( : immutable) ” - Wikipedia
  6. © 2020 Snowflake Inc. All Rights Reserved 11 'a' 'b'

    'c' Table Partitions UPDATE tab SET col = 'd' WHERE col = 'a'; 'a' 'b' 'c' Table Partitions 'd' v1 v2
  7. © 2020 Snowflake Inc. All Rights Reserved 13 / /

    : UNDO (MySQL, Oracle) UNDO (PostgreSQL) :
  8. © 2020 Snowflake Inc. All Rights Reserved 14 → /

    → Time Travel 90 / SELECT * FROM tab AT (TIMESTAMP => ...); SELECT * FROM tab BEFORE (STATEMENT => <query-id>); CREATE TABLE tab CLONE src AT (TIMESTAMP => ...); → Fail Safe 7 GC ...
  9. © 2020 Snowflake Inc. All Rights Reserved 15 Table SELECT

    col FROM tab; 'a' 'b' 'c' Partitions 'd' v1 v2 SELECT col FROM tab BEFORE (STATEMENT = ...); a b c b c d
  10. © 2020 Snowflake Inc. All Rights Reserved 17 ... /

    Zero Copy Cloning Data Sharing, Data Exchange ...
  11. © 2020 Snowflake Inc. All Rights Reserved 18 Zero Copy

    Cloning → Data Sharing ( ) → Data Exchange (1:1 ) → Snowflake Data Marketplace ( ) → Snowflake Data Service Marketplace ( ) → ...
  12. © 2020 Snowflake Inc. All Rights Reserved 20 Extensibility (

    ) Snowflake Snowflake extensible → (MP → S/C ) → (S/C → Zero Copy Cloning) → (Cloning → Data Sharing → ...) Snowflake