Snowflake Horizon Catalog と Apache Iceberg で作る オープンなデータ基盤

DevelopersIO 2026 Osaka #devio2026での登壇資料

Transcript

1. Snowflake Horizon Catalog Apache Iceberg
2. Kitagawa Zempei https://dev.classmethod.jp/author/kitagawa-zempei/ Snowflake
3. 1 2 Apache Iceberg 3 Snowflake Horizon Catalog 4 Snowflake Iceberg 5 03
5. Snowflake Spark Trino Snowflake Spark Trino Iceberg REST Catalog API 1 1 Iceberg Amazon S3 05
6. 3 Snowflake Apache Spark Trino DuckDB Snowflake Horizon Catalog Apache Iceberg Amazon S3 06
7. Apache Iceberg
8. Apache Iceberg Apache Iceberg 1 ACID Spark Trino Flink Presto 08
9. Iceberg Iceberg Iceberg Catalog metadata file metadata file s0 s1 manifest list manifest file manifest file data files Parquet https://iceberg.apache.org/spec/#overview 09
10. metadata file metadata file db1.table1 00003-<uuid>.metadata.json db1.table2 00012-<uuid>.metadata.json Amazon S3 db1/table1/metadata/ 00001-<uuid>.metadata.json 00002-<uuid>.metadata.json 00003-<uuid>.metadata.json db1.table3 00007-<uuid>.metadata.json metadata file 00003 00004-<uuid>.metadata.json metadata file 00004 00004 https://iceberg.apache.org/spec/#metastore-tables 10
11. Iceberg REST Catalog API HTTP Spark Trino API DuckDB PyIceberg Snowflake Iceberg REST Catalog API HTTP Horizon Catalog Apache Polaris AWS Glue 11
12. vended credentials Amazon S3 X-Iceberg-Access-Delegation S3 12
13. Snowflake Horizon Catalog
14. Snowflake Horizon Catalog Snowflake 1 Snowflake SQL API REST API Iceberg REST Catalog API MCP Apache Polaris https://www.snowflake.com/en/product/features/horizon/ 14
15. Snowflake Horizon Catalog Snowflake Polaris REST Spark Snowflake Horizon Catalog Trino Snowflake SQL DuckDB PyIceberg GRANT Iceberg REST /polaris/api/catalog vended credentials S3 Iceberg S3 15
16. Snowflake Iceberg
17. Snowflake Iceberg Snowflake Snowflake Managed Horizon Catalog Snowflake storage AWS Glue S3 Snowflake S3 Catalog Snowflake Horizon REST Snowflake catalog-linked database Snowflake cataloglinked database S3 17
18. Snowflake DuckDB Athena Spark Snowflake Athena DuckDB Snowflake Horizon Catalog AWS Amazon Athena PC DuckDB PAT PySpark INSERT SELECT Snowflake Snowsight vended credentials INSERT SELECT vended credentials Amazon S3 Snowflake Iceberg Athena PySpark engine version 3 DuckDB 1.5.4 https://dev.classmethod.jp/articles/snowflake-managed-iceberg-tables-write-from-athena/ https://dev.classmethod.jp/articles/snowflake-tables-iceberg-internal-storage-query-try/ 18
20. Athena Horizon Catalog REST type = rest Snowflake URL PAT warehouse uri = https://<account>.snowflakecomputing.com/polaris/api/catalog credential = <PAT> scope = session:role:<ROLE> warehouse = <DATABASE> S3 Athena header.X-Iceberg-Access-Delegation = vended-credentials Spark SELECT spark.sql.iceberg.vectorization.enabled = false 20
21. DuckDB PAT S3 Horizon Catalog /v1/oauth/tokens vended credentials TOKEN INSTALL iceberg; LOAD iceberg; CREATE SECRET horizon (TYPE iceberg, TOKEN '<access_token>'); ATTACH '<DATABASE>' AS horizon ( TYPE iceberg, SECRET horizon, ENDPOINT 'https://<account>.snowflakecomputing.com/polaris/api/catalog' ); SELECT * FROM horizon.<SCHEMA>.<TABLE>; scope session:role:<ROLE> PAT CLIENT_SECRET DuckDB Snowflake 21
22. Snowflake 2026 9 Snowflake scope session:role:<ROLE> GA vended credentials Snowflake Spark S3 Scan Plan API Iceberg 1.11 https://docs.snowflake.com/en/user-guide/tables-iceberg-enforce-access-policies-scan-plan-api 22
23. Snowflake S3 vended credentials IAM IAM Snowflake vended credentials IAM IAM Athena IAM Horizon Catalog 23
24. IAM { "Effect": "Deny", "Principal": "*", "Action": "s3:*", "Resource": ["arn:aws:s3:::example-iceberg-bucket", "arn:aws:s3:::example-iceberg-bucket/*"], "Condition": {"ArnNotLike": {"aws:PrincipalArn": [ "arn:aws:iam::111122223333:role/snowflake-volume", "arn:aws:iam::111122223333:role/your-admin" ]}} } 24
26. Snowflake AI Snowflake Snowflake Snowflake Cortex Agents Cortex Analyst Cortex Analyst Spark Trino DuckDB SQL Horizon Catalog Snowflake Iceberg REST Catalog API Iceberg S3 Snowflake 26
27. Snowflake S3 Open Data Sharing Iceberg REST Catalog API Snowflake Iceberg S3 external consumer PAT Snowflake S3 Iceberg REST Catalog API PAT Iceberg REST 27
28. Iceberg GA 2026 9 26 GA GA Snowflake Iceberg 2024-06 catalog-linked database 2025-10 Horizon 2026-02 Horizon 2026-05 Iceberg v3 2026-05 Snowflake storage 2026-06 Open Data Sharing Horizon Scan Plan API 2026-08 2026-08 2026-09 28
29. 1 Snowflake