ML パイプラインを実現 Google Cloud を使い、 大量のリアルタイム行動データ を 捌くデータシステムと多彩な ML システムをシンプルにつなぐ パイプラインを実現 Realizing “Human in the Loop” with Google Cloud ML End-to-End System
System Kubernetes Engine KARTE System 弊社開発者 Status Cloud Pub/Sub ML Pipeline AI Platform Pipelines Batch Train AI Platform Training Transform Train Inference KARTE Data BigQuery クライアント CI/CD Pipeline Github Actions cometML
Kubernetes Engine KARTE System 弊社開発者 Status Cloud Pub/Sub ML Pipeline AI Platform Pipelines Batch Train AI Platform Training Transform Train Inference KARTE Data BigQuery クライアント AI Platform Pipelines CI/CD Pipeline Github Actions cometML
System ML System kfp.Client.run_pipeline params = { start_date = “20200101”, end_date = “20200531”, …, } default の引数を設定すれば params には変更したい引数を入れるだけ ML Pipeline AI Platform Pipelines
kfp.Client.run_pipeline experiment_id = “development” experiment_id = “production” ML System ML Pipeline AI Platform Pipelines experiment_id の引数で分離 もちろん UI からも実行可能
Engine KARTE System ML System ML Pipeline AI Platform Pipelines Status Cloud Pub/Sub { model_id: “123456”, status: “SUCCEESS”, …. } ExitHandler 成功・失敗に関わらず 必ず実行 データ更新
Kubernetes Engine KARTE System 弊社開発者 Status Cloud Pub/Sub ML Pipeline AI Platform Pipelines Batch Train AI Platform Training Transform Train Inference KARTE Data BigQuery クライアント AI Platform Training cometML CI/CD Pipeline Github Actions
Train AI Platform Training High-Memory, GPU … etc node pool 作成時に固定 --master-machine-type=”n1-highmem-8”, --master-accelerator=”type=nvidia-tesla-k80,count-2” パイプラインごとに簡単に変更可能 Job High-Memory Pool Kubernetes Engine GPU Pool Kubernetes Engine
jobs submit training ML Pipeline AI Platform Pipelines Job Batch Train AI Platform Training Job High-Memory Pool Kubernetes Engine GPU Pool Kubernetes Engine
Kubernetes Engine KARTE System 弊社開発者 Status Cloud Pub/Sub ML Pipeline AI Platform Pipelines Batch Train AI Platform Training Transform Train Inference KARTE Data BigQuery クライアント Github Actions CI/CD Pipeline Github Actions cometML