Upgrade to Pro — share decks privately, control downloads, hide ads and more …

サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf...

Ken Hamada
November 03, 2017

サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf Tokyo 2017

サーバレスアーキテクチャによる時系列データベースの構築と監視

Ken Hamada

November 03, 2017
Tweet

More Decks by Ken Hamada

Other Decks in Technology

Transcript

  1. ࣗݾ঺հ גࣜձࣾ͸ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ MackerelνʔϜ ςοΫϦʔυ ޷͖: Rust, Haskell, Go, Scala,

    Vim ϓϩάϥϜϞάϞά itchyny.hatenablog.com lightline.vim, calendar.vim, etc. Thank you! ⅓ా ݈ Ken Hamada
  2. ࣌ܥྻσʔλϕʔε • ࣌ܥྻͷσʔλ (యܕతʹ͸਺஋σʔλ) ͷอଘʹಛԽ • αʔόʔϞχλϦϯάɾIoT༻్ • ॻ͖ࠐΈෛՙܰݮɾdisk༰ྔͷ࠷దԽ •

    ߴղ૾౓ɾ௕ظอଘɾେྔϝτϦοΫ 㱺 ෼ࢄDB • ྫ: Graphite, InfluxDB, Prometheus, OpenTSDB Mackerel͕࢖͍ͬͯ·ͨ͠
  3. GraphiteΛӡ༻͢Δ্Ͱͷ՝୊ ϋʔυ΢ΣΞͷݶք (disk༰ྔ) • ετϨʔδͷௐୡίετ εέʔϥϏϦςΟʔͷ֬อ͕ࠔ೉ • ϝτϦοΫͷ௕ظอଘ (1m:25h 㱺

    1m:400d+) σʔλϩετ଱ੑͷ௿͞ ϋοΫɾ࠷దԽ͠ʹ͘͞ Ϋϥ΢υΛ׆༻ͯ͠ӡ༻ίετ࡟ݮ͍ͨ͠ Application servers mackerel.io
  4. ৽࣌ܥྻDBͷઃܭ εέʔϥϏϦςΟʔɾӡ༻ίετɾௐୡίετ࡟ݮ • ϚωʔδυɾαʔϏεΛ࢖͏ 㱺 AWS DynamoDB DynamoDBͷॻ͖ࠐΈίετ (pricing) Λ཈͍͑ͨ

    • RedisΛΩϟογϡͱͯ͠ར༻ ݹ͍ϝτϦοΫσʔλ͸༰ྔ୯Ձͷ͍҆ετϨʔδʹҠಈ • DynamoDB 㱺 S3 ΁Ҡಈ σʔλϩετ଱ੑ • Amazon Kinesis Streams
  5. ہॴࢀরੑͱετϨʔδͷબ୒ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis

    ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ ಛੑͷҟͳΔετϨʔδΛ૊Έ߹ΘͤΔ͜ͱͰ ͔͔ΔίετΛ࠷దԽ͢Δ
  6. diamondߏ੒ਤ Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond: graphiteͷಉૉମ (ͱͯ΋ߗ͍) DynamoDB Kinesis Steams S3 Lambda

    Redis cluster Lambda TTL expired Application servers diamond-reader mackerel.io diamond-writer
  7. diamond-writer • ࣌ܥྻϝτϦοΫσʔλΛRedisɾDynamoDBʹॻ͖ࠐΉ • AWS Lambda (node.js) • RedisΛ࢖͍ॻ͖ࠐΈίετΛ࠷దԽ •

    ΞΠςϜʹTTLΛઃఆ • ݹ͍ϝτϦοΫ͸ࣗಈͰফ͑Δ DynamoDB Kinesis Steams Redis cluster Lambda
  8. Kinesis Streams→Lambda • Kinesis StreamsΛtriggerͱͯ͠LambdaΛىಈ • Lambda͕ॲཧͯ͠΋σʔλ͸Kinesis Streamsʹ࢒Δ • ো֐ൃੜ࣌఺͔Β࠶ॲཧͰ͖Δ

    • backup༻ʹผͷLambdaΛ࢖͏ • ࠷৽ͷϨίʔυΛॲཧ͠ͳ͕Β෮چ • Lambdaͷॲཧͷႈ౳ੑ͕ॏཁ Kinesis Steams Lambda Lambda backup TRIM_HORIZON AT_TIMESTAMP ো֐࣌ͷόοΫΞοϓܥ
  9. Lambda→Redis cluster • DynamoDBͷwrite-back cacheͱͯ͠RedisΛར༻ • EC2্ͰclusterΛ૊ΜͰ͍Δ • ౰ॳ͸ElastiCacheΛ૝ఆ͍ͯͨ͠ •

    εέʔϧ {Ξοϓ,Ξ΢τ} ͕ΦϯϥΠϯͰͰ͖ͳ͍ • Ͱ͖ΔΑ͏ʹͳͬͨΒҠߦݕ౼͠·͢ • ϝτϦοΫ͕Ұఆݸ਺Ҏ্ͨ·ͬͨΒDynamoDBʹॻ͖ࠐΉ • cron͸εέʔϧͤ͞ʹ͍͘ͷͰجຊతʹ͸࢖Θͳ͍ Lambda Redis cluster
  10. Lambda→DynamoDB • MetricName (partition key)ɾTimestamp (sort key) • ϝτϦοΫ஋͸Mapʹอଘ •

    BatchWriteItemͰ·ͱΊͯॻ͖ࠐΉ • Write capacity͸جຊతʹҰఆ • Read capacity͸ΞΫηε͕ूத͢Δ͜ͱ͕͋Δ DynamoDB Lambda
  11. Lambda ίʔυߏ੒ɾdeploy ϩδοΫ͸libσΟϨΫτϦʹ੾Γग़͢ • function handlerΛ࠷খʹอͭ (client஫ೖɾcallback) • ֤ϥΠϒϥϦʔΛखް͘ςετ apexͰdeploy

    • build hookͰlibσΟϨΫτϦͷίϐʔͱnpm install —production • σΟϨΫτϦ͝ͱzipʹͯ͠Ξοϓϩʔυͯ͘͠ΕΔ • zipΛ࡞Δͱ͖ʹsymlink΋ḷͬͯ͘ΕΔ functions diamond-writer lib diamondwriter.js index.js metriccache.js metriccache.spec.js package.json diamondwriter.spec.js diamond-writer-backup index.js
  12. CPU bound ͔ I/O bound • Kinesis Streams͕triggerͷ࣌ • Batch

    sizeΛ૿΍ͨ͠ͱ͖ͷdurationͷԠ౴ • I/OଟॏԽ͍ͯͨ͠ΒجຊҰఆͷ͸ͣ • ઢܗԠ౴ͳΒೖྗαΠζʹൺྫ͢ΔCPUॲཧ͕ࢧ഑త (͔΋) • Ұൠʹ͸…ʁ • ϝϞϦʔΛ্͛ͯCPUεϖοΫ্͕͕ͬͨ࣌ͷdurationͷԠ౴ • I/O͕ࢧ഑తͳΒCPU্͕͕ͬͯ΋ͦ͜·ͰԼ͕Βͳ͍͸ͣ • యܕతͳϦΫΤετͷlatency΍duration͔Β௚ײతʹ͸Θ͔Γͦ͏
  13. ύϑΥʔϚϯεܭଌͱվળ CPU boundͷͱ͖ • ࣌ؒΛଌͬͯCloudWatchϩάʹग़͢ (ૉ๿) • खݩͰܭଌ͢Δ • sjsp

    (Simple JavaScript Profiler) վળ • ϝϞϦʔΛ૿΍͢ (CPUεϖοΫ΋޲্) • ΞϧΰϦζϜɾॲཧ಺༰ͷվળ
  14. sjsp ੿࡞ “Simple JavaScript Profiler” • MackerelͷϑϩϯτΤϯυͷύϑΥʔϚϯεղੳͷͨΊʹ࡞ͬͨ • JavaScriptΛτϥϯεύΠϧ͢Δ͚ͩ (ϒϥ΢βʹґଘ͠ͳ͍)

    • Lambda function (node.js) ͷύϑΥʔϚϯεܭଌʹ΋࢖͑Δ CPU boundͳॏ͍ॲཧ͕෼͔Δ • ࣮ࡍdiamond-writerͷॏ͍ॲཧΛݟ͚ͭΔ͜ͱ͕Ͱ͖ͨ ʮ͍ͭ΋ͱಉ͡Α͏ʹύϑΥʔϚϯεܭଌ͢Δʯ
  15. Mackerel plugins ͸ͯͳʹ͓͚ΔAWSαʔϏεͷ؂ࢹ Monitoring host Mackerelͷpluginͱ AWSΠϯςάϨʔγϣϯͰ ϝτϦοΫΛऩूɾ ՄࢹԽ CloudWatch

    Application servers DB, Proxies, etc. #mackerel #bookmark #blog E-mail AWS Integration CloudWatchͷϝτϦοΫ΋Mackerelʹू໿ ϗετ؅ཧɾΞϥʔτɾ௨஌؅ཧΛҰݩԽ EC2 ΦϯϓϨͷϗετ΋ ͨ͘͞Μ͋Δ
  16. diamondߏ੒ਤ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda TTL

    expired Application servers diamond-reader mackerel.io diamond-writer
  17. Kinesis Streamsͷಛੑ Kinesis Steams Lambda #Shard Records PutRecords Partition Key

    γϟʔυΛݻఆ͍ͨ࣌͠͸ Partition KeyΛࢦఆ ϝτϦοΫ໊Λݩʹੜ੒ ਖ਼͘͠όϥ͚ͤ͞Δͷ͕େࣄ GetShardIterator GetRecords γϟʔυ਺ * ར༻࣌ؒ PUTϖΠϩʔυαΠζ σʔλอ࣋ظؒ (1d ~ 7d) ྉۚମܥ
  18. Kinesis Streamsͷ؂ࢹ I/O bytes͕Լ͕͍ͬͯΔ • PutRecordsʹࣦഊ͍ͯ͠Δ • ຊମͷΞϓϦέʔγϣϯΛٙ͏ • GetRecordsʹࣦഊ͍ͯ͠Δ

    • ޙஈͷLambdaΛٙ͏ Read delay্͕͕͍ͬͯΔ • Lambda͕ॲཧ͖͠Ε͍ͯͳ͍ • LambdaͷΤϥʔɾλΠϜΞ΢τ • γϟʔυ਺͕଍Γͳ͍ • γϟʔυ਺Λ૿΍ͯ͠
 throughputΛ্͛Δ Read delay I/O bytes
  19. Lambdaͷಛੑ FaaS (Function as a Service) ֤ίϯςφ͸جຊతʹ࠶ར༻͞ΕΔ • deploy΍ઃఆมߋͰ࡞Γ௚͞ΕΔ •

    client͸globalม਺ͰӬଓԽ ͦΕͧΕͷίϯςφ͸ಉ࣌ʹ1ͭॲཧ • ֤Kinesis shardʹ1ͭίϯςφཱ͕ͭ Kinesis Steams Lambda GetShardIterator GetRecords IteratorType Batch Size DynamoDB Redis cluster ֤container͸handlerΛ ಉ࣌ʹҰճ͔͠ॲཧ͠ͳ͍ ϝϞϦʔαΠζ (ॲཧೳྗ) ߹ܭ࣮ߦ࣌ؒ ϦΫΤετճ਺ ྉۚମܥ
  20. Lambdaͷ؂ࢹ Iterator Age (ms) ࣮ߦॲཧ࣌ؒ (ms) ࣮ߦճ਺ɾΤϥʔճ਺ Τϥʔճ਺্͕͕͍ͬͯΔ • CloudWatch

    LogsͰௐࠪ • apex logs —follow | grep Duration avg (ms) ্͕͕͍ͬͯΔ • ຊମଆͷෛՙɾKinesisνΣοΫ • DynamoDBͷRequest latency • ϝϞϦʔΛ૿΍ͦ͏ • CPUεϖοΫ΋ྉۚ΋͕͋Δ Iterator Age্͕͕͍ͬͯΔ • Τϥʔ͕ग़͍ͯΔ͔Ͳ͏͔ • ޙஈͷthroughput଍Γͳ͍ • DynamoDBͷCapacity্͛Δ • Kinesis shard਺Λ૿΍͢
  21. Lambdaͷ؂ࢹ ฏۉಉ࣌ॲཧ਺ = 1෼ؒͷ࣮ߦճ਺ * ฏۉ࣮ߦ࣌ؒ (ms) / 60,000 (ms)


    ≤ Kinesis shard਺ time 1 minute container processing idle Mackerel advanced graph Shard਺ͷ7ׂҎԼ(໨҆)ʹ཈͑Δ ShardΛ૿΍͢ͷ͸͕͔͔࣌ؒΔ ੵ෼஋͕՝ۚର৅ ύϑΥʔϚϯενϡʔχϯάͷࢦඪ Invocation count * Duration average (ms) / 60,000 (ms) ≤ Kinesis shard count
  22. DynamoDBͷ؂ࢹ Write capacity Read capacity Throttled requests Write Capacity্͕͕͍ͬͯΔ •

    LatencyͷѱԽɾ400Λฦ͢ • Throttled requests͕Ͱ͍ͯΕ͹
 Provisioned Write Capacity͋͛Δ Read Capacity্͕͕͍ͬͯΔ • Writeͱಉ༷ɾ400Λฦ͢ • Provisioned Read Capacity͋͛Δ Throttle events্͕͕͍ͬͯΔ • Τϥʔ400Λฦ͍ͯ͠Δঢ়ଶ • Provisioned Capacity͕଍Γͳ͍ • ϦΫΤετΛΏͬ͘Γʹ͢Δ • ϦΫΤετΛ·ͱΊΔ
  23. Redisͷ؂ࢹ CPU % Redis used memory Redis processed commands CPU

    %্͕͕͍ͬͯΔ • Redis͸1 core͔͠࢖͑ͳ͍ • readΫΤϦΛslaveʹ޲͚Δ • nodeΛ௥Ճͯ͠resharding memory࢖༻ྔ • ϋʔυ΢ΣΞͷݶք͕͋Δ • key͕ফ͑ͯ΋OSʹฦ͞ͳ͍ • redis-cli info | grep used_memory • resharding ࣮ߦͨ͠ίϚϯυ਺ • RedisͰॲཧͨ͠ྔͷࢦඪ ֤Πϯελϯεʹͯ RedisϓϥάΠϯͰՄࢹԽ
  24. ॲཧ਺ ؂ࢹ Lambda͕ॲཧͨ͠ϝτϦοΫͷ਺ / 1m • RedisɾDynamoDBʹ௥Ճͨ͠ϝτϦοΫ਺ • CloudWatchʹJSONͰ৘ใΛग़͢ •

    ผͷLambda functionͰҰ෼ؒͷϩάΛूܭ (cron trigger)
 㱺 Mackerelʹ౤ߘ • ॲཧ਺͕Լ͕ͬͨΒΞϥʔτ • Redisʹॲཧ਺༻ͷΩʔΛҰ෼͝ͱʹ࡞ͬͯincrͯ͠΋Α͍
  25. End-to-end ؂ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda

    TTL expired Application servers diamond-reader mackerel.io ϝτϦοΫΛ౤ߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ
  26. End-to-end ؂ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda

    TTL expired Application servers diamond-reader mackerel.io API ϝτϦοΫΛ౤ߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ Monitoring host