Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf...
Search
Ken Hamada
November 03, 2017
Technology
18
30k
サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf Tokyo 2017
サーバレスアーキテクチャによる時系列データベースの構築と監視
Ken Hamada
November 03, 2017
Tweet
Share
More Decks by Ken Hamada
See All by Ken Hamada
組織と技術の両輪で開発を加速させるkintoneチームの取り組み / JJUG CCC 2022 Fall Cybozu kintone
itchyny
1
2.5k
Mackerelチームにおけるインフラオーナーシップ / Hatena Pepabo tech conf 4 DevOps Kyoto
itchyny
0
5.7k
Other Decks in Technology
See All in Technology
AIエージェントを現場で使う / 2025.08.07 著者陣に聞く!現場で活用するためのAIエージェント実践入門(Findyランチセッション)
smiyawaki0820
6
600
Google Cloud で学ぶデータエンジニアリング入門 2025年版 #GoogleCloudNext / 20250805
kazaneya
PRO
11
2.8k
AIに目を奪われすぎて、周りの困っている人間が見えなくなっていませんか?
cap120
1
430
リリース2ヶ月で収益化した話
kent_code3
1
180
Claude Codeから我々が学ぶべきこと
s4yuba
7
1.9k
AI時代の経営、Bet AI Vision #BetAIDay
layerx
PRO
1
1.7k
金融サービスにおける高速な価値提供とAIの役割 #BetAIDay
layerx
PRO
1
720
生成AI導入の効果を最大化する データ活用戦略
ham0215
0
110
20250807_Kiroと私の反省会
riz3f7
0
140
LLMをツールからプラットフォームへ〜Ai Workforceの戦略〜 #BetAIDay
layerx
PRO
1
850
JAWS AI/ML #30 AI コーディング IDE "Kiro" を触ってみよう
inariku
3
270
Lambda management with ecspresso and Terraform
ijin
2
130
Featured
See All Featured
Optimising Largest Contentful Paint
csswizardry
37
3.4k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Unsuck your backbone
ammeep
671
58k
The Cult of Friendly URLs
andyhume
79
6.5k
A Modern Web Designer's Workflow
chriscoyier
695
190k
What's in a price? How to price your products and services
michaelherold
246
12k
Automating Front-end Workflow
addyosmani
1370
200k
4 Signs Your Business is Dying
shpigford
184
22k
Docker and Python
trallard
45
3.5k
Side Projects
sachag
455
43k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
283
13k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Transcript
αʔόϨεΞʔΩςΫνϟʹΑΔ ࣌ܥྻσʔλϕʔεͷߏஙͱࢹ גࣜձࣾͯͳɹ⅓ా ݈ 2017/11/03 Serverlessconf Tokyo
ࣗݾհ גࣜձࣾͯͳ ΞϓϦέʔγϣϯΤϯδχΞ MackerelνʔϜ ςοΫϦʔυ ͖: Rust, Haskell, Go, Scala,
Vim ϓϩάϥϜϞάϞά itchyny.hatenablog.com lightline.vim, calendar.vim, etc. Thank you! ⅓ా ݈ Ken Hamada
αʔόʔϨεͷίϯϙʔωϯτΛͬͯ αʔϏεΛӡ༻͍ͯ͠Δਓ ✋
αʔόʔϨεͷίϯϙʔωϯτΛ ͔ͬ͠Γࢹ͍ͯ͠Δਓ ✋
ࢹαʔϏεΛఏڙ͍ͯ͠Δࢲ͕ͨͪ ͲͷΑ͏ʹϚωʔδυαʔϏεΛ ࢹɾཧ͍ͯ͠Δ͔Λ͓͑͠·͢
࣍ Mackerelʹ͍ͭͯ ࣌ܥྻσʔλϕʔε Mackerelͷ৽࣌ܥྻσʔλϕʔεͷ։ൃ Lambda functionͷύϑΥʔϚϯεɾνϡʔχϯά ֤ίϯϙʔωϯτͷಛੑͱࢹ
Mackerelʹ͍ͭͯ
Mackerel mackerel.io
Mackerel SaaSͷαʔόʔࢹɾཧπʔϧ ͯͳࣾͷࢹγεςϜΛαʔϏεԽͯ͠ެ։ mackerel-agent͕ϝτϦοΫΛߘ͢Δpushܕ νϟοτπʔϧߏཧπʔϧͱΈ߹Θͤ Slack, HipChat, etc. Chef, Ansible,
etc.
Mackerelུ֓ਤ CPUɾMemory༻ OSใ ϝτϦοΫσʔλ ϝλσʔλ ϝʔϧ ཧը໘ νϟοτπʔϧ Ξϥʔτൃੜ
Mackerelͷάϥϑ ϝτϦοΫͷσʔλ ࣌ܥྻσʔλϕʔεʹ อଘ͞Ε͍ͯΔ ϗετใϝλσʔλ ϝΠϯͷσʔλϕʔε (PostgreSQL) ʹอଘ
࣌ܥྻσʔλϕʔε
࣌ܥྻσʔλϕʔε • ࣌ܥྻͷσʔλ (యܕతʹσʔλ) ͷอଘʹಛԽ • αʔόʔϞχλϦϯάɾIoT༻్ • ॻ͖ࠐΈෛՙܰݮɾdisk༰ྔͷ࠷దԽ •
ߴղ૾ɾظอଘɾେྔϝτϦοΫ 㱺 ࢄDB • ྫ: Graphite, InfluxDB, Prometheus, OpenTSDB Mackerel͕͍ͬͯ·ͨ͠
GraphiteΛӡ༻͢Δ্Ͱͷ՝ ϋʔυΣΞͷݶք (disk༰ྔ) • ετϨʔδͷௐୡίετ εέʔϥϏϦςΟʔͷ֬อ͕ࠔ • ϝτϦοΫͷظอଘ (1m:25h 㱺
1m:400d+) σʔλϩετੑͷ͞ ϋοΫɾ࠷దԽ͠ʹ͘͞ ΫϥυΛ׆༻ͯ͠ӡ༻ίετݮ͍ͨ͠ Application servers mackerel.io
৽࣌ܥྻDBͰୡ͍ͨ͜͠ͱ εέʔϥϏϦςΟʔ ӡ༻ίετɾϋʔυΣΞௐୡίετ͑Δ σʔλϩετੑ
৽࣌ܥྻDBͷઃܭ εέʔϥϏϦςΟʔɾӡ༻ίετɾௐୡίετݮ • ϚωʔδυɾαʔϏεΛ͏ 㱺 AWS DynamoDB DynamoDBͷॻ͖ࠐΈίετ (pricing) Λ͍͑ͨ
• RedisΛΩϟογϡͱͯ͠ར༻ ݹ͍ϝτϦοΫσʔλ༰ྔ୯Ձͷ͍҆ετϨʔδʹҠಈ • DynamoDB 㱺 S3 Ҡಈ σʔλϩετੑ • Amazon Kinesis Streams
ہॴࢀরੑͱετϨʔδͷબ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis
ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ
ہॴࢀরੑͱετϨʔδͷબ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis
ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ ಛੑͷҟͳΔετϨʔδΛΈ߹ΘͤΔ͜ͱͰ ͔͔ΔίετΛ࠷దԽ͢Δ
Aug. 2017 ৽࣌ܥྻDBʹҠߦ
diamondߏਤ Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond: graphiteͷಉૉମ (ͱͯߗ͍) DynamoDB Kinesis Steams S3 Lambda
Redis cluster Lambda TTL expired Application servers diamond-reader mackerel.io diamond-writer
Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond-writerͷ։ൃ
diamond-writer • ࣌ܥྻϝτϦοΫσʔλΛRedisɾDynamoDBʹॻ͖ࠐΉ • AWS Lambda (node.js) • RedisΛ͍ॻ͖ࠐΈίετΛ࠷దԽ •
ΞΠςϜʹTTLΛઃఆ • ݹ͍ϝτϦοΫࣗಈͰফ͑Δ DynamoDB Kinesis Steams Redis cluster Lambda
Kinesis Streams→Lambda • Kinesis StreamsΛtriggerͱͯ͠LambdaΛىಈ • Lambda͕ॲཧͯ͠σʔλKinesis StreamsʹΔ • োൃੜ͔࣌Β࠶ॲཧͰ͖Δ
• backup༻ʹผͷLambdaΛ͏ • ࠷৽ͷϨίʔυΛॲཧ͠ͳ͕Β෮چ • Lambdaͷॲཧͷႈੑ͕ॏཁ Kinesis Steams Lambda Lambda backup TRIM_HORIZON AT_TIMESTAMP ো࣌ͷόοΫΞοϓܥ
Lambda→Redis cluster • DynamoDBͷwrite-back cacheͱͯ͠RedisΛར༻ • EC2্ͰclusterΛΜͰ͍Δ • ॳElastiCacheΛఆ͍ͯͨ͠ •
εέʔϧ {Ξοϓ,Ξτ} ͕ΦϯϥΠϯͰͰ͖ͳ͍ • Ͱ͖ΔΑ͏ʹͳͬͨΒҠߦݕ౼͠·͢ • ϝτϦοΫ͕ҰఆݸҎ্ͨ·ͬͨΒDynamoDBʹॻ͖ࠐΉ • cronεέʔϧͤ͞ʹ͍͘ͷͰجຊతʹΘͳ͍ Lambda Redis cluster
Lambda→DynamoDB • MetricName (partition key)ɾTimestamp (sort key) • ϝτϦοΫMapʹอଘ •
BatchWriteItemͰ·ͱΊͯॻ͖ࠐΉ • Write capacityجຊతʹҰఆ • Read capacityΞΫηε͕ूத͢Δ͜ͱ͕͋Δ DynamoDB Lambda
Lambda ίʔυߏɾdeploy ϩδοΫlibσΟϨΫτϦʹΓग़͢ • function handlerΛ࠷খʹอͭ (clientೖɾcallback) • ֤ϥΠϒϥϦʔΛखް͘ςετ apexͰdeploy
• build hookͰlibσΟϨΫτϦͷίϐʔͱnpm install —production • σΟϨΫτϦ͝ͱzipʹͯ͠Ξοϓϩʔυͯ͘͠ΕΔ • zipΛ࡞Δͱ͖ʹsymlinkḷͬͯ͘ΕΔ functions diamond-writer lib diamondwriter.js index.js metriccache.js metriccache.spec.js package.json diamondwriter.spec.js diamond-writer-backup index.js
Lambda functionͷ ύϑΥʔϚϯεɾνϡʔχϯά
τϥϒϧൃੜʂ ϦΫΤετΛฒߦॲཧʹ͍ͯ͠Δͷʹɺ ఆΑΓύϑΥʔϚϯε͕ग़ͳ͍ʂ ࢥͬͨΑΓ͓ஈ͕ߴ͍
ܭଌͤΑʂ LambdaͷίϯςφʹϩάΠϯͰ͖ͳ͍ʂ topnetstattcpdumpଧͯͳ͍ʂ
͓͍ͪͭͯ CPU bound ͔ I/O bound ͔ Έ͖ΘΊ·͠ΐ͏
CPU bound ͔ I/O bound • Kinesis Streams͕triggerͷ࣌ • Batch
sizeΛ૿ͨ͠ͱ͖ͷdurationͷԠ • I/OଟॏԽ͍ͯͨ͠ΒجຊҰఆͷͣ • ઢܗԠͳΒೖྗαΠζʹൺྫ͢ΔCPUॲཧ͕ࢧత (͔) • Ұൠʹ…ʁ • ϝϞϦʔΛ্͛ͯCPUεϖοΫ্͕͕ͬͨ࣌ͷdurationͷԠ • I/O͕ࢧతͳΒCPU্͕͕ͬͯͦ͜·ͰԼ͕Βͳ͍ͣ • యܕతͳϦΫΤετͷlatencyduration͔ΒײతʹΘ͔Γͦ͏
ύϑΥʔϚϯεܭଌͱվળ CPU boundͷͱ͖ • ࣌ؒΛଌͬͯCloudWatchϩάʹग़͢ (ૉ) • खݩͰܭଌ͢Δ • sjsp
(Simple JavaScript Profiler) վળ • ϝϞϦʔΛ૿͢ (CPUεϖοΫ্) • ΞϧΰϦζϜɾॲཧ༰ͷվળ
sjsp ࡞ “Simple JavaScript Profiler” • MackerelͷϑϩϯτΤϯυͷύϑΥʔϚϯεղੳͷͨΊʹ࡞ͬͨ • JavaScriptΛτϥϯεύΠϧ͢Δ͚ͩ (ϒϥβʹґଘ͠ͳ͍)
• Lambda function (node.js) ͷύϑΥʔϚϯεܭଌʹ͑Δ CPU boundͳॏ͍ॲཧ͕͔Δ • ࣮ࡍdiamond-writerͷॏ͍ॲཧΛݟ͚ͭΔ͜ͱ͕Ͱ͖ͨ ʮ͍ͭͱಉ͡Α͏ʹύϑΥʔϚϯεܭଌ͢Δʯ
ύϑΥʔϚϯεܭଌͱվળ I/O boundͷͱ͖ • AWS X-RayͰॏ͍ϦΫΤετΛௐΔ վળ • ΫΤϦΛ·ͱΊΔɾϦΫΤετΛݮΒ͢ •
ΞΠςϜͷ·ͱΊ͔ͨΛม͑Δɾѹॖ͢Δ • Batch sizeΛ૿͢
AWS X-Ray ϚωʔδυαʔϏεͷϦΫΤετͷใΛऩूɾੳ AWSͷSDKʹΈࠐΉ • ؆୯ͳมߋͰτϨʔγϯάΛ࢝ΊΒΕΔ • I/O boundͷ߹ʹΛൃݟͰ͖Δ const
AWSXRay = require('aws-xray-sdk-core'); const AWS = AWSXRay.captureAWS(require('aws-sdk'));
֤ίϯϙʔωϯτͷಛੑͱࢹ
Mackerel plugins ͯͳʹ͓͚ΔAWSαʔϏεͷࢹ Monitoring host Mackerelͷpluginͱ AWSΠϯςάϨʔγϣϯͰ ϝτϦοΫΛऩूɾ ՄࢹԽ CloudWatch
Application servers DB, Proxies, etc. #mackerel #bookmark #blog E-mail AWS Integration CloudWatchͷϝτϦοΫMackerelʹू ϗετཧɾΞϥʔτɾ௨ཧΛҰݩԽ EC2 ΦϯϓϨͷϗετ ͨ͘͞Μ͋Δ
diamondߏਤ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda TTL
expired Application servers diamond-reader mackerel.io diamond-writer
Kinesis Streamsͷಛੑ Kinesis Steams Lambda #Shard Records PutRecords Partition Key
γϟʔυΛݻఆ͍ͨ࣌͠ Partition KeyΛࢦఆ ϝτϦοΫ໊Λݩʹੜ ਖ਼͘͠όϥ͚ͤ͞Δͷ͕େࣄ GetShardIterator GetRecords γϟʔυ * ར༻࣌ؒ PUTϖΠϩʔυαΠζ σʔλอ࣋ظؒ (1d ~ 7d) ྉۚମܥ
Kinesis Streamsͷࢹ I/O bytes͕Լ͕͍ͬͯΔ • PutRecordsʹࣦഊ͍ͯ͠Δ • ຊମͷΞϓϦέʔγϣϯΛٙ͏ • GetRecordsʹࣦഊ͍ͯ͠Δ
• ޙஈͷLambdaΛٙ͏ Read delay্͕͕͍ͬͯΔ • Lambda͕ॲཧ͖͠Ε͍ͯͳ͍ • LambdaͷΤϥʔɾλΠϜΞτ • γϟʔυ͕Γͳ͍ • γϟʔυΛ૿ͯ͠ throughputΛ্͛Δ Read delay I/O bytes
Lambdaͷಛੑ FaaS (Function as a Service) ֤ίϯςφجຊతʹ࠶ར༻͞ΕΔ • deployઃఆมߋͰ࡞Γ͞ΕΔ •
clientglobalมͰӬଓԽ ͦΕͧΕͷίϯςφಉ࣌ʹ1ͭॲཧ • ֤Kinesis shardʹ1ͭίϯςφཱ͕ͭ Kinesis Steams Lambda GetShardIterator GetRecords IteratorType Batch Size DynamoDB Redis cluster ֤containerhandlerΛ ಉ࣌ʹҰճ͔͠ॲཧ͠ͳ͍ ϝϞϦʔαΠζ (ॲཧೳྗ) ߹ܭ࣮ߦ࣌ؒ ϦΫΤετճ ྉۚମܥ
Lambdaͷࢹ Iterator Age (ms) ࣮ߦॲཧ࣌ؒ (ms) ࣮ߦճɾΤϥʔճ Τϥʔճ্͕͕͍ͬͯΔ • CloudWatch
LogsͰௐࠪ • apex logs —follow | grep Duration avg (ms) ্͕͕͍ͬͯΔ • ຊମଆͷෛՙɾKinesisνΣοΫ • DynamoDBͷRequest latency • ϝϞϦʔΛ૿ͦ͏ • CPUεϖοΫྉ͕ۚ͋Δ Iterator Age্͕͕͍ͬͯΔ • Τϥʔ͕ग़͍ͯΔ͔Ͳ͏͔ • ޙஈͷthroughputΓͳ͍ • DynamoDBͷCapacity্͛Δ • Kinesis shardΛ૿͢
Lambdaͷࢹ ฏۉಉ࣌ॲཧ = 1ؒͷ࣮ߦճ * ฏۉ࣮ߦ࣌ؒ (ms) / 60,000 (ms)
≤ Kinesis shard time 1 minute container processing idle Mackerel advanced graph Shardͷ7ׂҎԼ(҆)ʹ͑Δ ShardΛ૿͢ͷ͕͔͔࣌ؒΔ ੵ͕՝ۚର ύϑΥʔϚϯενϡʔχϯάͷࢦඪ Invocation count * Duration average (ms) / 60,000 (ms) ≤ Kinesis shard count
DynamoDBͷಛੑ DynamoDB Lambda diamond-reader BatchWriteItem BatchGetItem ϚωʔδυͳNoSQLσʔλϕʔε ʮΫϥυ൛ͷࢄσʔλϕʔεʯ ΦϯϥΠϯͰΩϟύγςΟʔΛมߋ 3ͭͷϨϓϦΧʹΑΔߴՄ༻ੑ
{Read,Write} ΩϟύγςΟ σʔλసૹྔ σʔλอଘ༰ྔ ྉۚମܥ
DynamoDBͷࢹ Write capacity Read capacity Throttled requests Write Capacity্͕͕͍ͬͯΔ •
LatencyͷѱԽɾ400Λฦ͢ • Throttled requests͕Ͱ͍ͯΕ Provisioned Write Capacity͋͛Δ Read Capacity্͕͕͍ͬͯΔ • Writeͱಉ༷ɾ400Λฦ͢ • Provisioned Read Capacity͋͛Δ Throttle events্͕͕͍ͬͯΔ • Τϥʔ400Λฦ͍ͯ͠Δঢ়ଶ • Provisioned Capacity͕Γͳ͍ • ϦΫΤετΛΏͬ͘Γʹ͢Δ • ϦΫΤετΛ·ͱΊΔ
Redisͷࢹ CPU % Redis used memory Redis processed commands CPU
%্͕͕͍ͬͯΔ • Redis1 core͔͑͠ͳ͍ • readΫΤϦΛslaveʹ͚Δ • nodeΛՃͯ͠resharding memory༻ྔ • ϋʔυΣΞͷݶք͕͋Δ • key͕ফ͑ͯOSʹฦ͞ͳ͍ • redis-cli info | grep used_memory • resharding ࣮ߦͨ͠ίϚϯυ • RedisͰॲཧͨ͠ྔͷࢦඪ ֤Πϯελϯεʹͯ RedisϓϥάΠϯͰՄࢹԽ
diamond-readerͷಛੑ GoݴޠͰॻ͔ΕͨwebΞϓϦέʔγϣϯ ෳͷετϨʔδ͔ΒϝτϦοΫΛಡΈࠐΉ ΞΠςϜ͝ͱʹgoroutineɾI/OΛଟॏԽ DynamoDB S3 Redis cluster Application servers
diamond-reader
diamond-readerͷࢹ golang-stats-api-handlerͱ mackerel-plugin-gostatsͰ GoͷwebΞϓϦέʔγϣϯͷ ϞχλϦϯάΛ࢝ΊΒΕΔΑʂ GC ώʔϓ ϝϞϦʔ mallocɾfree goroutineɾCGO
࣌ܥྻσʔλϕʔε͕ਖ਼͘͠ಈ͍͍ͯΔͱ 1. ϝτϦοΫΛਖ਼͘͠ετϨʔδʹอଘ 2. ͦͷϝτϦοΫΛಡΈऔΔ͜ͱ͕Ͱ͖Δ
ॲཧ ࢹ Lambda͕ॲཧͨ͠ϝτϦοΫͷ / 1m • RedisɾDynamoDBʹՃͨ͠ϝτϦοΫ • CloudWatchʹJSONͰใΛग़͢ •
ผͷLambda functionͰҰؒͷϩάΛूܭ (cron trigger) 㱺 Mackerelʹߘ • ॲཧ͕Լ͕ͬͨΒΞϥʔτ • Redisʹॲཧ༻ͷΩʔΛҰ͝ͱʹ࡞ͬͯincrͯ͠Α͍
End-to-end ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda
TTL expired Application servers diamond-reader mackerel.io ϝτϦοΫΛߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ
End-to-end ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda
TTL expired Application servers diamond-reader mackerel.io API ϝτϦοΫΛߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ Monitoring host
End-to-end ࢹ ࢹ༻ϗετ͔ΒμϛʔϝτϦοΫΛߘ sleep͠ͳ͕ΒɺऔಘͰ͖Δ·Ͱͷ࣌ؒΛܭଌ ͔͔ͬͨ࣌ؒΛMackerelʹߘɾࢹ ܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔ͔Λ֬ೝ͢ΔE2Eॏཁʂ
ࢹΛ࡞Δ ίϯϙʔωϯτΛΈ߹Θͤͨ࣌ શମͷܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔͱͲ͏͍͏͜ͱ͔ ܥશମͷՄ༻ੑͷతͳূڌ ܥΛ֎͔Βݟͨ࣌ͷಈ͖Λܭଌɾࢹ͠·͠ΐ͏
ʮࢹΛ͢Δʯͱ ϝτϦοΫʹᮢΛઃఆ͢Δ͜ͱ ͚ͩͰͳ͍
ීஈ͔ΒோΊΔ Kinesis Iterator Age Lambda࣮ߦճ Lambdaॲཧ࣌ؒ ߹ܭॲཧ DynamoDB Write E2E
MackerelͷάϥϑϘʔυ
ࢹ͢Δͱ͍͏͜ͱ ฏৗঢ়ଶΛΓ ఆٛ͢Δͱ͍͏͜ͱ
·ͱΊ ϝτϦοΫΛՄࢹԽͯ͠ோΊΑ͏ ࢹͷجૅฏৗঢ়ଶΛΔ͜ͱ ܥશମͷՄ༻ੑΛࢹ͠Α͏ ʮࢹΛ࡞Δʯ
We are hiring! σΟϨΫλʔ CRE (Customer Reliability Engineer) WebΞϓϦέʔγϣϯΤϯδχΞ WebΦϖϨʔγϣϯΤϯδχΞ
iPhoneɾAndroidΞϓϦΤϯδχΞ
Thank you!