Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf...
Search
Ken Hamada
November 03, 2017
Technology
18
29k
サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf Tokyo 2017
サーバレスアーキテクチャによる時系列データベースの構築と監視
Ken Hamada
November 03, 2017
Tweet
Share
More Decks by Ken Hamada
See All by Ken Hamada
組織と技術の両輪で開発を加速させるkintoneチームの取り組み / JJUG CCC 2022 Fall Cybozu kintone
itchyny
1
2.2k
Mackerelチームにおけるインフラオーナーシップ / Hatena Pepabo tech conf 4 DevOps Kyoto
itchyny
0
5.5k
Other Decks in Technology
See All in Technology
よくわからんサービスについての問い合わせが来たときの強い味方 Amazon Q について
kazzpapa3
0
220
Autify Company Deck
autifyhq
1
39k
Fargateを使った研修の話
takesection
0
110
いまならこう作りたい AWSコンテナ[本格]入門ハンズオン 〜2024年版 ハンズオンの構想〜
horsewin
9
2.1k
WINTICKETアプリで実現した高可用性と高速リリースを支えるエコシステム / winticket-eco-system
cyberagentdevelopers
PRO
1
190
AWSコンテナ本出版から3年経った今、もし改めて執筆し直すなら / If I revise our container book
iselegant
15
4k
現地でMeet Upをやる場合の注意点〜反省点を添えて〜
shotashiratori
0
520
フルカイテン株式会社 採用資料
fullkaiten
0
36k
話題のGraphRAG、その可能性と課題を理解する
hide212131
4
1.5k
マネジメント視点でのre:Invent参加 ~もしCEOがre:Inventに行ったら~
kojiasai
0
460
Product Engineer Night #6プロダクトエンジニアを育む仕組み・施策
hacomono
PRO
1
470
生成AIとAWS CDKで実現! 自社ブログレビューの効率化
ymae
2
330
Featured
See All Featured
Into the Great Unknown - MozCon
thekraken
31
1.5k
It's Worth the Effort
3n
183
27k
Side Projects
sachag
452
42k
Music & Morning Musume
bryan
46
6.1k
Rails Girls Zürich Keynote
gr2m
93
13k
Large-scale JavaScript Application Architecture
addyosmani
510
110k
StorybookのUI Testing Handbookを読んだ
zakiyama
26
5.2k
Bash Introduction
62gerente
608
210k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
355
29k
Adopting Sorbet at Scale
ufuk
73
9k
Faster Mobile Websites
deanohume
304
30k
Facilitating Awesome Meetings
lara
49
6k
Transcript
αʔόϨεΞʔΩςΫνϟʹΑΔ ࣌ܥྻσʔλϕʔεͷߏஙͱࢹ גࣜձࣾͯͳɹ⅓ా ݈ 2017/11/03 Serverlessconf Tokyo
ࣗݾհ גࣜձࣾͯͳ ΞϓϦέʔγϣϯΤϯδχΞ MackerelνʔϜ ςοΫϦʔυ ͖: Rust, Haskell, Go, Scala,
Vim ϓϩάϥϜϞάϞά itchyny.hatenablog.com lightline.vim, calendar.vim, etc. Thank you! ⅓ా ݈ Ken Hamada
αʔόʔϨεͷίϯϙʔωϯτΛͬͯ αʔϏεΛӡ༻͍ͯ͠Δਓ ✋
αʔόʔϨεͷίϯϙʔωϯτΛ ͔ͬ͠Γࢹ͍ͯ͠Δਓ ✋
ࢹαʔϏεΛఏڙ͍ͯ͠Δࢲ͕ͨͪ ͲͷΑ͏ʹϚωʔδυαʔϏεΛ ࢹɾཧ͍ͯ͠Δ͔Λ͓͑͠·͢
࣍ Mackerelʹ͍ͭͯ ࣌ܥྻσʔλϕʔε Mackerelͷ৽࣌ܥྻσʔλϕʔεͷ։ൃ Lambda functionͷύϑΥʔϚϯεɾνϡʔχϯά ֤ίϯϙʔωϯτͷಛੑͱࢹ
Mackerelʹ͍ͭͯ
Mackerel mackerel.io
Mackerel SaaSͷαʔόʔࢹɾཧπʔϧ ͯͳࣾͷࢹγεςϜΛαʔϏεԽͯ͠ެ։ mackerel-agent͕ϝτϦοΫΛߘ͢Δpushܕ νϟοτπʔϧߏཧπʔϧͱΈ߹Θͤ Slack, HipChat, etc. Chef, Ansible,
etc.
Mackerelུ֓ਤ CPUɾMemory༻ OSใ ϝτϦοΫσʔλ ϝλσʔλ ϝʔϧ ཧը໘ νϟοτπʔϧ Ξϥʔτൃੜ
Mackerelͷάϥϑ ϝτϦοΫͷσʔλ ࣌ܥྻσʔλϕʔεʹ อଘ͞Ε͍ͯΔ ϗετใϝλσʔλ ϝΠϯͷσʔλϕʔε (PostgreSQL) ʹอଘ
࣌ܥྻσʔλϕʔε
࣌ܥྻσʔλϕʔε • ࣌ܥྻͷσʔλ (యܕతʹσʔλ) ͷอଘʹಛԽ • αʔόʔϞχλϦϯάɾIoT༻్ • ॻ͖ࠐΈෛՙܰݮɾdisk༰ྔͷ࠷దԽ •
ߴղ૾ɾظอଘɾେྔϝτϦοΫ 㱺 ࢄDB • ྫ: Graphite, InfluxDB, Prometheus, OpenTSDB Mackerel͕͍ͬͯ·ͨ͠
GraphiteΛӡ༻͢Δ্Ͱͷ՝ ϋʔυΣΞͷݶք (disk༰ྔ) • ετϨʔδͷௐୡίετ εέʔϥϏϦςΟʔͷ֬อ͕ࠔ • ϝτϦοΫͷظอଘ (1m:25h 㱺
1m:400d+) σʔλϩετੑͷ͞ ϋοΫɾ࠷దԽ͠ʹ͘͞ ΫϥυΛ׆༻ͯ͠ӡ༻ίετݮ͍ͨ͠ Application servers mackerel.io
৽࣌ܥྻDBͰୡ͍ͨ͜͠ͱ εέʔϥϏϦςΟʔ ӡ༻ίετɾϋʔυΣΞௐୡίετ͑Δ σʔλϩετੑ
৽࣌ܥྻDBͷઃܭ εέʔϥϏϦςΟʔɾӡ༻ίετɾௐୡίετݮ • ϚωʔδυɾαʔϏεΛ͏ 㱺 AWS DynamoDB DynamoDBͷॻ͖ࠐΈίετ (pricing) Λ͍͑ͨ
• RedisΛΩϟογϡͱͯ͠ར༻ ݹ͍ϝτϦοΫσʔλ༰ྔ୯Ձͷ͍҆ετϨʔδʹҠಈ • DynamoDB 㱺 S3 Ҡಈ σʔλϩετੑ • Amazon Kinesis Streams
ہॴࢀরੑͱετϨʔδͷબ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis
ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ
ہॴࢀরੑͱετϨʔδͷબ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis
ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ ಛੑͷҟͳΔετϨʔδΛΈ߹ΘͤΔ͜ͱͰ ͔͔ΔίετΛ࠷దԽ͢Δ
Aug. 2017 ৽࣌ܥྻDBʹҠߦ
diamondߏਤ Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond: graphiteͷಉૉମ (ͱͯߗ͍) DynamoDB Kinesis Steams S3 Lambda
Redis cluster Lambda TTL expired Application servers diamond-reader mackerel.io diamond-writer
Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond-writerͷ։ൃ
diamond-writer • ࣌ܥྻϝτϦοΫσʔλΛRedisɾDynamoDBʹॻ͖ࠐΉ • AWS Lambda (node.js) • RedisΛ͍ॻ͖ࠐΈίετΛ࠷దԽ •
ΞΠςϜʹTTLΛઃఆ • ݹ͍ϝτϦοΫࣗಈͰফ͑Δ DynamoDB Kinesis Steams Redis cluster Lambda
Kinesis Streams→Lambda • Kinesis StreamsΛtriggerͱͯ͠LambdaΛىಈ • Lambda͕ॲཧͯ͠σʔλKinesis StreamsʹΔ • োൃੜ͔࣌Β࠶ॲཧͰ͖Δ
• backup༻ʹผͷLambdaΛ͏ • ࠷৽ͷϨίʔυΛॲཧ͠ͳ͕Β෮چ • Lambdaͷॲཧͷႈੑ͕ॏཁ Kinesis Steams Lambda Lambda backup TRIM_HORIZON AT_TIMESTAMP ো࣌ͷόοΫΞοϓܥ
Lambda→Redis cluster • DynamoDBͷwrite-back cacheͱͯ͠RedisΛར༻ • EC2্ͰclusterΛΜͰ͍Δ • ॳElastiCacheΛఆ͍ͯͨ͠ •
εέʔϧ {Ξοϓ,Ξτ} ͕ΦϯϥΠϯͰͰ͖ͳ͍ • Ͱ͖ΔΑ͏ʹͳͬͨΒҠߦݕ౼͠·͢ • ϝτϦοΫ͕ҰఆݸҎ্ͨ·ͬͨΒDynamoDBʹॻ͖ࠐΉ • cronεέʔϧͤ͞ʹ͍͘ͷͰجຊతʹΘͳ͍ Lambda Redis cluster
Lambda→DynamoDB • MetricName (partition key)ɾTimestamp (sort key) • ϝτϦοΫMapʹอଘ •
BatchWriteItemͰ·ͱΊͯॻ͖ࠐΉ • Write capacityجຊతʹҰఆ • Read capacityΞΫηε͕ूத͢Δ͜ͱ͕͋Δ DynamoDB Lambda
Lambda ίʔυߏɾdeploy ϩδοΫlibσΟϨΫτϦʹΓग़͢ • function handlerΛ࠷খʹอͭ (clientೖɾcallback) • ֤ϥΠϒϥϦʔΛखް͘ςετ apexͰdeploy
• build hookͰlibσΟϨΫτϦͷίϐʔͱnpm install —production • σΟϨΫτϦ͝ͱzipʹͯ͠Ξοϓϩʔυͯ͘͠ΕΔ • zipΛ࡞Δͱ͖ʹsymlinkḷͬͯ͘ΕΔ functions diamond-writer lib diamondwriter.js index.js metriccache.js metriccache.spec.js package.json diamondwriter.spec.js diamond-writer-backup index.js
Lambda functionͷ ύϑΥʔϚϯεɾνϡʔχϯά
τϥϒϧൃੜʂ ϦΫΤετΛฒߦॲཧʹ͍ͯ͠Δͷʹɺ ఆΑΓύϑΥʔϚϯε͕ग़ͳ͍ʂ ࢥͬͨΑΓ͓ஈ͕ߴ͍
ܭଌͤΑʂ LambdaͷίϯςφʹϩάΠϯͰ͖ͳ͍ʂ topnetstattcpdumpଧͯͳ͍ʂ
͓͍ͪͭͯ CPU bound ͔ I/O bound ͔ Έ͖ΘΊ·͠ΐ͏
CPU bound ͔ I/O bound • Kinesis Streams͕triggerͷ࣌ • Batch
sizeΛ૿ͨ͠ͱ͖ͷdurationͷԠ • I/OଟॏԽ͍ͯͨ͠ΒجຊҰఆͷͣ • ઢܗԠͳΒೖྗαΠζʹൺྫ͢ΔCPUॲཧ͕ࢧత (͔) • Ұൠʹ…ʁ • ϝϞϦʔΛ্͛ͯCPUεϖοΫ্͕͕ͬͨ࣌ͷdurationͷԠ • I/O͕ࢧతͳΒCPU্͕͕ͬͯͦ͜·ͰԼ͕Βͳ͍ͣ • యܕతͳϦΫΤετͷlatencyduration͔ΒײతʹΘ͔Γͦ͏
ύϑΥʔϚϯεܭଌͱվળ CPU boundͷͱ͖ • ࣌ؒΛଌͬͯCloudWatchϩάʹग़͢ (ૉ) • खݩͰܭଌ͢Δ • sjsp
(Simple JavaScript Profiler) վળ • ϝϞϦʔΛ૿͢ (CPUεϖοΫ্) • ΞϧΰϦζϜɾॲཧ༰ͷվળ
sjsp ࡞ “Simple JavaScript Profiler” • MackerelͷϑϩϯτΤϯυͷύϑΥʔϚϯεղੳͷͨΊʹ࡞ͬͨ • JavaScriptΛτϥϯεύΠϧ͢Δ͚ͩ (ϒϥβʹґଘ͠ͳ͍)
• Lambda function (node.js) ͷύϑΥʔϚϯεܭଌʹ͑Δ CPU boundͳॏ͍ॲཧ͕͔Δ • ࣮ࡍdiamond-writerͷॏ͍ॲཧΛݟ͚ͭΔ͜ͱ͕Ͱ͖ͨ ʮ͍ͭͱಉ͡Α͏ʹύϑΥʔϚϯεܭଌ͢Δʯ
ύϑΥʔϚϯεܭଌͱվળ I/O boundͷͱ͖ • AWS X-RayͰॏ͍ϦΫΤετΛௐΔ վળ • ΫΤϦΛ·ͱΊΔɾϦΫΤετΛݮΒ͢ •
ΞΠςϜͷ·ͱΊ͔ͨΛม͑Δɾѹॖ͢Δ • Batch sizeΛ૿͢
AWS X-Ray ϚωʔδυαʔϏεͷϦΫΤετͷใΛऩूɾੳ AWSͷSDKʹΈࠐΉ • ؆୯ͳมߋͰτϨʔγϯάΛ࢝ΊΒΕΔ • I/O boundͷ߹ʹΛൃݟͰ͖Δ const
AWSXRay = require('aws-xray-sdk-core'); const AWS = AWSXRay.captureAWS(require('aws-sdk'));
֤ίϯϙʔωϯτͷಛੑͱࢹ
Mackerel plugins ͯͳʹ͓͚ΔAWSαʔϏεͷࢹ Monitoring host Mackerelͷpluginͱ AWSΠϯςάϨʔγϣϯͰ ϝτϦοΫΛऩूɾ ՄࢹԽ CloudWatch
Application servers DB, Proxies, etc. #mackerel #bookmark #blog E-mail AWS Integration CloudWatchͷϝτϦοΫMackerelʹू ϗετཧɾΞϥʔτɾ௨ཧΛҰݩԽ EC2 ΦϯϓϨͷϗετ ͨ͘͞Μ͋Δ
diamondߏਤ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda TTL
expired Application servers diamond-reader mackerel.io diamond-writer
Kinesis Streamsͷಛੑ Kinesis Steams Lambda #Shard Records PutRecords Partition Key
γϟʔυΛݻఆ͍ͨ࣌͠ Partition KeyΛࢦఆ ϝτϦοΫ໊Λݩʹੜ ਖ਼͘͠όϥ͚ͤ͞Δͷ͕େࣄ GetShardIterator GetRecords γϟʔυ * ར༻࣌ؒ PUTϖΠϩʔυαΠζ σʔλอ࣋ظؒ (1d ~ 7d) ྉۚମܥ
Kinesis Streamsͷࢹ I/O bytes͕Լ͕͍ͬͯΔ • PutRecordsʹࣦഊ͍ͯ͠Δ • ຊମͷΞϓϦέʔγϣϯΛٙ͏ • GetRecordsʹࣦഊ͍ͯ͠Δ
• ޙஈͷLambdaΛٙ͏ Read delay্͕͕͍ͬͯΔ • Lambda͕ॲཧ͖͠Ε͍ͯͳ͍ • LambdaͷΤϥʔɾλΠϜΞτ • γϟʔυ͕Γͳ͍ • γϟʔυΛ૿ͯ͠ throughputΛ্͛Δ Read delay I/O bytes
Lambdaͷಛੑ FaaS (Function as a Service) ֤ίϯςφجຊతʹ࠶ར༻͞ΕΔ • deployઃఆมߋͰ࡞Γ͞ΕΔ •
clientglobalมͰӬଓԽ ͦΕͧΕͷίϯςφಉ࣌ʹ1ͭॲཧ • ֤Kinesis shardʹ1ͭίϯςφཱ͕ͭ Kinesis Steams Lambda GetShardIterator GetRecords IteratorType Batch Size DynamoDB Redis cluster ֤containerhandlerΛ ಉ࣌ʹҰճ͔͠ॲཧ͠ͳ͍ ϝϞϦʔαΠζ (ॲཧೳྗ) ߹ܭ࣮ߦ࣌ؒ ϦΫΤετճ ྉۚମܥ
Lambdaͷࢹ Iterator Age (ms) ࣮ߦॲཧ࣌ؒ (ms) ࣮ߦճɾΤϥʔճ Τϥʔճ্͕͕͍ͬͯΔ • CloudWatch
LogsͰௐࠪ • apex logs —follow | grep Duration avg (ms) ্͕͕͍ͬͯΔ • ຊମଆͷෛՙɾKinesisνΣοΫ • DynamoDBͷRequest latency • ϝϞϦʔΛ૿ͦ͏ • CPUεϖοΫྉ͕ۚ͋Δ Iterator Age্͕͕͍ͬͯΔ • Τϥʔ͕ग़͍ͯΔ͔Ͳ͏͔ • ޙஈͷthroughputΓͳ͍ • DynamoDBͷCapacity্͛Δ • Kinesis shardΛ૿͢
Lambdaͷࢹ ฏۉಉ࣌ॲཧ = 1ؒͷ࣮ߦճ * ฏۉ࣮ߦ࣌ؒ (ms) / 60,000 (ms)
≤ Kinesis shard time 1 minute container processing idle Mackerel advanced graph Shardͷ7ׂҎԼ(҆)ʹ͑Δ ShardΛ૿͢ͷ͕͔͔࣌ؒΔ ੵ͕՝ۚର ύϑΥʔϚϯενϡʔχϯάͷࢦඪ Invocation count * Duration average (ms) / 60,000 (ms) ≤ Kinesis shard count
DynamoDBͷಛੑ DynamoDB Lambda diamond-reader BatchWriteItem BatchGetItem ϚωʔδυͳNoSQLσʔλϕʔε ʮΫϥυ൛ͷࢄσʔλϕʔεʯ ΦϯϥΠϯͰΩϟύγςΟʔΛมߋ 3ͭͷϨϓϦΧʹΑΔߴՄ༻ੑ
{Read,Write} ΩϟύγςΟ σʔλసૹྔ σʔλอଘ༰ྔ ྉۚମܥ
DynamoDBͷࢹ Write capacity Read capacity Throttled requests Write Capacity্͕͕͍ͬͯΔ •
LatencyͷѱԽɾ400Λฦ͢ • Throttled requests͕Ͱ͍ͯΕ Provisioned Write Capacity͋͛Δ Read Capacity্͕͕͍ͬͯΔ • Writeͱಉ༷ɾ400Λฦ͢ • Provisioned Read Capacity͋͛Δ Throttle events্͕͕͍ͬͯΔ • Τϥʔ400Λฦ͍ͯ͠Δঢ়ଶ • Provisioned Capacity͕Γͳ͍ • ϦΫΤετΛΏͬ͘Γʹ͢Δ • ϦΫΤετΛ·ͱΊΔ
Redisͷࢹ CPU % Redis used memory Redis processed commands CPU
%্͕͕͍ͬͯΔ • Redis1 core͔͑͠ͳ͍ • readΫΤϦΛslaveʹ͚Δ • nodeΛՃͯ͠resharding memory༻ྔ • ϋʔυΣΞͷݶք͕͋Δ • key͕ফ͑ͯOSʹฦ͞ͳ͍ • redis-cli info | grep used_memory • resharding ࣮ߦͨ͠ίϚϯυ • RedisͰॲཧͨ͠ྔͷࢦඪ ֤Πϯελϯεʹͯ RedisϓϥάΠϯͰՄࢹԽ
diamond-readerͷಛੑ GoݴޠͰॻ͔ΕͨwebΞϓϦέʔγϣϯ ෳͷετϨʔδ͔ΒϝτϦοΫΛಡΈࠐΉ ΞΠςϜ͝ͱʹgoroutineɾI/OΛଟॏԽ DynamoDB S3 Redis cluster Application servers
diamond-reader
diamond-readerͷࢹ golang-stats-api-handlerͱ mackerel-plugin-gostatsͰ GoͷwebΞϓϦέʔγϣϯͷ ϞχλϦϯάΛ࢝ΊΒΕΔΑʂ GC ώʔϓ ϝϞϦʔ mallocɾfree goroutineɾCGO
࣌ܥྻσʔλϕʔε͕ਖ਼͘͠ಈ͍͍ͯΔͱ 1. ϝτϦοΫΛਖ਼͘͠ετϨʔδʹอଘ 2. ͦͷϝτϦοΫΛಡΈऔΔ͜ͱ͕Ͱ͖Δ
ॲཧ ࢹ Lambda͕ॲཧͨ͠ϝτϦοΫͷ / 1m • RedisɾDynamoDBʹՃͨ͠ϝτϦοΫ • CloudWatchʹJSONͰใΛग़͢ •
ผͷLambda functionͰҰؒͷϩάΛूܭ (cron trigger) 㱺 Mackerelʹߘ • ॲཧ͕Լ͕ͬͨΒΞϥʔτ • Redisʹॲཧ༻ͷΩʔΛҰ͝ͱʹ࡞ͬͯincrͯ͠Α͍
End-to-end ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda
TTL expired Application servers diamond-reader mackerel.io ϝτϦοΫΛߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ
End-to-end ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda
TTL expired Application servers diamond-reader mackerel.io API ϝτϦοΫΛߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ Monitoring host
End-to-end ࢹ ࢹ༻ϗετ͔ΒμϛʔϝτϦοΫΛߘ sleep͠ͳ͕ΒɺऔಘͰ͖Δ·Ͱͷ࣌ؒΛܭଌ ͔͔ͬͨ࣌ؒΛMackerelʹߘɾࢹ ܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔ͔Λ֬ೝ͢ΔE2Eॏཁʂ
ࢹΛ࡞Δ ίϯϙʔωϯτΛΈ߹Θͤͨ࣌ શମͷܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔͱͲ͏͍͏͜ͱ͔ ܥશମͷՄ༻ੑͷతͳূڌ ܥΛ֎͔Βݟͨ࣌ͷಈ͖Λܭଌɾࢹ͠·͠ΐ͏
ʮࢹΛ͢Δʯͱ ϝτϦοΫʹᮢΛઃఆ͢Δ͜ͱ ͚ͩͰͳ͍
ීஈ͔ΒோΊΔ Kinesis Iterator Age Lambda࣮ߦճ Lambdaॲཧ࣌ؒ ߹ܭॲཧ DynamoDB Write E2E
MackerelͷάϥϑϘʔυ
ࢹ͢Δͱ͍͏͜ͱ ฏৗঢ়ଶΛΓ ఆٛ͢Δͱ͍͏͜ͱ
·ͱΊ ϝτϦοΫΛՄࢹԽͯ͠ோΊΑ͏ ࢹͷجૅฏৗঢ়ଶΛΔ͜ͱ ܥશମͷՄ༻ੑΛࢹ͠Α͏ ʮࢹΛ࡞Δʯ
We are hiring! σΟϨΫλʔ CRE (Customer Reliability Engineer) WebΞϓϦέʔγϣϯΤϯδχΞ WebΦϖϨʔγϣϯΤϯδχΞ
iPhoneɾAndroidΞϓϦΤϯδχΞ
Thank you!