Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf Tokyo 2017
Search
Ken Hamada
November 03, 2017
Technology
18
28k
サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf Tokyo 2017
サーバレスアーキテクチャによる時系列データベースの構築と監視
Ken Hamada
November 03, 2017
Tweet
Share
More Decks by Ken Hamada
See All by Ken Hamada
組織と技術の両輪で開発を加速させるkintoneチームの取り組み / JJUG CCC 2022 Fall Cybozu kintone
itchyny
1
2.1k
Mackerelチームにおけるインフラオーナーシップ / Hatena Pepabo tech conf 4 DevOps Kyoto
itchyny
0
5.4k
Other Decks in Technology
See All in Technology
JAWS-UG Bedrock Claude Night
yamahiro
3
580
20240418_Google ColabにLLMが搭載されたようなのでPython x データ分析の勉強方法を考えてみる
doradora09
0
130
エンジニア候補者向け資料2024.04.24.pdf
macloud
0
3.3k
ここが嬉しいABAC ここが辛いよABAC #再解説+補足編
masahirokawahara
1
270
Google Cloud Next '24でブログを10本書いた方法と勉強会を沸かせた方法
yasumuusan
0
290
Postman v10リリース後を振り返る / Looking back at Postman v10 after release
yokawasa
1
160
反実仮想機械学習とは何か
usaito
PRO
11
4.6k
コンテナセキュリティの基本と脅威への対策
kyohmizu
3
760
レガシーをぶっ壊せ。AEONで始めるDevRelの話 / Qiita Night 2024-2-22
aeonpeople
3
1.3k
Tellus の衛星データを見てみよう #mf_fukuoka
kongmingstrap
0
190
サーバー間 GraphQL と webmock-graphql の話 / server-to-server graphql and webmock-graphql
qsona
2
190
DevOpsメトリクスとアウトカムの接続にトライ!開発プロセスを通して計測できるメトリクスの活用方法
ham0215
2
240
Featured
See All Featured
Building Adaptive Systems
keathley
31
1.9k
The Cult of Friendly URLs
andyhume
74
5.7k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
78
42k
Code Review Best Practice
trishagee
55
15k
Principles of Awesome APIs and How to Build Them.
keavy
121
16k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
2
1.3k
The Illustrated Children's Guide to Kubernetes
chrisshort
31
46k
Product Roadmaps are Hard
iamctodd
44
9.7k
In The Pink: A Labor of Love
frogandcode
138
21k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
17
1.4k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
20
1.9k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
14
1.5k
Transcript
αʔόϨεΞʔΩςΫνϟʹΑΔ ࣌ܥྻσʔλϕʔεͷߏஙͱࢹ גࣜձࣾͯͳɹ⅓ా ݈ 2017/11/03 Serverlessconf Tokyo
ࣗݾհ גࣜձࣾͯͳ ΞϓϦέʔγϣϯΤϯδχΞ MackerelνʔϜ ςοΫϦʔυ ͖: Rust, Haskell, Go, Scala,
Vim ϓϩάϥϜϞάϞά itchyny.hatenablog.com lightline.vim, calendar.vim, etc. Thank you! ⅓ా ݈ Ken Hamada
αʔόʔϨεͷίϯϙʔωϯτΛͬͯ αʔϏεΛӡ༻͍ͯ͠Δਓ ✋
αʔόʔϨεͷίϯϙʔωϯτΛ ͔ͬ͠Γࢹ͍ͯ͠Δਓ ✋
ࢹαʔϏεΛఏڙ͍ͯ͠Δࢲ͕ͨͪ ͲͷΑ͏ʹϚωʔδυαʔϏεΛ ࢹɾཧ͍ͯ͠Δ͔Λ͓͑͠·͢
࣍ Mackerelʹ͍ͭͯ ࣌ܥྻσʔλϕʔε Mackerelͷ৽࣌ܥྻσʔλϕʔεͷ։ൃ Lambda functionͷύϑΥʔϚϯεɾνϡʔχϯά ֤ίϯϙʔωϯτͷಛੑͱࢹ
Mackerelʹ͍ͭͯ
Mackerel mackerel.io
Mackerel SaaSͷαʔόʔࢹɾཧπʔϧ ͯͳࣾͷࢹγεςϜΛαʔϏεԽͯ͠ެ։ mackerel-agent͕ϝτϦοΫΛߘ͢Δpushܕ νϟοτπʔϧߏཧπʔϧͱΈ߹Θͤ Slack, HipChat, etc. Chef, Ansible,
etc.
Mackerelུ֓ਤ CPUɾMemory༻ OSใ ϝτϦοΫσʔλ ϝλσʔλ ϝʔϧ ཧը໘ νϟοτπʔϧ Ξϥʔτൃੜ
Mackerelͷάϥϑ ϝτϦοΫͷσʔλ ࣌ܥྻσʔλϕʔεʹ อଘ͞Ε͍ͯΔ ϗετใϝλσʔλ ϝΠϯͷσʔλϕʔε (PostgreSQL) ʹอଘ
࣌ܥྻσʔλϕʔε
࣌ܥྻσʔλϕʔε • ࣌ܥྻͷσʔλ (యܕతʹσʔλ) ͷอଘʹಛԽ • αʔόʔϞχλϦϯάɾIoT༻్ • ॻ͖ࠐΈෛՙܰݮɾdisk༰ྔͷ࠷దԽ •
ߴղ૾ɾظอଘɾେྔϝτϦοΫ 㱺 ࢄDB • ྫ: Graphite, InfluxDB, Prometheus, OpenTSDB Mackerel͕͍ͬͯ·ͨ͠
GraphiteΛӡ༻͢Δ্Ͱͷ՝ ϋʔυΣΞͷݶք (disk༰ྔ) • ετϨʔδͷௐୡίετ εέʔϥϏϦςΟʔͷ֬อ͕ࠔ • ϝτϦοΫͷظอଘ (1m:25h 㱺
1m:400d+) σʔλϩετੑͷ͞ ϋοΫɾ࠷దԽ͠ʹ͘͞ ΫϥυΛ׆༻ͯ͠ӡ༻ίετݮ͍ͨ͠ Application servers mackerel.io
৽࣌ܥྻDBͰୡ͍ͨ͜͠ͱ εέʔϥϏϦςΟʔ ӡ༻ίετɾϋʔυΣΞௐୡίετ͑Δ σʔλϩετੑ
৽࣌ܥྻDBͷઃܭ εέʔϥϏϦςΟʔɾӡ༻ίετɾௐୡίετݮ • ϚωʔδυɾαʔϏεΛ͏ 㱺 AWS DynamoDB DynamoDBͷॻ͖ࠐΈίετ (pricing) Λ͍͑ͨ
• RedisΛΩϟογϡͱͯ͠ར༻ ݹ͍ϝτϦοΫσʔλ༰ྔ୯Ձͷ͍҆ετϨʔδʹҠಈ • DynamoDB 㱺 S3 Ҡಈ σʔλϩετੑ • Amazon Kinesis Streams
ہॴࢀরੑͱετϨʔδͷબ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis
ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ
ہॴࢀরੑͱετϨʔδͷબ සൟʹࢀর͞ΕΔ ৽͍͠ϝτϦοΫ ݹ͍ ͋·Γࢀর͞Εͳ͍ ಡΈॻ͖ίετ อ࣋ίετ DynamoDB S3 Redis
ϨΠςϯγ εέʔϥϏϦςΟʔ ༰ྔ୯Ձ ಛੑͷҟͳΔετϨʔδΛΈ߹ΘͤΔ͜ͱͰ ͔͔ΔίετΛ࠷దԽ͢Δ
Aug. 2017 ৽࣌ܥྻDBʹҠߦ
diamondߏਤ Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond: graphiteͷಉૉମ (ͱͯߗ͍) DynamoDB Kinesis Steams S3 Lambda
Redis cluster Lambda TTL expired Application servers diamond-reader mackerel.io diamond-writer
Mackerelͷ৽࣌ܥྻσʔλϕʔε diamond-writerͷ։ൃ
diamond-writer • ࣌ܥྻϝτϦοΫσʔλΛRedisɾDynamoDBʹॻ͖ࠐΉ • AWS Lambda (node.js) • RedisΛ͍ॻ͖ࠐΈίετΛ࠷దԽ •
ΞΠςϜʹTTLΛઃఆ • ݹ͍ϝτϦοΫࣗಈͰফ͑Δ DynamoDB Kinesis Steams Redis cluster Lambda
Kinesis Streams→Lambda • Kinesis StreamsΛtriggerͱͯ͠LambdaΛىಈ • Lambda͕ॲཧͯ͠σʔλKinesis StreamsʹΔ • োൃੜ͔࣌Β࠶ॲཧͰ͖Δ
• backup༻ʹผͷLambdaΛ͏ • ࠷৽ͷϨίʔυΛॲཧ͠ͳ͕Β෮چ • Lambdaͷॲཧͷႈੑ͕ॏཁ Kinesis Steams Lambda Lambda backup TRIM_HORIZON AT_TIMESTAMP ো࣌ͷόοΫΞοϓܥ
Lambda→Redis cluster • DynamoDBͷwrite-back cacheͱͯ͠RedisΛར༻ • EC2্ͰclusterΛΜͰ͍Δ • ॳElastiCacheΛఆ͍ͯͨ͠ •
εέʔϧ {Ξοϓ,Ξτ} ͕ΦϯϥΠϯͰͰ͖ͳ͍ • Ͱ͖ΔΑ͏ʹͳͬͨΒҠߦݕ౼͠·͢ • ϝτϦοΫ͕ҰఆݸҎ্ͨ·ͬͨΒDynamoDBʹॻ͖ࠐΉ • cronεέʔϧͤ͞ʹ͍͘ͷͰجຊతʹΘͳ͍ Lambda Redis cluster
Lambda→DynamoDB • MetricName (partition key)ɾTimestamp (sort key) • ϝτϦοΫMapʹอଘ •
BatchWriteItemͰ·ͱΊͯॻ͖ࠐΉ • Write capacityجຊతʹҰఆ • Read capacityΞΫηε͕ूத͢Δ͜ͱ͕͋Δ DynamoDB Lambda
Lambda ίʔυߏɾdeploy ϩδοΫlibσΟϨΫτϦʹΓग़͢ • function handlerΛ࠷খʹอͭ (clientೖɾcallback) • ֤ϥΠϒϥϦʔΛखް͘ςετ apexͰdeploy
• build hookͰlibσΟϨΫτϦͷίϐʔͱnpm install —production • σΟϨΫτϦ͝ͱzipʹͯ͠Ξοϓϩʔυͯ͘͠ΕΔ • zipΛ࡞Δͱ͖ʹsymlinkḷͬͯ͘ΕΔ functions diamond-writer lib diamondwriter.js index.js metriccache.js metriccache.spec.js package.json diamondwriter.spec.js diamond-writer-backup index.js
Lambda functionͷ ύϑΥʔϚϯεɾνϡʔχϯά
τϥϒϧൃੜʂ ϦΫΤετΛฒߦॲཧʹ͍ͯ͠Δͷʹɺ ఆΑΓύϑΥʔϚϯε͕ग़ͳ͍ʂ ࢥͬͨΑΓ͓ஈ͕ߴ͍
ܭଌͤΑʂ LambdaͷίϯςφʹϩάΠϯͰ͖ͳ͍ʂ topnetstattcpdumpଧͯͳ͍ʂ
͓͍ͪͭͯ CPU bound ͔ I/O bound ͔ Έ͖ΘΊ·͠ΐ͏
CPU bound ͔ I/O bound • Kinesis Streams͕triggerͷ࣌ • Batch
sizeΛ૿ͨ͠ͱ͖ͷdurationͷԠ • I/OଟॏԽ͍ͯͨ͠ΒجຊҰఆͷͣ • ઢܗԠͳΒೖྗαΠζʹൺྫ͢ΔCPUॲཧ͕ࢧత (͔) • Ұൠʹ…ʁ • ϝϞϦʔΛ্͛ͯCPUεϖοΫ্͕͕ͬͨ࣌ͷdurationͷԠ • I/O͕ࢧతͳΒCPU্͕͕ͬͯͦ͜·ͰԼ͕Βͳ͍ͣ • యܕతͳϦΫΤετͷlatencyduration͔ΒײతʹΘ͔Γͦ͏
ύϑΥʔϚϯεܭଌͱվળ CPU boundͷͱ͖ • ࣌ؒΛଌͬͯCloudWatchϩάʹग़͢ (ૉ) • खݩͰܭଌ͢Δ • sjsp
(Simple JavaScript Profiler) վળ • ϝϞϦʔΛ૿͢ (CPUεϖοΫ্) • ΞϧΰϦζϜɾॲཧ༰ͷվળ
sjsp ࡞ “Simple JavaScript Profiler” • MackerelͷϑϩϯτΤϯυͷύϑΥʔϚϯεղੳͷͨΊʹ࡞ͬͨ • JavaScriptΛτϥϯεύΠϧ͢Δ͚ͩ (ϒϥβʹґଘ͠ͳ͍)
• Lambda function (node.js) ͷύϑΥʔϚϯεܭଌʹ͑Δ CPU boundͳॏ͍ॲཧ͕͔Δ • ࣮ࡍdiamond-writerͷॏ͍ॲཧΛݟ͚ͭΔ͜ͱ͕Ͱ͖ͨ ʮ͍ͭͱಉ͡Α͏ʹύϑΥʔϚϯεܭଌ͢Δʯ
ύϑΥʔϚϯεܭଌͱվળ I/O boundͷͱ͖ • AWS X-RayͰॏ͍ϦΫΤετΛௐΔ վળ • ΫΤϦΛ·ͱΊΔɾϦΫΤετΛݮΒ͢ •
ΞΠςϜͷ·ͱΊ͔ͨΛม͑Δɾѹॖ͢Δ • Batch sizeΛ૿͢
AWS X-Ray ϚωʔδυαʔϏεͷϦΫΤετͷใΛऩूɾੳ AWSͷSDKʹΈࠐΉ • ؆୯ͳมߋͰτϨʔγϯάΛ࢝ΊΒΕΔ • I/O boundͷ߹ʹΛൃݟͰ͖Δ const
AWSXRay = require('aws-xray-sdk-core'); const AWS = AWSXRay.captureAWS(require('aws-sdk'));
֤ίϯϙʔωϯτͷಛੑͱࢹ
Mackerel plugins ͯͳʹ͓͚ΔAWSαʔϏεͷࢹ Monitoring host Mackerelͷpluginͱ AWSΠϯςάϨʔγϣϯͰ ϝτϦοΫΛऩूɾ ՄࢹԽ CloudWatch
Application servers DB, Proxies, etc. #mackerel #bookmark #blog E-mail AWS Integration CloudWatchͷϝτϦοΫMackerelʹू ϗετཧɾΞϥʔτɾ௨ཧΛҰݩԽ EC2 ΦϯϓϨͷϗετ ͨ͘͞Μ͋Δ
diamondߏਤ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda TTL
expired Application servers diamond-reader mackerel.io diamond-writer
Kinesis Streamsͷಛੑ Kinesis Steams Lambda #Shard Records PutRecords Partition Key
γϟʔυΛݻఆ͍ͨ࣌͠ Partition KeyΛࢦఆ ϝτϦοΫ໊Λݩʹੜ ਖ਼͘͠όϥ͚ͤ͞Δͷ͕େࣄ GetShardIterator GetRecords γϟʔυ * ར༻࣌ؒ PUTϖΠϩʔυαΠζ σʔλอ࣋ظؒ (1d ~ 7d) ྉۚମܥ
Kinesis Streamsͷࢹ I/O bytes͕Լ͕͍ͬͯΔ • PutRecordsʹࣦഊ͍ͯ͠Δ • ຊମͷΞϓϦέʔγϣϯΛٙ͏ • GetRecordsʹࣦഊ͍ͯ͠Δ
• ޙஈͷLambdaΛٙ͏ Read delay্͕͕͍ͬͯΔ • Lambda͕ॲཧ͖͠Ε͍ͯͳ͍ • LambdaͷΤϥʔɾλΠϜΞτ • γϟʔυ͕Γͳ͍ • γϟʔυΛ૿ͯ͠ throughputΛ্͛Δ Read delay I/O bytes
Lambdaͷಛੑ FaaS (Function as a Service) ֤ίϯςφجຊతʹ࠶ར༻͞ΕΔ • deployઃఆมߋͰ࡞Γ͞ΕΔ •
clientglobalมͰӬଓԽ ͦΕͧΕͷίϯςφಉ࣌ʹ1ͭॲཧ • ֤Kinesis shardʹ1ͭίϯςφཱ͕ͭ Kinesis Steams Lambda GetShardIterator GetRecords IteratorType Batch Size DynamoDB Redis cluster ֤containerhandlerΛ ಉ࣌ʹҰճ͔͠ॲཧ͠ͳ͍ ϝϞϦʔαΠζ (ॲཧೳྗ) ߹ܭ࣮ߦ࣌ؒ ϦΫΤετճ ྉۚମܥ
Lambdaͷࢹ Iterator Age (ms) ࣮ߦॲཧ࣌ؒ (ms) ࣮ߦճɾΤϥʔճ Τϥʔճ্͕͕͍ͬͯΔ • CloudWatch
LogsͰௐࠪ • apex logs —follow | grep Duration avg (ms) ্͕͕͍ͬͯΔ • ຊମଆͷෛՙɾKinesisνΣοΫ • DynamoDBͷRequest latency • ϝϞϦʔΛ૿ͦ͏ • CPUεϖοΫྉ͕ۚ͋Δ Iterator Age্͕͕͍ͬͯΔ • Τϥʔ͕ग़͍ͯΔ͔Ͳ͏͔ • ޙஈͷthroughputΓͳ͍ • DynamoDBͷCapacity্͛Δ • Kinesis shardΛ૿͢
Lambdaͷࢹ ฏۉಉ࣌ॲཧ = 1ؒͷ࣮ߦճ * ฏۉ࣮ߦ࣌ؒ (ms) / 60,000 (ms)
≤ Kinesis shard time 1 minute container processing idle Mackerel advanced graph Shardͷ7ׂҎԼ(҆)ʹ͑Δ ShardΛ૿͢ͷ͕͔͔࣌ؒΔ ੵ͕՝ۚର ύϑΥʔϚϯενϡʔχϯάͷࢦඪ Invocation count * Duration average (ms) / 60,000 (ms) ≤ Kinesis shard count
DynamoDBͷಛੑ DynamoDB Lambda diamond-reader BatchWriteItem BatchGetItem ϚωʔδυͳNoSQLσʔλϕʔε ʮΫϥυ൛ͷࢄσʔλϕʔεʯ ΦϯϥΠϯͰΩϟύγςΟʔΛมߋ 3ͭͷϨϓϦΧʹΑΔߴՄ༻ੑ
{Read,Write} ΩϟύγςΟ σʔλసૹྔ σʔλอଘ༰ྔ ྉۚମܥ
DynamoDBͷࢹ Write capacity Read capacity Throttled requests Write Capacity্͕͕͍ͬͯΔ •
LatencyͷѱԽɾ400Λฦ͢ • Throttled requests͕Ͱ͍ͯΕ Provisioned Write Capacity͋͛Δ Read Capacity্͕͕͍ͬͯΔ • Writeͱಉ༷ɾ400Λฦ͢ • Provisioned Read Capacity͋͛Δ Throttle events্͕͕͍ͬͯΔ • Τϥʔ400Λฦ͍ͯ͠Δঢ়ଶ • Provisioned Capacity͕Γͳ͍ • ϦΫΤετΛΏͬ͘Γʹ͢Δ • ϦΫΤετΛ·ͱΊΔ
Redisͷࢹ CPU % Redis used memory Redis processed commands CPU
%্͕͕͍ͬͯΔ • Redis1 core͔͑͠ͳ͍ • readΫΤϦΛslaveʹ͚Δ • nodeΛՃͯ͠resharding memory༻ྔ • ϋʔυΣΞͷݶք͕͋Δ • key͕ফ͑ͯOSʹฦ͞ͳ͍ • redis-cli info | grep used_memory • resharding ࣮ߦͨ͠ίϚϯυ • RedisͰॲཧͨ͠ྔͷࢦඪ ֤Πϯελϯεʹͯ RedisϓϥάΠϯͰՄࢹԽ
diamond-readerͷಛੑ GoݴޠͰॻ͔ΕͨwebΞϓϦέʔγϣϯ ෳͷετϨʔδ͔ΒϝτϦοΫΛಡΈࠐΉ ΞΠςϜ͝ͱʹgoroutineɾI/OΛଟॏԽ DynamoDB S3 Redis cluster Application servers
diamond-reader
diamond-readerͷࢹ golang-stats-api-handlerͱ mackerel-plugin-gostatsͰ GoͷwebΞϓϦέʔγϣϯͷ ϞχλϦϯάΛ࢝ΊΒΕΔΑʂ GC ώʔϓ ϝϞϦʔ mallocɾfree goroutineɾCGO
࣌ܥྻσʔλϕʔε͕ਖ਼͘͠ಈ͍͍ͯΔͱ 1. ϝτϦοΫΛਖ਼͘͠ετϨʔδʹอଘ 2. ͦͷϝτϦοΫΛಡΈऔΔ͜ͱ͕Ͱ͖Δ
ॲཧ ࢹ Lambda͕ॲཧͨ͠ϝτϦοΫͷ / 1m • RedisɾDynamoDBʹՃͨ͠ϝτϦοΫ • CloudWatchʹJSONͰใΛग़͢ •
ผͷLambda functionͰҰؒͷϩάΛूܭ (cron trigger) 㱺 Mackerelʹߘ • ॲཧ͕Լ͕ͬͨΒΞϥʔτ • Redisʹॲཧ༻ͷΩʔΛҰ͝ͱʹ࡞ͬͯincrͯ͠Α͍
End-to-end ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda
TTL expired Application servers diamond-reader mackerel.io ϝτϦοΫΛߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ
End-to-end ࢹ DynamoDB Kinesis Steams S3 Lambda Redis cluster Lambda
TTL expired Application servers diamond-reader mackerel.io API ϝτϦοΫΛߘ͔ͯ͠ΒάϥϑͰݟΕΔ·Ͱͷ࣌ؒ Monitoring host
End-to-end ࢹ ࢹ༻ϗετ͔ΒμϛʔϝτϦοΫΛߘ sleep͠ͳ͕ΒɺऔಘͰ͖Δ·Ͱͷ࣌ؒΛܭଌ ͔͔ͬͨ࣌ؒΛMackerelʹߘɾࢹ ܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔ͔Λ֬ೝ͢ΔE2Eॏཁʂ
ࢹΛ࡞Δ ίϯϙʔωϯτΛΈ߹Θͤͨ࣌ શମͷܥͱͯ͠ਖ਼͘͠ಈ͍͍ͯΔͱͲ͏͍͏͜ͱ͔ ܥશମͷՄ༻ੑͷతͳূڌ ܥΛ֎͔Βݟͨ࣌ͷಈ͖Λܭଌɾࢹ͠·͠ΐ͏
ʮࢹΛ͢Δʯͱ ϝτϦοΫʹᮢΛઃఆ͢Δ͜ͱ ͚ͩͰͳ͍
ීஈ͔ΒோΊΔ Kinesis Iterator Age Lambda࣮ߦճ Lambdaॲཧ࣌ؒ ߹ܭॲཧ DynamoDB Write E2E
MackerelͷάϥϑϘʔυ
ࢹ͢Δͱ͍͏͜ͱ ฏৗঢ়ଶΛΓ ఆٛ͢Δͱ͍͏͜ͱ
·ͱΊ ϝτϦοΫΛՄࢹԽͯ͠ோΊΑ͏ ࢹͷجૅฏৗঢ়ଶΛΔ͜ͱ ܥશମͷՄ༻ੑΛࢹ͠Α͏ ʮࢹΛ࡞Δʯ
We are hiring! σΟϨΫλʔ CRE (Customer Reliability Engineer) WebΞϓϦέʔγϣϯΤϯδχΞ WebΦϖϨʔγϣϯΤϯδχΞ
iPhoneɾAndroidΞϓϦΤϯδχΞ
Thank you!