Share
Embed
How mixi2 Uses TiDB for SNS Scalability and Performance
Developers Summit 2025 登壇資料
kanmo
February 14, 2025
Transcript
29 slides
-
mixi2ʹ͍ͭͯ • จςΩετSNS • ʮϑΥϩʔʯλΠϜϥΠϯͷଞɺSNS mixiʹ͋ͬͨίϛϡχςΟػೳΑΓ ϦΞϧλΠϜͳίϛϡχέʔγϣϯ ͚ͷΠϕϯτػೳ • αʔόGoɺϞόΠϧΫϥΠΞϯτ
FlutterΛ࠾༻ • ॳظͷػೳ։ൃ4ਓ(ϦϦʔε࣌7ਓ) -
DBͷબఆ SNSΛ։ൃ͢Δͱ͖ʹٻΊͨཁ݅ • ϦΫΤετɺϢʔβʔͷ૿ՃʹରԠͰ͖ΔεέʔϥϏϦςΟ • োൃੜ࣌ͷϑΣΠϧΦʔόʔ • ʢαʔϏεͱͯ͠ڐ༰Ͱ͖ΔʣϨΠςϯγʔ • τϥϯβΫγϣϯ
• ϝϯςφϯεΦϖϨʔγϣϯͷ༰қ͞ • ίετ໘ NewSQLʹ͍ͭͯಋೖݕ౼Λͨ͠ -
NewSQLͷݕ౼ ͦΕͧΕͷಛੑΛൺֱ͠ɺࠓճTiDBΛ࠾༻ τϥϯβΫ γϣϯ ϨΠςϯγʔ োੑ ӡ༻ίετ Πϯϑϥඅ༻ 3%# 4IBSEJOH
̋ʢ˚ʣ ̋ ˚ ˚ ̋ /FX42- ̋ ˚ ̋ ̋ ˚ -
TiDBͷಛ • ׂ͝ͱͷίϯϙʔωϯτ͕ࢄஔ͞Ε ͨΞʔΩςΫνϟ • TiDB, TiKV, PD • ֤ίϯϙʔωϯτԽ͢Δ͜ͱʹΑΓ
ύϑΥʔϚϯεͱোੑΛߴΊΒΕΔ • SQL LayerͷTiDBαʔόਫฏํʹ ֦ுՄೳ • ΫϥελΛཧ͢ΔPDαʔόɺ Λ૿͢͜ͱͰՄ༻ੑΛߴΊΒΕΔ ʢ࠷̏ʣ Ҿ༻ݩ: https://docs.pingcap.com/tidb/stable/tidb-architecture -
TiDB Storage (TiKV) • σʔλΛ֨ೲ͢ΔͷTiKVίϯϙʔωϯτ • σʔλʢϦʔδϣϯͱ͍͏୯ҐʣΛࣗಈͰෳ ͷϨϓϦΧͰཧ͞ΕΔ • ಉ͡σʔλ͕ࡾͭίϐʔ͞ΕΔ
• ߴՄ༻ੑΛ࣋ͪɺࣗಈϑΣΠϧΦʔόʔΛα ϙʔτ͢Δ • TiKVϊʔυΛΫϥελʹՃ͢Δ͜ͱͰ༰ྔ Λ૿͢͜ͱ͕Ͱ͖Δ • RDBͰߦ͍ͬͯͨσʔλංେԽʹ͏γϟʔ σΟϯάɺϚελʔԽͳͲ͕ෆཁʹͳΔ Ҿ༻ݩ: https://docs.pingcap.com/tidb/stable/tidb-storage -
Backup • όοΫΞοϓˍϦετΞ • σʔλόοΫΞοϓฐࣾαʔϏεӡ ༻ʹ͓͍ͯඞਢཁ݅ • εφοϓγϣοτͱ૿ϩά͕औΕΔ • όοΫΞοϓ͔Βղੳ༻σʔλͷߏங
• Dumpͷෛՙ͕ແࢹͰ͖ͳ͍͘Β͍ʹ σʔλ͕૿Ճͨ͠߹ɺόοΫΞοϓ ͔Β෮ݩͨ͠Ϋϥελ͔ΒDWH༻ͷ σʔλ͕औΕΔΑ͏ʹ͢Δ Starting component br: /.tiup/components/br/v8.1.0/br restore full --pd 10.30.2.92:2379 --storage s3://kan-tidb-test-tiup-backup/ snapshot-20240712-3 --send-credentials-to-tikv=false Detail BR log in /tmp/br.log.2024-07-12T08.13.06Z Full Restore <------------------------------------------------------------------- ----------------------------------------------------------------> 100.00% -
TiUP • ΫϥελʔͷߏஙɺScale out/Scale inɺBackupଐͷϕϯνπʔϧͳͲ ͯ͢ͷૢ࡞ͰtiupίϚϯυ͕ඇৗʹ ͍͔ͬͨ͢ • υΩϡϝϯτ๛ •
ϦϦʔεޙͷӡ༻ʹߩݙͯ͘͠Εͦ͏ tiup cluster scale-out test-cluster scaleout-tikv.yaml tikv_servers: - host: 10.20.1.206 config: server.labels: rack: r4 1. Ճ͢ΔίϯϙʔωϯτΛyamlʹఆٛ͢Δ 2. tiupίϚϯυΛ࣮ߦ͢Δ -
Home Timeline Timeline • ࣄલʹϗʔϜλΠϜϥΠϯΛ࡞͢Δ • ߘऀͷϑΥϩϫʔ/ίϛϡχςΟΛ୳ ͠ɺϑΥϩϫʔຖͷλΠϜϥΠϯςʔ ϒϧʹϙετΛॻ͖ࠐΉ ϙετIDΛ֤Ϣʔβʔ͝ͱʹ৴͍ͯ͠Δ
Post API Redis Redis Redis Find Followers Find Community Members Timeline API -
Timeline Tables (Redis) • ϢʔβʔͷϗʔϜλΠϜϥΠϯΛࣄલʹredis ʹ࡞͢Δ • ͜ͷใΛݩʹDBͷใͱΈ߹Θͤͯ ϗʔϜλΠϜϥΠϯΛߏங͢Δ •
ҰఆͷϙετใΛϢʔβʔຖʹอ࣋͢Δ • ࢀর͞ΕΔՄೳੑ͕͘ͳͬͨϙετҰ ఆִؒͰআ͍ͯ͠Δ post_id user: A post_id post_id post_id user: B post_id post_id post_id user: C post_id post_id Redis Sorted Set -
Timeline Tables (DB) • TiDBʹશͯͷϙετͱϢʔβʔλΠϜ ϥΠϯͷϙετΛه • ϗʔϜλΠϜϥΠϯDBʹ࡞ΒΕ ͳ͍ •
ϗʔϜλΠϜϥΠϯΛ࡞ΔͨΊͷ ใʢϑΥϩϫʔɺίϛϡχςΟϢʔ βʔʣ͕ଘࡏ͢Δ posts post_id user_id community_id text user_posts post_id user_id community_ users community_id user_id followings following_id user_id -
ෛՙςετ ϦϦʔεલʹෛՙݕূΛ࣮ࢪɻAPIͷϨΠςϯγʔͷඪΛ100msʹઃఆ • Writeੑೳ • දతͳRPC͕10000RPSΛग़ͤΔͱ͜Ζ·Ͱݕূ • Readੑೳ • ߘ͕ϑΥϩϫʔͷλΠϜϥΠϯʹදࣔ͞ΕΔ·Ͱͷ͕࣌ؒ5ඵ͔Β10ඵʹͳΔ͜ͱ
Λඪʹௐ • αʔϏε໊͔Β͋ΔఔϢʔβʔ͕དྷΔ͜ͱ͕ఆͰ͖ͨ -
Locust - Boomer • ෛՙςετ༻ͷڥΛ༻ҙͯ͠LocustΛ͍ݕূ • ฒྻΛେ͖͍ͨ͘͠ͷͰgoͷworker࣮ ͷboomerΛ༻ͨ͠ • ߈ܸΫϥΠΞϯτͷεέʔϧΞτɺεέʔ
ϧΞοϓΛසൟʹߦ͏ͨΊɺk8s্Ͱಈ͔ͨ͠ • γφϦΦϢʔβʔ࡞͔ΒೝূΛߦ͍ɺϑΥ ϩʔɺϙετɺϥΠΫͳͲͷجຊతͳಈ࡞Λશͯ ࣮ࢪ -
ෛՙςετ • ॻ͖ࠐΈੑೳTiDB, TiKVͷεέʔϧΞτͰఆ ௨Γʹੑೳ্ͨ͠ • Ұ෦ɺॏ͍ॲཧʢݕࡧΠϯσοΫεͷॻ͖ࠐ ΈͳͲʣΛඇಉظॲཧʹมߋ • pprofΛ͍ॏ͍ॲཧΛ୳ͨ͠
• ಡΈࠐΈॲཧআॲཧࠓճͷඪΛୡ͢ Δ্Ͱͳ͠ • ߴෛՙ࣌ʹෳͷϑΥϩϫʔ͕ಉ࣌ʹϑΥϩʔ͢ Δɺ͍͍ͶΛ͢ΔέʔεͰසൟϩοΫͪͷλΠϜ Ξτ͕ൃੜ -
• ϦϦʔεఆΛ্ճΔϦΫΤετ͕དྷͨ ͕ɺ֤ίϯϙʔωϯτΛεέʔϧΞτ͢Δ ͜ͱͰແࣄՔಇ͠ଓ͚Δ͜ͱ͕Ͱ͖ͨ • ͓ਖ਼݄ͳͲҰ࣌తͳෛՙ͕ݟࠐ·ΕΔ߹ εέʔϧΞτ͢Δ͜ͱͰͳ͘ӡ༻Λଓ ͚Δ͜ͱ͕Ͱ͖͍ͯΔ • ࠓ·ͰαʔϏεແఀࢭͰӡ༻
• TiDBΠϯελϯεͷϊʔυো࣌αʔϏε Өڹͳ͔ͬͨ ϦϦʔε -
Slow Query • Secondary IndexΛͬͨΫΤϦͰIndexͷઌ ಄ΧϥϜͰରΛߜΓɺ̎൪ͷΧϥϜͰ ιʔτ͢Δ • In۟ͷཁૉ͕Ұͭͷ߹sort͕ޮ͍͕ͨɺෳ ཁૉࢦఆ͞ΕΔͱsort͕ޮ͔ͳ͍
• ର͕ेສ݅ʹͳΔ͜ͱ͋ΓSlow Queryʹͳͬͯ͠·ͬͨ KEY `idx_1` (`community_id`,`time_series_id`,`user_id`) /* ҎԼͷΑ͏ͳΫΤϦͷ߹ʹkeep order:falseͱͳΔ */ explain analyze select /*+ USE_INDEX(`community_posts`, idx_1) */ * from community_posts where community_id IN (‘aaaa’, ‘bbbb’, ‘cccc’) and time_series_id < 1736935091883343234 order by time_series_id ASC LIMIT 30; +----------------------------------+---------+---------+ | id | estRows | actRows | +----------------------------------+---------+---------+ | TopN_9 | 30.00 | 30 | | └ ─ IndexLookUp_16 | 30.00 | 30 | | ├ ─ TopN_15(Build) | 30.00 | 30 | | │ └ ─ IndexRangeScan_13 | 1317.05 | 1265 | | └ ─ TableRowIDScan_14(Probe) | 30.00 | 30 | +----------------------------------+---------+---------+ -
Slow Query total_keys: εΩϟϯ͞ΕͨKeyͷ߹ܭɻݹ͍όʔδϣϯΛؚΉ total_process_keys: ॲཧ͞ΕͨKeyɻݹ͍όʔδϣϯΛؚ·ͳ͍ delete_skipped_count: আϚʔΧʔ͕͍ͭͨKeyͷ߹ܭ key_skipped_count: εΩοϓͨ͠Keyͷ߹ܭ
• ظؒͰUPDATEΛ܁Γฦ͔͢DELETE͢Δ έʔε • ෳͷόʔδϣϯ͕ҙਤͤͣੵ͞Εͨ • ಉ͡ςʔϒϧͷϨίʔυΛݕࡧ͢Δͱ͖ ʹSlow Query͕ൃੜ scan_detail: { total_process_keys: 1, total_process_keys_size: 102, total_keys: 26150, get_snapshot_time: 10.5µs, rocksdb: { delete_skipped_count: 179504, key_skipped_count: 205653, . . . } -
ϨϓϦΧ εέδϡʔϧ Մ༻ੑͷ্ͷͨΊɺτϙϩδϥϕϧͷػೳΛ ͍TiKVϊʔυΛҟͳΔϥοΫʹஔ͍ͯ͠Δ EC2 PlacementGroupͰ࡞ͨ͠ύʔςΟγϣ ϯΛPD͕ཧղͰ͖ΔΑ͏ʹ location-labelsͷઃ ఆΛߦ͏ $
tiup ctl:v8.3.0 pd config set location-labels rack -u 10.20.3.13:2379 { "max-replicas": 3, "location-labels": "rack", "strictly-match-label": "false", "enable-placement-rules": "true", "enable-placement-rules-cache": "false", "isolation-level": "" } -
Dumpling • tiup dumplingίϚϯυͰDBσʔλΛS3ʹΤ Ϋεϙʔτ(CSV) • ECSͷεέδϡʔϧυλεΫͱͯ͠ఆظత ʹ࣮ߦ͢Δ • Dumpͨ͠σʔλΛղੳ༻DBʢBigQuery)
ʹऔΓࠐΉ $ tiup dumpling -u username -P 4000 -h 10.20.1.244 -- filetype csv -t 8 -o “s3://dump-bucket/20250214" -r 20000 -F 256MiB --csv-output-dialect bigquery --csv-null-value "" --escape-backslash -
Monitoring TiDB Dashboard - Ϋϥελશମͷ࣮ߦεςʔλ εʢQPSɺ࣮ߦ࣌ؒͳͲʣͱSQL࣮ߦ࣌ؒɺε ϩʔΫΤϦΛ֬ೝ͢Δ Grafana - TiDB,
TiKV, PDͷओཁͳϝτϦΫεͷ ࢹ - ϦʔδϣϯɺҟৗͳϦʔδϣϯ - εΩοϓ͞ΕͨΩʔɺআͣΈΩʔͷ -
·ͱΊͱࠓޙͷల • গͳ͍ਓͷ։ൃνʔϜ͕ͩɺTiDB ClusterΛ༻ͨ͠αʔϏε։ൃͱӡ༻Λߦ͍͑ͯΔ • TiDBແఀࢭͰӡ༻ΛܧଓɺٸܹͳෛՙʹεέʔϧΞτͰରԠ͢Δ͜ͱ͕Ͱ͖ͨ • υΩϡϝϯτ͕๛ʹ͋Δ͜ͱࢹπʔϧ͕Ұ௨Γଐ͢Δ͜ͱগਓνʔϜʹ ͋Γ͕͍ͨ •
ࠓޙͷల • TiCDCΛ͍νΣϯδΠϕϯτΛϝοηʔδͱͯ͠ૹ৴͠ඇಉظॲཧʹ͍͍ͨʢSQS ͱඇಉظδϣϒͷ෦Λஔ͖͑ΔͨΊʣ