Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
堅牢なTCPサーバを作るために - katsubushiの知見から/kamakura.go#5
Search
FUJIWARA Shunichiro
June 22, 2019
Technology
12
7k
堅牢なTCPサーバを作るために - katsubushiの知見から/kamakura.go#5
kamakura.go #5
FUJIWARA Shunichiro
June 22, 2019
Tweet
Share
More Decks by FUJIWARA Shunichiro
See All by FUJIWARA Shunichiro
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
fujiwara3
3
510
監視のこれまでとこれから/sakura monitoring seminar 2025
fujiwara3
11
4.6k
k6による負荷試験 入門から日常的な実践まで/Re:TechTalk #01
fujiwara3
2
88
困難を「一般解」で解く
fujiwara3
10
3.7k
「隙間家具OSS」に至る道/Fujiwara Tech Conference 2025
fujiwara3
7
12k
alecthomas/kong はいいぞ / kamakura.go#7
fujiwara3
1
1k
ISUCONに強くなるかもしれない日々の過ごしかた/Findy ISUCON 2024-11-14
fujiwara3
11
1.4k
「最高のチューニング」をしないために / hack@delta 24.10
fujiwara3
21
4.4k
AWS Lambdaで実現するスケーラブルで低コストなWebサービス構築/YAPC::Hakodate2024
fujiwara3
10
6.5k
Other Decks in Technology
See All in Technology
セキュアなAI活用のためのLiteLLMの可能性
tk3fftk
1
330
アクセスピークを制するオートスケール再設計: 障害を乗り越えKEDAで実現したリソース管理の最適化
myamashii
1
660
セキュアな社内Dify運用と外部連携の両立 ~AIによるAPIリスク評価~
zozotech
PRO
0
120
Deep Security Conference 2025:生成AI時代のセキュリティ監視 /dsc2025-genai-secmon
mizutani
4
2.8k
[SRE NEXT] ARR150億円_エンジニア140名_27チーム_17プロダクトから始めるSLO.pdf
satos
5
3k
サービスを止めるな! DDoS攻撃へのスマートな備えと最前線の事例
coconala_engineer
1
180
SREの次のキャリアの道しるべ 〜SREがマネジメントレイヤーに挑戦して、 気づいたこととTips〜
coconala_engineer
1
4.3k
対話型音声AIアプリケーションの信頼性向上の取り組み
ivry_presentationmaterials
3
1k
サイバーエージェントグループのSRE10年の歩みとAI時代の生存戦略
shotatsuge
4
1k
CDKコード品質UP!ナイスな自作コンストラクタを作るための便利インターフェース
harukasakihara
2
230
AIでテストプロセス自動化に挑戦する
sakatakazunori
1
530
SRE with AI:実践から学ぶ、運用課題解決と未来への展望
yoshiiryo1
0
300
Featured
See All Featured
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Writing Fast Ruby
sferik
628
62k
Optimising Largest Contentful Paint
csswizardry
37
3.3k
Six Lessons from altMBA
skipperchong
28
3.9k
Java REST API Framework Comparison - PWX 2021
mraible
31
8.7k
How GitHub (no longer) Works
holman
314
140k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Unsuck your backbone
ammeep
671
58k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.9k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Transcript
ݎ࿚ͳTCPαʔόΛ࡞ΔͨΊʹ ʙkatsubushiͷݟ͔Βʙ 2019-06-22 Kamakura.go #5 @fujiwara
@fujiwara SRE(૯෦) github.com/fujiwara sfujiwara.hatenablog.com WEB+DB Press vol.111 SREνʔϜ࿈ࡌ Perl Hackers
HubͰAWS X-RayͷΛॻ ͖·ͨ͠
katsubushi - ࢄϢχʔΫID࠾൪ػ • ϢχʔΫͳ࣌ܥྻॱͷInt64 IDΛൃߦ͢Δ • ࢄڥ(ෳ)ͰඃΒͣಈ͘ • Memcached
ProtocolΛ࣮͍ͯ͠Δ • Goͷಠ࣮ࣗϛυϧΣΞ github.com/kayac/go-katsubushi
ݎ࿚ͳαʔόʁ ݎ࿚ is ... • ϦιʔεϦʔΫ͠ͳ͍ • མͪͳ͍ • (ϓϩμΫτͷੑ্࣭)
ID͕ܾͯ͠ඃΒͳ͍ • →γϟʔσΟϯά͞ΕͨผDBʹಉҰID͕ൃߦ͞ΕͨΒࢮ
memcached protocol ͷ࠾༻ཧ༝ • ܰྔ • HTTPͷΑ͏ʹ༨ܭͳϔομ͕ͳ͍ • Int64 Λ1ݸฦ͢ͷʹϔομ͕Կඦbyteͱ͔ͪΐͬͱ…
• ΫϥΠΞϯτ͕ߴ • ࣌ओʹ༻͍ͯͨ͠PerlʹCache::Memcached::Fastͱ ͍͏ߴͳ࣮ IDେྔʹൃߦ͢Δ͜ͱ͕͋ΔͷͰ͍ܰͷ͕Α͍
HTTPͳΒͱ͔͘memcached protocolΛॲཧ͢Δαʔό طଘͷϑϨʔϜϫʔΫ͕ͳ͍ ॻ͚͍͍ΑͶɺ؆୯ͩ͠
Go Ͱ TCP αʔό net package Ͱ؆୯ʹͰ͖Δ Listen, Acceptͯ͠ConnΛgoroutineͰॲཧ͢Δ͚ͩ ln,
err := net.Listen("tcp", ":8080") if err != nil { // handle error } for { conn, err := ln.Accept() if err != nil { // handle error } go handleConnection(conn) }
Go Ͱ TCP Echo αʔόɺ؆୯ func handleConnection(conn net.Conn) { defer
conn.Close() // ൈ͚Δͱ͖ʹஅ b := bufio.NewReader(conn) for { line, err := b.ReadString('\n') // 1ߦಡΉ if err != nil { return } _, err = io.WriteString(conn, line) // ΦϜฦ͠ if err != nil { return } } }
͔͠͠ݱ࣮໘
ϦιʔεϦʔΫ͠ͳ͍
ແ௨৴λΠϜΞτ • ܨ͗ͬͺͳ͠Ͱ͍ͳ͘ͳΔΫϥΠΞϯτରࡦ • ΫϥΠΞϯτ͕TCPΛ͖Ε͍ʹஅ͢ΔͱݶΒͳ͍ • αʔόଆͰΒͳ͍ͱίωΫγϣϯ(goroutine)͕ϦʔΫ͢Δ
net/Conn.SetDeadline ίωΫγϣϯʹࢦఆͨ࣌͠ࠁ·Ͱ௨৴͕ͳ͔ͬͨΒ ΤϥʔʹͳΔΑ͏ઃఆ͢Δ ௨৴ཱ֬ޙɺಡΈॻ͖Ͱ͖ͨλΠϛϯάͰ࠶ઃఆ → ແ௨৴λΠϜΞτ͕࣮Ͱ͖Δ type Conn interface {
SetDeadline(t time.Time) error }
for { conn, err := ln.Accept() if err != nil
{ // handle error } conn.SetDeadline(time.Now().Add(idleTimeout)) go handleConnection(conn) } func handleConnection(conn net.Conn) { defer conn.Close() buf := bufio.NewReader(conn) for { line, _ := buf.ReadString('\n') // 1ߦಡΉ conn.SetDeadline(time.Now().Add(idleTimeout)) // ^λΠϜΞτԆ
ϓϩηεऴྃ࣌ʹஅ αʔόͪΌΜͱஅ͠ͳ͍ͱߦّ͕ѱ͍ΑͶ? func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ϓϩηε͕γάφϧΛड͚Δͱ͜ͷctx͕Done conn.Close() // ͪΌΜͱΔʂ }() // ...
͜͜ͰͰ͢ ͜ͷίʔυͰͳʹ͔͕ϦʔΫ͠·͢ func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ϓϩηε͕γάφϧΛड͚Δͱ͜ͷctx͕Done conn.Close() // ͪΌΜͱΔʂ }() // ...
goroutineϦʔΫ handleConn͕returnͯ͠goroutine͕ࢮͳͳ͍ func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ͜Ε͕ݺΕΔ(=ϓϩηεऴྃ)·Ͱ conn.Close() // goroutine ͕ੜ͖Δ }() // ...
मਖ਼ํ๏1 ࢠ context Λ࡞ͬͯ defer cancel() ࣗൃతऴྃ࣌ɺऴྃ࣌ͲͪΒͰίωΫγϣϯஅͱ goroutineऴ͕ྃͰ͖ΔͷͰϦʔΫ͠ͳ͍ func (app
*App) handleConn(ctx context.Context, conn net.Conn) { ctx2, cancel := context.WithCancel(ctx) defer cancel() go func() { <-ctx2.Done() conn.Close() }() // ... 1 Fix goroutine leak. #28 https://github.com/kayac/go-katsubushi/pull/28/files
མͪͳ͍
ʮ͜ͷ katsubushi ಈ͍ͯΔ͔ͳ?ʯ $ curl localhost:11212 VALUE / 0 18
591644124189298688 VALUE HTTP/1.1 0 18 591644124189298689 END ERROR ERROR ERROR ʮIDฦ͖͚ͬͯͨͲͳʹ͜Ε??ʯ ʮ͋ɺcurlͰୟ͍ͯͨɻɻɻʯ
Կ͕ى͖͍͔ͯͨ curl katsubushi ʹ HTTP ͰϦΫΤετͨ͠(વ) $ curl -v
127.0.0.1:11212 > GET / HTTP/1.1 > Host: 127.0.0.1:11212 > User-Agent: curl/7.65.1 > Accept: */* > (ۭߦ) HTTPͷϦΫΤετ memcached protocol ͰղऍͰ͖Δʂ memcached protocol Ͱͷऔಘ: GET key1 key2 HTTP Ͱ GET ϦΫΤετͷ1ߦ: GET / HTTP/1.1
ͭ·Γ͜͏͍͏Ϩεϙϯε͕ฦΔ VALUE / 0 18 # / ͱ͍͏keyͷ(flag=0, 18byte) 591644124189298688
VALUE HTTP/1.1 0 18 # HTTP/1.1 ͱ͍͏keyͷ(flag=0, 18byte) 591644124189298689 END # HTTPͷ1ߦʹϨεϙϯε͕ฦͤͨΑʂ ERROR # Host: ͱ͔Βͳ͍! ERROR # User-Agnet: ͱ͔Βͳ͍!! ERROR # Accept: ͱ͔Βͳ͍!!!
Ұํͦͷ͜Ζ HTTP Λ৯ͨ katsubushi panic: runtime error: invalid memory
address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x134807f] goroutine 20 [running]: github.com/kayac/go-katsubushi.(*App).handleConn(0xc00012a000, 0x14a3de0, 0xc0000ac038) /Users/fujiwara/src/github.com/kayac/go-katsubushi/app.go:160 +0x2cf created by github.com/kayac/go-katsubushi.(*App).Listen /Users/fujiwara/src/github.com/kayac/go-katsubushi/app.go:131 +0x2ed ʮ͠ɺࢮΜͰΔ…ʯ
Կ͕ى͖͍͔ͯͨ ಡΈऔͬͨϦΫΤετߦ([]byte)ΛίϚϯυʹ࣮ͯ͠ߦ HTTPϔομ͓ΘΓͷۭߦΛಡΜͩͱ͜ΖͰ… cmd, err := app.BytesToCmd(scanner.Bytes()) if err !=
nil { return } err = cmd.Execute(app, conn) // ͜͜Ͱࢮ
൜ਓ func (app *App) BytesToCmd(data []byte) (cmd MemdCmd, err error)
{ if len(data) == 0 { // ۭߦͳͷͰ 0 byte return nil, nil // ← ͍ͭ͜ } foo, err := Foo() ͱ͍͏ΠϯλʔϑΣʔεΛΈͨ ී௨ͷGopher͕ߟ͑Δ͜ͱ ʮerr != nil ͳΒ foo == nil ͔͔ͩΒؾΛ͚ͭΑ͏ʯ ʮerr == nil ͳΒ foo != nil ͩΑͶʂʯ
nil, nil Λฦ͢ͷ͕ѱ͍ ॻ͍ͨͷͲ͚ͬͪࣗͩͲ݀ʹམͪͨ ௨ৗmemcached protocolͰϦΫΤετʹۭߦདྷͳ͍ ຊ൪ӡ༻தͷϓϩηε͕ଈࢮͯ͠ͼͬ͘Γͨ͠ katsubushiΫϥΠΞϯτϥΠϒϥϦfailoverΛ࣮͍ͯͨ͠ daemontools Ͱ͙͢ʹ࠶ىಈ͞Εͨ
ͷͰக໋ইʹͳΒͳ͔ͬͨ γεςϜશମͰݎ࿚ʹ͍͖ͯ͠·͠ΐ͏…
ରԠ2 func (app *App) BytesToCmd(data []byte) (cmd MemdCmd, err error)
{ if len(data) == 0 { - return nil, nil + return nil, errors.New("No command") Βͳ͍ίϚϯυ͕དྷͨΒଈ࠲ʹଓΛͬͨ΄͏͕͍͍͔ 2 Never die #8 https://github.com/kayac/go-katsubushi/pull/8
༨ஊ1 ࣮ curl telnet Ͱ͖·͢ $ curl telnet://127.0.0.1:11212 GET
id VALUE id 0 18 591655476635111424 END quit
༨ஊ2 ʮ1ߦΛղऍͯ͠key͕ / HTTP/1.[01] ͳΒHTTPɻɻɻʯ ʮͭ·Γಉ͡ϙʔτͰ memcached ͱ HTTP ྆ରԠ
αʔό͕ॻ͚ΔͷͰ!??ʯ ్த·Ͱॻ͍ͯਖ਼ؾʹͬͯʹ
(ϓϩμΫτͷੑ্࣭) ID͕ܾͯ͠ඃΒͳ͍
katsubushi ID ࣌ࠁϕʔε +-+-----------------------------------------+----------+------------+ | | Timestamp | WorkerID |
Sequence | +-+-----------------------------------------+----------+------------+ |0|00001001001001010100110010111011011100100|0000000001|000000000000| +-+-----------------------------------------+----------+------------+ | | 78,560,982,756ms since epoch | 1| 0| +-+-----------------------------------------+----------+------------+ = 2017-06-28T06:29:42.756Z ࣌ࠁ͕Δͱಉ͡ID͕ൃߦ͞ΕΔՄೳੑ͕͋Δ
monotonic time https://golang.org/doc/go1.9#monotonic-time Go 1.9ͰOS࣌ࠁ͕ͬͯ time.Now() Βͳ͍ monotonic time͕࣮͞Εͨ 2017−01−01ͷӞඵൃੜ࣌
Cloudflare͕࣌ࠁͷר͖ΓͰো3 ࣌ࠁ͕ͬͨ݁Ռ math/rand.Int63n ʹෛͷ͕Γpanic 3 How and why the leap second affected Cloudflare DNS https://blog.cloudflare.com/how-and-why-the-leap-second-affected- cloudflare-dns/
monotonic time Ҏલͷ katsubushi ts := g.timestamp() // for rewind
of server clock if ts < g.lastTimestamp { return 0, errors.New("system clock was rollbacked") } // ུ g.lastTimestamp = ts ࣌ࠁ͕ר͖ͬͨΒΤϥʔʹͯ͠ޚ ID͕ඃΔ͙Β͍ͳΒൃߦͰ͖ͳ͍ํ͕Ϛγ
monotonic time ରԠ4 katsubushi ID ͷ࣌ࠁ෦ 2015-01-01T00:00:00 UTC Λىݯͱͨ͠ܦաඵ var
Epoch = time.Date(2015, 1, 1, 0, 0, 0, 0, time.UTC) d := time.Now().Sub(Epoch) ͜ͷΑ͏ʹɺ࣌ࢦఆͰ࡞͔ͬͨΒܭࢉ͢Δͱmonotonicʹ ͳΒͳ͍ 4 use Monotonic time #21 https://github.com/kayac/go-katsubushi/pull/21
monotonic time ରԠ now := time.Now() startedAt := now offset
:= now.Sub(Epoch) // ىಈ࣌ͷ࣌ࠁͱepochͷࠩΛܭࢉ͓ͯ͘͠ // ... d := time.Now().Sub(startedAt) + offset time.Now() Ͱ࡞ͬͨಉ࢜Λൺֱͯ͠offsetΛௐ͢Ε ר͖Βͳ͍
࣌ࠁͷଞʹඃΔՄೳੑ͕͋Δ෦͕… +-+-----------------------------------------+----------+------------+ | | Timestamp | WorkerID | Sequence |
+-+-----------------------------------------+----------+------------+ |0|00001001001001010100110010111011011100100|0000000001|000000000000| +-+-----------------------------------------+----------+------------+ | | 78,560,982,756ms since epoch | 1| 0| +-+-----------------------------------------+----------+------------+ = 2017-06-28T06:29:42.756Z WorkerID = ಉ࣌ʹಈ࡞͍ͯ͠ΔϓϩηεؒͰҰҙͷID
RedisΛͬͯҰҙੑΛ୲อ͢Δ Ҏલͷൃදʹ͋ΔͷͰͲ͏ͧ https://speakerdeck.com/fujiwara3/katsubushi?slide=79
ࢹେࣄ • github.com/fukata/golang-stats-api-handler • GoͷϝτϦΫεΛऔΔ • mackerel-plugin-gostats ͋ΔΑ • net/http/pprof
ͰϓϩϑΝΠϧऔಘͷޱΛ։͚Δ
·ͱΊ • GoͰTCPαʔόΛ࡞Δͷ؆୯ • Ͱ࣮ӡ༻Ͱݎ࿚ͳͷΛ࡞Δʹ͍Ζ͍Ζ͋Δ • Ұݟ؆୯ʹݟ͑ΔͷͰɺࣗલ࣮পͷୈҰา