Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
堅牢なTCPサーバを作るために - katsubushiの知見から/kamakura.go#5
Search
FUJIWARA Shunichiro
June 22, 2019
Technology
12
7.1k
堅牢なTCPサーバを作るために - katsubushiの知見から/kamakura.go#5
kamakura.go #5
FUJIWARA Shunichiro
June 22, 2019
Tweet
Share
More Decks by FUJIWARA Shunichiro
See All by FUJIWARA Shunichiro
alecthomas/kong はいいぞ
fujiwara3
6
1.8k
ecspressoの設計思想に至る道 / sekkeinight2025
fujiwara3
12
2.9k
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
fujiwara3
3
1.2k
監視のこれまでとこれから/sakura monitoring seminar 2025
fujiwara3
11
5.3k
k6による負荷試験 入門から日常的な実践まで/Re:TechTalk #01
fujiwara3
2
160
困難を「一般解」で解く
fujiwara3
10
3.8k
「隙間家具OSS」に至る道/Fujiwara Tech Conference 2025
fujiwara3
7
13k
alecthomas/kong はいいぞ / kamakura.go#7
fujiwara3
1
1.2k
ISUCONに強くなるかもしれない日々の過ごしかた/Findy ISUCON 2024-11-14
fujiwara3
11
1.4k
Other Decks in Technology
See All in Technology
OpenAI gpt-oss ファインチューニング入門
kmotohas
2
900
ユニットテストに対する考え方の変遷 / Everyone should watch his live coding
mdstoy
0
120
「AI駆動PO」を考えてみる - 作る速さから価値のスループットへ:検査・適応で未来を開発 / AI-driven product owner. scrummat2025
yosuke_nagai
4
540
PythonとLLMで挑む、 4コマ漫画の構造化データ化
esuji5
1
130
PLaMoの事後学習を支える技術 / PFN LLMセミナー
pfn
PRO
9
3.8k
SwiftUIのGeometryReaderとScrollViewを基礎から応用まで学び直す:設計と活用事例
fumiyasac0921
0
130
全てGoで作るP2P対戦ゲーム入門
ponyo877
3
1.3k
Exadata Database Service on Dedicated Infrastructure(ExaDB-D) UI スクリーン・キャプチャ集
oracle4engineer
PRO
2
5.4k
extension 現場で使えるXcodeショートカット一覧
ktombow
0
200
VCC 2025 Write-up
bata_24
0
180
DataOpsNight#8_Terragruntを用いたスケーラブルなSnowflakeインフラ管理
roki18d
1
320
生成AIで「お客様の声」を ストーリーに変える 新潮流「Generative ETL」
ishikawa_satoru
1
290
Featured
See All Featured
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
657
61k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.5k
Unsuck your backbone
ammeep
671
58k
Writing Fast Ruby
sferik
629
62k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.6k
How to train your dragon (web standard)
notwaldorf
96
6.3k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Imperfection Machines: The Place of Print at Facebook
scottboms
269
13k
For a Future-Friendly Web
brad_frost
180
9.9k
Code Review Best Practice
trishagee
72
19k
4 Signs Your Business is Dying
shpigford
185
22k
Scaling GitHub
holman
463
140k
Transcript
ݎ࿚ͳTCPαʔόΛ࡞ΔͨΊʹ ʙkatsubushiͷݟ͔Βʙ 2019-06-22 Kamakura.go #5 @fujiwara
@fujiwara SRE(૯෦) github.com/fujiwara sfujiwara.hatenablog.com WEB+DB Press vol.111 SREνʔϜ࿈ࡌ Perl Hackers
HubͰAWS X-RayͷΛॻ ͖·ͨ͠
katsubushi - ࢄϢχʔΫID࠾൪ػ • ϢχʔΫͳ࣌ܥྻॱͷInt64 IDΛൃߦ͢Δ • ࢄڥ(ෳ)ͰඃΒͣಈ͘ • Memcached
ProtocolΛ࣮͍ͯ͠Δ • Goͷಠ࣮ࣗϛυϧΣΞ github.com/kayac/go-katsubushi
ݎ࿚ͳαʔόʁ ݎ࿚ is ... • ϦιʔεϦʔΫ͠ͳ͍ • མͪͳ͍ • (ϓϩμΫτͷੑ্࣭)
ID͕ܾͯ͠ඃΒͳ͍ • →γϟʔσΟϯά͞ΕͨผDBʹಉҰID͕ൃߦ͞ΕͨΒࢮ
memcached protocol ͷ࠾༻ཧ༝ • ܰྔ • HTTPͷΑ͏ʹ༨ܭͳϔομ͕ͳ͍ • Int64 Λ1ݸฦ͢ͷʹϔομ͕Կඦbyteͱ͔ͪΐͬͱ…
• ΫϥΠΞϯτ͕ߴ • ࣌ओʹ༻͍ͯͨ͠PerlʹCache::Memcached::Fastͱ ͍͏ߴͳ࣮ IDେྔʹൃߦ͢Δ͜ͱ͕͋ΔͷͰ͍ܰͷ͕Α͍
HTTPͳΒͱ͔͘memcached protocolΛॲཧ͢Δαʔό طଘͷϑϨʔϜϫʔΫ͕ͳ͍ ॻ͚͍͍ΑͶɺ؆୯ͩ͠
Go Ͱ TCP αʔό net package Ͱ؆୯ʹͰ͖Δ Listen, Acceptͯ͠ConnΛgoroutineͰॲཧ͢Δ͚ͩ ln,
err := net.Listen("tcp", ":8080") if err != nil { // handle error } for { conn, err := ln.Accept() if err != nil { // handle error } go handleConnection(conn) }
Go Ͱ TCP Echo αʔόɺ؆୯ func handleConnection(conn net.Conn) { defer
conn.Close() // ൈ͚Δͱ͖ʹஅ b := bufio.NewReader(conn) for { line, err := b.ReadString('\n') // 1ߦಡΉ if err != nil { return } _, err = io.WriteString(conn, line) // ΦϜฦ͠ if err != nil { return } } }
͔͠͠ݱ࣮໘
ϦιʔεϦʔΫ͠ͳ͍
ແ௨৴λΠϜΞτ • ܨ͗ͬͺͳ͠Ͱ͍ͳ͘ͳΔΫϥΠΞϯτରࡦ • ΫϥΠΞϯτ͕TCPΛ͖Ε͍ʹஅ͢ΔͱݶΒͳ͍ • αʔόଆͰΒͳ͍ͱίωΫγϣϯ(goroutine)͕ϦʔΫ͢Δ
net/Conn.SetDeadline ίωΫγϣϯʹࢦఆͨ࣌͠ࠁ·Ͱ௨৴͕ͳ͔ͬͨΒ ΤϥʔʹͳΔΑ͏ઃఆ͢Δ ௨৴ཱ֬ޙɺಡΈॻ͖Ͱ͖ͨλΠϛϯάͰ࠶ઃఆ → ແ௨৴λΠϜΞτ͕࣮Ͱ͖Δ type Conn interface {
SetDeadline(t time.Time) error }
for { conn, err := ln.Accept() if err != nil
{ // handle error } conn.SetDeadline(time.Now().Add(idleTimeout)) go handleConnection(conn) } func handleConnection(conn net.Conn) { defer conn.Close() buf := bufio.NewReader(conn) for { line, _ := buf.ReadString('\n') // 1ߦಡΉ conn.SetDeadline(time.Now().Add(idleTimeout)) // ^λΠϜΞτԆ
ϓϩηεऴྃ࣌ʹஅ αʔόͪΌΜͱஅ͠ͳ͍ͱߦّ͕ѱ͍ΑͶ? func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ϓϩηε͕γάφϧΛड͚Δͱ͜ͷctx͕Done conn.Close() // ͪΌΜͱΔʂ }() // ...
͜͜ͰͰ͢ ͜ͷίʔυͰͳʹ͔͕ϦʔΫ͠·͢ func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ϓϩηε͕γάφϧΛड͚Δͱ͜ͷctx͕Done conn.Close() // ͪΌΜͱΔʂ }() // ...
goroutineϦʔΫ handleConn͕returnͯ͠goroutine͕ࢮͳͳ͍ func (app *App) handleConn(ctx context.Context, conn net.Conn) {
defer conn.Close() // ͜ͷίωΫγϣϯͷॲཧΛࣗൃతʹΔͱ͖ go func() { <-ctx.Done() // ͜Ε͕ݺΕΔ(=ϓϩηεऴྃ)·Ͱ conn.Close() // goroutine ͕ੜ͖Δ }() // ...
मਖ਼ํ๏1 ࢠ context Λ࡞ͬͯ defer cancel() ࣗൃతऴྃ࣌ɺऴྃ࣌ͲͪΒͰίωΫγϣϯஅͱ goroutineऴ͕ྃͰ͖ΔͷͰϦʔΫ͠ͳ͍ func (app
*App) handleConn(ctx context.Context, conn net.Conn) { ctx2, cancel := context.WithCancel(ctx) defer cancel() go func() { <-ctx2.Done() conn.Close() }() // ... 1 Fix goroutine leak. #28 https://github.com/kayac/go-katsubushi/pull/28/files
མͪͳ͍
ʮ͜ͷ katsubushi ಈ͍ͯΔ͔ͳ?ʯ $ curl localhost:11212 VALUE / 0 18
591644124189298688 VALUE HTTP/1.1 0 18 591644124189298689 END ERROR ERROR ERROR ʮIDฦ͖͚ͬͯͨͲͳʹ͜Ε??ʯ ʮ͋ɺcurlͰୟ͍ͯͨɻɻɻʯ
Կ͕ى͖͍͔ͯͨ curl katsubushi ʹ HTTP ͰϦΫΤετͨ͠(વ) $ curl -v
127.0.0.1:11212 > GET / HTTP/1.1 > Host: 127.0.0.1:11212 > User-Agent: curl/7.65.1 > Accept: */* > (ۭߦ) HTTPͷϦΫΤετ memcached protocol ͰղऍͰ͖Δʂ memcached protocol Ͱͷऔಘ: GET key1 key2 HTTP Ͱ GET ϦΫΤετͷ1ߦ: GET / HTTP/1.1
ͭ·Γ͜͏͍͏Ϩεϙϯε͕ฦΔ VALUE / 0 18 # / ͱ͍͏keyͷ(flag=0, 18byte) 591644124189298688
VALUE HTTP/1.1 0 18 # HTTP/1.1 ͱ͍͏keyͷ(flag=0, 18byte) 591644124189298689 END # HTTPͷ1ߦʹϨεϙϯε͕ฦͤͨΑʂ ERROR # Host: ͱ͔Βͳ͍! ERROR # User-Agnet: ͱ͔Βͳ͍!! ERROR # Accept: ͱ͔Βͳ͍!!!
Ұํͦͷ͜Ζ HTTP Λ৯ͨ katsubushi panic: runtime error: invalid memory
address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x134807f] goroutine 20 [running]: github.com/kayac/go-katsubushi.(*App).handleConn(0xc00012a000, 0x14a3de0, 0xc0000ac038) /Users/fujiwara/src/github.com/kayac/go-katsubushi/app.go:160 +0x2cf created by github.com/kayac/go-katsubushi.(*App).Listen /Users/fujiwara/src/github.com/kayac/go-katsubushi/app.go:131 +0x2ed ʮ͠ɺࢮΜͰΔ…ʯ
Կ͕ى͖͍͔ͯͨ ಡΈऔͬͨϦΫΤετߦ([]byte)ΛίϚϯυʹ࣮ͯ͠ߦ HTTPϔομ͓ΘΓͷۭߦΛಡΜͩͱ͜ΖͰ… cmd, err := app.BytesToCmd(scanner.Bytes()) if err !=
nil { return } err = cmd.Execute(app, conn) // ͜͜Ͱࢮ
൜ਓ func (app *App) BytesToCmd(data []byte) (cmd MemdCmd, err error)
{ if len(data) == 0 { // ۭߦͳͷͰ 0 byte return nil, nil // ← ͍ͭ͜ } foo, err := Foo() ͱ͍͏ΠϯλʔϑΣʔεΛΈͨ ී௨ͷGopher͕ߟ͑Δ͜ͱ ʮerr != nil ͳΒ foo == nil ͔͔ͩΒؾΛ͚ͭΑ͏ʯ ʮerr == nil ͳΒ foo != nil ͩΑͶʂʯ
nil, nil Λฦ͢ͷ͕ѱ͍ ॻ͍ͨͷͲ͚ͬͪࣗͩͲ݀ʹམͪͨ ௨ৗmemcached protocolͰϦΫΤετʹۭߦདྷͳ͍ ຊ൪ӡ༻தͷϓϩηε͕ଈࢮͯ͠ͼͬ͘Γͨ͠ katsubushiΫϥΠΞϯτϥΠϒϥϦfailoverΛ࣮͍ͯͨ͠ daemontools Ͱ͙͢ʹ࠶ىಈ͞Εͨ
ͷͰக໋ইʹͳΒͳ͔ͬͨ γεςϜશମͰݎ࿚ʹ͍͖ͯ͠·͠ΐ͏…
ରԠ2 func (app *App) BytesToCmd(data []byte) (cmd MemdCmd, err error)
{ if len(data) == 0 { - return nil, nil + return nil, errors.New("No command") Βͳ͍ίϚϯυ͕དྷͨΒଈ࠲ʹଓΛͬͨ΄͏͕͍͍͔ 2 Never die #8 https://github.com/kayac/go-katsubushi/pull/8
༨ஊ1 ࣮ curl telnet Ͱ͖·͢ $ curl telnet://127.0.0.1:11212 GET
id VALUE id 0 18 591655476635111424 END quit
༨ஊ2 ʮ1ߦΛղऍͯ͠key͕ / HTTP/1.[01] ͳΒHTTPɻɻɻʯ ʮͭ·Γಉ͡ϙʔτͰ memcached ͱ HTTP ྆ରԠ
αʔό͕ॻ͚ΔͷͰ!??ʯ ్த·Ͱॻ͍ͯਖ਼ؾʹͬͯʹ
(ϓϩμΫτͷੑ্࣭) ID͕ܾͯ͠ඃΒͳ͍
katsubushi ID ࣌ࠁϕʔε +-+-----------------------------------------+----------+------------+ | | Timestamp | WorkerID |
Sequence | +-+-----------------------------------------+----------+------------+ |0|00001001001001010100110010111011011100100|0000000001|000000000000| +-+-----------------------------------------+----------+------------+ | | 78,560,982,756ms since epoch | 1| 0| +-+-----------------------------------------+----------+------------+ = 2017-06-28T06:29:42.756Z ࣌ࠁ͕Δͱಉ͡ID͕ൃߦ͞ΕΔՄೳੑ͕͋Δ
monotonic time https://golang.org/doc/go1.9#monotonic-time Go 1.9ͰOS࣌ࠁ͕ͬͯ time.Now() Βͳ͍ monotonic time͕࣮͞Εͨ 2017−01−01ͷӞඵൃੜ࣌
Cloudflare͕࣌ࠁͷר͖ΓͰো3 ࣌ࠁ͕ͬͨ݁Ռ math/rand.Int63n ʹෛͷ͕Γpanic 3 How and why the leap second affected Cloudflare DNS https://blog.cloudflare.com/how-and-why-the-leap-second-affected- cloudflare-dns/
monotonic time Ҏલͷ katsubushi ts := g.timestamp() // for rewind
of server clock if ts < g.lastTimestamp { return 0, errors.New("system clock was rollbacked") } // ུ g.lastTimestamp = ts ࣌ࠁ͕ר͖ͬͨΒΤϥʔʹͯ͠ޚ ID͕ඃΔ͙Β͍ͳΒൃߦͰ͖ͳ͍ํ͕Ϛγ
monotonic time ରԠ4 katsubushi ID ͷ࣌ࠁ෦ 2015-01-01T00:00:00 UTC Λىݯͱͨ͠ܦաඵ var
Epoch = time.Date(2015, 1, 1, 0, 0, 0, 0, time.UTC) d := time.Now().Sub(Epoch) ͜ͷΑ͏ʹɺ࣌ࢦఆͰ࡞͔ͬͨΒܭࢉ͢Δͱmonotonicʹ ͳΒͳ͍ 4 use Monotonic time #21 https://github.com/kayac/go-katsubushi/pull/21
monotonic time ରԠ now := time.Now() startedAt := now offset
:= now.Sub(Epoch) // ىಈ࣌ͷ࣌ࠁͱepochͷࠩΛܭࢉ͓ͯ͘͠ // ... d := time.Now().Sub(startedAt) + offset time.Now() Ͱ࡞ͬͨಉ࢜Λൺֱͯ͠offsetΛௐ͢Ε ר͖Βͳ͍
࣌ࠁͷଞʹඃΔՄೳੑ͕͋Δ෦͕… +-+-----------------------------------------+----------+------------+ | | Timestamp | WorkerID | Sequence |
+-+-----------------------------------------+----------+------------+ |0|00001001001001010100110010111011011100100|0000000001|000000000000| +-+-----------------------------------------+----------+------------+ | | 78,560,982,756ms since epoch | 1| 0| +-+-----------------------------------------+----------+------------+ = 2017-06-28T06:29:42.756Z WorkerID = ಉ࣌ʹಈ࡞͍ͯ͠ΔϓϩηεؒͰҰҙͷID
RedisΛͬͯҰҙੑΛ୲อ͢Δ Ҏલͷൃදʹ͋ΔͷͰͲ͏ͧ https://speakerdeck.com/fujiwara3/katsubushi?slide=79
ࢹେࣄ • github.com/fukata/golang-stats-api-handler • GoͷϝτϦΫεΛऔΔ • mackerel-plugin-gostats ͋ΔΑ • net/http/pprof
ͰϓϩϑΝΠϧऔಘͷޱΛ։͚Δ
·ͱΊ • GoͰTCPαʔόΛ࡞Δͷ؆୯ • Ͱ࣮ӡ༻Ͱݎ࿚ͳͷΛ࡞Δʹ͍Ζ͍Ζ͋Δ • Ұݟ؆୯ʹݟ͑ΔͷͰɺࣗલ࣮পͷୈҰา