Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
pt&Goroutines
Search
monochromegane
May 31, 2014
Technology
20
15k
pt&Goroutines
pt(the_platinum_searcher) を高速化するために Goroutines まわりで試したことを発表しました。
http://connpass.com/event/6370/
monochromegane
May 31, 2014
Tweet
Share
More Decks by monochromegane
See All by monochromegane
不確実性下における目的と手段の統合的探索に向けた連続腕バンディットの応用 / iot70_gp_rff_mab
monochromegane
1
51
なめらかなシステムと運用維持の終わらぬ未来 / dicomo2025_coherently_fittable_system
monochromegane
0
1.4k
ベクトル検索システムの気持ち
monochromegane
35
11k
Go言語での実装を通して学ぶ、高速なベクトル検索を支えるクラスタリング技術/fukuokago-kmeans
monochromegane
1
190
Go言語でターミナルフレンドリーなAIコマンド、afaを作った/fukuokago20_afa
monochromegane
2
270
多様かつ継続的に変化する環境に適応する情報システム/thesis-defense-presentation
monochromegane
1
950
Online Nonstationary and Nonlinear Bandits with Recursive Weighted Gaussian Process
monochromegane
0
590
AIを前提とした体験の実現に向けて/toward_ai_based_experiences
monochromegane
2
980
Go言語でMac GPUプログラミング
monochromegane
1
640
Other Decks in Technology
See All in Technology
【CEDEC2025】大規模言語モデルを活用したゲーム内会話パートのスクリプト作成支援への取り組み
cygames
PRO
1
560
経験がないことを言い訳にしない、 AI時代の他領域への染み出し方
parayama0625
0
280
人に寄り添うAIエージェントとアーキテクチャ #BetAIDay
layerx
PRO
2
560
2025新卒研修・HTML/CSS #弁護士ドットコム
bengo4com
2
4.2k
会社もクラウドも違うけど 通じたコスト削減テクニック/Cost optimization strategies effective regardless of company or cloud provider
aeonpeople
2
410
20250728 MCP, A2A and Multi-Agents in the future
yoshidashingo
1
170
生成AIによる情報システムへのインパクト
taka_aki
1
220
反脆弱性(アンチフラジャイル)とデータ基盤構築
cuebic9bic
2
130
ビジネス文書に特化した基盤モデル開発 / SaaSxML_Session_2
sansan_randd
0
190
【CEDEC2025】『ウマ娘 プリティーダービー』における映像制作のさらなる高品質化へ!~ 豊富な素材出力と制作フローの改善を実現するツールについて~
cygames
PRO
0
120
Claude Codeが働くAI中心の業務システム構築の挑戦―AIエージェント中心の働き方を目指して
os1ma
9
1.2k
製造業の課題解決に向けた機械学習の活用と、製造業特化LLM開発への挑戦
knt44kw
0
110
Featured
See All Featured
Reflections from 52 weeks, 52 projects
jeffersonlam
351
21k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Statistics for Hackers
jakevdp
799
220k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Become a Pro
speakerdeck
PRO
29
5.5k
Music & Morning Musume
bryan
46
6.7k
Optimising Largest Contentful Paint
csswizardry
37
3.4k
Fireside Chat
paigeccino
37
3.6k
How to Think Like a Performance Engineer
csswizardry
25
1.8k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.5k
Transcript
pt &Goroutine - GoCon 2014 spring -
MIYAKE Yusuke (@monochromegane)
GMO Pepabo, Inc.
grep ͯ͠·͔͢ʁ
grep?
ack?
ag?
pt The Platinum Searcher
Written in Golang
Mac OSX Linux Windows
UTF-8 EUC-JP Shift-JIS
AND
fast ! ack go 6.24s user 1.06s system 99% cpu
7.304 total # ack ag go 0.88s user 1.39s system 221% cpu 1.027 total # ag pt go 1.09s user 1.01s system 235% cpu 0.892 total # pt
How?
Goroutine & Channel
͍ͬ͠ΐʹߴԽͯ͠Έ·͠ΐ͏
1. ϑΝΠϧΛݕࡧͯ͠(find) 2. จࣈྻΛݕࡧͯ͠(grep) 3. ݁ՌΛදࣔ͢Δ(print) ύλʔϯݕࡧͱ
Approach-0 ! ॱ൪ʹ
find
find grep
find grep print
// find find := find.Find{Option: self.Option} find.Do(self.Root) ! // grep
grep := grep.Grep{ Files: find.Files, // result Pattern: self.Pattern, Option: self.Option} grep.Do() ! // print print := print.Print{ Matches: grep.Matches, // result Pattern: self.Pattern, Option: self.Option} print.Do()
> the_simple_searcher go $GOROOT > /dev/null
0.79 seconds
Approach-1 ! ฒߦʹ
Goroutine
• GoݴޠͰฒߦॲཧΛ࣮ݱ͢Δ • εϨουɺίϧʔνϯͱҧ͏ • Concurrency(ฒߦ)ͱParallelism(ฒྻ) • ܰྔ • go
f()
find grep print go go go
Channel
• Goroutineؒͷϝοηʔδϯά • ͷૹड৴ • όοϑΝʹΑΔϒϩοΫ
find grep print go go go
$IBOFM find grep print $IBOFM go go go
$IBOFM find grep print $IBOFM go go go
// channel files := make(chan *string, self.Option.Cap) matches := make(chan
*grep.Match, self.Option.Cap) done := make(chan bool) ! // find find := find.Find{Files: files, Option: self.Option} go find.Do(self.Root) ! // grep grep := grep.Grep{ Files: files, Matches: matches, Pattern: self.Pattern, Option: self.Option} go grep.Do() ! // print print := print.Print{ Done: done, Matches: matches, Pattern: self.Pattern, Option: self.Option} go print.Do() ! <-done // block
walkFunc := func(path string, info os.FileInfo, err error) error {
if info.IsDir() { return nil } self.Files <- &path // send return nil } ! filepath.Walk(root, walkFunc) close(self.Files) // close
for file := range self.Files { // receive ( <-self.Files
) fh, err := os.Open(*file) if err != nil { panic(err) } ! f := bufio.NewReader(fh) ! var buf []byte var lineNum = 1 for { buf, _, err = f.ReadLine() if err != nil { break } line := string(buf) if strings.Contains(line, self.Pattern) { self.Matches <- &Match{*file, lineNum, line} // send } lineNum++ } fh.Close() } close(self.Matches) // close
for match := range self.Matches { // receive fmt.Printf("%s:%d:%s\n", match.Path,
match.Num, match.Match) } self.Done <- true // send
> the_simple_searcher go $GOROOT > /dev/null
0.79 -> 0.87 seconds
?
buffer
• Channelͷड༰ྔ • ch := make(chan ܕ, ༰ྔ) • ༰ྔ·Ͱड
• ༰ྔ͑Δͱૹ৴ଆडͪ • ड৴͢Δͱ༰ྔ͕ͻͱۭͭ͘ • ༰ྔ͕0ͷ߹ɺৗʹͭ
// channel with buffer files := make(chan *string, self.Option.Cap) matches
:= make(chan *grep.Match, self.Option.Cap) done := make(chan bool) // always wait
> the_simple_searcher go $GOROOT > /dev/null
0.79 -> 0.8 seconds
Approach-2 ! ͬͱฒߦʹ
$IBOFM find grep print $IBOFM go go go
$IBOFM find grep print $IBOFM go go go grep grep
grep
var wg sync.WaitGroup for file := range self.Files { wg.Add(1)
// goroutineͷىಈΛΠϯΫϦϝϯτ (தུ) go func(self *Grep, file *string) { defer wg.Done() // goroutine͕ྃͨ͠ΒىಈΛσΫϦϝϯτ for { ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ(தུ) } fh.Close() ! }(self, file) // ΫϩʔδϟΛgoroutineʹ͢Δͱ͖มͷڞ༗ʹҙ ! } wg.Wait() // ෆಛఆͷgoroutine͕શͯऴྃ͢ΔͷΛͭ close(self.Matches)
> the_simple_searcher go $GOROOT > /dev/null
panic ! too many open files
var wg sync.WaitGroup sem := make(chan bool, self.Option.Cap) // ىಈ͢ΔgoroutineͷΛ੍ޚ͢Δchannel
for file := range self.Files { sem <- true // goroutineͷىಈ(channelͷbuffer)͕͍ͬͺ͍ͳΒͭ wg.Add(1) (தུ) go func(self *Grep, file *string) { defer wg.Done() for { ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ(தུ) } fh.Close() <-sem // ಉ࣌ىಈchannelͷbufferʹۭ͖Λͭ͘Δ ! }(self, file) ! } wg.Wait() close(self.Matches)
> the_simple_searcher go $GOROOT > /dev/null
0.79 -> 0.8 seconds
Approach-3 ! ฒྻʹ
GOMAXPROCS
• Goroutineͷฒྻ • σϑΥϧτ1 • runtime.NumCPU()ͰίΞΛऔಘ • runtime.GOMAXPROCS()ͰฒྻΛઃఆ
> the_simple_searcher go $GOROOT > /dev/null
0.79 -> 0.55 ! seconds
benchmark ! • Mac OSX(10.9.3) • CPU: 2.5GHz Core i5(2Core)
• Memory: 8GB • Go: 1.2.2
#V⒎FS (0."9130$4 "QQSPBDI
ฒߦԽͯ͠ͳ͍ͷͰ ฒྻԽͯ͠มΘΒͣ ίΞҎ্ͷࢦఆ ޮՌͳ͠ ଌఆͯ͠ௐ͠ͳ͍ͱ ߹ʹΑͬͯ͘ͳΔ #V⒎FS (0."9130$4 "QQSPBDI
–Rob Pike • Concurrency is powerful. • Concurrency is not
parallelism. • Concurrency enables parallelism. • Concurrency makes parallelism (and scaling and everything else) easy.
એ ϖύϘͰΤϯδχΞΛืू͍ͯ͠·͢ɻ ڞʹαʔϏεΛੜΈग़͠ҭͯͯ͘ΕΔ৽͍ؒ͠ Λ͍ͬͯ·͢ɻ ! http://pepabo.com/recruit/career/engineer/
͓ΘΓ