Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Lazy Enumeration
Search
Eric Hodel
April 27, 2016
Programming
0
110
Lazy Enumeration
An introduction to lazy enumeration in ruby
Eric Hodel
April 27, 2016
Tweet
Share
More Decks by Eric Hodel
See All by Eric Hodel
Building maintainable command-line tools with MRuby
drbrain
0
560
Introduction to Rake
drbrain
0
260
Lessons in Mentorship
drbrain
1
160
Open Source Maintenance — Ruby on Ales 2014
drbrain
1
84
Open Source Maintenance — RailsClub Moscow
drbrain
1
140
drbdump
drbrain
2
430
Other Decks in Programming
See All in Programming
Quarto Clean Theme
nicetak
0
220
個人開発で使ってるやつを紹介する回
yohfee
1
730
のびしろを広げる巻き込まれ力:偶然を活かすキャリアの作り方/oso2024
takahashiikki
1
160
MLOps in Mercari Group’s Trust and Safety ML Team
cjhj
1
130
ActiveRecordの力でDBのメタデータを迅速に解析する
lnit
5
1.9k
CSC509 Lecture 05
javiergs
PRO
0
180
自分だけの世界を創るクリエイティブコーディング / Creative Coding: Creating Your Own World
chobishiba
2
2.1k
CSC509 Lecture 06
javiergs
PRO
0
130
Vertical Architectures for Scalable Angular Applications
manfredsteyer
PRO
0
180
今日で分かる!カスタムコップの作り方
krpk1900
2
170
CSC509 Lecture 07
javiergs
PRO
0
130
NEWTにおけるiOS18対応の進め方
ryu1sazae
0
330
Featured
See All Featured
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.5k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
26
1.4k
Music & Morning Musume
bryan
46
6.1k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
4
250
How STYLIGHT went responsive
nonsquared
95
5.1k
Optimizing for Happiness
mojombo
375
69k
Intergalactic Javascript Robots from Outer Space
tanoku
268
27k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
31
2.6k
Teambox: Starting and Learning
jrom
132
8.7k
YesSQL, Process and Tooling at Scale
rocio
167
14k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
9
630
Side Projects
sachag
452
42k
Transcript
Lazy Enumera,on Eric Hodel —
[email protected]
Loop values = [1, 2, 3, 4] doubles = []
index = 0 while index < values.length do doubles << values[index] * 2 index += 1 end
Enumera,ng Where am I? values[index] Am I done? index <
values.length What’s next? index += 1
Enumerator API Where am I? next #=> nil or exception
if done Am I done? nil, StopIteration What’s next? handled for you
External Enumerator result = db_conn.exec ‘SELECT * FROM orders’ while
order = result.next do # … end
Internal Enumerator result = db_conn.exec ‘SELECT * FROM orders’ result.each_row
do |order| # … end
External vs Internal You write loop Impera,ve C, Ruby Loop
built-in Func,onal Scheme, Ruby
Eager Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value
= orders.map { |order| # 10,000 values order[‘total’] }.reduce { |sum, order_total| sum + order_total }
Eager Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value
= orders.map { |order| # 100,000,000 values order[‘total’] }.reduce { |sum, order_total| sum + order_total }
100,000,000 Objects >> a = Array.new 100_000_000 >> ObjectSpace.memsize_of a
=> 800000040 800MB 400ms
Lazy Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value
= orders.lazy.map { |order| order[‘total’] }.reduce { |sum, order_total| sum + order_total } 10MB similar ,me
Eager Processing 100M 100M .map .map 100M .map 100M
Lazy Processing 1 1 .map .map 1 .map 1 2
2 2 2 3 3 3 3 … … … … 100M 100M 100M 100M
How does lazy work? Fibers!
Fiber? •Story line for a program •One Fiber runs at
a ,me •Scheduled by author •“Corou,ne”
Hierarchy Process ↳Thread ↳Fiber OS scheduled Manually scheduled
Scheduling Fibers resume(input) #=> output Run a specific Fiber Fiber.yield(value)
Return output to #resume
Ac,ve Fiber 1 1 .map .map 1 .map 1 2
2 2 2 3 3 3 3 … … … … 100M 100M 100M 100M Fiber Fiber Fiber Fiber
Example of Fiber ⃠
Lazy Enumera,on •Reduces memory used •Great for huge data sets
•Processes one at a ,me •Uses Fiber (corou,ne)