Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How to Use In-Memory Streams
Search
HayaoSuzuki
August 29, 2020
Technology
1
5.4k
How to Use In-Memory Streams
PyCon JP 2020
HayaoSuzuki
August 29, 2020
Tweet
Share
More Decks by HayaoSuzuki
See All by HayaoSuzuki
Tasting "Python Distilled"
hayaosuzuki
0
300
Let's implement useless Python objects
hayaosuzuki
0
1.9k
How to Write Robust Python Code
hayaosuzuki
5
4.4k
Unknown Evolution of the Built-in Function pow
hayaosuzuki
0
1.4k
Python for Everyday
hayaosuzuki
1
2.2k
Do you know cmath module?
hayaosuzuki
0
3.3k
Elementary Number Theory with Python
hayaosuzuki
1
3.5k
Django QuerySet "ARE" Patterns
hayaosuzuki
0
3.3k
A Modernization of Legacy Django Based Applications
hayaosuzuki
1
7.8k
Other Decks in Technology
See All in Technology
ウォンテッドリーのアラート設計と Datadog 移行での知見
donkomura
0
160
データモデリング通り #2オンライン勉強会 ~方法論の話をしよう~
datayokocho
0
190
20250818_KGX・One Hokkaidoコラボイベント
tohgeyukihiro
0
110
結局QUICで通信は速くなるの?
kota_yata
8
7.5k
Autonomous Database Serverless 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
18
52k
AIドリブンのソフトウェア開発 - うまいやり方とまずいやり方
okdt
PRO
7
100
プロダクトエンジニアリングで開発の楽しさを拡張する話
barometrica
0
210
テストを実行してSorbetのsigを書こう!
sansantech
PRO
1
130
事業特性から逆算したインフラ設計
upsider_tech
0
240
LLM 機能を支える Langfuse / ClickHouse のサーバレス化
yuu26
9
2.7k
プロジェクトマネジメントは不確実性との対話だ
hisashiwatanabe
0
160
EKS Pod Identity における推移的な session tags
z63d
1
160
Featured
See All Featured
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
Making the Leap to Tech Lead
cromwellryan
134
9.5k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.8k
How to train your dragon (web standard)
notwaldorf
96
6.2k
A designer walks into a library…
pauljervisheath
207
24k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
890
Rebuilding a faster, lazier Slack
samanthasiow
83
9.1k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
We Have a Design System, Now What?
morganepeng
53
7.7k
How to Ace a Technical Interview
jacobian
279
23k
Transcript
ΠϯϝϞϦʔετϦʔϜ׆༻ज़ How to Use In-Memory Streams Hayao Suzuki PyCon JP
2020 August 29, 2020
ൃදʹࡍͯ͠ GitHub ʹࢿྉ͕͋Γ·͢ › https://github.com/HayaoSuzuki/pyconjp2020 Twitter ͷϋογϡλά › #pyconjp_1 PyCon
JP Fellow Slack › #jp-2020-track-1 2 / 27
Who am I ? ͓લ୭Α Name Hayao Suzukiʢླɹॣʣ Twitter @CardinalXaro
Work Python Programmer at iRidge, Inc. 3 / 27
Who am I ? Technical Reviewer › Effective Python ୈ
2 ൛ (O’Reilly Japan) › ಈֶ͔ͯ͠Ϳྔࢠίϯϐϡʔλϓϩάϥϛϯά (O’Reilly Japan) https://xaro.hatenablog.jp/ ʹϦετ͕͋Γ·͢ɻ 4 / 27
Who am I ? Selected Talks › ϨΨγʔ Django ΞϓϦέʔγϣϯͷݱԽ
(DjangoCongress JP 2018) › SymPy ʹΑΔࣜॲཧ (PyCon JP 2018) › Python ͱָ͠Ήॳ (PyCon mini Hiroshima 2019) › ܅ cmath Λ͍ͬͯΔ͔ (PyCon mini Shizuoka 2020) https://xaro.hatenablog.jp/ ʹϦετ͕͋Γ·͢ɻ 5 / 27
ࠓͷඪ ͜Μͳ՝Λղܾ͍ͨ͠ʂ › Πϯλʔωοτܦ༝Ͱ GB αΠζͷσʔλΛऔಘ͠ɺCSV ϑΝΠϧʹՃ͢Δ › Ϋϥυ্ʹߏஙͨ͠طଘͷγεςϜʹՃ͢ΔܗͰ࣮͢Δ ›
ຖ࣮ߦ͢Δ ΫϥυαʔϏεैྔ՝ۚ ͳΔ͘ਝʹॲཧ͍ͨ͠ʂ 6 / 27
ࠓͷඪ ॲཧͷྲྀΕ › Πϯλʔωοτܦ༝Ͱ GB αΠζͷσʔλΛऔಘ͢Δ › GB αΠζͷσʔλΛ
CSV ϑΝΠϧʹՃ͢Δ › CSV ϑΝΠϧΛ ZIP ѹॖ͢Δ › ZIP ѹॖσʔλΛΫϥυετϨʔδʹΞοϓϩʔυ͢Δ ੳ › σʔλαΠζ͕େ͖͍ › σʔλͷՃ୯७ͳॲཧ 7 / 27
ࠓͷඪ ϘτϧωοΫͲ͔͜ › ZIP ѹॖͦΕ΄ͲେมͰͳ͍ › σʔλՃ୯७ͳॲཧ › ϘτϧωοΫ I/O
ॲཧʹ͋Γͦ͏ Կͱ͔ͯ͠ I/O ॲཧΛਝʹॲཧ͍ͨ͠ʂʂʂ 8 / 27
Today’s Theme In-Memory Streams 9 / 27
Stream? ͦͦετϦʔϜͬͯԿʁ ετϦʔϜϑΝΠϧΦϒδΣΫτͰ͋Δɻ 10 / 27
File Object? ϑΝΠϧΦϒδΣΫτͬͯԿʁ › read() write() ͳͲͷϝιουΛ࣋ͭΦϒδΣΫτ › σΟεΫ্ͷϑΝΠϧผͷॴʹ͋ΔετϨʔδɺೖग़ྗػثͱ
ΓͱΓ͕Ͱ͖Δ 11 / 27
File Object? ϑΝΠϧΦϒδΣΫτͨͪ › ੜόΠφϦϑΝΠϧ › όοϑΝ͖όΠφϦϑΝΠϧ › ςΩετϑΝΠϧ 12
/ 27
͍ํ ςΩετϑΝΠϧ f = open("myfile.txt", "r") όοϑΝ͖όΠφϦ f = open("myfile.jpg",
"rb") 13 / 27
open ؔͷཪଆ open ԿΛ͍ͯ͠Δͷ͔ʁ OS ͷγεςϜίʔϧ API ΛݺͿ 14 /
27
open ؔͷཪଆ ྫɿCSV ʹՃ͢Δ with open("events.csv", "w") as csv_file: fieldnames
= ["title", "started_at", "ended_at"] writer = csv.DictWriter(csv_file, fieldnames) writer.writeheader() writer.writerows(events) 15 / 27
open ؔͷཪଆ ྫɿWindows › CreateFileʢϑΝΠϧͷΞΫηεݖऔಘʣ › QueryAllInformationFileʢϑΝΠϧใͷऔಘʣ › WriteFileʢϑΝΠϧॻ͖ࠐΉʣ ›
CloseFileʢϑΝΠϧΛด͡Δʣ Process Monitor ܦ༝Ͱ֬ೝͨ͠ɻ 16 / 27
open ؔͷཪଆ ྫɿUbuntu on WSL › openat ʢϑΝΠϧͷΦʔϓϯʣ › fstatʢϑΝΠϧใͷऔಘʣ
› ioctlʢσόΠε੍ޚʣ › lseekʢϑΝΠϧͷγʔΫʣ › writeʢϑΝΠϧॻ͖ࠐΉʣ › closeʢϑΝΠϧΛด͡Δʣ strace ܦ༝Ͱ֬ೝͨ͠ɻ 17 / 27
࠷ޙʹস͏ͷ୭ͩ ࠷ऴతͳՌͲ͜ʹஔ͘ʁ › ϑΝΠϧΛϩʔΧϧʹอଘ͢Δͷ͕ΰʔϧͰͳ͍ › ϑΝΠϧΛ AWS S3 ͳͲͷ֎෦ʹஔ͖͍ͨ ϩʔΧϧσόΠεʹϑΝΠϧΛॻ͖ࠐΈͨ͘ͳ͍ʂ
18 / 27
Today’s Theme In-Memory Streams 19 / 27
ΠϯϝϞϦʔετϦʔϜ ΠϯϝϞϦʔετϦʔϜͱ › str bytes ΛϑΝΠϧΦϒδΣΫτͷΑ͏ʹѻ͑Δ › ಡΈॻ͖ՄೳɺϥϯμϜΞΫηεՄೳ 20
/ 27
StringIO StringIO ςΩετϑΝΠϧͷͨΊͷΠϯϝϞϦετϦʔϜ ྫɿCSV Λ StringIO ͰऔΓѻ͏ import io with
io.StringIO() as csv_file: fieldnames = ["title", "started_at", "ended_at"] writer = csv.DictWriter(csv_file, fieldnames) writer.writeheader() writer.writerows(events) 21 / 27
BytesIO BytesIO όοϑΝ͖όΠφϦϑΝΠϧͷͨΊͷΠϯϝϞϦετϦʔϜ ྫɿPNG Λ BytesIO ͰऔΓѻ͏ import io with
io.BytesIO(png_bytes) as f: png_header = f.read(8) print(png_header) # b'\x89PNG\r\n\x1a\n' 22 / 27
෮शɿࠓͷඪ ॲཧͷྲྀΕ › Πϯλʔωοτܦ༝Ͱ GB αΠζͷσʔλΛऔಘ͢Δ › GB αΠζͷσʔλΛ
CSV ϑΝΠϧʹՃ͢Δ › CSV ϑΝΠϧΛ ZIP ѹॖ͢Δ › ZIP ѹॖσʔλΛΫϥυετϨʔδʹΞοϓϩʔυ͢Δ 23 / 27
σʔλΛΠϯλʔωοτܦ༝Ͱऔಘ͢Δ ྫɿConnpass API Λίʔϧ͢Δ with urllib.request.urlopen(url) as response: events =
json.load(response)["events"] 24 / 27
σʔλΛՃ͢Δ ྫɿAPI ͷऔಘ݁ՌΛ CSV ʹ͢Δ with io.StringIO() as ts: header
= ["title", "started_at", "ended_at"] writer = csv.DictWriter(ts, fieldnames=header) writer.writeheader() writer.writerows(events) 25 / 27
σʔλΛѹॖ&Ξοϓϩʔυ ྫɿZIP ʹѹॖͯ͠ AWS S3 ʹΞοϓϩʔυ with io.BytesIO() as bs:
with zipfile.ZipFile(bytes_stream, "w") as zf: zf.writestr("events.csv", ts.getvalue()) bs.seek(0) # ϑΝΠϧγʔΫ͕ϙΠϯτ s3.upload_fileobj(bs, "bucket", "events.zip") 26 / 27
Conclusion ·ͱΊ › io ϞδϡʔϧʹΠϯϝϞϦʔετϦʔϜؚ͕·ΕΔɻ › str bytes ΛϑΝΠϧΦϒδΣΫτͷΑ͏ʹѻ͏͜ͱ͕Ͱ͖Δɻ
› ௨ৗͷ open ͱҟͳΓγεςϜίʔϧ͕ݺΕͳ͍ɻ › σΟεΫͷ I/O ΛݮΒ͍ͨ͠ɺ·ͨͰ͖ͳ͍ঢ়گԼͰͷར༻ ͕࠷దͰ͋Δɻ io ϞδϡʔϧΛօ༷ͷಓ۩ശʹೖΕ͍ͯͩ͘͞ʂ 27 / 27