How to Use In-Memory Streams

PyCon JP 2020

Transcript

1. ΠϯϝϞϦʔετϦʔϜ׆༻ज़ How to Use In-Memory Streams Hayao Suzuki PyCon JP 2020 August 29, 2020
2. ൃදʹࡍͯ͠ GitHub ʹࢿྉ͕͋Γ·͢ › https://github.com/HayaoSuzuki/pyconjp2020 Twitter ͷϋογϡλά › #pyconjp_1 PyCon JP Fellow Slack › #jp-2020-track-1 2 / 27
3. Who am I ? ͓લ୭Α Name Hayao Suzukiʢླ໦ɹॣʣ Twitter @CardinalXaro Work Python Programmer at iRidge, Inc. 3 / 27
4. Who am I ? Technical Reviewer › Effective Python ୈ 2 ൛ (O’Reilly Japan) › ಈֶ͔ͯ͠Ϳྔࢠίϯϐϡʔλϓϩάϥϛϯά (O’Reilly Japan) https://xaro.hatenablog.jp/ ʹϦετ͕͋Γ·͢ɻ 4 / 27
5. Who am I ? Selected Talks › ϨΨγʔ Django ΞϓϦέʔγϣϯͷݱ୅Խ (DjangoCongress JP 2018) › SymPy ʹΑΔ਺ࣜॲཧ (PyCon JP 2018) › Python ͱָ͠Ήॳ౳੔਺࿦ (PyCon mini Hiroshima 2019) › ܅͸ cmath Λ஌͍ͬͯΔ͔ (PyCon mini Shizuoka 2020) https://xaro.hatenablog.jp/ ʹϦετ͕͋Γ·͢ɻ 5 / 27
6. ࠓ೔ͷ໨ඪ ͜Μͳ՝୊Λղܾ͍ͨ͠ʂ › Πϯλʔωοτܦ༝Ͱ਺ GB αΠζͷσʔλΛऔಘ͠ɺCSV ϑΝΠϧʹՃ޻͢Δ › Ϋϥ΢υ্ʹߏஙͨ͠طଘͷγεςϜʹ௥Ճ͢ΔܗͰ࣮૷͢Δ › ຖ೔࣮ߦ͢Δ Ϋϥ΢υαʔϏε͸ैྔ՝ۚ ͳΔ΂͘ਝ଎ʹॲཧ͍ͨ͠ʂ 6 / 27
7. ࠓ೔ͷ໨ඪ ॲཧͷྲྀΕ › Πϯλʔωοτܦ༝Ͱ਺ GB αΠζͷσʔλΛऔಘ͢Δ › ਺ GB αΠζͷσʔλΛ CSV ϑΝΠϧʹՃ޻͢Δ › CSV ϑΝΠϧΛ ZIP ѹॖ͢Δ › ZIP ѹॖσʔλΛΫϥ΢υετϨʔδʹΞοϓϩʔυ͢Δ ෼ੳ › σʔλαΠζ͕େ͖͍ › σʔλͷՃ޻͸୯७ͳॲཧ 7 / 27
8. ࠓ೔ͷ໨ඪ ϘτϧωοΫ͸Ͳ͔͜ › ZIP ѹॖ͸ͦΕ΄ͲେมͰ͸ͳ͍ › σʔλՃ޻͸୯७ͳॲཧ › ϘτϧωοΫ͸ I/O ॲཧʹ͋Γͦ͏ Կͱ͔ͯ͠ I/O ॲཧΛਝ଎ʹॲཧ͍ͨ͠ʂʂʂ 8 / 27
9. Today’s Theme In-Memory Streams 9 / 27
10. Stream? ͦ΋ͦ΋ετϦʔϜͬͯԿʁ ετϦʔϜ͸ϑΝΠϧΦϒδΣΫτͰ͋Δɻ 10 / 27
11. File Object? ϑΝΠϧΦϒδΣΫτͬͯԿʁ › read() ΍ write() ͳͲͷϝιουΛ࣋ͭΦϒδΣΫτ › σΟεΫ্ͷϑΝΠϧ΍ผͷ৔ॴʹ͋ΔετϨʔδɺೖग़ྗػثͱ ΍ΓͱΓ͕Ͱ͖Δ 11 / 27
12. File Object? ϑΝΠϧΦϒδΣΫτͨͪ › ੜόΠφϦϑΝΠϧ › όοϑΝ෇͖όΠφϦϑΝΠϧ › ςΩετϑΝΠϧ 12 / 27
13. ࢖͍ํ ςΩετϑΝΠϧ f = open("myfile.txt", "r") όοϑΝ෇͖όΠφϦ f = open("myfile.jpg", "rb") 13 / 27
14. open ؔ਺ͷཪଆ open ͸ԿΛ͍ͯ͠Δͷ͔ʁ OS ͷγεςϜίʔϧ API ΛݺͿ 14 / 27
15. open ؔ਺ͷཪଆ ྫɿCSV ʹՃ޻͢Δ with open("events.csv", "w") as csv_file: fieldnames = ["title", "started_at", "ended_at"] writer = csv.DictWriter(csv_file, fieldnames) writer.writeheader() writer.writerows(events) 15 / 27
16. open ؔ਺ͷཪଆ ྫɿWindows › CreateFileʢϑΝΠϧͷΞΫηεݖऔಘʣ › QueryAllInformationFileʢϑΝΠϧ৘ใͷऔಘʣ › WriteFileʢϑΝΠϧ΁ॻ͖ࠐΉʣ › CloseFileʢϑΝΠϧΛด͡Δʣ Process Monitor ܦ༝Ͱ֬ೝͨ͠ɻ 16 / 27
17. open ؔ਺ͷཪଆ ྫɿUbuntu on WSL › openat ʢϑΝΠϧͷΦʔϓϯʣ › fstatʢϑΝΠϧ৘ใͷऔಘʣ › ioctlʢσόΠε੍ޚʣ › lseekʢϑΝΠϧͷγʔΫʣ › writeʢϑΝΠϧ΁ॻ͖ࠐΉʣ › closeʢϑΝΠϧΛด͡Δʣ strace ܦ༝Ͱ֬ೝͨ͠ɻ 17 / 27
18. ࠷ޙʹস͏ͷ͸୭ͩ ࠷ऴతͳ੒Ռ෺͸Ͳ͜ʹஔ͘ʁ › ϑΝΠϧΛϩʔΧϧʹอଘ͢Δͷ͕ΰʔϧͰ͸ͳ͍ › ϑΝΠϧΛ AWS S3 ͳͲͷ֎෦ʹஔ͖͍ͨ ϩʔΧϧσόΠεʹϑΝΠϧΛॻ͖ࠐΈͨ͘ͳ͍ʂ 18 / 27
19. Today’s Theme In-Memory Streams 19 / 27
20. ΠϯϝϞϦʔετϦʔϜ ΠϯϝϞϦʔετϦʔϜͱ͸ › str ΍ bytes ΛϑΝΠϧΦϒδΣΫτͷΑ͏ʹѻ͑Δ › ಡΈॻ͖ՄೳɺϥϯμϜΞΫηεՄೳ 20 / 27
21. StringIO StringIO ςΩετϑΝΠϧͷͨΊͷΠϯϝϞϦετϦʔϜ ྫɿCSV Λ StringIO ͰऔΓѻ͏ import io with io.StringIO() as csv_file: fieldnames = ["title", "started_at", "ended_at"] writer = csv.DictWriter(csv_file, fieldnames) writer.writeheader() writer.writerows(events) 21 / 27
22. BytesIO BytesIO όοϑΝ෇͖όΠφϦϑΝΠϧͷͨΊͷΠϯϝϞϦετϦʔϜ ྫɿPNG Λ BytesIO ͰऔΓѻ͏ import io with io.BytesIO(png_bytes) as f: png_header = f.read(8) print(png_header) # b'\x89PNG\r\n\x1a\n' 22 / 27
23. ෮शɿࠓ೔ͷ໨ඪ ॲཧͷྲྀΕ › Πϯλʔωοτܦ༝Ͱ਺ GB αΠζͷσʔλΛऔಘ͢Δ › ਺ GB αΠζͷσʔλΛ CSV ϑΝΠϧʹՃ޻͢Δ › CSV ϑΝΠϧΛ ZIP ѹॖ͢Δ › ZIP ѹॖσʔλΛΫϥ΢υετϨʔδʹΞοϓϩʔυ͢Δ 23 / 27
24. σʔλΛΠϯλʔωοτܦ༝Ͱऔಘ͢Δ ྫɿConnpass API Λίʔϧ͢Δ with urllib.request.urlopen(url) as response: events = json.load(response)["events"] 24 / 27
25. σʔλΛՃ޻͢Δ ྫɿAPI ͷऔಘ݁ՌΛ CSV ʹ͢Δ with io.StringIO() as ts: header = ["title", "started_at", "ended_at"] writer = csv.DictWriter(ts, fieldnames=header) writer.writeheader() writer.writerows(events) 25 / 27
26. σʔλΛѹॖ&Ξοϓϩʔυ ྫɿZIP ʹѹॖͯ͠ AWS S3 ʹΞοϓϩʔυ with io.BytesIO() as bs: with zipfile.ZipFile(bytes_stream, "w") as zf: zf.writestr("events.csv", ts.getvalue()) bs.seek(0) # ϑΝΠϧγʔΫ͕ϙΠϯτ s3.upload_fileobj(bs, "bucket", "events.zip") 26 / 27
27. Conclusion ·ͱΊ › io Ϟδϡʔϧʹ͸ΠϯϝϞϦʔετϦʔϜؚ͕·ΕΔɻ › str ΍ bytes ΛϑΝΠϧΦϒδΣΫτͷΑ͏ʹѻ͏͜ͱ͕Ͱ͖Δɻ › ௨ৗͷ open ͱҟͳΓγεςϜίʔϧ͕ݺ͹Εͳ͍ɻ › σΟεΫ΁ͷ I/O ΛݮΒ͍ͨ͠ɺ·ͨ͸Ͱ͖ͳ͍ঢ়گԼͰͷར༻ ͕࠷దͰ͋Δɻ io ϞδϡʔϧΛօ༷ͷಓ۩ശʹೖΕ͍ͯͩ͘͞ʂ 27 / 27