Better docker image+

Transcript

1. Better Docker Image+ Bonfire Backend #2 #yjbonfire @orisano
2. ྑ͍Docker Imageͱ͸Կ͔
3. ݟͯΘ͔Γ΍͍͢ खݩͰߴ଎ʹbuild͞ΕΔ CI্Ͱߴ଎ʹbuild͞ΕΔ ߴ଎ʹdeploy͞ΕΔ
4. ࠓ೔࿩͍ͨ͜͠ͱ
5. ͲͷΑ͏ʹ଎͘͢Δ͔ ͲͷΑ͏ʹখ͘͢͞Δ͔
6. ͲͷΑ͏ʹ଎͘͢Δ͔ ͲͷΑ͏ʹখ͘͢͞Δ͔
7. ͲͷΑ͏ʹ଎͘͢Δ͔ • ΠϝʔδΛখ͘͢͞Δ • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ • cacheΛޮ͔ͤΔ • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ • ඞཁͳ͍εςʔδΛbuild͠ͳ͍
8. ͲͷΑ͏ʹ଎͘͢Δ͔ • ΠϝʔδΛখ͘͢͞Δ • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ • cacheΛޮ͔ͤΔ • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ • ඞཁͳ͍εςʔδΛbuild͠ͳ͍
9. imageΛখ͘͢͞Δͷ͸ docker push͢Δͱ͖ͷ଎౓ docker pull͢Δͱ͖ͷ଎౓ ͷߴ଎Խʹͭͳ͕Δ
10. Ͳͷ༷ʹখ͘͢͞Δ͔͸ ͋ͱͰ
11. ͲͷΑ͏ʹ଎͘͢Δ͔ • ΠϝʔδΛখ͘͢͞Δ • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ • cacheΛޮ͔ͤΔ • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ • ඞཁͳ͍εςʔδΛbuild͠ͳ͍
12. ίϚϯυͦͷ΋ͷΛ଎͘͢Δ
13. URLʹର͢ΔADDΛ࢖Θͳ͍
14. URLʹର͢ΔADD͸ جຊతʹμ΢ϯϩʔυ͢Δ
15. ஗͍
16. ΞΫηε͍ͯ͠Δઌͷ ίϯςϯπ͕ႈ౳ͳΒ wget + gzip + tarͰे෼
17. `RUN wget`ʹ͢Δ͜ͱͰ cache͕ޮ͘
18. ႈ౳Ͱͳ͍Ϧιʔεʹ ґଘ͢ΔͷΛۃྗ΍ΊΔ
19. build context ͷసૹྔΛߟ͑Δ
20. ϞϊϨϙʹ͢Δͱ build contextେ͖͘ͳΓ͕ͪ
21. ϞϊϨϙͷ৔߹͸ .dockerignore Λࣗಈੜ੒͠Α͏
22. github.com/orisano/dignore • ࢦఆ͞ΕͨdirectoryҎ֎ͷdirectoryΛignore • ࢦఆ͞Εͨdirectoryʹdockerignore͕͋Ε͹ ల։ • ͜Ε͚ͩʂ
23. buildͷલ޻ఔͰ buildʹඞཁͳ͍αʔϏε͸ Ignore͠Α͏
24. buildkitΛ࢖͍ͬͯΔͱ ࠩ෼సૹͯ͘͠ΕͨΓ͢Δ
25. COPYΛҙࣝͨ͠ directoryߏ଄
26. COPYͷҾ਺͕ directoryͷ৔߹͸ ର৅ͷdirectoryʹ த਎Λશ෦ίϐʔͯ͠͠·͏
27. ಛఆͷσΟϨΫτϦ͚ͩ ίϐʔͨ͘͠ͳ͍ ͱ͍͏͕೉͍͠
28. dockerignore Ͱআ֎͢Ε͹ྑ͘ͳ͍ʁ
29. ͍͍͑
30. vendorͳͲΛ௚઀؅ཧͯ͠ build࣌ʹdownload͠ͳ͍ ৔߹ʹຊ౰ʹਏ͍
31. 2ճҎ্ॏ͍directoryΛ COPYͨ͘͠ͳ͍
32. ͦ͏ͳΒͳ͍ͨΊͷ directoryߏ଄ʹ͓ͯ͜͠͏
33. ਓ͕ؒॻ͍ͨ΋ͷ͕ ೖ͍ͬͯΔdirectory ֎෦ʹґଘ͍ͯ͠Δ΋ͷ ࣗಈੜ੒෺͕ೖ͍ͬͯΔ directoryΛ෼཭͠Α͏
34. buildkitͩͱࠩ෼సૹ
35. ͍ͣΕʹͤΑ ෼཭͓͍ͯͨ͠ํ͕ cacheʹ༏͍͠
36. RUNͷத਎Λ଎͘͢Δ
37. GitHub͔Β࣮ߦϑΝΠϧΛ curl or wgetͰऔಘ͢Δ ͕஗͍
38. Ͳ͏ʹ͔଎͘Ͱ͖ͳ͍͔ curl -vvvΛ͏ͬͯΈͨ
39. GitHub Release͸S3Ͱ͋Δ S3͸Accept-Ranges: bytes ͱ͍͏͜ͱ͕Θ͔ͬͨ
40. Accept-Ranges: bytes ͬͯͳΜͩΖ͏
41. RFC7233, Range Requests ൣғΛࢦఆͯ͠ downloadͰ͖Δ
42. ͭ·Γ ฒྻμ΢ϯϩʔυ͕Մೳ
43. github.com/orisano/rget
45. ؀ڥʹΑΓ·͕͢ 5min -> 2min
46. Dockerfileͷ޻෉Ҏ֎ʹ΋ ଎͘͢ΔΞϓϩʔν͸͋Δ
47. ͲͷΑ͏ʹ଎͘͢Δ͔ • ΠϝʔδΛখ͘͢͞Δ • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ • cacheΛޮ͔ͤΔ • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ • ඞཁͳ͍εςʔδΛbuild͠ͳ͍
48. cacheΛޮ͔ͤΔ
49. docker build͢ΔϚγϯ͕ ಉҰͷ৔߹ ಉҰͰͳ͍৔߹
50. docker build͢ΔϚγϯ͕ ಉҰͷ৔߹ ಉҰͰͳ͍৔߹
51. cacheͷ࢓૊ΈΛཧղ͢Δ
52. RUN͸ จࣈྻ͕มΘΒͳ͍ݶΓ جຊతʹcache͞ΕΔ
53. COPY, ADD͢ΔϑΝΠϧͷ ಺༰͕มΘͬͨ৔߹ Ҏ߱ͷRUNͷcache͕ഁغ
54. ͳͷͰ lockfileͳͲΛઌʹίϐʔͯ͠ install͚ͩͯ͠cacheͤ͞Δ
57. docker build͢ΔϚγϯ͕ ಉҰͷ৔߹ ಉҰͰͳ͍৔߹
58. CI্Ͱbuild͢Δͱ͖ͳͲ cache͕ͳ͍͜ͱ͕ଟ͍
59. docker save & load or docker pull docker build —cache-from Λ࢖͏
60. image͕େ͖͍/layer͕ଟ͍ ৔߹ buildͷ΄͏͕ૣ͍͜ͱ΋͋Δ
61. image͕େ͖͍/layer͕ଟ͍ ৔߹ buildͷ΄͏͕ૣ͍͜ͱ΋͋Δ ܭଌܾͯ͠ΊΔ
62. ͲͷΑ͏ʹ଎͘͢Δ͔ • ΠϝʔδΛখ͘͢͞Δ • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ • cacheΛޮ͔ͤΔ • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ • ඞཁͳ͍εςʔδΛbuild͠ͳ͍
63. ґଘͷͳ͍εςʔδΛ ฒྻͰ࣮ߦ͢Δ
64. buildkitΛ࢖͍ͬͯͩ͘͞ʂ github.com/moby/buildkit export DOCKER_BUILDKIT=1
65. ͲͷΑ͏ʹ଎͘͢Δ͔ • ΠϝʔδΛখ͘͢͞Δ • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ • cacheΛޮ͔ͤΔ • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ • ඞཁͳ͍εςʔδΛbuild͠ͳ͍
66. ඞཁͳ͍εςʔδΛ build͠ͳ͍
67. —targetΛ࢖͑͹Α͍ͷͰ͸ʁ
68. —target͸ ࢦఆͨ͠εςʔδҎલΛ શ෦build͢Δ
69. multi stage buildΛ ౿ΈࠐΜͩ࢖͍ํ͍ͯ͠Δͱ ૺ۰͕ͪ͠
70. multi stage buildͷ ౿ΈࠐΜͩ࢖͍ํʹ͍ͭͯ͸
71. https://speakerdeck.com/orisano/multi-stage-builds-patterns-and-practice
72. ͓ͦΒ͘buildkitͰͰ͖Δ͕ buildkit͕࢖͑ͳ͍৔߹ʹ
73. github.com/orisano/targd • DockerfileͷASTΛऔಘ • ࢦఆ͞ΕͨεςʔδͷґଘεςʔδΛநग़ • ґଘεςʔδ͚ͩͷDockerfileΛग़ྗ
74. github.com/orisano/targd
75. ͲͷΑ͏ʹ଎͘͢Δ͔ ͲͷΑ͏ʹখ͘͢͞Δ͔
76. ͲͷΑ͏ʹখ͘͢͞Δ͔ • multi stage buildΛ࢖͏ • RUNΛ·ͱΊΔ(?) • ͳͥେ͖͍͔Λ஌Δ
77. ͲͷΑ͏ʹখ͘͢͞Δ͔ • multi stage buildΛ࢖͏ • RUNΛ·ͱΊΔ(?) • ͳͥେ͖͍͔Λ஌Δ
78. multi stage buildΛ࢖͏
79. multi stage buildͰ ͋Γ͕ͪͳٙ໰
80. ࠷ऴతͳΠϝʔδ͕ খ͘͞ͳΔ͔Β ͦΕ·Ͱͷεςʔδ͸ ࠷దԽ͠ͳͯ͘ྑ͍ʁ
82. ݸਓతʹ͸No
83. moby/issues/34715 —cache-formͩͱ multi stage buildͷ લஈͷεςʔδͷcache͕ ޮ͔ͳ͍
84. ࠷ऴεςʔδ͔͠ pushͯ͠ͳ͍͔Β౰ͨΓલ
85. CI্Ͱͷbuild͸ —cache-fromΛ࢖͏ ࠷ऴεςʔδ͚ͩcache͢Δ ҙຯ͸ͳ͍
86. multi stage build࣌ʹ cacheΛޮ͔͍ͤͨ৔߹͸ લͷεςʔδ΋ ໌ࣔతʹpush͢Δ͔͠ͳ͍
87. ͜ΕΛखಈͰ΍Δͱ cache-from஍ࠈʹͳΔ
88. github.com/orisano/castage • DockerfileͷASTΛऔಘ • εςʔδҰཡΛऔಘ • docker pullΛੜ੒ • docker buildΛcache-from෇͖Ͱੜ੒
89. ݁ہpush͢ΔͷͰ push/pullͷίετ͕͔͔Δ ͢΂ͯͷεςʔδΛฏ౳ʹ খ͘͢͞΂͖ (ݸਓͷݟղͰ͢)
90. ͲͷΑ͏ʹখ͘͢͞Δ͔ • multi stage buildΛ࢖͏ • RUNΛ·ͱΊΔ(?) • ͳͥେ͖͍͔Λ஌Δ
91. RUNΛ·ͱΊΔ(?)
92. Ͳ͔ͬʔ;͍͊Δ͸ 1ͭͷRUNʹ શ෦ॻ͘ͱྑ͍Β͍͠(?)
93. github.com/orisano/minid • DockerfileͷASTΛऔಘ • ࿈ଓ͢ΔRUN, COPY, ADDΛ࿈݁͢Δ • ݁ՌͷDockerfileΛग़ྗ
94. github.com/orisano/minid
95. αΠζ͕খ͘͞ͳͬͨ
96. Ͱ΋ͳΜͰʁ
97. ϨΠϠʔͷΦʔόʔϔου͕ ݮΔ͔Βখ͘͞ͳΔʁ
98. ͍͍͑
99. ·ͣ Ͳ͏͍͏ܗͰอଘ͞ΕͯΔ͔ Λ஌Δ
100. moby/image/spec/v1.md
101. AUFS
102. http://docs.docker.jp/engine/userguide/storagedriver/aufs-driver.html
103. ࡟আ͸ whiteoutϑΝΠϧͷ௥Ճ Ҡಈ͸ opaqueϑΝΠϧͷ௥Ճ ʴ ҠಈઌͷϑΝΠϧࠩ෼
104. Ұ౓Ͱ΋ RUN,COPY,ADDΛލ͙ͱ imageʹ͸࢒ͬͯ͠·͏
105. ػցతʹͰ΋ ҰͭͷRUNʹ·ͱΊΔ͜ͱͰ ༨ܭͳ΋ͷΛ࡟ݮͰ͖Δ
106. 1ͭͷϨΠϠʔʹ ·ͱΊΔ͜ͱ͸ ຊ౰ʹਖ਼͍͠ͷ͔
107. ϨΠϠʔΛผ͚Δ͜ͱʹΑΓ ฒྻμ΢ϯϩʔυͷԸܙ cacheͷ༗ޮ׆༻
108. ϨΠϠʔΛผ͚Δ͜ͱʹΑΓ ฒྻμ΢ϯϩʔυͷԸܙ cacheͷ༗ޮ׆༻ ܭଌܾͯ͠ΊΔ
109. ͲͷΑ͏ʹখ͘͢͞Δ͔ • multi stage buildΛ࢖͏ • RUNΛ·ͱΊΔ(?) • ͳͥେ͖͍͔Λ஌Δ
110. ͳͥେ͖͍͔Λ஌Δ
111. ͍͔ͳΔνϡʔχϯάͰ΋ ܭଌͤͣʹ΍ͬͯ͸͍͚ͳ͍
112. docker history
113. docker history
114. ͲͷϨΠϠʔ͕ େ͖͍͔͸Θ͔Δ͚Ͳ ͳͥେ͖͍͔͸෼͔ΓͮΒ͍
115. ҙ֎ʹίϚϯυ͕ Ͳ͏͍͏ϑΝΠϧΛ࡞Δͷ͔ ஌Βͳ͍
116. github.com/orisano/dlayer • docker saveͰಘΒΕͨtarΛղੳ • layerͱcmdͷରԠΛͱΔ • layerͷࠩ෼ΛϑΝΠϧαΠζ͕େ͖͍ॱʹද ࣔ
117. github.com/orisano/dlayer
118. ࣮ࡍʹdlayerΛ࢖ͬͯ golangެࣜalpineΠϝʔδͷ αΠζΛݮΒͯ͠Έͨ
119. docker-library/golang/pull/232
121. golang:1.11͔Βtoolchainͷ αΠζ͕Ͱ͔͘ͳͬͨ SSAͰͷ࠷దԽ͕૿͔͑ͨΒʁ
122. 253MB
123. 305MB
124. github.com/wagoodman/dive
125. ࡞ͬͨπʔϧ • github.com/orisano/rget • github.com/orisano/targd • github.com/orisano/minid • github.com/orisano/dlayer • ྑ͍ͱࢥͬͨΒελʔ͍͚ͯͨͩ͠ΔͱྭΈʹͳΓ·͢
126. ࡞ͬͨπʔϧ • github.com/orisano/dignore • github.com/orisano/castage • ྑ͍ͱࢥͬͨΒελʔ͍͚ͯͨͩ͠ΔͱྭΈʹͳΓ·͢
127. ·ͱΊ • ίϚϯυΛ଎͘Ͱ͖ͳ͍͔ߟ͑Δ • cacheΛཧղͯ͠༗ޮʹ׆͔͢ (CI or ϩʔΧϧ) • multi stage buildΛ࢖͏ (ॏ͍stageΛcache͢Δ) • layerΛখ͘͢͞ΔͨΊʹ෼ੳ͢Δ • buildkitΛ࢖͏!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
128. rget͜΅Ε࿩
129. github.com/orisano/rget
130. Ͱ΋ ͜ΕͰຊ౰ʹ͍͍ΜͩΖ͏͔
131. rgetͷbinaryΛdownload ͢ΔͷͰ͸ෛ͚ͨؾ͕͢Δ()
132. shell͚ͩͰ ࣮ݱͰ͖ͳ͍ͩΖ͏͔
133. ϙʔλϒϧੑ͸͍Βͳ͍ͷͰ ओઓ৔ͷalpineʹߜΔ
134. alpine(busybox)͸ xargs͕ೖ͍ͬͯΔʂʂʂ
135. xargsΛ࢖͏͜ͱͰ ฒྻԽ ಉ࣌઀ଓ਺੍ݶ ͕࣮ݱͰ͖Δ
136. GitHub Release͸ ؆୯ʹHEAD͕Ͱ͖ͳ͍ͷͰ GETͰbodyΛແࢹͯ͠ Content-LengthΛऔಘͰ͖Δ
137. seqͰ chunkͷrangeΛੜ੒Ͱ͖Δ
138. wget͸206 Partial Content ͕͏·͘ѻ͑ͳ͍ͷͰμϝ
139. ࢓ํͳ͍ͷͰ curl -RͰdownload
140. ࠷ޙʹcatͯ͠chunkΛ݁߹
141. shellͰ΋࣮ݱͰ͖Δʂ
142. ͔͠͠ curl͸alpineඪ४Ͱ͸ͳ͍
143. apk add —no-cache curl ͸ෛ͚ͨؾ͕͢Δ()
144. ͔͠͠ wget͸206͕ॲཧͰ͖ͳ͍
145. ఘΊΒΕͳ͍ͷͰ busyboxͷwgetͷ ιʔείʔυΛಡΉ
146. busybox/networking/wget.c
147. busybox/networking/wget.c
148. -OͰࢦఆͨ͠ϑΝΠϧ͕ଘࡏ (ϑΝΠϧαΠζ͕1Ҏ্) -cΛࢦఆ͍ͯ͠Δ ্هͷ৚݅Λຬͨ͢ͱ͖ 206Λॲཧͯ͘͠ΕΔ
149. ͭ·Γ ద౰ͳ1byteͷϑΝΠϧʹ -cͰࢦఆ͢Δ͜ͱͰճආՄೳ
150. ઌ಄ͷ1byteΛऔΓআ͚Ε͹ alpineඪ४ͷΈͰ࣮ݱՄೳ
151. tailͰ΋Մೳ͕ͩ஗͍ dd skip=1 iflag=skip_bytes ͕ߴ଎
152. alpineඪ४ͷΈͰ࣮ݱ
153. https://github.com/orisano/rget/blob/master/rget.sh