Mobile Image Processing at London Web Perf Meetup, 2016

Loading images isn’t as simple as you think. We often discuss image weight and download time, but mobile devices deal with many other challenges, such as decode times in underpowered CPUs, limited CPU/GPU transfers, and memory consumption. In this talk we’ll explain those limitations, their impact, and how you can best handle them on your own site.

Presented at London Web Perf Meetup in London, UK on Feb 11, 2015.

Transcript

1. MOBILE IMAGE PROCESSING https://flic.kr/p/qRJKUz Tim Kadlec | @tkadlec
2. https://flic.kr/p/aShfSP
4. https://flic.kr/p/oPn6ZV
6. 10% of photos ever 2012 Source: http://time.com/3445111/#1
7. Source: http://time.com/3445111/#1 1839 2012
8. Source: http://time.com/3445111/#1 1839 2012 2011 172 years, 90%
9. Source: http://time.com/3445111/#1 1839 2012 2011 1 year, 10%
10. Source: http://time.com/3445111/#1 1839 2012 2011 1 year, 10%
11. 0.1 Source: http://www.kpcb.com/internet-trends
12. 1.8 billion photos PER DAY Source: http://www.kpcb.com/internet-trends
16. 360px x 510px
17. 1300px x 1024px
18. Request Decode Copy to GPU Display
19. Request Copy to GPU Display Decode
20. Request Copy to GPU Display Decode
21. Source: May 1st run of http://httparchive.org 46% JPG 29% PNG 23% GIF 2% Other
22. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding
23. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding
24. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding
25. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding
26. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding
27. YUV to RGB Chroma Upsampling iDCT / Dequantization Huffman Decoding
35. Resized: 7.86ms
36. Resized: 7.86ms Double: 26.34ms
37. Resized: 7.86ms Double: 26.34ms Large: 266.49ms
38. Resized: 7.86ms Double: +235.1% Large: +3291.7%
40. 2GB RAM Quad-core 1.5 GHz Krait CPU Adreno 320 GPU
41. Resized: 30.38ms
42. Resized: 30.38ms Double: 102.77ms
43. Resized: 30.38ms Double: 102.77ms Large (6x): 1,534.99ms
44. Resized: 30.38ms Double: +238.3% Large (6x): +4952.6%
47. RESIZE IMAGES
49. “HUGER”
50. ROOTER
51. ROUTER
52. COLOUR
53. COLOUR
54. Request Display Decode Copy to GPU
55. Request Display Decode Copy to GPU
56. Request Copy to GPU Display Store in memory Decode
59. R: 45 G: 48 B: 46 A: 1 R: 177 G: 167 B: 157 A: 1 R: 190 G: 187 B: 181 A: 1
60. W x H x 4
61. 1300px x 1024px
62. 1300 x 1024 x 4
63. 5,324,800 1300 x 1024 x 4
64. 5.1 MB 1300 x 1024 x 4
68. 360 x 510 x 4
69. 734,400
70. - 734,400 5,324,800 4,590,400
71. 4.38 MB
72. “…25% of new Android phones have only 512MB of RAM.” Jen Fitzpatrick VP of product management for Google Maps
73. 270px x 936px 987 kb
74. RESIZE IMAGES
75. COLOR
76. 600px x 600px 200px x 200px
77. 550px x 550px 150px x 150px
78. (600 x 600 - 550 x 550) x 4 = 230,000 wasted bytes
79. (200 x 200 - 150 x 150) x 4 = 70,000 wasted bytes 230,000 wasted bytes
80. 70,000 wasted bytes 230,000 wasted bytes
81. 700px 1200px 200px
82. 1200px 200px 700px
83. MEMORY BATTERY LIFE OF DEVICE CPU/GPU OVERHEAD
85. iDCT / Dequantization Huffman Decoding Chroma Upsampling YUV to RGB
86. iDCT / Dequantization Huffman Decoding Chroma Upsampling YUV to RGB
88. Luma (light) Chroma (color)
89. Y CB CR
90. Chroma Subsampling
91. J : a : b
93. 4 : a : b
97. 4 : 4 : 4
99. 4 4
100. 4 4 4 8 total
104. 4 : 2 : 2
106. 4 2
107. 2 2 4 4 total (50% savings)
112. 4 : 1 : 1
114. 4 1
115. 4 1 1 2 total (75% savings)
120. 4 : 2 : 0
122. 4 2
123. 4 2 0 2 total (75% savings)
129. Request Decode Copy to GPU Display
130. Request Decode Copy to GPU Display Decode
131. (W x H x 3) - (W x H x SUBSAMPLE_LEVEL x 2)
132. RGBA 5,324,800 4:4:4 3,993,600 4:2:2 2,662,400 4:2:0 1,996,800 1300px x 1024px
133. 62.5% memory savings 4:2:0 Source: http://time.com/3445111/#1
134. https://flic.kr/p/aaThYm
135. https://flic.kr/p/aaThYm
136. Source: May 1st run of http://calendar.perfplanet.com/2015/why-arent-your-images-using-chroma-subsampling/ 39.3% 4:2:0 0.9% Other 59.7% 4:4:4
143. > 26MB
144. Browser OS
145. Browser OS Psst….need some memory?
146. Browser OS Yeah, I’ll take some of that.
147. iDCT / Dequantization Huffman Decoding Chroma Upsampling YUV to RGB
149. <meta name=“viewport" content=“width=device-width, minimum-scale=1.0” />
151. Resized: 30.38ms
152. Resized: 5.62ms
153. Resized: 5.62ms Double: 102.77ms
154. Resized: 5.62ms Double: 27.74ms
155. Resized: 5.62ms Double: 27.74ms Large (6x): 1534.99ms
156. Resized: 5.62ms Double: 27.74ms Large (6x): 462.11ms
157. Resized: 5.62ms Double: +393.6% Large (6x): +8122.6%
159. <meta name=“viewport” content=“initial-scale = 1.0, maximum-scale = 1.0” />
160. <meta name=“viewport” content=“initial-scale = 1.0, maximum-scale = 1.0, width=device-width” />
161. MICRO-OPTIMIZATIONS?
169. “…as bandwidth grows, and as processing power grows, and as browsers get better we just keep filling everything up. ” Jeff Veen
170. NETWORK MEMORY BATTERY LIFE OF DEVICE CPU/GPU OVERHEAD
171. RESIZE IMAGES 1
172. RESIZE IMAGES 1 2 SMALL BREAKPOINTS FOR LARGE IMAGES
173. RESIZE IMAGES 1 2 META VIEWPORT IS YOUR FRIEND 3 SMALL BREAKPOINTS FOR LARGE IMAGES
174. RESIZE IMAGES 1 2 META VIEWPORT IS YOUR FRIEND 3 USE 4:2:0 SUBSAMPLING 4 SMALL BREAKPOINTS FOR LARGE IMAGES
175. THANK YOU Tim Kadlec | @tkadlec