Mobile Image Processing at Velocity SC, 2015

Loading images isn’t as simple as you think. We often discuss image weight and download time, but mobile devices deal with many other challenges, such as decode times in underpowered CPUs, limited CPU/GPU transfers, and memory consumption. In this talk we’ll explain those limitations, their impact, and how you can best handle them on your own site.

Presented at Velocity SC 2015 in Santa Clara, CA on May 29, 2015.

Transcript

1. MOBILE IMAGE PROCESSING https://flic.kr/p/qRJKUz Tim Kadlec | @tkadlec
2. https://flic.kr/p/aShfSP
4. https://flic.kr/p/oPn6ZV
6. 10% of photos ever 2012 Source: http://time.com/3445111/#1
7. Source: http://time.com/3445111/#1 1839 2012 2011 172 years, 90%
8. Source: http://time.com/3445111/#1 1839 2012 2011 1 year, 10%
9. billion photos PER DAY Source: http://www.kpcb.com/internet-trends 1.8
10. Source: May 1st run of http://httparchive.org 789kb Everything else 1310kb Images
11. Source: May 1st run of http://httparchive.org 36.5% Everything else 62.4% Images
14. 360px x 202px
15. 1280px x 520px
16. Request Decode Copy to GPU Display
17. Request Copy to GPU Display Decode
18. 1280px x 520px
19. Source: May 1st run of http://httparchive.org 46% JPG 29% PNG 23% GIF 2% Other
20. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding
21. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding
22. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding
23. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding
24. RGB to YUV Chroma Subsampling DCT / Quantization Huffman Encoding
25. YUV to RGB Chroma Upsampling iDCT / Dequantization Huffman Decoding
35. Resized: 7.86ms
36. Resized: 7.86ms Double: 26.34ms
37. Resized: 7.86ms Double: 26.34ms Large: 266.49ms
38. Resized: 7.86ms Double: +235.1% Large: +3291.7%
40. 2GB RAM Quad-core 1.5 GHz Krait CPU Adreno 320 GPU
41. Resized: 30.38ms
42. Resized: 30.38ms Double: 102.77ms
43. Resized: 30.38ms Double: 102.77ms Large (6x): 1,534.99ms
44. Resized: 30.38ms Double: +238.3% Large (6x): +4952.6%
46. RESIZE IMAGES
47. RESIZE IMAGES
48. Request Display Decode Copy to GPU
49. Request Copy to GPU Display Store in memory Decode
51. R: 65 G: 47 B: 35 A: 1 R: 26 G: 26 B: 29 A: 1 R: 30 G: 29 B: 32 A: 1
52. W x H x 4
53. 1280px x 520px
54. 1280 x 520 x 4
55. 1280 x 520 x 4 2,662,400
56. 2.54 MB 1280 x 520 x 4
59. “…25% of new Android phones have only 512MB of RAM.” Jen Fitzpatrick VP of product management for Google Maps
60. 360 x 202 x 4
61. 360 x 202 x 4 290,880
62. - 290,880 2,662,400 2,371,520
63. 2.26 MB
64. 270px x 936px 987 kb
65. RESIZE IMAGES
66. 600px x 600px 200px x 200px
67. 550px x 550px 150px x 150px
68. (600 x 600 - 550 x 550) x 4 = 230,000 wasted bytes
69. (200 x 200 - 150 x 150) x 4 = 70,000 wasted bytes 230,000 wasted bytes
70. 70,000 wasted bytes 230,000 wasted bytes
71. 700px 1200px 200px
72. 1200px 200px 700px
73. MEMORY BATTERY LIFE OF DEVICE CPU/GPU OVERHEAD
75. iDCT / Dequantization Huffman Decoding Chroma Upsampling YUV to RGB
77. Luma (light) Chroma (color)
78. Y CB CR
79. Chroma Subsampling
80. J : a : b
82. 4 : a : b
86. 4 : 4 : 4
88. 4 4
89. 4 4 4 8 total
93. 4 : 2 : 2
95. 4 2
96. 2 2 4 4 total (50% savings)
101. 4 : 1 : 1
103. 4 1
104. 4 1 1 2 total (75% savings)
109. 4 : 2 : 0
111. 4 2
112. 4 2 0 2 total (75% savings)
117. Request Decode Copy to GPU Display
118. Request Decode Copy to GPU Display Decode
119. (W x H x 3) - (W x H x SUBSAMPLE_LEVEL x 2)
120. RGBA 2,662,400 4:4:4 1,996,800 4:2:2 1,331,200 4:2:0 998,400 1280px x 520px
121. 62.5% memory savings 4:2:0 Source: http://time.com/3445111/#1
126. > 26MB
127. Browser OS
128. Browser OS Psst….need some memory?
129. Browser OS Yeah, I’ll take some of that.
130. iDCT / Dequantization Huffman Decoding Chroma Upsampling YUV to RGB
132. <meta name=“viewport" content=“width=device-width, minimum-scale=1.0” />
134. Resized: 30.38ms
135. Resized: 5.62ms
136. Resized: 5.62ms Double: 102.77ms
137. Resized: 5.62ms Double: 27.74ms
138. Resized: 5.62ms Double: 27.74ms Large (6x): 1534.99ms
139. Resized: 5.62ms Double: 27.74ms Large (6x): 462.11ms
140. Resized: 5.62ms Double: +393.6% Large (6x): +8122.6%
141. MICRO-OPTIMIZATIONS?
142. “…as bandwidth grows, and as processing power grows, and as browsers get better we just keep filling everything up. ” Jeff Veen
143. NETWORK MEMORY BATTERY LIFE OF DEVICE CPU/GPU OVERHEAD
144. RESIZE IMAGES 1
145. RESIZE IMAGES 1 2 SMALL BREAKPOINTS FOR LARGE IMAGES
146. RESIZE IMAGES 1 2 META VIEWPORT IS YOUR FRIEND 3 SMALL BREAKPOINTS FOR LARGE IMAGES
147. RESIZE IMAGES 1 2 META VIEWPORT IS YOUR FRIEND 3 USE 4:2:0 SUBSAMPLING 4 SMALL BREAKPOINTS FOR LARGE IMAGES
148. THANK YOU Tim Kadlec | @tkadlec