Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A LUT(lot) of image Filters on Android

A LUT(lot) of image Filters on Android

In this talk, we will learn how to create image filters with LUTs (Lookup Tables) using Renderscript on Android. We will briefly look into various approaches one can take for creating filters on Android. We will then learn about the basics of LUTs and how we can use them with Renderscript for creating filters. Finally, we will focus on the challenges that we faced while implementing filters on Android with the Renderscript framework. We will try to use graphical representations wherever possible to explain all these concepts.

ragdroid

July 02, 2019
Tweet

More Decks by ragdroid

Other Decks in Technology

Transcript

  1. Presets { "saturation": 0, "sharpness": 73, "luminance": 50, "vignetteAmount": 10,

    "vignetteRadius": 1.0, "shadows": 0.0, "contrast": 23, ... }
  2. LUT (Lookup Table) • Replaces runtime computation with simpler array

    indexing operation • Different Types: ✦ 1D ✦ 3 x 1D ✦ 3D
  3. 1D LUT (Lookup Table) 0 0 1 85 2 169

    3 255 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 1D Identity LUT (index, color)
  4. 3 x 0 1 2 3 0 1 2 3

    0 1 2 3 1D LUT (Lookup Table) 1D Identity LUT 3 x 0 0 1 85 2 169 3 255
  5. 3D LUT (Lookup Table) 0 0 0 0 0 1

    0 0 2 0 0 3 0 1 0 0 1 1 0 1 2 0 1 3 0 2 0 0 2 1 0 2 2 0 2 3 0 3 0 0 3 1 0 3 2 0 3 3 B G R 1 0 0 1 0 1 1 0 2 1 0 3 1 1 0 1 1 1 1 1 2 1 1 3 1 2 0 1 2 1 1 2 2 1 2 3 1 3 0 1 3 1 1 3 2 1 3 3 2 0 0 2 0 1 2 0 2 2 0 3 2 1 0 2 1 1 2 1 2 2 1 3 2 2 0 2 2 1 2 2 2 2 2 3 2 3 0 2 3 1 2 3 2 2 3 3 3 0 0 3 0 1 3 0 2 3 0 3 3 1 0 3 1 1 3 1 2 3 1 3 3 2 0 3 2 1 3 2 2 3 2 3 3 3 0 3 3 1 3 3 2 3 3 3 3D Identity LUT (4 x 4 x 4) B G R B G R B G R
  6. 3D LUT (Lookup Table) • 4 x 4 x 4

    is not enough • Practically 64 x 64 x 64 LUT images are used • 64 LUT images can map 262144 (64 ^3) colors
  7. Designers Pipeline • Create Filters • Start with Identity LUT

    image • Apply effects to it • Export and create new LUTs
  8. • getPixel() and setPixel() • Not Fast • Efficient Image

    Processing on Android - Nicolas Roard • Practical Image Processing in Android - Rebecca Franks easyLUT Library
  9. Image Processing • In Java - getPixel() and setPixel() •

    Native Code (JNI calls to C/C++) • OpenGL • RenderScript
  10. ScriptIntrinsic3DLUT “Intrinsic for converting RGB to RGBA by using a

    3D lookup table. The incoming r,g,b values are use as normalized x,y,z coordinates into a 3D allocation. The 8 nearest values are sampled and linearly interpolated. The result is placed in the output.”
  11. How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million

    params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()
  12. How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million

    params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()
  13. How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million

    params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()
  14. How to create Allocation? createCubemapFromBitmap(Renderscript rs, Bitmap b) createCubemapFromCubeFaces(some million

    params) createFromBitmap() createFromBitmapResource() createFromString() createSized() createTyped()
  15. Cube { int[] cube rSize int gSize int bSize int

    } LUT3DParams - silvaren/easyrs
  16. Cube Creation for (r in 0 until dimension) { for

    (g in 0 until dimension) { val p = r + g * w for (b in 0 until dimension) { val newPixel = pixels[p + b * h] lut[i++] = newPixel } } }
  17. ScriptIntrinsic3DLUT “Intrinsic for converting RGB to RGBA by using a

    3D lookup table. The incoming r,g,b values are use as normalized x,y,z coordinates into a 3D allocation. The 8 nearest values are sampled and linearly interpolated. The result is placed in the output.”
  18. Custom Script For each pixel value (R,G,B) find the corresponding

    value in LUT image and use that value instead
  19. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
  20. LUTFilter • Blue : Square within the LUT • Green

    and Red : (x, y) within a Square
  21. LUTFilter • Example : Input Pixel (127, 127, 127) •

    64 LUTs can not process all 256 colors so we divide by 4. • divide (127, 127, 127) by 4 = (31, 31, 31) ★ Blue : Square within the LUT : 31st square ★ Green and Red : (x, y) within a Square : (31, 31) within the square
  22. x Y

  23. Droidcon Berlin 2019 @ragdroid LUTFilter x = b % 8

    * 64 + r y = floor(b/8.0f) * 64 + g
  24. Droidcon Berlin 2019 @ragdroid LUTFilter out.a = in.a; out.rgb =

    rsGetElementAt_uchar4(lut, x, y); Annd it works!
  25. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
  26. Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha dest.r = dest.r *

    (256 - src.a) + src.r * src.a dest.g = dest.g * (256 - src.a) + src.g * src.a dest.b = dest.b * (256 - src.a) + src.b * src.a
  27. Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha dest.r = dest.r *

    (256 - src.a) + src.premultipliedR dest.g = dest.g * (256 - src.a) + src.premultipliedG dest.b = dest.b * (256 - src.a) + src.premultipliedB
  28. Droidcon Berlin 2019 @ragdroid Pre-multiplied alpha (Issue) • When we

    lookup input pixel in LUT • Wrong value lookup
  29. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
  30. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
  31. Droidcon Berlin 2019 @ragdroid Un-premultiply Alpha float div = in.a

    / 256.0; modifiedR = clamp(in.r / div, 0.0, 255.0); modifiedG = clamp(in.g / div, 0.0, 255.0); modifiedB = clamp(in.b / div, 0.0, 255.0);
  32. Droidcon Berlin 2019 @ragdroid LUTFilter x = modifiedB % 8

    * 64 + modifiedR y = floor(modifiedB/8.0f) * 64 + modifiedG
  33. Banding Input Image LUT R : 0 - 255 G:

    0 - 255 B: 0 - 255 R : 0 - 63 G: 0 - 63 B: 0 - 63 (255 ^ 3 = 16581375 colors) (64 ^ 3 = 262144 colors)
  34. Droidcon Berlin 2019 @ragdroid Banding x = (b / 4.0)

    % 8 * 64 + r/4.0 y = floor((b/4.0)/8.0f) * 64 + g/4.0
  35. 64 LUT Limitations • 64 LUT images can map 262144

    (64 ^3) colors • Jpg / png images have 16777216 (256 ^3) colors! • Lossy Technique
  36. Droidcon Berlin 2019 @ragdroid Interpolation “ Interpolation is a method

    of constructing new data points within the range of A discrete set of known data points. ” — Wikipedia
  37. Droidcon Berlin 2019 @ragdroid Trilinear Interpolation • Source: Computational Color

    Technology, Henry R. Kang • Wikipedia : https://en.wikipedia.org/wiki/Trilinear_interpolation
  38. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap
  39. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap Interpolate
  40. Droidcon Berlin 2019 @ragdroid LUTFilter (HALD) x = g %

    8 * 64 + r y = floor(g/8.0f) + b * 8
  41. Input Bitmap Input LUT Pixel Value (R,G, B) Find corresponding

    pixel in LUT image Un-premultiply Output Bitmap Repeat for all pixels in image Save updated pixel value in Output Bitmap Interpolate
  42. References • Computational Color Technology, Henry R. Kang (Interpolation) •

    easyLUT Library • silvaren/easyrs (LUT3DParams) • Renderscript docs • The Curious case of Android premultiplied Alpha (pre-multiplied Alpha
  43. What Next? • Building Filters with Go - Wayne Ashley

    Berry • .cube file format • Improve speed even further • Better Interpolation