Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical Image Processing on Android - DevFest...

Practical Image Processing on Android - DevFest Ukraine 2018

Images can add colour and add character to our apps. People love using images to depict parts of their everyday lives. But when you start working with them as a developer, your opinion on dealing with images may change. How can I easily add image effects to my app?

In this session, you will learn how to process images on Android. Working with ColorFilters and Renderscript can provide your app with functionality to allow users to adjust their images to their liking. You will also learn the basics of displaying and working with images on Android.

Rebecca Franks

October 13, 2018
Tweet

More Decks by Rebecca Franks

Other Decks in Programming

Transcript

  1. Challenges • Process multiple images • Complex image processing •

    Storing filters • Custom Canvas Sizes • An iOS app that had the filters already
  2. Mistakes we made Rendering project size vs display size ⏳

    Loading bitmaps synchronously Many different Image processing tools/libraries
  3. What is a Bitmap? 2D Array of Pixels Pixel =

    [R, G, B, A] (red, green, blue, alpha) Width Height
  4. How do I load up a Bitmap? • Manually with

    BitmapFactory • Glide • Picasso • Fresco • …
  5. val options = BitmapFactory.Options() options.inJustDecodeBounds = true BitmapFactory.decodeResource(res, resId, options)

    options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight) options.inJustDecodeBounds = false return BitmapFactory.decodeResource(res, resId, options) BitmapFactory https://developer.android.com/topic/performance/graphics/load-bitmap
  6. OutOfMemory Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 16777228 byte

    allocation with 4194304 free bytes and 4MB until OOM at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java) at android.graphics.Bitmap.nativeCreate(Bitmap.java) at android.graphics.Bitmap.createBitmap(Bitmap.java:977) at android.graphics.Bitmap.createBitmap(Bitmap.java:948) at android.graphics.Bitmap.createBitmap(Bitmap.java:915)
  7. Bitmap too large Fatal Exception: java.lang.RuntimeException: Canvas: trying to draw

    too large(283435200bytes) bitmap. at android.view.DisplayListCanvas.throwIfCannotDraw(DisplayListCanvas .java:260) at android.graphics.Canvas.drawBitmap(Canvas.java:1373)
  8. You need to scale your images Height = 1920px Width

    = 700px 1920px * 700px = 1 344 000 * 4 bytes per pixel = 5 376 000 bytes = 5.375 mb Height = 128px Width = 50px = 0.0256mb
  9. How do I scale down my image? val options =

    BitmapFactory.Options() options.inSampleSize = calculateInSampleSize(options, reqWidth, reqHeight) BitmapFactory.decodeResource(res, resId, options)
  10. Applying Image Effects • getPixel() & setPixel() • Glide Image

    Transformations / GPUImage • ColorMatrix • Renderscript • OpenGL • Native (C++) • … ⛔ ⛔
  11. setPixel() override fun filter(bitmap: Bitmap): Bitmap { val newBitmap =

    bitmap.copy(Bitmap.Config.ARGB_8888, true) for (i in 0 until bitmap.width){ for (j in 0 until bitmap.height){ newBitmap.setPixel(i, j, Color.BLUE) } } return newBitmap }
  12. Pros % • Easy to understand • Manipulate per Pixel

    manually Cons & • Suuuuuper Slow • CPU Execution • No parallel execution
  13. What is a Color Matrix Filter? Modifies the color of

    each pixel drawn with a certain Paint.
  14. ColorMatrixColorFilter 4 * 5 matrix applied to each pixel a,

    b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t
  15. ColorMatrix 4 * 5 matrix applied to each pixel R’

    = a*R + b*G + c*B + d*A + e; G’ = f*R + g*G + h*B + i*A + j; B’ = k*R + l*G + m*B + n*A + o; A’ = p*R + q*G + r*B + s*A + t; a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t [R’, G’, B’, A’]
  16. ColorMatrix Identity Matrix - same image in as out 1,

    0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0 R, G, B, A * = R, G, B, A
  17. ColorMatrixColorFilter - Invert Image val canvas = Canvas(bitmap) val paint

    = Paint() paint.colorFilter = ColorMatrixColorFilter( ColorMatrix(floatArrayOf( -1f, 0f, 0f, 0f, 255f, 0f, -1f, 0f, 0f, 255f, 0f, 0f, -1f, 0f, 255f, 0f, 0f, 0f, 1f, 0f ))) canvas.drawBitmap(original, 0, 0, paint)
  18. Other ColorFilters - LightingColorFilter - PorterDuffColorFilter More info: Manipulating images

    and Drawables with Android’s ColorFilter - Nick Rout bit.ly/2Cwvx8t
  19. Pros % • Easy-ish to understand • Fast (GPU) •

    Write in Kotlin/ Java • Understand the exact effect being applied Cons & • Can’t do all effects like Sharpen, Unsharp mask etc
  20. Pros % • Easy to understand • Fast (GPU) •

    Nice API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you
  21. Pros % • Easy to understand • Fast • Nice

    API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you • Dependant on Third Party
  22. Pros % • Easy to understand • Fast • Nice

    API • Java/Kotlin Cons & • Can’t do everything • Values are abstracted away from you • Dependant on Third Party • Dependent on OpenGL code
  23. What is Renderscript? • Framework on Android for computationally expensive

    tasks • Parallelises tasks across CPU and GPU • Language: C99-derived language for writing high-performance compute code
  24. #pragma version(1) #pragma rs java_package_name(app.rigs.renderscriptfun) #pragma rs_fp_full static float bright

    = 0.f; void setBright(float v) { bright = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255);
  25. static float bright = 0.f; void setBright(float v) { bright

    = 255.f / (255.f - v); } void exposure(const uchar4 *in, uchar4 *out) { out->r = clamp((int)(bright * in->r), 0, 255); out->g = clamp((int)(bright * in->g), 0, 255); out->b = clamp((int)(bright * in->b), 0, 255); }
  26. class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context)

    private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap)
  27. class ExposureFilter(val context: Context) { private val renderScript = RenderScript.create(context)

    private val script = ScriptC_exposure(renderScript) fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap = Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut)
  28. fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap =

    Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }
  29. fun filter(exposure: Float, bitmap: Bitmap): Bitmap { val outputBitmap =

    Bitmap.createBitmap(bitmap) val tmpIn = Allocation.createFromBitmap(renderScript, bitmap) val tmpOut = Allocation.createFromBitmap(renderScript, outputBitmap) script.invoke_setBright(exposure) script.forEach_exposure(tmpIn, tmpOut) tmpOut.copyTo(outputBitmap) tmpIn.destroy() tmpOut.destroy() return outputBitmap }
  30. 0, -1, 0 -1, 5, -1, 0, -1, 0 Sharpen

    an Image Sharpen Kernel 2, 30, 1 2, 40, 20, 4, 20, 5 Pixel in an Image with surrounding pixel values Resulting Pixel value = (0 * 2) + (-1 * 30) + (0 * 1) +(-1 * 2) + (5 * 40) + (-1* 20) +(0 * 4) + (-1 * 20) + (0 * 5) = 128
  31. class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript =

    RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) }
  32. class SharpenImageFilter(context: Context, val multiplier: Float) { val renderScript =

    RenderScript.create(context) val convolution = ScriptIntrinsicConvolve3x3.create(renderScript, Element.U8_4(renderScript)) override fun applyFilter(bitmap: Bitmap): Bitmap { val matrixSharpen = floatArrayOf( 0f, -multiplier, 0f, -multiplier, 1 + 4 * multiplier, -multiplier, 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients)
  33. 0f, -multiplier, 0f) return applyFilterConvolve(bitmap, matrixSharpen) } fun applyFilterConvolve(bitmapIn: Bitmap,

    coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy()
  34. fun applyFilterConvolve(bitmapIn: Bitmap, coefficients: FloatArray): Bitmap { val bitmapOut: Bitmap

    = Bitmap.createBitmap(bitmapIn.width, bitmapIn.height, bitmapIn.config) val allocationIn = Allocation.createFromBitmap(renderScript, bitmapIn) val allocationOut = Allocation.createFromBitmap(renderScript, bitmapOut) convolution.setInput(allocationIn) convolution.setCoefficients(coefficients) convolution.forEach(allocationOut) allocationOut.copyTo(bitmapOut) allocationIn.destroy() allocationOut.destroy() return bitmapOut } fun cleanup(){ renderScript.destroy() } }
  35. Further Examples • Efficient Image Processing - Nicolas Roard https://youtu.be/j2szUDjScOw

    • Basic Renderscript - https:// github.com/googlesamples/android- BasicRenderScript • Gallery2 - Renderscript Examples https://android.googlesource.com/ platform/packages/apps/Gallery2
  36. Pros % • Fast • Highly customisable • Parallel execution

    on GPU • Interactions between layers isn’t complex Cons & • Need to understand a new language • Manage your own memory • Native Libs
  37. How do I choose? ColorFilter / Renderscript / OpenGL? +

    ImageFilterView or Glide Transformations