Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to effectively scan barcodes on Android

How to effectively scan barcodes on Android

Android does not provide a native solution to detect barcodes in your application. However, some barcode image processing libraries are available but it is not always easy to use it for top performance.

Through this talk, we will see how to implement barcode detection in your Android application from camera management to optimization of barcode recognition thanks to machine learning.

Renaud MATHIEU

January 29, 2020
Tweet

More Decks by Renaud MATHIEU

Other Decks in Programming

Transcript

  1. Thesis of Wallace Flint 1932 Norman Woodland and Bernard Silver

    patented the barcode idea 1949 Philco purchased the barcode patent then sold it to RCA 1952 UPC 1973 First produc scanned at a check-out counter 1974
  2. Philco purchased the barcode patent then sold it to RCA

    1952 UPC 1973 First product scanned at a check-out counter 1974 EAN 1975 DataMatrix 1993 QR Code 1994
  3. Linear Stacked 2D EAN8 EAN13 UPC Code 11 Code 39

    Code 93 PDF 417 Code 49 DataMatrix QR Code Aztec
  4. •Add the digits in the odd-numbered positions together and multiply

    the total by three 6+2+7+9+1+6=31 31x3=93 •Add the digits in the even-numbered positions 9+7+1+8+1=26 •Add the two results together 93+26=119 •Now what single digit number makes the total a multiple of 10? That’s the check digit. 119 + 1 = 120 Data = 69277198116
  5. implementation 'androidx.lifecycle:lifecycle-runtime-ktx:2.2.0-rc03' implementation 'androidx.camera:camera-core:1.0.0-alpha09' implementation 'androidx.camera:camera-camera2:1.0.0-alpha09' implementation 'androidx.camera:camera-lifecycle:1.0.0-alpha02' implementation 'androidx.camera:camera-view:1.0.0-alpha05'

    implementation 'com.google.firebase:firebase-analytics:17.2.2' implementation 'com.google.firebase:firebase-ml-vision:24.0.1' implementation 'com.google.firebase:firebase-ml-vision-barcode-model:16.0.2' implementation 'com.otaliastudios:cameraview:2.4.0' implementation 'com.google.zxing:core:3.3.3' implementation 'io.reactivex.rxjava2:rxjava:2.2.9' implementation 'io.reactivex.rxjava2:rxandroid:2.1.1'
  6. enum class Symbology(val value: String) { AZTEC("aztec"), CODE128("code128"), CODE39("code39"), CODE93("code93"),

    CODABAR("codabar"), DATAMATRIX("datamatrix"), EAN13("ean13"), EAN8("ean8"), ITF14("itf14"), QR("qr"), UPCA("upca"), UPCE("upce"), PDF417("pdf417"); } Models
  7. class OCameraView @JvmOverloads constructor( context: Context, attrs: AttributeSet? = null

    ) : CameraView(context, attrs) { private var isPaused: Boolean = false private val frameProcessor = FrameProcessor { if (!isPaused) { "// I’ve got a new frame } } init { setLifecycleOwner(context as LifecycleOwner) }
  8. } init { setLifecycleOwner(context as LifecycleOwner) } @OnLifecycleEvent(Lifecycle.Event.ON_RESUME) private fun

    resumeScan() { isPaused = false addFrameProcessor(frameProcessor) } @OnLifecycleEvent(Lifecycle.Event.ON_PAUSE) private fun pauseScan() { isPaused = true removeFrameProcessor(frameProcessor) }
  9. !!/** * MultiFormatReader is a convenience class and the main

    entry point into the library for most uses. * By default it attempts to decode all barcode formats that the library supports. Optionally, you * can provide a hints object to request different behavior, for example only decoding QR codes. * * @author Sean Owen * @author [email protected] (Daniel Switkin) !*/ public final class MultiFormatReader implements Reader { … }
  10. !!/** * Decode an image using the state set up

    by calling setHints() previously. Continuous scan * clients will get a <b>large!</b> speed increase by using this instead of decode(). * * @param image The pixel data to decode * @return The contents of the image * @throws NotFoundException Any errors which occurred !*/ public Result decodeWithState(BinaryBitmap image) throws NotFoundException { "// Make sure to set up the default state so we don't crash if (readers "== null) { setHints(null); } return decodeInternal(image); }
  11. !!/** * This class is the core bitmap class used

    by ZXing to represent 1 bit data. Reader objects * accept a BinaryBitmap and attempt to decode it. * * @author [email protected] (Daniel Switkin) !*/ public final class BinaryBitmap { private final Binarizer binarizer; private BitMatrix matrix; public BinaryBitmap(Binarizer binarizer) { if (binarizer "== null) { throw new IllegalArgumentException("Binarizer must be non-null."); } this.binarizer = binarizer; }
  12. !!/** * This class implements a local thresholding algorithm, which

    while slower than the * GlobalHistogramBinarizer, is fairly efficient for what it does. It is designed for * high frequency images of barcodes with black data on white backgrounds. For this application, * it does a much better job than a global blackpoint with severe shadows and gradients. * However it tends to produce artifacts on lower frequency images and is therefore not * a good general purpose binarizer for uses outside ZXing. * * This class extends GlobalHistogramBinarizer, using the older histogram approach for 1D readers, * and the newer local approach for 2D readers. 1D decoding using a per-row histogram is already * inherently local, and only fails for horizontal gradients. We can revisit that problem later, * but for now it was not a win to use local blocks for 1D. * * This Binarizer is the default for the unit tests and the recommended class for library users. * * @author [email protected] (Daniel Switkin) !*/ public final class HybridBinarizer extends GlobalHistogramBinarizer { public HybridBinarizer(LuminanceSource source) { super(source); } }
  13. !!/** * The purpose of this class hierarchy is to

    abstract different bitmap implementations across * platforms into a standard interface for requesting greyscale luminance values. The interface * only provides immutable methods; therefore crop and rotation create copies. This is to ensure * that one Reader does not modify the original luminance source and leave it in an unknown state * for other Readers in the chain. * * @author [email protected] (Daniel Switkin) !*/ public abstract class LuminanceSource { private final int width; private final int height; protected LuminanceSource(int width, int height) { this.width = width; this.height = height; }
  14. !!/** * This object extends LuminanceSource around an array of

    YUV data returned from the camera driver, * with the option to crop to a rectangle within the full data. This can be used to exclude * superfluous pixels around the perimeter and speed up decoding. * * It works for any pixel format where the Y channel is planar and appears first, including * YCbCr_420_SP and YCbCr_422_SP. * * @author [email protected] (Daniel Switkin) !*/ public final class PlanarYUVLuminanceSource extends LuminanceSource {
  15. private fun detectInImageSingleOrientation( data: ByteArray, width: Int, height: Int, rotation:

    Int ): Result? { val source = getPlanarYUVLuminanceSource(data, width, height, rotation) val bitmap = BinaryBitmap(HybridBinarizer(source)) var result: Result? = null try { result = multiFormatReader.decodeWithState(bitmap) } catch (re: NotFoundException) { } finally { multiFormatReader.reset() } return result }
  16. private fun detectInImage(data: ByteArray, width: Int, height: Int): Observable<Result> =

    Observable.create { emitter "-> val result = detectInImageSingleOrientation(data, width, height, ROTATION_90) "?: detectInImageSingleOrientation(data, width, height, ROTATION_0) if (result "!= null) { emitter.onNext(result) } }
  17. class ZXingImageAnalyzer : BarcodeAnalyzer { companion object { const val

    ROTATION_0 = 0 const val ROTATION_90 = 90 } private val multiFormatReader: MultiFormatReader = MultiFormatReader() val onFrameProcessorPublisher: PublishSubject<Frame> = PublishSubject.create() override fun onBarcodeScanned(): Observable<BarcodeModel> = onFrameProcessorPublisher .flatMap { detectInImage(it.data, it.size.width, it.size.height) } .map { BarcodeModel(it.text, Symbology.CODE128) }
  18. class OCameraView @JvmOverloads constructor( context: Context, attrs: AttributeSet? = null

    ) : CameraView(context, attrs), BarcodeProcessor { private var isPaused: Boolean = false private val zxingImageAnalyzer = ZXingImageAnalyzer() private val frameProcessor = FrameProcessor { if (!isPaused) { zxingImageAnalyzer.onFrameProcessorPublisher.onNext(it) } }
  19. The CameraX library is in alpha stage, as its API

    surfaces aren't yet finalized.
  20. CameraX is an addition to Jetpack that makes it easier

    to leverage the capabilities of Camera2 APIs.
  21. !!/** * Convenience class for generating a pre-populated Camera2 {@link

    CameraXConfig}. !*/ public final class Camera2Config { private Camera2Config() { } !!/** * Creates a {@link CameraXConfig} containing the default Camera2 implementation for CameraX. !*/ @NonNull public static CameraXConfig defaultConfig() { "// Create the camera factory for creating Camera2 camera objects CameraFactory.Provider cameraFactoryProvider = Camera2CameraFactory"::new; "// Create the DeviceSurfaceManager for Camera2 CameraDeviceSurfaceManager.Provider surfaceManagerProvider = Camera2DeviceSurfaceManager"::new; "// Create default configuration factory UseCaseConfigFactory.Provider configFactoryProvider = context "-> { ExtendableUseCaseConfigFactory factory = new ExtendableUseCaseConfigFactory(); factory.installDefaultProvider( ImageAnalysisConfig.class, new ImageAnalysisConfigProvider(context));
  22. CameraDeviceSurfaceManager.Provider surfaceManagerProvider = Camera2DeviceSurfaceManager"::new; "// Create default configuration factory UseCaseConfigFactory.Provider

    configFactoryProvider = context "-> { ExtendableUseCaseConfigFactory factory = new ExtendableUseCaseConfigFactory(); factory.installDefaultProvider( ImageAnalysisConfig.class, new ImageAnalysisConfigProvider(context)); factory.installDefaultProvider( ImageCaptureConfig.class, new ImageCaptureConfigProvider(context)); factory.installDefaultProvider( VideoCaptureConfig.class, new VideoCaptureConfigProvider(context)); factory.installDefaultProvider( PreviewConfig.class, new PreviewConfigProvider(context)); return factory; }; CameraXConfig.Builder appConfigBuilder = new CameraXConfig.Builder() .setCameraFactoryProvider(cameraFactoryProvider) .setDeviceSurfaceManagerProvider(surfaceManagerProvider) .setUseCaseConfigFactoryProvider(configFactoryProvider); return appConfigBuilder.build(); } }
  23. class XCameraView @JvmOverloads constructor( context: Context, attrs: AttributeSet? = null

    ) : PreviewView(context, attrs), BarcodeProcessor { private var cameraProviderFuture: ListenableFuture<ProcessCameraProvider> = ProcessCameraProvider.getInstance(context) private val cameraSelector: CameraSelector = CameraSelector.Builder() .requireLensFacing(CameraSelector.LENS_FACING_BACK) .build() init { cameraProviderFuture.addListener( Runnable { bindToLifecycle(cameraProviderFuture.get()) }, ContextCompat.getMainExecutor(context) ) }
  24. public Camera bindToLifecycle(@NonNull LifecycleOwner lifecycleOwner, @NonNull CameraSelector cameraSelector, @NonNull UseCase""...

    useCases) { return CameraX.bindToLifecycle(lifecycleOwner, cameraSelector, useCases); }
  25. private fun bindToLifecycle(cameraProvider: ProcessCameraProvider) { val preview: Preview = Preview.Builder()

    .setTargetName("Preview") .build() preview.previewSurfaceProvider = previewSurfaceProvider cameraProvider.bindToLifecycle( context as LifecycleOwner, cameraSelector, preview ) }
  26. ML Kit's barcode scanning API •Reads most standard formats •Automatic

    format detection •Extracts structured data •Works with any orientation •Runs on the device
  27. class FirebaseImageAnalyzer : BarcodeAnalyzer { private val onBarcodeScannedPublisher: PublishSubject<BarcodeModel> =

    PublishSubject.create() override fun onBarcodeScanned(): Observable<BarcodeModel> = onBarcodeScannedPublisher
  28. class FirebaseImageAnalyzer : BarcodeAnalyzer { private val options = FirebaseVisionBarcodeDetectorOptions.Builder()

    .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_ALL_FORMATS) .build() private val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) private val onBarcodeScannedPublisher: PublishSubject<BarcodeModel> = PublishSubject.create() override fun onBarcodeScanned(): Observable<BarcodeModel> = onBarcodeScannedPublisher
  29. class FirebaseImageAnalyzer : ImageAnalysis.Analyzer, BarcodeAnalyzer { private val options =

    FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_ALL_FORMATS) .build() private val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) private val onBarcodeScannedPublisher: PublishSubject<BarcodeModel> = PublishSubject.create() override fun analyze(image: ImageProxy) { } override fun onBarcodeScanned(): Observable<BarcodeModel> = onBarcodeScannedPublisher private fun degreesToFirebaseRotation(degrees: Int): Int = when (degrees) { 0 "-> FirebaseVisionImageMetadata.ROTATION_0 90 "-> FirebaseVisionImageMetadata.ROTATION_90 180 "-> FirebaseVisionImageMetadata.ROTATION_180 270 "-> FirebaseVisionImageMetadata.ROTATION_270 else "-> throw Exception("Rotation must be 0, 90, 180, or 270.") }
  30. private fun bindToLifecycle(cameraProvider: ProcessCameraProvider) { val preview: Preview = Preview.Builder()

    .setTargetName("Preview") .build() preview.previewSurfaceProvider = previewSurfaceProvider val imageAnalysis = ImageAnalysis.Builder() .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) .setTargetResolution((Size(1280, 720))) .build() imageAnalysis.setAnalyzer( Executors.newSingleThreadExecutor(), firebaseImageAnalyzer ) cameraProvider.bindToLifecycle( context as LifecycleOwner, cameraSelector, preview, imageAnalysis ) }
  31. override fun analyze(image: ImageProxy) { val mediaImage = image.image val

    imageRotation = degreesToFirebaseRotation(image.imageInfo.rotationDegrees) if (mediaImage "!= null) { val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { barcodes "-> if (barcodes.isEmpty()) return@addOnSuccessListener for (barcode in barcodes) { barcode.displayValue"?.let { value "-> val model = BarcodeModel( value = value, symbology = barcodeFormatToSymbology(barcode.format) ) onBarcodeScannedPublisher.onNext(model) } } } .addOnFailureListener { onBarcodeScannedPublisher.onError(it) } } image.close() }
  32. override fun analyze(image: ImageProxy) { val mediaImage = image.image val

    imageRotation = degreesToFirebaseRotation(image.imageInfo.rotationDegrees) if (mediaImage "!= null) { val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { barcodes "-> if (barcodes.isEmpty()) return@addOnSuccessListener for (barcode in barcodes) { barcode.displayValue"?.let { value "-> val model = BarcodeModel( value = value, symbology = barcodeFormatToSymbology(barcode.format) ) onBarcodeScannedPublisher.onNext(model) } } } .addOnFailureListener { onBarcodeScannedPublisher.onError(it) } } image.close() }
  33. override fun analyze(image: ImageProxy) { val mediaImage = image.image val

    imageRotation = degreesToFirebaseRotation(image.imageInfo.rotationDegrees) if (mediaImage "!= null) { val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { barcodes "-> if (barcodes.isEmpty()) return@addOnSuccessListener for (barcode in barcodes) { barcode.displayValue"?.let { value "-> val model = BarcodeModel( value = value, symbology = barcodeFormatToSymbology(barcode.format) ) onBarcodeScannedPublisher.onNext(model) } } } .addOnFailureListener { onBarcodeScannedPublisher.onError(it) } } image.close() }
  34. override fun analyze(image: ImageProxy) { val mediaImage = image.image val

    imageRotation = degreesToFirebaseRotation(image.imageInfo.rotationDegrees) if (mediaImage "!= null) { val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { barcodes "-> if (barcodes.isEmpty()) return@addOnSuccessListener for (barcode in barcodes) { barcode.displayValue"?.let { value "-> val model = BarcodeModel( value = value, symbology = barcodeFormatToSymbology(barcode.format) ) onBarcodeScannedPublisher.onNext(model) } } } .addOnFailureListener { onBarcodeScannedPublisher.onError(it) } } image.close() }
  35. !!/** * Only deliver the latest image to the analyzer,

    dropping images as they arrive. * * <p>This strategy ignores the value set by {@link Builder#setImageQueueDepth(int)}. * Only one image will be delivered for analysis at a time. If more images are produced * while that image is being analyzed, they will be dropped and not queued for delivery. * Once the image being analyzed is closed by calling {@link ImageProxy#close()}, the * next latest image will be delivered. * * <p>Internally this strategy may make use of an internal {@link Executor} to receive * and drop images from the producer. A performance-tuned executor will be created * internally unless one is explicitly provided by * {@link Builder#setBackgroundExecutor(Executor)}. In order to * ensure smooth operation of this backpressure strategy, any user supplied * {@link Executor} must be able to quickly respond to tasks posted to it, so setting * the executor manually should only be considered in advanced use cases. * * @see Builder#setBackgroundExecutor(Executor) !*/ public static final int STRATEGY_KEEP_ONLY_LATEST = 0;
  36. class ScanActivity : AppCompatActivity() { companion object { const val

    VIBRATION_DURATION: Long = 250 } private val compositeDisposable = CompositeDisposable() override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setContentView(R.layout.activity_main) compositeDisposable.add(previewView.onBarcodeScanned() .subscribeOn(Schedulers.io()) .observeOn(AndroidSchedulers.mainThread()) .subscribe( { vibrate() }, { Log.e("ScanActivity", "Scanning error", it) } ) ) }
  37. <?xml version="1.0" encoding="utf-8"?> <androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http:"// schemas.android.com/apk/res/android" xmlns:app="http:"//schemas.android.com/apk/res-auto" xmlns:tools="http:"//schemas.android.com/tools" android:layout_width="match_parent" android:layout_height="match_parent"

    tools:context=".ScanActivity"> <com.alephom.android.scan.views.XCameraView android:id="@+id/previewView" android:layout_width="wrap_content" android:layout_height="wrap_content" app:layout_constraintBottom_toBottomOf="parent" app:layout_constraintEnd_toEndOf="parent" app:layout_constraintStart_toStartOf="parent" app:layout_constraintTop_toTopOf="parent" "/> "</androidx.constraintlayout.widget.ConstraintLayout>
  38. <?xml version="1.0" encoding="utf-8"?> <androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http:"// schemas.android.com/apk/res/android" xmlns:app="http:"//schemas.android.com/apk/res-auto" xmlns:tools="http:"//schemas.android.com/tools" android:layout_width="match_parent" android:layout_height="match_parent"

    tools:context=".ScanActivity"> <com.alephom.android.scan.views.OCameraView android:id="@+id/previewView" android:layout_width="wrap_content" android:layout_height="wrap_content" app:cameraAudio="off" app:layout_constraintBottom_toBottomOf="parent" app:layout_constraintEnd_toEndOf="parent" app:layout_constraintStart_toStartOf="parent" app:layout_constraintTop_toTopOf="parent" "/> "</androidx.constraintlayout.widget.ConstraintLayout>
  39. Reading barcodes with neural networks Department of Electrical Engineering Linköping

    University SE-581 83 Linköping, Sweden Copyright © 2017 Fredrik Fridborn