Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Providing Better Feedback in Real-time Object D...

Providing Better Feedback in Real-time Object Detection Apps

lightning talk at #tryswiftconf

Shinichi Goto

March 02, 2017
Tweet

More Decks by Shinichi Goto

Other Decks in Technology

Transcript

  1. Providing Be,er Feedback in Real-&me Object Detec&on Apps try! Swi)

    Tokyo 2017 Lightning Talk shingt (Shinichi Goto) @ Wantedly, Inc. ɹ
  2. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer

    sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) updateCircleLayersView(with: cardFeatures) ... } }
  3. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer

    sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) ! // Detect cards updateCircleLayersView(with: cardFeatures) ... } }
  4. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer

    sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) updateCircleLayersView(with: cardFeatures) ! // Add CALayers ... } }
  5. Tracking • Assigning consistent labels to objects in videos •

    Generic trackers available in OpenCV • e.g. TLD, KCF, etc. • Performance issues on older iOS devices
  6. Tracking (our case) • Combina)on of lightweight informa)on: • dHash

    (perceptual hash based on gradient) • Posi)on • Op)cal flow
  7. extension DetectionViewController: AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput( _ captureOutput: AVCaptureOutput!, didOutputSampleBuffer

    sampleBuffer: CMSampleBuffer!, ...) { ... guard let frame = sampleBuffer.toUIImage() else { return } let cardFeatures = detector.cardFeatures(in: frame) // // Now we can animate circle layers // from previous positions to current positions // updateCircleLayersView(with: cardFeatures) ! ... } }
  8. Face Tracking using CoreImage open class CIFaceFeature : CIFeature {

    open var bounds: CGRect { get } open var trackingID: Int32 { get } ! ... }
  9. Face Tracking using CoreImage open class CIFaceFeature : CIFeature {

    open var bounds: CGRect { get } open var trackingID: Int32 { get } ! ... } Sample including anima.on: h2ps:/ /github.com/shingt/FaceTracker- sample (Instead of our product code)
  10. Summary • In real-*me object detec*on apps • Difficulty in

    giving feedback differs depending a situa*on • Tracking can improve feedback in some cases