Adaptive Fusion of RGB/NIR Signals based on Face/Background Cross-spectral Analysis for Heart Rate Estimation

Transcript

1. ADAPTIVE FUSION OF RGB/NIR SIGNALS BASED ON FACE/BACKGROUND CROSS-SPECTRAL ANALYSIS FOR HEART RATE ESTIMATION Kosuke Kurihara 1 , Daisuke Sugimura 2 and Takayuki Hamamoto 1 1 Graduate School of Engineering , Tokyo University of Science 2 Department of Computer Science , Tsuda University 3. Experimental results ▮ Experimental setting • 11 subjects • Reference HR : measured using a Pulse Oximeter • Tested with 4 Real RGB/NIR videos : S1 : Bright scene (600 lx) S2 : Low-light scene (0.4 lx) S3 : Varying illumination (1 lx) - frequency near to HR S4: Varying illumination (1 lx) - like a theater S1 S2 S3 S4 3.9 14.4 22.7 25.7 [1] (NIR) 6.7 6.3 4.2 6.8 4.9 7.0 22.6 14.1 Ours 1.9 2.7 2.5 2.6 ▮ Results [1] A. Lam et al., IEEE ICCV 2015 [2] S. Kado et al., IEEE EMBC 2018 [3] E. B. Blackford et al., SPIE BiOS 2018 References ▮ Quantitative evaluation • Mean absolute error (bpm) Video RGB/NIR camera Subject S3 Movie RGB/NIR camera Subject S4 1. Background ▮ Related Works • Using RGB and near-infrared (NIR) videos with NIR flash [2] → Can utilize stable NIR signals independent to illumination variations ▮ Problem : Less robustness against varying illuminations • In Controlled (Bright) scenes : NIR << RGB ∵Unstable NIR reflections owing to light absorption characteristics of vessel [3] • In Uncontrolled scenes : RGB << NIR ∵ Dominant color cast by background illuminations Video Temporal variations R G B N Blood vessel Skin Pulsation HR → It is necessary to adaptively use RGB/NIR signals considering illumination conditions 〇 RGB video × NIR video Controlled scenes × RGB video 〇 NIR video Uncontrolled scenes ▮ Objective • Remote Heart rate (HR) estimation using temporal variations in pixel values owing to cardiac pulse 2. Proposed method ▮ Key idea : Adaptive fusion of RGB/NIR signals considering illuminations conditions • Measuring signal correlations between face and background regions using cross-spectral analysis → Assess which of RGB and NIR signals are more reliable for HR estimation » Controlled scenes : Less correlations between face and background RGB signals → RGB signals are more reliable for HR estimation » Uncontrolled scenes : High correlations due to varying background illuminations → NIR signals are more reliable for HR estimation ▮ Pipeline of our method Background Face ( HR signal ) Controlled scenes : Less correlations Background Face Uncontrolled scenes : High correlations Likelihood State transition Posterior at 𝜏 − 1 Time-series filtering for latent HR ℎ𝜏 RGB/NIR reliability NIR Likelihood RGB Likelihood Adaptive fusion of RGB/NIR likelihood NIR flash Subject RGB/NIR camera RGB NIR RGB video NIR video RGB/NIR reliability computation 𝛽 back by face/background cross-spectral analysis Face Background RGB video Calculation of cross-power spectrum • Calculate the cross-spectrum strength 𝑚 in the HR range of normal person for RGB/NIR reliability computation • High (less) strength → high (less) face/background correlation RGB/NIR reliability : Strength in HR range : Frequency HR range cross-power spectrum Posterior at 𝜏−1 Posterior at 𝜏 Time-series filtering HR at 𝜏 Adaptive Fusion HR Likelihood at 𝜏 NIR Likelihood Computation RGB Likelihood Computation Face/Background Cross-spectral analysis MAP NIR video RGB video Patch Selection [1] (RGB) [2] (RGB/NIR)