Physiological Modeling With Multispectral Imaging for Heart Rate Estimation

Transcript

1. Physiological Modeling With Multispectral Imaging for Heart Rate Estimation Kosuke Kurihara 1 Yoshihiro Maeda 2 Daisuke Sugimura 3 Takayuki Hamamoto 1 1 Tokyo University of Science 2 Tokyo Metropolitan University 3 Shibaura Institute of Technology ICIP 2024 TP1.L5.5
2. ⚫ Video-based non-contact heart rate (HR) estimation • Attracts research attention because of its convenience ⚫ Principle: color change analysis on the facial videos • Blood volume pulse (BVP) associated with the cardiac pulse causes skin color changes → BVP signal and HR can be estimated Non-contact HR estimation using cameras 1 HR: 68 bpm RGB video Temporal variations G B R BVP signal Blood vessel Skin Cardiac pulse Analysis time→
3. ⚫ Video-based HR estimation performance is limited ⚫ BVP signal is low signal-to-noise ratio 1. BVP signal is subtle signal (typically less than 2 bits) 2. Large noise due to fluctuations of ambient illumination Problem of video-based HR estimation 2 Camera Ambient light 1. Subtle BVP 2. Illumination noise Illumination Small amplitude Large amplitude BVP
4. ⚫ Modeling spatio-temporal BVP characteristics improve BVP signal extraction performance [1] • BVP has quasi-periodic temporal characteristics → BVP can be modeled using conservative systems • BVP propagates over the facial region with the same timing → BVP can be modeled as spatially similar Previous method 1: Physiological modeling of BVP 3 video observation time Temporal characteristic Cardiac pulse Spatial characteristic BVP propagation spatial [1] Kurihara+, Spatio-Temporal Structure Extraction of Blood Volume Pulse Using Dynamic Mode Decomposition for Heart Rate Estimation, Access 2023
5. ⚫ Using RGB and near-infrared (NIR) videos • Captured with invisible NIR light → Stable NIR signals independent to illumination variations ⚫ Flexible integration of RGB/NIR videos based on background illumination analysis [2] → Improve performance in a various illumination scene Previous method 2: Ambient illumination modeling 4 RGB video NIR video Controlled illumination RGB video NIR video Uncontrolled illumination [2] Kurihara+, Non-Contact Heart Rate Estimation via Adaptive RGB/NIR Signal Fusion, TIP 2021
6. ⚫ Perform physiological and illumination modeling ⚫ Comprehensive modeling can enhance HR estimation performance Our key idea 5 RGB/NIR camera Ambient Physiological modeling Illumination modeling RGB NIR NIR light
7. ⚫ Bayesian inference-based HR estimation incorporating physiological and illumination model Overview of our algorithm 6 NIR video RGB video Flexible integration based on illumination model RGB/NIR weight calculation patch group NIR HR likelihood computation based on physiological model RGB HR likelihood computation based on physiological model Integrated HR likelihood HR posterior at 𝜏-1 HR posterior at 𝜏 MAP estimation State transition
8. ⚫ Extract RGB/NIR signals from the facial patches • RGB signals: project onto the chrominance space → Mitigate influences of the noise components Obtaining multiple observation signal 7 RGB video RGB Observation RGB signal ←patch ←patch Chrominance analysis[6] B R G NIR video NIR Observation ←patch time→ time→ time→ patches
9. ⚫ HR estimation method based on spatio-temporal BVP modeling (BVPDMD) [1] • Dynamic Mode Decomposition incorporating BVP dynamics → Enable accurate BVP signal extraction HR candidate extraction: overview 8 BVP Temporal dynamics BVP Spatial dynamics spatial time temporal freq. amplitude BVPDMD Mode spectral peak analysis BVP signal & HR candidate Select mode conservative spatially aligned Observation signal [1] Kurihara+, Spatio-Temporal Structure Extraction of Blood Volume Pulse Using Dynamic Mode Decomposition for Heart Rate Estimation, Access 2023
10. ⚫ Perform BVPDMD to neighbor patch group • Obtain RGB/NIR HR candidate & confidence score Calculating HR candidates & confidence scores 9 RGB Observation NIR Observation time→ time→ ←patch ←patch neighbor patch group #1 BVPDMD RGB HR candidate & confidence score #1 … BVPDMD NIR HR candidate & confidence score #1 … temporal freq. amplitude BVPDMD Mode spectral HR candidates BVP signal & HR candidate Confidence score Selected mode
11. ⚫ Compute RGB/NIR HR likelihood using HR candidates & confidence score • Perform weighted kernel density estimation RGB/NIR HR likelihood computation 10 time→ time→ ←patch ←patch … #1 #2 Weighted kernel density estimation RGB HR likelihood … #1 #2 Weighted kernel density estimation NIR HR likelihood RGB NIR
12. ⚫ Correlation analysis between face/background • Assess whether RGB or NIR signals is more reliable [2] » Uncontrolled scene : High correlations • NIR is reliable because of stable NIR light » Controlled scene : Less correlations • RGB is reliable due to light absorption of skin and blood Flexible RGB/NIR integration: overview 11 Face Back Controlled : Less correlations → RGB is reliable Face Back Uncontrolled : High correlations → NIR is reliable [2] Kurihara+, Non-Contact Heart Rate Estimation via Adaptive RGB/NIR Signal Fusion, TIP 2021
13. ⚫ Calculate weight for RGB/NIR integration • Perform Pearson correlation analysis on RGB time series signals in face/background regions • If correlation is high: RGB facial signal is deteriorated by varying illumination → Assign more confidence on NIR likelihood Flexible RGB/NIR integration: calculate flexible weight 12 RGB likelihood (priority in controlled) NIR likelihood (priority in uncontrolled) Flexibly integrated likelihood RGB video G face G back Pearson Correlation Correlation analysis × ×
14. ⚫ Infer latent HR based on MAP estimation • Calculate HR posterior using HR likelihood and prior based on particle filter framework HR estimation based on MAP inference 13 HR HR posterior at 𝜏 − 1 HR posterior at 𝜏 HR prior at 𝜏 State transition RGB/NIR integrated HR likelihood Apply Bayes' theorem MAP estimation
15. ⚫ Experimental setting (4 scenes, 11 subjects) • Bright (600 lx), Low (0.4 lx), Varying (1 lx), Theater (1 lx) ⚫ Comparison methods: • RGB method [3,4,5] • NIR method [6,7] • RGB/NIR method [8,9] Experiments using real data 14 Low High Theater RGB NIR Varying During experiment (Theater) Video [3] Kumar+, DistancePPG: Robust non-contact vital signs monitoring using a camera, Biomedical Optics Express 2015 [4] Tulyakov+, Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions, CVPR 2016 [5] Yu+, Physformer: Facial video-based physiological measurement with temporal difference transformer, CVPR 2022 [6] Martinez+, Non-contact photoplethysmogram and instantaneous heart rate estimation from infrared face video, ICIP 2019 [7] Nowara+, Sparseppg: Towards driver monitoring using camera-based vital signs estimation in near-infrared, CVPRW 2018 [8] Kado+, Remote heart rate measurement from RGB-NIR video based on spatial and spectral face patch selection, EMBC 2018 [9] Liu+, Information-enhanced network for noncontact heart rate estimation from facial videos, TCSVT 2023 RGB/NIR camera subject
16. ⚫ Evaluation metrics: • Mean absolute error [bpm] between ground-truth HRs and estimated HRs Quantitative comparison 15 Bright Low Vary Theater Ave. RGB Distance [3] 5.2 157.3 28.5 35.2 56.6 SAMC [4] 4.1 87.1 23.1 26.9 35.3 Physformer [5] 19.5 74.0 27.9 53.8 43.8 NIR NatalIR [6] 27.6 24.2 26.0 28.1 26.5 SparsePPG [7] 45.4 46.4 59.7 22.6 43.5 RGB/ NIR Random [8] 5.6 13.5 22.3 17.7 14.8 MDEF [9] 40.3 39.5 30.0 38.0 37.0 Ours 2.2 5.5 6.4 6.8 5.2
17. ⚫ Non-contact HR estimation based on physiological and illumination modeling • Comprehensive modeling can enhance performance Summary 16 RGB/NIR camera Ambient Illumination modeling RGB NIR NIR light Physiological modeling