Share
Embed
Through the Looking Glass: Diminishing Occlusions in Robot Vision Systems with Mirror Reflections
IROS 2021
Yoshioka Lab (Keio CSG)
October 26, 2021
Transcript
24 slides
-
1 of 57 Through the Looking Glass: Diminishing Occlusions in
Robot Vision Systems with Mirror Reflections Kentaro Yoshioka1 2, Hidenori Okuni1, Tuan Thanh Ta1 and Akihide Sai1 1. Toshiba, Japan 2. Now Assistant Prof. at Keio University, Japan -
2 of 57 Motivation • Occlusion stand as a large
challenge in robot vision – Danger of overlooking or even damaging the target – Occlusions are even more challenging when multiple robots work under the same environment Depth sensor Occluded! Boxes: Detection results of Mask-RCNN -
3 of 57 Motivation • Occlusion stand as a large
challenge in robot vision – Danger of overlooking or even damaging the target – Occlusions are even more challenging when multiple robots work under the same environment Depth sensor Occluded! Boxes: Detection results of Mask-RCNN -
4 of 57 Reducing occlusions in robot vision systems [Ref.]
D. Holz, IROS 2015. [Ref.] A. Zeng, ICRA 2018. Installing sensor at End-effector Work-efficiency ☺ Cost Installing multiple sensors ☺ Work-efficiency Cost -
5 of 57 Reducing occlusions in robot vision systems Trade-off
between work-efficiency and sensor cost [Ref.] D. Holz, IROS 2015. [Ref.] A. Zeng, ICRA 2018. Installing sensor at End-effector Work-efficiency ☺ Cost Installing multiple sensors ☺ Work-efficiency Cost -
6 of 57 Tilt-type reflection sensing Depth sensor Tilt unit
c c Occluded! c c Tilt! Occlusions diminished Mirror • First robot vision system which diminishes occlusions by sensor tilting and mirror reflections – Objective: Eliminate occlusions under the robot arm – Non-line-of-sight (NLoS) sensing by mirror Direct sensing Reflection sensing -
7 of 57 Tilt-type reflection sensing Depth sensor Tilt unit
c c Occluded! c c Tilt! Occlusions diminished Mirror • First robot vision system which diminishes occlusions by sensor tilting and mirror reflections – Pros: Low-cost hardware (mirror+tilt-unit) – Pros: Tilting faster than end-effector configuration -
8 of 57 Tilt-type reflection sensing Direct sensing Direct sensing
depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit c c -
9 of 57 Tilt-type reflection sensing Direct sensing Direct sensing
depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit c c Tomato missing ! -
10 of 57 Tilt-type reflection sensing Direct sensing Direct sensing
depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Detected occlusion area Depth sensor Tilt unit c c -
11 of 57 Tilt-type reflection sensing Direct sensing Direct sensing
depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Detected occlusion area Target Xt “Virtual” target Xtv Depth sensor Tilt unit c c 𝑯𝒎 −𝟏 -
12 of 57 Tilt-type reflection sensing Direct sensing Direct sensing
depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit Detected occlusion area Target Xt “Virtual” target Xtv c c qtilt 𝑯𝒎 −𝟏 -
13 of 57 Tilt-type reflection sensing Direct sensing Direct sensing
depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Tilt angle Virtual image correction Camera trans. Mirror Depth sensor Tilt unit Detected occlusion area Target Xt “Virtual” target Xtv c c qtilt Occlusion areas detected on-the-fly Adaptively remove occlusions 𝑯𝒎 −𝟏 -
14 of 57 Tilt-type reflection sensing Direct sensing Direct sensing
depth data Reflection sensing depth data Occlusion detection Calculate tilt angle Reflection sensing + Virtual image correction Camera trans. Mirror Depth sensor Tilt unit Detected occlusion area Target Xt “Virtual” target Xtv c c qtilt Concatenated results Tomato recovered ! Occlusion areas detected on-the-fly Adaptively remove occlusions 𝑯𝒎 −𝟏 -
15 of 57 Occlusion detection • Occlusions detected based on
height-thresholds – Simple, yet fast and effective rule-based approach. • DNN methods are more general but time-consuming – Sensing time = 𝟐 × 𝒕𝒔𝒆𝒏𝒔𝒐𝒓 + 𝒕𝒕𝒊𝒍𝒕 + 𝒕𝒅𝒆𝒕𝒆𝒄𝒕 Centroid Area[pix2] (-0.2, -0.1) 1400 (0.3, -0.1) 30 Occlusion centroid Sensor noise Segment area which exceeds height threshold Find contours and convert to centroid and area Largest area classified as occlusions Target Xt -
16 of 57 Occlusion detection • Occlusions detected based on
height-thresholds – Simple, yet fast and effective rule-based approach. • DNN methods are more general but time-consuming – Sensing time = 𝟐 × 𝒕𝒔𝒆𝒏𝒔𝒐𝒓 + 𝒕𝒕𝒊𝒍𝒕 + 𝒕𝒅𝒆𝒕𝒆𝒄𝒕 Centroid Area[pix2] (-0.2, -0.1) 1400 (0.3, -0.1) 30 Occlusion centroid Sensor noise Segment area which exceeds height threshold Find contours and convert to centroid and area Largest area classified as occlusions Target Xt -
17 of 57 Calculating optimal θtilt • θtilt calculated based
on centroid of the occlusion area – 𝑿𝒕𝒗 = 𝑯𝒎 −𝟏𝑿𝒕 – 𝛉𝒕𝒊𝒍𝒕 = 𝐚𝐫𝐜𝐭𝐚𝐧(𝒚𝒕𝒗−𝒚𝒔 𝒙𝒕𝒗−𝒙𝒔 ) Mirror Depth sensor Tilt unit Xt “Virtual” Xtv c c qtilt 𝑯𝒎 −𝟏 -
18 of 57 Calculating optimal θtilt • θtilt calculated based
on centroid of the occlusion area – 𝑿𝒕𝒗 = 𝑯𝒎 −𝟏𝑿𝒕 – 𝛉𝒕𝒊𝒍𝒕 = 𝐚𝐫𝐜𝐭𝐚𝐧(𝒚𝒕𝒗−𝒚𝒔 𝒙𝒕𝒗−𝒙𝒔 ) 𝑯𝒎 −𝟏 Householder transformation Mirror Depth sensor Tilt unit Xt “Virtual” Xtv c c qtilt -
19 of 57 Meas. errors for reflection sensing • Meas.
error is proportional to the working distance(WD) – Reflection sensing’s WD is × 𝟏 𝒄𝒐𝒔𝜽𝒕𝒊𝒍𝒕 longer than direct sensing – Error of our sensor (EnsensoN35) was proportional to the square of the distance. • e.g. at 𝜽𝒕𝒊𝒍𝒕=45deg., meas. error become x2 larger -
20 of 57 Experiment Setup Depth sensor (ENSENSO N35) Tilt
unit (FLIR PUT-E46) Off-the-shelf Mirror Robot arm (DENSO COBOTTA) -
21 of 57 Two sensing schemes are rapidly switched by
tilting the zenith sensor. Occlusions are greatly reduced, which boosts detection accuracy. Direct only Direct + Reflection Speed: x2 Experiments -
22 of 57 Reflection sensing results are almost equal to
results of two sensors. The evaluated mAP was almost the same (1% difference). Results with direct + reflection Results with Two sensors Experiments -
23 of 57 Our proposed system detects the occlusion in
the scene and decides scan angles adaptively. Sensor rearrangements are not required after changes in robot operation or layout. Adaptive sensing Fixed angles Speed: x2 Experiments -
24 of 57 Conclusion • Proposed Tilt-type mirror reflection sensing
– Showed that occlusions can be adaptively eliminated with mirror reflections and sensor tilting. – Perceptions close to a two-sensor setup were obtained. • Future works – Adaptation to complex environments with multiple robots – Extension to multiple mirrors