[IROS2020] Non-overlapping RGB-D Camera Network Calibration with Monocular Visual Odometry

Non-overlapping RGB-D Camera Network Calibration with Monocular Visual Odometry
Kenji Koide and Emanuele Menegatti
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2020)

Transcript

1. Non-overlapping RGB-D Camera Network Calibration with Monocular Visual Odometry Kenji Koide* and Emanuele Menegatti** * National Institute of Advanced Industrial Science and Technology , Japan ** University of Padova, Italy Code available at: https://github.com/koide3/sparse_dynamic_calibration
2. Camera Network [Cho 2017] [Karasev, 2017] [Byeon, 2015] • Surveillance • Behavior analysis • HMI systems
3. Camera network-based HMI framework http://openptrack.org/ UCLA Remap Used for R&D and education projects - People position and skeleton tracking - Face and gesture recognition - Arbitrary object tracking
4. Camera Network Calibration
5. Camera Network Calibration /world
6. Camera Network Calibration /world
7. Camera Network Calibration /world
8. Traditional Calibration Method M. Munaro et al., “OpenPTrack: Open source multi-camera calibration and people tracking for RGB-D camera networks”, RAS 2016 [Munaro 2016] 1. Print out a large chessboard 2. Show the board to several cameras 3. Move the board to a new position 4. Repeat 2 and 3 5. Optimize the pose graph
9. Traditional Calibration Method M. Munaro et al., “OpenPTrack: Open source multi-camera calibration and people tracking for RGB-D camera networks”, RAS 2016 [Munaro 2016] 1. Print out a large chessboard 2. Show the board to several cameras 3. Move the board to a new position 4. Repeat 2 and 3 5. Optimize the pose graph
10. Traditional Calibration Method M. Munaro et al., “OpenPTrack: Open source multi-camera calibration and people tracking for RGB-D camera networks”, RAS 2016 [Munaro 2016] 1. Print out a large chessboard 2. Show the board to several cameras 3. Move the board to a new position 4. Repeat 2 and 3 5. Optimize the pose graph
11. Traditional Calibration Method M. Munaro et al., “OpenPTrack: Open source multi-camera calibration and people tracking for RGB-D camera networks”, RAS 2016 [Munaro 2016] 1. Print out a large chessboard 2. Show the board to several cameras 3. Move the board to a new position 4. Repeat 2 and 3 5. Optimize the pose graph
12. Traditional Calibration Method M. Munaro et al., “OpenPTrack: Open source multi-camera calibration and people tracking for RGB-D camera networks”, RAS 2016 [Munaro 2016] 1. Print out a large chessboard 2. Show the board to several cameras 3. Move the board to a new position 4. Repeat 2 and 3 5. Optimize the pose graph
13. Traditional Calibration Method M. Munaro et al., “OpenPTrack: Open source multi-camera calibration and people tracking for RGB-D camera networks”, RAS 2016 [Munaro 2016] 1. Print out a large chessboard 2. Show the board to several cameras 3. Move the board to a new position 4. Repeat 2 and 3 5. Optimize the pose graph
14. Traditional Calibration Method M. Munaro et al., “OpenPTrack: Open source multi-camera calibration and people tracking for RGB-D camera networks”, RAS 2016 [Munaro 2016] 1. Print out a large chessboard 2. Show the board to several cameras 3. Move the board to a new position 4. Repeat 2 and 3 5. Optimize the pose graph
15. Problems 1. Limited camera arrangement and calibration accuracy 2. Requires large overlap between cameras 3. Impossible to calibrate non-overlapping cameras
16. Problems 1. Limited camera arrangement and calibration accuracy 2. Requires large overlap between cameras 3. Impossible to calibrate non-overlapping cameras A few meters distance CAM1 CAM2 A0 pattern becomes very small in a distant camera view • Distance between cameras must be < 5m • Deteriorated PnP pose estimation accuracy
17. Problems 1. Limited camera arrangement and calibration accuracy 2. Requires large overlap between cameras 3. Impossible to calibrate non-overlapping cameras A few meters distance CAM1 CAM2 A0 pattern becomes very small in a distant camera view • Distance between cameras must be < 5m • Deteriorated PnP pose estimation accuracy
18. Problems 1. Limited camera arrangement and calibration accuracy 2. Requires large overlap between cameras 3. Impossible to calibrate non-overlapping cameras A few meters distance CAM1 CAM2 A0 pattern becomes very small in a distant camera view • Distance between cameras must be < 5m • Deteriorated PnP pose estimation accuracy Overlap (Accuracy) Coverage Tradeoff Small overlap results in limited pattern pose variation Worse calibration accuracy
19. Problems 1. Limited camera arrangement and calibration accuracy 2. Requires large overlap between cameras 3. Impossible to calibrate non-overlapping cameras A few meters distance CAM1 CAM2 A0 pattern becomes very small in a distant camera view • Distance between cameras must be < 5m • Deteriorated PnP pose estimation accuracy Overlap (Accuracy) Coverage Tradeoff Small overlap results in limited pattern pose variation Worse calibration accuracy All cameras must have common view Inter-room camera arrangement is difficult
20. VO-based Online Calibration We use visual odometry on a dynamic support camera to “bridge” separated camera views 𝐶0 𝐶1 𝐶2 𝑀0 𝑀1 𝑀2 1. Place fiducial tags (e.g., Apriltag) 2. Show tags to the dynamic camera 3. Estimate 𝐶𝑖 , 𝑀𝑗 , 𝑉𝑘
21. VO-based Online Calibration We use visual odometry on a dynamic support camera to “bridge” separated camera views 𝐶0 𝐶1 𝐶2 𝑀0 𝑀1 𝑀2 𝑉0 𝑉𝑡 1. Place fiducial tags (e.g., Apriltag) 2. Show tags to the dynamic camera 3. Estimate 𝐶𝑖 , 𝑀𝑗 , 𝑉𝑘
22. VO-based Online Calibration We use visual odometry on a dynamic support camera to “bridge” separated camera views 𝐶0 𝐶1 𝐶2 𝑀0 𝑀1 𝑀2 𝑉0 𝑉𝑡 1. Place fiducial tags (e.g., Apriltag) 2. Show tags to the dynamic camera 3. Estimate 𝐶𝑖 , 𝑀𝑗 , 𝑉𝑘 Landmark-based Pose Graph SLAM
23. VO-based Online Calibration We use visual odometry on a dynamic support camera to “bridge” separated camera views 𝐶0 𝐶1 𝐶2 𝑀0 𝑀1 𝑀2 𝑉0 𝑉𝑡 1. Place fiducial tags (e.g., Apriltag) 2. Show tags to the dynamic camera 3. Estimate 𝐶𝑖 , 𝑀𝑗 , 𝑉𝑘 Landmark-based Pose Graph SLAM
24. VO-based Online Calibration We use visual odometry on a dynamic support camera to “bridge” separated camera views 𝐶0 𝐶1 𝐶2 𝑀0 𝑀1 𝑀2 𝑉0 𝑉𝑡 1. Place fiducial tags (e.g., Apriltag) 2. Show tags to the dynamic camera 3. Estimate 𝐶𝑖 , 𝑀𝑗 , 𝑉𝑘 Landmark-based Pose Graph SLAM
25. VO-based Online Calibration We use visual odometry on a dynamic support camera to “bridge” separated camera views 𝐶0 𝐶1 𝐶2 𝑀0 𝑀1 𝑀2 𝑉0 𝑉𝑡 1. Place fiducial tags (e.g., Apriltag) 2. Show tags to the dynamic camera 3. Estimate 𝐶𝑖 , 𝑀𝑗 , 𝑉𝑘 Landmark-based Pose Graph SLAM
26. VO-based Online Calibration We use visual odometry on a dynamic support camera to “bridge” separated camera views 𝐶0 𝐶1 𝐶2 𝑀0 𝑀1 𝑀2 𝑉0 𝑉𝑡 1. Place fiducial tags (e.g., Apriltag) 2. Show tags to the dynamic camera 3. Estimate 𝐶𝑖 , 𝑀𝑗 , 𝑉𝑘 Landmark-based Pose Graph SLAM
27. VO-based Online Calibration We use visual odometry on a dynamic support camera to “bridge” separated camera views 1. Place fiducial tags (e.g., Apriltag) 2. Show tags to the dynamic camera 3. Estimate 𝐶𝑖 , 𝑀𝑗 , 𝑉𝑘 𝐶0 𝐶1 𝐶2 𝑀0 𝑀1 𝑀2 𝑉0 𝑉𝑡 Landmark-based Pose Graph SLAM
28. Inter-room Camera Network Calibration Visual odometry Fiducial tags Graph optimization : Direct Sparse Odometry [Engel, 2017] : Apriltag2 [Wang, 2016] : Levenberg-Marquardt in g2o [Kuemmerle, 2011] Start
29. Depth-based Calibration Refinement Online calibration bridges separated cameras But calibration accuracy relies on VO... can be deteriorated in feature-less env Online calibration bridges separated cameras
30. Depth-based Calibration Refinement Online calibration bridges separated cameras But calibration accuracy relies on VO... can be deteriorated in feature-less env Online calibration bridges separated cameras Use depth information to refine the result
31. Depth-based Calibration Refinement Online calibration bridges separated cameras But calibration accuracy relies on VO... can be deteriorated in feature-less env Online calibration bridges separated cameras • Floor plane constraint • ICP constraint Use depth information to refine the result
32. Depth-based Constraints Floor plane constraint - Detect the floor plane using RANSAC - Align the camera poses such that the detected floor planes become identical (or parallel)
33. Depth-based Constraints Floor plane constraint - Detect the floor plane using RANSAC - Align the camera poses such that the detected floor planes become identical (or parallel)
34. Depth-based Constraints Floor plane constraint - Detect the floor plane using RANSAC - Align the camera poses such that the detected floor planes become identical (or parallel)
35. Depth-based Constraints Floor plane constraint - Detect the floor plane using RANSAC - Align the camera poses such that the detected floor planes become identical (or parallel) ICP constraint - Add constraints between closest points - Optimize the graph - Remove ICP constraints - Repeat the above process
36. Depth-based Constraints Floor plane constraint - Detect the floor plane using RANSAC - Align the camera poses such that the detected floor planes become identical (or parallel) ICP constraint - Add constraints between closest points - Optimize the graph - Remove ICP constraints - Repeat the above process
37. Depth-based Constraints Floor plane constraint - Detect the floor plane using RANSAC - Align the camera poses such that the detected floor planes become identical (or parallel) ICP constraint - Add constraints between closest points - Optimize the graph - Remove ICP constraints - Repeat the above process
38. Depth-based Constraints Floor plane constraint - Detect the floor plane using RANSAC - Align the camera poses such that the detected floor planes become identical (or parallel) ICP constraint - Add constraints between closest points - Optimize the graph - Remove ICP constraints - Repeat the above process Constraints created in the preceded steps are kept No large overlap is required
39. Simulation Experiment Gazebo simulator Lenz distortion noise is injected to simulate deteriorated visual odometry Arrangement1 Arrangement2
40. Simulation Experiment Separated camera groups are bridged by VO vertices via tag vertices Depth-based refinement results Before After Constructed pose graph
41. Simulation Experiment Arrangement1 (Sim1) Arrangement2 (Sim2)
42. Simulation Experiment Arrangement1 (Sim1) Arrangement2 (Sim2)
43. Simulation Experiment Arrangement1 (Sim1) Arrangement2 (Sim2)
44. Simulation Experiment Arrangement1 (Sim1) Arrangement2 (Sim2) Depth-based refinement effectively compensates for VO errors
45. Experiment in a real environment GT measured using FARO Arrangement1 • ~ 6mm accuracy achieved • Traditional one deteriorated with small overlap • Proposed method retained the accuracy
46. Experiment in a real environment Arrangement1 (Small overlap) • ~ 6mm accuracy achieved • Traditional one deteriorated with small overlap • Proposed method retained the accuracy GT measured using FARO
47. Experiment in a real environment Arrangement1 (Small overlap) • ~ 6mm accuracy achieved • Traditional one deteriorated with small overlap • Proposed method retained the accuracy GT measured using FARO Traditional Proposed w/o refinement Proposed w/ refinement Green Orange : Table surface : Floor surface
48. Experiment in a real environment Arrangement1 • ~ 6mm accuracy achieved • Traditional one deteriorated with small overlap • Proposed method retained the accuracy Arrangement2 (Inter-room) • Traditional one is no longer able to calibrate • Proposed method showed a high accuracy thanks to the VO-based bridging • Refinement step further improved accuracy GT measured using FARO
49. Integration of Static and Dynamic Camera
50. Conclusion Visual odometry-based non-overlapping camera network calibration is proposed