CV_5_3dVision

Computer vision 3d vision (Week 8-11) NHSM - 4th year
- Spring 2025 - Prof. Mohammed Hachama [email protected] http://hachama.github.io/home/

Outline Image formation and Camera Calibration Image formation From World
to Pixel coordinates Camera calibration 3d vision 3D Reconstruction Stereo vision Binocular stereo NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 2/20

Image formation NHSM - 4th year: Computer vision - 3d
(Week 8-11) - M. Hachama ([email protected]) 3/20

Image Formation Image formation: using a film • Place a
piece of film in front of an object • → Do we get a reasonable image? NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 5/20

Image Formation Image formation: Pinhole camera • Add a barrier
to block off most of the rays • Ideally: only one ray of light reaches each point on the film. • Reduces blurring • Opening = aperture NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 5/20

Image Formation Image formation (Pinhole camera): Effects of the Aperture
Size • Making aperture bigger makes the image blurry. • Making aperture very small → Less light gets through (must increase the exposure) + Diffraction effects. NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 5/20

Image Formation Image formation using a converging lens • A
lens focuses light onto the film • Rays passing through the Optical Center are not deviated NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 5/20

Image Formation Image formation using a converging lens • All
rays parallel to the Optical Axis converge at the Focal Point. NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 5/20

Image Formation In focus • Point at a specific distance
from the lens are ”in focus” • Other points project to a ”blur circle” in the image NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 5/20

Image Formation Projective Geometry • What is preserved: Straight lines
are still straight NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 5/20

Image Formation Projective Geometry • What is lost?: Length, Angles
NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 5/20

Image Formation Vanishing points and lines •World parallel lines intersect
in the image at a ”vanishing points” NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 5/20

Image Formation Vanishing points and lines •World parallel lines intersect
in the image at a ”vanishing points” •Parallel planes in the world intersect in the image at a ”vanishing line” NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 5/20

Camera parameters Focus and depth of field • Depth of
field (DOF): the distance between the nearest and farthest objects in a scene that appear acceptably sharp. • A lens can precisely focus at only one distance. The sharpness decreases gradually on each side of the focused distance. NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 6/20

Camera parameters Focus and depth of field • How does
the aperture affect the depth of field? • A smaller aperture increases the depth of field but reduces the amount of light into the camera NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 6/20

Camera parameters Field of view • Angular measure of portion
of 3d space seen by the camera • Smaller FOV = larger Focal Length NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 6/20

From World to Pixel coordinates Perspective camera model • Camera
and world coordinates NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 8/20

From World to Pixel coordinates Perspective camera model • From
the Camera frame to the image plane • 1. Convert world pt Pw to camera pt Pc (Extrinsic params. [R|T]) • 2. Convert Pc to image-plane coord. (x, y) (Intrinsic params. f , ...) • 3. Convert Pc to (discretised) pixel coordinates (u, v) NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 8/20

From World to Pixel coordinates Perspective camera model • 1.
Convert world point Pw to camera point Pc    Xc Yc Zc    =    r11 r12 r13 r21 r22 r23 r31 r32 r33    R    Xw Yw Zw    +    t1 t2 t3    T    Xc Yc Zc    =    r11 r12 r13 |t1 r21 r22 r23 |t2 r31 r32 r33 |t3         Xw Yw Zw 1      = [R|T]      Xw Yw Zw 1      NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 8/20

Convert Pc to image-plane coordinates (x, y) • Pc = (Xc , 0, Zc )T projects to p = (x, y) onto the image plane • From similar triangles: x f = Xc Zc =⇒ x = fXc Zc • In the general case: y f = Yc Zc =⇒ y = fYc Zc NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 8/20

Convert Pc to (discretised) pixel coordinates (u, v) • The pixel coord. of the camera optical center: O = (u0, v0 ) • Scale factors ku , kv for the pixel-size in both dimensions u = u0 + ku x =⇒ u = u0 + ku fXc ZC , v = v0 + kv fYc ZC NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 8/20

From World to Pixel coordinates Perspective camera model • Homogeneous
coordinates:    λu λv λ    =    ku f 0 u0 0 kv f v0 0 0 1       Xc Yc Zc    =    αu 0 u0 0 αv v0 0 0 1    K    Xc Yc Zc    • αu and αv are the Focal length in pixels and K is the ”Calibration matrix” or ”Matrix of Intrinsic Parameters”.    λu λv λ    = K[R|T]      Xw Yw Zw 1      NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 8/20

From World to Pixel coordinates Perspective camera model • Lens
distortion: from the undistorted coordinates (u, v) to the observable distorted coordinates (ud , vd ) NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 8/20

From World to Pixel coordinates Perspective camera model • Lens
distortion: from the undistorted coordinates (u, v) to the observable distorted coordinates (ud , vd ) • Simple quadratic model (works for most lenses): ud vd = (1 + k1 r2) u − u0 v − v0 + u0 v0 where r2 = (u − u0 )2 + (v − v0 )2. NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 8/20

Camera calibration Calibration methods • We suppose that radial distortions
are already corrected • Estimation of the camera parameters (K, R, T) that satisfy the perspective projection equation ˜ p =    ˜ u ˜ v ˜ w    = λ    u v 1    = K[R|T]      Xw Yw Zw 1      = M      Xw Yw Zw 1      • Method 1: Direct Linear Transform (DLT) From 3D objects • Method 2: Direct Linear Transform (DLT) From planar grids NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 10/20

Camera calibration Method 1: Direct Linear Transform (DLT) From 3D
objects • From n points, we can stack all these equations into: QM = 0 NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 11/20

objects • From n points, we can stack all these equations into: QM = 0 • Q(2n×12) should have rank 11 to have a unique (up to a scale) non-trivial solution M. • Each 3D-to-2D point correspondence → 2 independent equations. • → 6 point correspondences are needed. • minimize ∥QM∥2 s.t. ∥M∥2= 1 =⇒ SVD: The solution is the eigenvector corresponding to the smallest eigenvalue of QT Q. NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 11/20

objects • From n points, we can stack all these equations into: QM = 0 • From M, we can recover the intrinsic and extrinsic parameters M = K(R|T) • Enforce the constraint that R is orthogonal (RT R = I), we can use the QR factorization NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 11/20

Camera calibration Method 2: Direct Linear Transform (DLT) From planar
grids • On few images of a planar grid (e.g., a chessboard) at different orientations, find relations between points without knowing their 3D coordinates. NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 12/20

grids • Since the points lie on a plane, we have Zw = 0    λu λv λ    =    αu 0 u0 0 αv v0 0 0 1       r11 r12 r13 |t1 r21 r22 r23 |t2 r31 r32 r33 |t3         Xw Yw 0 1      =⇒    λu λv λ    =    αu 0 u0 0 αv v0 0 0 1       r11 r12 |t1 r21 r22 |t2 r31 r32 |t3    H    Xw Yw 1    H is called Homography. NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 12/20

grids • For n points        X1 w Y 1 w 1 0 0 0 −u1 X1 w −u1 Y 1 w −u1 0 0 0 X1 w Y 1 w 1 −v1 X1 w −v1 Y 1 w −v1 . . . . . . . . . Xn w Y n w 1 0 0 0 −un Xn w −un Y n w −un 0 0 0 Xn w Y n w 1 −vn Xn w −vn Y n w −vn           h1 h2 h3    =         0 0 . . . 0 0         =⇒ Q.H = 0 NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 12/20

grids • Q2n×9 should have rank 8 to have a unique (up to a scale) non-trivial solution H • Each point correspondence provides 2 independent equations • Thus, a minimum of 4 non-collinear points is required • Over-determined solution: n ≥ 4 points: SVD • Solving for (K, R, T): H decomposition    h11 h12 h13 h21 h22 h23 h31 h32 h33    =    αu 0 u0 0 αv v0 0 0 1       r11 r12 t1 r21 r22 t2 r31 r32 t3    NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 12/20

grids • Plane-based self calibration: Zhang’s method (1999) • 1. Images I1, ...IM are taken under different views. • 2. From each i, estimate the associated homography Hi . Thus, the intrinsic and extrinsic parameters are computed. • 3. Refining all parameters (Global optimization) E = M i=1 N j=1 (ui,j − T(Pi,j W , Ri , Ti , αu, αv , u0, v0 ))2 NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 12/20

3D vision NHSM - 4th year: Computer vision - 3d
(Week 8-11) - M. Hachama ([email protected]) 13/20

3D Reconstruction 3D Reconstruction NHSM - 4th year: Computer vision
- 3d (Week 8-11) - M. Hachama ([email protected]) 15/20

3D Reconstruction 3D Reconstruction from one image • Shape from
shading: The brightness (intensity) of a pixel in an image is related to the angle between the surface normal and the light source direction. NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 15/20

3D Reconstruction 3D Reconstruction from multiple images • Photometric stereo
from shading: Uses multiple images with known lighting to improve robustness. NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 15/20

3D Reconstruction 3D Reconstruction from multiple images • 3D reconstruction
from multiple views: • Assumptions: K, T and R are known. • Goal: Recover the 3D structure from images • Structure From Motion: • Assumptions: none (K, T, and R are unknown). • Goal: Recover simultaneously 3D scene structure and camera poses (up to scale) from multiple images NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 15/20

Stereo vision Stereo vision • Stereopsys: the brain allows us
to see the left and right retinal images as a single 3D image NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 17/20

Stereo vision Stereo vision • Suppose we have two calibrated
cameras and a pair of corresp. pixels • Find 3D location of point! NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 17/20

Stereo vision Triangulation NHSM - 4th year: Computer vision -
3d (Week 8-11) - M. Hachama ([email protected]) 17/20

Stereo vision Triangulation • Requires: camera pose (calibration)+point correspondence NHSM
- 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 17/20

Stereo vision Triangulation • Suppose we have two calibrated cameras
and a pair of corresp. pixels • Find 3D location of point! NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 17/20

Stereo vision Triangulation • Two image points   
xi yi 1    = x(1) img = P(1)xw = P(1)      X Y Z 1      xi ≈ P(i) 11 X + P(i) 12 Y + P(i) 13 Z + P(i) 14 P(i) 31 X + P(i) 32 Y + P(i) 33 Z + P(1) 34 = ¯ xi yi ≈ P(i) 21 X + P(i) 22 Y + P(i) 23 Z + P(i) 24 P(1) 31 X + P(i) 32 Y + P(i) 33 Z + P(i) 34 = ¯ yi • Optimization min X,Y ,Z E(X, Y , Z) = 2 i=1 (xi − ¯ xi )2 + (yi − ¯ yi )2 . NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 17/20

Binocular stereo Binocular stereo • Stereo vision: use corresponding image
locations to find world coordinates NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 19/20

locations to find world coordinates • Find pairs of corresponding pixels? NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 19/20

locations to find world coordinates • Find pairs of corresponding pixels? • Binocular stereo = using two cameras (or eyes) capturing images from slightly different viewpoints. NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 19/20

Binocular stereo Binocular stereo • General case: cameras can be
arbitrary locations and orientations NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 19/20

Binocular stereo Binocular stereo with rectified cameras • Special case:
Parallel cameras translated along the X-axis NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 19/20

Binocular stereo Binocular stereo with rectified cameras • Special case:
Parallel cameras translated along the X-axis • Rectified cameras: examples NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 19/20

Binocular stereo Binocular stereo with rectified cameras • Stereo with
rectified cameras (short displacement) NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 19/20

rectified cameras (short displacement) Correspondence problem is easier: searching along a particular row. NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 19/20

rectified cameras (short displacement) • Without loss of generality, assume origin is at pinhole of 1st camera.    λx1 λy1 λ    =    X Y Z    ,    λx2 λy2 λ    =    X + tx Y Z    =⇒      x1 = X Z , x2 = X+tx Z = x1 + tx Z , y1 = y2 = Y Z NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 19/20

rectified cameras (short displacement) • Translation between cameras is tx → baseline • X coordinate differs by tx Z → disparity. disparity = baseline/depth • Cross-Correlation NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 19/20

Cameras rectification Camera rectification • Camera rectification (or stereo rectification)
= transforming a pair of stereo images so that corresponding points in both images lie along the same horizontal scanline. • Can we rotate / translate cameras? • Do we need to know the 3D structure of the world to do this? NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 20/20

Cameras rectification Rectification: Epipolar geometry • A point in one
view ”generates” an epipolar line in the other view • The corresponding point lies on this line NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 20/20

Cameras rectification Rectification: Epipolar geometry • Epipolar lines • Rectified
images: Epipolar lines are ...? NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 20/20

Cameras rectification Stereo rectification • Example NHSM - 4th year:
Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 20/20

Cameras rectification Stereo rectification • Example: remove radial distortion NHSM
- 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 20/20

Cameras rectification Stereo rectification • Example; compute homographies and rectify
NHSM - 4th year: Computer vision - 3d (Week 8-11) - M. Hachama ([email protected]) 20/20

CV_5_3dVision

CV_5_3dVision

More Decks by Mohammed Hachama

Other Decks in Science

Featured

Transcript