Stereo Camera#

This module provides the StereoCamera class, which contains functionality for working with a horizontal stereo camera setup.

The horizontal stereo camera setup is assumed to be calibrated and rectified such that the setup can be described by two camera matrices:

The left rectified camera matrix:

\[\begin{split}P_0 = \begin{bmatrix} fx & 0 & cx & 0 \\ 0 & fy & cy & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix}\end{split}\]

The right rectified camera matrix:

\[\begin{split}P_1 = \begin{bmatrix} fx & 0 & cx & tx * fx \\ 0 & fy & cy & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix}\end{split}\]
where:

  • \(fx\) is the focal length in the x-direction in pixels.

  • \(fy\) is the focal length in the y-direction in pixels.

  • \(cx\) is the x-coordinate of the principal point in pixels.

  • \(cy\) is the y-coordinate of the principal point in pixels.

  • \(tx\) is the horizontal baseline in metric units.

These camera matrices are obtained by calibrating your stereo camera setup, which can be done in OpenCV.

The StereoCamera allows you to convert disparity maps to real-world 3D geometry represented by a point cloud.

This is done by forming the \(Q\) matrix.

Using the pinhole camera model, a point \([X, Y, Z, 1]\) in world coordinates projects to pixels \((u, v)\) in the left and right camera frames respectively:

\[\begin{split}\begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = P_0 * \begin{bmatrix} X \\ Y \\ Z \\ 1 \end{bmatrix} \\ \begin{bmatrix} u-d \\ v \\ 1 \end{bmatrix} = P_1 * \begin{bmatrix} X \\ Y \\ Z \\ 1 \end{bmatrix}\end{split}\]

where \(d\) is the disparity between the pixels in the left and right images.

Combining these two expressions lets us write them as one matrix multiplication:

\[\begin{split}\begin{bmatrix} u \\ v \\ u-d \\ 1 \end{bmatrix} = \begin{bmatrix} fx & 0 & cx_{left} & 0 \\ 0 & fy & cy & 0 \\ fx & 0 & cx_{right} & fx * tx \\ 0 & 0 & 1 & 0 \end{bmatrix} \begin{bmatrix} X \\ Y \\ Z \\ 1 \end{bmatrix}\end{split}\]

Subtracting the first row from the third and inverting the expression gives:

\[\begin{split}\begin{bmatrix} u \\ v \\ d \\ 1 \end{bmatrix} = \begin{bmatrix} fy * tx & 0 & 0 & -fy * cx * tx \\ 0 & fx * tx & 0 & -fx * cy * tx \\ 0 & 0 & 0 & fx * fy * tx \\ 0 & 0 & -fy & fy * (cx_{left} -cx_{right}) \end{bmatrix} \begin{bmatrix} X \\ Y \\ Z \\ 1 \end{bmatrix}\end{split}\]

where \(Q\) is

\[\begin{split}Q = \begin{bmatrix} fy * tx & 0 & 0 & -fy * cx * tx \\ 0 & fx * tx & 0 & -fx * cy * tx \\ 0 & 0 & 0 & fx * fy * tx \\ 0 & 0 & -fy & fy * (cx_{left} -cx_{right}) \end{bmatrix}\end{split}\]

The \(Q\) that StereoCamera builds is exactly this matrix with \(tx = P_1[0, 3] / fx\), which the constructor requires to be strictly negative (a zero baseline or swapped cameras raise). The class’s tx attribute is the positive baseline \(-P_1[0, 3] / fx\), so do not substitute it for \(tx\) above.

The \(Q[3, 3]\) term is zero on every rig StereoCamera accepts outside torch.export: the constructor rejects differing \(cx_{left}\) and \(cx_{right}\) (#4270).

Assuming \(fx = fy\) you can further reduce this to:

\[\begin{split}Q = \begin{bmatrix} 1 & 0 & 0 & -cx \\ 0 & 1 & 0 & -cy \\ 0 & 0 & 0 & fx \\ 0 & 0 & -1/tx & (cx_{left} - cx_{right}) / tx \end{bmatrix}\end{split}\]

but the implementation uses the general \(Q\) matrix.

Using the \(Q\) matrix we can obtain the 3D points by:

\[\begin{split}\begin{bmatrix} X \\ Y \\ Z \\ W \end{bmatrix} = Q * \begin{bmatrix} u \\ v \\ disparity(u, v) \\ 1 \end{bmatrix}\end{split}\]

where the Euclidean 3D point is \((X/W, Y/W, Z/W)\).

class kornia.geometry.camera.stereo.StereoCamera(rectified_left_camera, rectified_right_camera)[source]#

Represent a horizontal stereo camera setup for depth estimation.

Convention:

  • the two arguments are the rectified projection matrices of the left and the right camera, each of shape \((B, 3, 4)\): [[fx, 0, cx, 0], [0, fy, cy, 0], [0, 0, 1, 0]] for the left camera, and the same matrix with -tx * fx in the last column for the right one. The constructor requires the two to be equal outside that last column.

  • the baseline is read back as the attribute tx = -P_right[0, 3] / fx, which must be strictly positive for every rig in the batch: a zero baseline and swapped cameras both raise. A non-finite P_right[0, 3] is not screened, and these checks and the equal-intrinsics check are skipped under torch.export. The Stereo Camera page’s symbol tx is the negation of this attribute; Q is the page’s matrix evaluated at the page’s tx.

  • Q[0, 0] carries fy and Q[1, 1] carries fx; with the divide by W = -fy * disparity the result is X = (u - cx) Z / fx, Y = (v - cy) Z / fy, Z = fx * tx / disparity.

  • u is the column index and v the row index, as in cv2.reprojectImageTo3D.

  • a disparity map is channels-last, \((B, H, W, 1)\) (a \((B, 1, H, W)\) map is rejected), and the point cloud is \((B, H, W, 3)\).

  • pixel centres are integers; see the Convention block on PinholeCamera.

Warning

A differing cx is rejected, although Q[3, 3] = fy * (cx_left - cx_right) exists for that case, so it is zero on every rig the constructor accepts outside torch.export: #4270. A zero disparity (a point at infinity) makes W = 0; the divide is then skipped and a finite placeholder, behind the camera on a real rig, is returned and not flagged as invalid: #4555.

Warning

The module-level reproject_disparity_to_3D() is rendered on Stereo Camera but is in no __all__, so it is not reachable as kornia.geometry.reproject_disparity_to_3D. Tracked as #4275.

Parameters:

  • rectified_left_camera (Tensor) – The rectified left camera projection matrix of shape \((B, 3, 4)\).

  • rectified_right_camera (Tensor) – The rectified right camera projection matrix of shape \((B, 3, 4)\).

__init__(rectified_left_camera, rectified_right_camera)[source]#

Class representing a horizontal stereo camera setup.

Parameters:

  • rectified_left_camera (Tensor) – The rectified left camera projection matrix of shape \((B, 3, 4)\)

  • rectified_right_camera (Tensor) – The rectified right camera projection matrix of shape \((B, 3, 4)\)

property Q: Tensor#

The Q matrix of the horizontal stereo setup.

This matrix is used for reprojecting a disparity torch.Tensor to the corresponding point cloud. Note that this is in a general form that allows different focal lengths in the x and y direction.

Returns:

The Q matrix of shape \((B, 4, 4)\).

property batch_size: int#

Return the batch size of the storage.

Returns:

scalar with the batch size

property cx_left: Tensor#

Return the x-coordinate of the principal point for the left camera.

Returns:

torch.Tensor of shape \((B)\)

property cx_right: Tensor#

Return the x-coordinate of the principal point for the right camera.

Returns:

torch.Tensor of shape \((B)\)

property cy: Tensor#

Return the y-coordinate of the principal point.

Note that the y-coordinate of the principal points is assumed to be equal for the left and right camera.

Returns:

torch.Tensor of shape \((B)\)

property fx: Tensor#

Return the focal length in the x-direction.

Note that the focal lengths of the rectified left and right camera are assumed to be equal.

Returns:

torch.Tensor of shape \((B)\)

property fy: Tensor#

Returns the focal length in the y-direction.

Note that the focal lengths of the rectified left and right camera are assumed to be equal.

Returns:

torch.Tensor of shape \((B)\)

reproject_disparity_to_3D(disparity_tensor)[source]#

Reproject the disparity torch.Tensor to a 3D point cloud.

See the Convention block on StereoCamera.

Parameters:

disparity_tensor (Tensor) – Disparity torch.Tensor of shape \((B, H, W, 1)\).

Return type:

Tensor

Returns:

The 3D point cloud of shape \((B, H, W, 3)\)

property tx: Tensor#

The horizontal baseline between the two cameras.

Returns:

torch.Tensor of shape \((B)\)

kornia.geometry.camera.stereo.reproject_disparity_to_3D(disparity_tensor, Q_matrix)[source]#

Reproject the disparity torch.Tensor to a 3D point cloud.

See the Convention block on StereoCamera.

Parameters:

  • disparity_tensor (Tensor) – Disparity torch.Tensor of shape \((B, H, W, 1)\).

  • Q_matrix (Tensor) – torch.Tensor of Q matrices of shapes \((B, 4, 4)\).

Return type:

Tensor

Returns:

The 3D point cloud of shape \((B, H, W, 3)\)