kornia.geometry.depth#

kornia.geometry.depth.depth_from_disparity(disparity, baseline, focal)[source]#

Compute depth from disparity.

Convention:

the depth is baseline * focal / disparity, elementwise: baseline is the distance between the two camera centres and focal the focal length in pixels, so a disparity of 2 with a baseline of 0.5 and a focal length of 100 gives a depth of 25.
baseline and focal are each a scalar (Python number, 0-d or \((1,)\) tensor) shared by the whole disparity, or a \((B,)\) tensor paired with disparity[b], which then needs a leading batch axis of size B (otherwise ShapeError). The two may mix.
disparity is \((*, H, W)\) and the result has its shape. Its sign is not checked.

Warning

The divisor is disparity + 1e-8, so a zero disparity (no stereo match) returns a large finite depth set by the epsilon (5e9 for baseline 0.5, focal 100) rather than inf, except where the epsilon underflows: #4272.

Parameters:

disparity (Tensor) – Disparity tensor of shape \((*, H, W)\).
baseline (float | Tensor) – Distance between the two lenses, as an int, float, or tensor of shape () or (1,), or of shape (B,) for one value per batch element.
focal (float | Tensor) – Focal length, as an int, float, or tensor of shape () or (1,), or of shape (B,) for one value per batch element.

Return type:

Tensor

Returns:

Depth map of the shape \((*, H, W)\).

Example

>>> disparity = torch.rand(4, 1, 4, 4)
>>> baseline = torch.rand(1)
>>> focal = torch.rand(1)
>>> depth_from_disparity(disparity, baseline, focal).shape
torch.Size([4, 1, 4, 4])

kornia.geometry.depth.depth_to_3d(depth, camera_matrix, normalize_points=False)[source]#

Compute a 3d point per pixel given its depth value and the camera intrinsics.

Note

kornia.geometry.depth.depth_to_3d_v2() computes the same points without building a meshgrid, in the \((B, H, W, 3)\) layout.

Convention:

depth is the camera-frame z of each pixel and the result the camera-frame point ((u - cx) z / fx, (v - cy) z / fy, z), channels-first \((B, 3, H, W)\), with u the column and v the row (integer pixel centres; see PinholeCamera). It equals depth_to_3d_v2() after permute(0, 2, 3, 1).
normalize_points=True reads depth as the Euclidean ray length instead of z.
an integer depth is not rejected; the result takes the floating dtype of camera_matrix.

Parameters:

depth (Tensor) – image tensor containing a depth value per pixel with shape \((B, 1, H, W)\).
camera_matrix (Tensor) – tensor containing the camera intrinsics with shape \((B, 3, 3)\).
normalize_points (bool, optional) – whether to normalise the pointcloud. This must be set to True when the depth is represented as the Euclidean ray length from the camera position. Default: False

Return type:

Tensor

Returns:

tensor with a 3d point per pixel of the same resolution as the input \((B, 3, H, W)\).

Example

>>> depth = torch.rand(1, 1, 4, 4)
>>> K = torch.eye(3)[None]
>>> depth_to_3d(depth, K).shape
torch.Size([1, 3, 4, 4])

kornia.geometry.depth.depth_to_3d_v2(depth, camera_matrix, normalize_points=False, xyz_grid=None)[source]#

Compute a 3d point per pixel given its depth value and the camera intrinsics.

Note

This is an alternative implementation of kornia.geometry.depth.depth_to_3d() that does not require the creation of a meshgrid.

Convention:

depth is the camera-frame z of each pixel and the result the camera-frame point ((u - cx) z / fx, (v - cy) z / fy, z), channels-last \((*, H, W, 3)\), with u the column and v the row (integer pixel centres; see PinholeCamera). It equals depth_to_3d() after permute(0, 2, 3, 1).
without xyz_grid, camera_matrix must be exactly \((B, 3, 3)\) (another shape raises ShapeError); with xyz_grid it is never read.
normalize_points=True reads depth as the Euclidean ray length instead of z.
xyz_grid replaces the grid construction; pass what unproject_meshgrid() returns for the same camera.

Parameters:

depth (Tensor) – image tensor containing a depth value per pixel with shape \((*, H, W)\).
camera_matrix (Tensor) – tensor containing the camera intrinsics with shape \((B, 3, 3)\).
normalize_points (bool, optional) – whether to normalise the pointcloud. This must be set to True when the depth is represented as the Euclidean ray length from the camera position. Default: False
xyz_grid (Optional[Tensor], optional) – explicit xyz point values. Default: None

Return type:

Tensor

Returns:

tensor with a 3d point per pixel of the same resolution as the input, \((*, H, W, 3)\), whose leading dimensions are the broadcast of depth’s and the grid’s: the Example broadcasts a \((4, 4)\) depth against a \((2, 3, 3)\) camera to \((2, 4, 4, 3)\), and a \((B, T, H, W)\) depth with an xyz_grid of shape \((B, 1, H, W, 3)\) returns \((B, T, H, W, 3)\).

Example

>>> depth = torch.rand(4, 4)
>>> K = torch.eye(3).repeat(2,1,1)
>>> depth_to_3d_v2(depth, K).shape
torch.Size([2, 4, 4, 3])

kornia.geometry.depth.unproject_meshgrid(height, width, camera_matrix, normalize_points=False, device=None, dtype=None)[source]#

Compute a 3d point per pixel given its depth value and the camera intrinsics.

Tip

This function should be used in conjunction with kornia.geometry.depth.depth_to_3d_v2() to cache the meshgrid computation when warping multiple frames with the same camera intrinsics.

Convention:

the result is the camera-frame ray through each pixel at depth 1, \((B, H, W, 3)\), so multiplying it by a \((B, H, W)\) depth map reproduces depth_to_3d_v2() (its xyz_grid cache). Pixel centres are integers; see PinholeCamera.
camera_matrix must be \((B, 3, 3)\); an unbatched matrix or extra leading dimensions raise ShapeError.
normalize_points=True returns the unit ray instead of the ray whose z is 1.

Parameters:

height (int) – height of image.
width (int) – width of image.
camera_matrix (Tensor) – tensor containing the camera intrinsics with shape \((B, 3, 3)\).
normalize_points (bool, optional) – whether to normalize the pointcloud. This must be set to True when the depth is represented as the Euclidean ray length from the camera position. Default: False
device (Optional[device], optional) – device to place the result on. Default: None
dtype (Optional[dtype], optional) – dtype of the result. Default: None

Return type:

Tensor

Returns:

tensor with a 3d point per pixel, with shape \((B, H, W, 3)\).

kornia.geometry.depth.depth_to_normals(depth, camera_matrix, normalize_points=False)[source]#

Compute the normal surface per pixel.

Convention:

both spatial dimensions must be at least 2: a surface normal requires two tangent directions.
the normal is the unit cross product d/dx x d/dy of the unprojected point cloud, so a fronto-parallel plane gets (0, 0, 1), pointing away from the camera; a depth growing along the columns tilts it towards -x, one growing along the rows towards -y.
the arguments mean what they mean for depth_to_3d(), and the result has its \((B, 3, H, W)\) layout.

Parameters:

depth (Tensor) – image tensor containing a depth value per pixel with shape \((B, 1, H, W)\), with \(H \geq 2\) and \(W \geq 2\).
camera_matrix (Tensor) – tensor containing the camera intrinsics with shape \((B, 3, 3)\).
normalize_points (bool, optional) – whether to normalize the pointcloud. This must be set to True when the depth is Default: False
position. (represented as the Euclidean ray length from the camera)

Return type:

Tensor

Returns:

tensor with a normal surface vector per pixel of the same resolution as the input \((B, 3, H, W)\).

Raises:

ShapeError – if either spatial dimension of depth is smaller than 2.

Example

>>> depth = torch.rand(1, 1, 4, 4)
>>> K = torch.eye(3)[None]
>>> depth_to_normals(depth, K).shape
torch.Size([1, 3, 4, 4])

kornia.geometry.depth.depth_from_plane_equation(plane_normals, plane_offsets, points_uv, camera_matrix, eps=1e-8)[source]#

Compute depth values from plane equations and pixel coordinates.

Convention:

the plane is given in Hessian form \(n \cdot X = d\), with the normal plane_normals and the offset plane_offsets in the camera frame: n = (0, 0, 1) with d = 2 is the plane z = 2, and every pixel on it has depth 2.
points_uv are pixels (integer centres; see PinholeCamera), normalized here with camera_matrix.
the result is the camera-frame z of each pixel, \((B, N)\): a list of depths, not a map.
a ray-plane dot product inside \((-eps, eps)\) is replaced by \(\pm\) eps with its sign (+eps for an exact zero), so a grazing ray returns the large signed depth d / eps rather than inf. In float16 a positive eps is floored at the smallest subnormal, 2**-24, because the default 1e-8 rounds to zero there; d / 2**-24 is still inf in float16 once |d| exceeds about 3.9e-3, so pass a larger eps for a finite depth (eps=1e-4 keeps |d| <= 6.5 finite).

Parameters:

plane_normals (torch.Tensor) – Plane normal vectors of shape (B, 3).
plane_offsets (torch.Tensor) – Plane offsets of shape (B, 1).
points_uv (torch.Tensor) – Pixel coordinates of shape (B, N, 2).
camera_matrix (torch.Tensor) – Camera intrinsic matrix of shape (B, 3, 3).
eps (float, optional) – epsilon for numerical stability. Default: 1e-8

Returns:

Computed depth values at the given pixels, shape (B, N).

Return type:

torch.Tensor

kornia.geometry.depth.warp_frame_depth(image_src, depth_dst, src_trans_dst, camera_matrix, normalize_points=False)[source]#

Warp a tensor from a source to destination frame by the depth in the destination.

Compute 3d points from the depth, transform them using given transformation, then project the point cloud to an image plane.

Convention:

the depth belongs to the destination frame (depth_dst) and the image to the source frame (image_src), and src_trans_dst maps destination-frame points into the source frame. With fx = fy = 1 and a unit depth, a \(+1\) translation in x samples image_src one pixel to the right of each destination pixel.
the unprojection is depth_to_3d_v2() and the reprojection project_points(), whose conventions apply.
the sampling is bilinear grid_sample with a fixed align_corners=True and padding_mode="zeros"; the output is \((B, D, H, W)\) with image_src’s channels.

Warning

DepthWarper performs the same warp under the opposite naming: this function’s dst (holding the depth) is that class’s src, and image_src is its patch_dst: #4273.

Parameters:

image_src (Tensor) – image tensor in the source frame with shape \((B,D,H,W)\).
depth_dst (Tensor) – depth tensor in the destination frame with shape \((B,1,H,W)\).
src_trans_dst (Tensor) – transformation matrix from destination to source with shape \((B,4,4)\).
camera_matrix (Tensor) – tensor containing the camera intrinsics with shape \((B,3,3)\).
normalize_points (bool, optional) – whether to normalize the pointcloud. This must be set to True when the depth is represented as the Euclidean ray length from the camera position. Default: False

Return type:

Tensor

Returns:

image_src resampled onto the destination pixel grid, with shape \((B,D,H,W)\).

kornia.geometry.depth.depth_warp(pinhole_dst, pinhole_src, depth_src, patch_dst, height, width, align_corners=True)[source]#

Warp a tensor from destination frame to reference given the depth in the reference frame.

See the Convention block on DepthWarper.

This function is that class’s functional form: it constructs a DepthWarper, calls its compute_projection_matrix and returns its output, which is equal to the class’s bit for bit under the default mode and padding_mode, the only ones this function offers (align_corners is forwarded).

Return type:: Tensor

Example

>>> # pinholes camera models
>>> pinhole_dst = PinholeCamera(torch.randn(1, 4, 4), torch.randn(1, 4, 4),
... torch.tensor([32]), torch.tensor([32]))
>>> pinhole_src = PinholeCamera(torch.randn(1, 4, 4), torch.randn(1, 4, 4),
... torch.tensor([32]), torch.tensor([32]))
>>> # warp the destination frame to reference by depth
>>> depth_src = torch.ones(1, 1, 32, 32)  # Nx1xHxW
>>> image_dst = torch.rand(1, 3, 32, 32)  # NxCxHxW
>>> image_src = depth_warp(pinhole_dst, pinhole_src, depth_src, image_dst, 32, 32)  # NxCxHxW

class kornia.geometry.depth.DepthWarper(pinhole_dst, height, width, mode='bilinear', padding_mode='zeros', align_corners=True)[source]#

Warp a patch by depth.

\[ \begin{align}\begin{aligned}P_{src}^{\{dst\}} = K_{dst} * T_{src}^{\{dst\}}\\\begin{split}I_{src} = \\omega(I_{dst}, P_{src}^{\{dst\}}, D_{src})\end{split}\end{aligned}\end{align} \]

Convention:

the depth lives in the source frame and the image in the destination frame: forward() takes (depth_src, patch_dst), samples patch_dst at the pixels the source-frame depth projects to, and returns a \((B, C, H, W)\) tensor carrying patch_dst’s channel count.
this is a two-step API: compute_projection_matrix() has to be called first. Until it has been, warp_grid() and forward() raise ValueError and compute_subpixel_step() raises RuntimeError.
compute_projection_matrix() stores K_dst @ (E_dst @ inverse(E_src)), the matrix above, from the constructor’s pinhole_dst and that method’s pinhole_src (both PinholeCamera, world-to-camera extrinsics). Both cameras use only the top-left 3x3 block of their intrinsics, embedded in homogeneous 4x4 form; the stored fourth row and column are ignored.
the grid attribute is the homogeneous integer-centre pixel grid (pixel (0, 0) is (0, 0, 1)); warp_grid() returns sampling positions in grid_sample’s align_corners=True normalized coordinates of the destination image.
align_corners (default True), mode and padding_mode are forwarded to grid_sample. depth_warp() is the functional form and exposes align_corners only.

Warning

warp_frame_depth() performs the same warp under the opposite naming: this class’s src (holding the depth) is that function’s dst, and patch_dst is its image_src; reading “dst” as “dst” across the two gives the inverse warp: #4273. The two agree to rounding for float32/float64 away from z = 0, and split at z = 0, where cam2pixel sends every pixel with a nonzero projected numerator far outside the image: #4267.

Parameters:

pinhole_dst (PinholeCamera) – the pinhole model for the destination frame.
height (int) – the height of the image to warp.
width (int) – the width of the image to warp.
mode (str, optional) – interpolation mode to calculate output values 'bilinear' | 'nearest'. Default: "bilinear"
padding_mode (str, optional) – padding mode for outside grid values 'zeros' | 'border' | 'reflection'. Default: "zeros"
align_corners (bool, optional) – interpolation flag. Default: True

compute_projection_matrix(pinhole_src)[source]#

Compute the projection matrix from the source to destination frame.

See the Convention block on DepthWarper.

Return type:: DepthWarper

compute_subpixel_step()[source]#

Compute the inverse depth step for sub pixel accurate sampling of the depth cost volume, per camera.

See the Convention block on DepthWarper.

Szeliski, Richard, and Daniel Scharstein. “Symmetric sub-pixel stereo matching.” European Conference on Computer Vision. Springer Berlin Heidelberg, 2002.

Return type:: Tensor

forward(depth_src, patch_dst)[source]#

Warp a tensor from destination frame to reference given the depth in the reference frame.

See the Convention block on DepthWarper.

Parameters:

depth_src (Tensor) – the depth in the reference frame. The tensor must have a shape \((B, 1, H, W)\).
patch_dst (Tensor) – the patch in the destination frame. The tensor must have a shape \((B, C, H, W)\).

Return type:

Tensor

Returns:

the warped patch from destination frame to reference.

Shape:

Input: \((B, 1, H, W)\) and \((B, C, H, W)\).
Output: \((B, C, H, W)\) where C = number of channels.

Example

>>> # pinholes camera models
>>> pinhole_dst = PinholeCamera(torch.randn(1, 4, 4), torch.randn(1, 4, 4),
... torch.tensor([32]), torch.tensor([32]))
>>> pinhole_src = PinholeCamera(torch.randn(1, 4, 4), torch.randn(1, 4, 4),
... torch.tensor([32]), torch.tensor([32]))
>>> # create the depth warper, compute the projection matrix
>>> warper = DepthWarper(pinhole_dst, 32, 32)
>>> _ = warper.compute_projection_matrix(pinhole_src)
>>> # warp the destination frame to reference by depth
>>> depth_src = torch.ones(1, 1, 32, 32)  # Nx1xHxW
>>> image_dst = torch.rand(1, 3, 32, 32)  # NxCxHxW
>>> image_src = warper(depth_src, image_dst)  # NxCxHxW

warp_grid(depth_src)[source]#

Compute a grid for warping a given the depth from the reference pinhole camera.

See the Convention block on DepthWarper.

The function compute_projection_matrix has to be called beforehand in order to have precomputed the relative projection matrices encoding the relative pose and the intrinsics between the reference and a non reference camera.

Return type:: Tensor