kornia.geometry.depth#

depth_to_3d() (channels first) and depth_to_3d_v2() (channels last, optionally reusing rays from unproject_meshgrid()) turn a depth map into 3D points, and depth_to_normals() into surface normals. Get depth from a stereo disparity map with depth_from_disparity(), or for pixels on a known plane with depth_from_plane_equation(). Render an image from another viewpoint by depth with warp_frame_depth() (a 4x4 pose and one camera matrix), or with DepthWarper and depth_warp() (two pinhole camera models).

kornia.geometry.depth.depth_from_disparity(disparity, baseline, focal)[source]#

Convert a rectified stereo disparity map to depth.

depth = depth_from_disparity(disparity, baseline, focal) returns baseline * focal / disparity for every pixel of a (*, H, W) disparity map, with baseline the distance between the two camera centres and focal the focal length in pixels, so the depth comes out in the unit of baseline. A zero disparity gives a large finite depth, not inf, except in float16 (see below). To go from disparity straight to 3D points with a calibrated stereo rig, use reproject_disparity_to_3D(), which takes a (B, H, W, 1) disparity.

Convention:
  • the exact formula is baseline * focal / (disparity + 1e-8): a disparity of 2 with a baseline of 0.5 and a focal length of 100 gives a depth of 25. The 1e-8 lowers the depth by a relative 1e-8 / disparity. Float32 rounds that away from a disparity of 0.25 up; below it the bias shows, 1e-6 relative (10 float32 ulps) at a disparity of 0.01.

  • a zero disparity (no stereo match, or a point at infinity) gives a finite far depth, baseline * focal / 1e-8 (5e9 for baseline 0.5 and focal 100), not inf. Such a point warps as a point at infinity, along the rotation-only homography, in warp_frame_depth(), whereas an inf depth would turn into NaN in depth_to_3d() wherever a ray component is zero. The far depth is a placeholder, not a measurement: mask disparity > 0 to tell matched pixels apart.

  • in float16 the 1e-8 rounds to 0 and the far depth would overflow anyway, so a zero disparity gives inf. A float32 far depth turns back into inf wherever the warp casts it to float16, as a float16 pose or float16 autocast does, so run the whole warp (depth, image, pose and camera matrix) in float32, outside float16 autocast, when zero disparities flow into it.

  • baseline and focal are each a scalar (Python number, 0-d or (1,) tensor) shared by the whole disparity, or a (B,) tensor paired with disparity[b], which then needs a leading batch axis of size B (otherwise ShapeError). The two may mix.

  • disparity is (*, H, W) and the result has its shape. Its sign is not checked.

Parameters:
  • disparity (Tensor) – Disparity tensor of shape (*, H, W).

  • baseline (float | Tensor) – Distance between the two lenses, as an int, float, or tensor of shape () or (1,), or of shape (B,) for one value per batch element.

  • focal (float | Tensor) – Focal length, as an int, float, or tensor of shape () or (1,), or of shape (B,) for one value per batch element.

Return type:

Tensor

Returns:

Depth map of the shape (*, H, W).

See also

Example

>>> disparity = torch.rand(4, 1, 4, 4)
>>> baseline = torch.rand(1)
>>> focal = torch.rand(1)
>>> depth_from_disparity(disparity, baseline, focal).shape
torch.Size([4, 1, 4, 4])
kornia.geometry.depth.depth_to_3d(depth, camera_matrix, normalize_points=False)[source]#

Unproject a depth map to a 3D point per pixel, channels first.

points = depth_to_3d(depth, camera_matrix) takes a (B, 1, H, W) map of camera-frame depths and (B, 3, 3) pixel intrinsics, and returns the camera-frame point of every pixel as a (B, 3, H, W) tensor. For a channels-last (*, H, W, 3) result, or to reuse the pixel rays across frames, use depth_to_3d_v2() on depth[:, 0]. To unproject selected pixels rather than a whole map, use unproject_points().

Convention:
  • pixel (u, v), with u the column and v the row, maps to ((u - cx) z / fx, (v - cy) z / fy, z). Pixel centres are integers, as in PinholeCamera. depth_to_3d(depth, K).permute(0, 2, 3, 1) equals depth_to_3d_v2(depth[:, 0], K).

  • normalize_points=True reads depth as the Euclidean ray length instead of z.

  • an integer depth is not rejected; the result takes the floating dtype of camera_matrix. The pixel grid is built in promote_types(depth.dtype, camera_matrix.dtype) when either input is floating (the default floating dtype when both are integer), so half-precision depth does not round the pixel indices below the intrinsics’ precision.

Parameters:
  • depth (Tensor) – image tensor containing a depth value per pixel with shape (B, 1, H, W).

  • camera_matrix (Tensor) – tensor containing the camera intrinsics with shape (B, 3, 3).

  • normalize_points (bool, optional) – whether to normalize the pointcloud. This must be set to True when the depth is represented as the Euclidean ray length from the camera position. Default: False

Return type:

Tensor

Returns:

tensor with a 3d point per pixel of the same resolution as the input (B, 3, H, W).

See also

Example

>>> depth = torch.rand(1, 1, 4, 4)
>>> K = torch.eye(3)[None]
>>> depth_to_3d(depth, K).shape
torch.Size([1, 3, 4, 4])
kornia.geometry.depth.depth_to_3d_v2(depth, camera_matrix, normalize_points=False, xyz_grid=None)[source]#

Unproject a depth map to a 3D point per pixel, channels last.

points = depth_to_3d_v2(depth, camera_matrix) takes a (*, H, W) map of camera-frame depths and (B, 3, 3) pixel intrinsics, and returns the camera-frame point of every pixel as a (*, H, W, 3) tensor. The (B, H, W, 3) rays of the B cameras broadcast against depth by torch’s rules, so the camera batch lines up with the axis of depth just before H, not with its first axis: an (H, W) depth gives one map per camera, a (B, H, W) depth pairs camera b with depth[b], and a (B, T, H, W) depth pairs camera t with depth[:, t] when T == B. With T == 1 it pairs every camera with every map and returns (B, B, H, W, 3), so pass depth[:, 0]; any other T raises, unless there is a single camera, which then serves every map. For one camera per (B, T, H, W) batch element, pass xyz_grid=unproject_meshgrid(H, W, camera_matrix)[:, None]; the same grid also saves rebuilding the pixel rays for many frames of one camera. For a channels-first (B, 3, H, W) result, use depth_to_3d().

Convention:
  • pixel (u, v), with u the column and v the row, maps to ((u - cx) z / fx, (v - cy) z / fy, z). Pixel centres are integers, as in PinholeCamera. depth_to_3d_v2(depth[:, 0], K) equals depth_to_3d(depth, K).permute(0, 2, 3, 1).

  • without xyz_grid, camera_matrix must be exactly (B, 3, 3) (another shape raises ShapeError); with xyz_grid only its shape is checked.

  • normalize_points=True reads depth as the Euclidean ray length instead of z. With xyz_grid it is ignored: build the grid with the same normalize_points.

  • without xyz_grid, the pixel grid is built in promote_types(depth.dtype, camera_matrix.dtype) when either input is floating (the default floating dtype when both are integer), so half-precision depth does not round the pixel indices below the intrinsics’ precision.

Parameters:
  • depth (Tensor) – image tensor containing a depth value per pixel with shape (*, H, W).

  • camera_matrix (Tensor) – tensor containing the camera intrinsics with shape (B, 3, 3).

  • normalize_points (bool, optional) – whether to normalize the pointcloud. This must be set to True when the depth is represented as the Euclidean ray length from the camera position. Default: False

  • xyz_grid (Optional[Tensor], optional) – explicit xyz point values. Default: None

Return type:

Tensor

Returns:

tensor with a 3d point per pixel of the same resolution as the input, (*, H, W, 3), whose leading dimensions are the broadcast of depth’s and the grid’s: the Example broadcasts a (4, 4) depth against a (2, 3, 3) camera to (2, 4, 4, 3), and a (B, T, H, W) depth with an xyz_grid of shape (B, 1, H, W, 3) returns (B, T, H, W, 3).

See also

Example

>>> depth = torch.rand(4, 4)
>>> K = torch.eye(3).repeat(2,1,1)
>>> depth_to_3d_v2(depth, K).shape
torch.Size([2, 4, 4, 3])
kornia.geometry.depth.unproject_meshgrid(height, width, camera_matrix, normalize_points=False, device=None, dtype=None)[source]#

Compute the camera ray through every pixel, to reuse across frames with depth_to_3d_v2.

rays = unproject_meshgrid(H, W, camera_matrix) returns the camera-frame ray of each pixel as a (B, H, W, 3) tensor, scaled so that z is 1 by default (unit length with normalize_points=True). It takes no depth: multiply the rays by depth[..., None] for a (B, H, W) depth map, or pass them to depth_to_3d_v2() as xyz_grid, to get the 3D points of many frames from one camera without rebuilding the grid each time.

Convention:
  • pixel centres are integers; see PinholeCamera.

  • camera_matrix must be (B, 3, 3); an unbatched matrix or extra leading dimensions raise ShapeError.

Parameters:
  • height (int) – height of image.

  • width (int) – width of image.

  • camera_matrix (Tensor) – tensor containing the camera intrinsics with shape (B, 3, 3).

  • normalize_points (bool, optional) – whether to normalize the pointcloud. This must be set to True when the depth is represented as the Euclidean ray length from the camera position. Default: False

  • device (Optional[device], optional) – device to place the result on. Default: None

  • dtype (Optional[dtype], optional) – dtype of the pixel grid; the output dtype follows promotion with camera_matrix. Default: None

Return type:

Tensor

Returns:

the ray through each pixel, with z equal to 1 (or unit length with normalize_points=True), with shape (B, H, W, 3).

See also

kornia.geometry.depth.depth_to_normals(depth, camera_matrix, normalize_points=False)[source]#

Estimate a unit surface normal per pixel from a depth map.

normals = depth_to_normals(depth, camera_matrix) takes the same (B, 1, H, W) depth map and (B, 3, 3) intrinsics as depth_to_3d(), differentiates the unprojected points with a Sobel spatial_gradient(), and returns the normalized cross product of the two tangents as a (B, 3, H, W) tensor. Normals point away from the camera: a fronto-parallel plane gets (0, 0, 1).

Convention:
  • both spatial dimensions must be at least 2: a surface normal requires two tangent directions.

  • a depth growing along the columns tilts the normal towards -x, one growing along the rows towards -y.

  • float16 and bfloat16 inputs are widened to float32 first. The result has the promoted dtype of depth and camera_matrix, or the default floating dtype when both are integer, and it does not follow an enclosing autocast region: its dtype and values are the same inside autocast as outside.

Parameters:
  • depth (Tensor) – image tensor containing a depth value per pixel with shape (B, 1, H, W), with \(H \geq 2\) and \(W \geq 2\).

  • camera_matrix (Tensor) – tensor containing the camera intrinsics with shape (B, 3, 3).

  • normalize_points (bool, optional) – whether to normalize the pointcloud. This must be set to True when the depth is represented as the Euclidean ray length from the camera position. Default: False

Return type:

Tensor

Returns:

tensor with a normal surface vector per pixel of the same resolution as the input (B, 3, H, W).

Raises:

ShapeError – if either spatial dimension of depth is smaller than 2.

See also

Example

>>> depth = torch.rand(1, 1, 4, 4)
>>> K = torch.eye(3)[None]
>>> depth_to_normals(depth, K).shape
torch.Size([1, 3, 4, 4])
kornia.geometry.depth.depth_from_plane_equation(plane_normals, plane_offsets, points_uv, camera_matrix, eps=1e-8)[source]#

Compute the depth of given pixels that lie on a known plane.

depth = depth_from_plane_equation(plane_normals, plane_offsets, points_uv, camera_matrix) intersects the camera ray of each pixel in points_uv of shape (B, N, 2) with the camera-frame plane \(n \cdot X = d\) and returns the z of each intersection as a (B, N) tensor: a list of depths, not a depth map.

Convention:
  • the plane is in Hessian form, with the normal plane_normals of shape (B, 3) and the offset plane_offsets of shape (B, 1): n = (0, 0, 1) with d = 2 is the plane z = 2, and every pixel on it has depth 2.

  • points_uv are pixels (integer centres; see PinholeCamera), normalized here with camera_matrix.

  • a ray-plane dot product inside \((-eps, eps)\) is replaced by \(\pm\) eps with its sign (+eps for an exact zero), so a grazing ray returns the large signed depth d / eps rather than inf. In float16 a positive eps is floored at the smallest subnormal, 2**-24, because the default 1e-8 rounds to zero there; d / 2**-24 is still inf in float16 once |d| exceeds about 3.9e-3, so pass a larger eps for a finite depth (eps=1e-4 keeps |d| <= 6.5 finite).

Parameters:
  • plane_normals (torch.Tensor) – Plane normal vectors of shape (B, 3).

  • plane_offsets (torch.Tensor) – Plane offsets of shape (B, 1).

  • points_uv (torch.Tensor) – Pixel coordinates of shape (B, N, 2).

  • camera_matrix (torch.Tensor) – Camera intrinsic matrix of shape (B, 3, 3).

  • eps (float, optional) – epsilon for numerical stability. Default: 1e-8

Returns:

Computed depth values at the given pixels, shape (B, N).

Return type:

torch.Tensor

See also

  • unproject_points(): turn these pixels and depths into 3D points.

  • depth_from_disparity(): a whole depth map from a rectified stereo disparity map, not from a plane.

  • Hyperplane: a plane object, \(n \cdot x + d = 0\), with the offset of the opposite sign: for a batch of planes, pass plane.normal.data and -plane.offset.data[:, None].

kornia.geometry.depth.warp_frame_depth(image_src, depth_dst, src_trans_dst, camera_matrix, normalize_points=False)[source]#

Render an image from another viewpoint, given that viewpoint’s depth map and the relative pose.

image_dst = warp_frame_depth(image_src, depth_dst, src_trans_dst, camera_matrix) shows the source image (B, D, H, W) as the destination camera sees it. Each destination pixel is unprojected with its depth from depth_dst of shape (B, 1, H, W), moved into the source frame by the (B, 4, 4) pose src_trans_dst, projected with the (B, 3, 3) intrinsics camera_matrix that both views share, and sampled bilinearly. DepthWarper and depth_warp() do the same warp from two PinholeCamera models; for a planar scene, warp_perspective() with a homography needs no depth.

Convention:
  • src_trans_dst maps destination-frame points into the source frame. With fx = fy = 1 and a unit depth, a translation of 1 along x samples image_src one pixel to the right of each destination pixel.

  • the unprojection is depth_to_3d_v2() and the reprojection project_points(), whose conventions apply.

  • the sampling is bilinear grid_sample with a fixed align_corners=True and padding_mode="zeros"; the output keeps image_src’s channels.

  • DepthWarper and depth_warp() name the frames the other way round: this function’s dst (the frame holding the depth) is their src, and image_src is their patch_dst.

  • a caller’s autocast region does not change the sampling grid, outside TorchScript.

Parameters:
  • image_src (Tensor) – image tensor in the source frame with shape (B, D, H, W).

  • depth_dst (Tensor) – depth tensor in the destination frame with shape (B, 1, H, W).

  • src_trans_dst (Tensor) – transformation matrix from destination to source with shape (B, 4, 4).

  • camera_matrix (Tensor) – tensor containing the camera intrinsics with shape (B, 3, 3).

  • normalize_points (bool, optional) – whether to normalize the pointcloud. This must be set to True when the depth is represented as the Euclidean ray length from the camera position. Default: False

Return type:

Tensor

Returns:

image_src resampled onto the destination pixel grid, with shape (B, D, H, W).

A float16/bfloat16 call with every input in that dtype runs in float32 on every device and returns the input dtype: half-precision pixel coordinates and a half-precision grid are off by a pixel or more at 256 px.

See also

kornia.geometry.depth.depth_warp(pinhole_dst, pinhole_src, depth_src, patch_dst, height, width, align_corners=True)[source]#

Render an image from another viewpoint by depth, given two PinholeCamera models (one call).

image = depth_warp(pinhole_dst, pinhole_src, depth_src, patch_dst, H, W) samples the destination image patch_dst of shape (B, C, H, W) at the pixels where the source-frame depth depth_src projects, so the result shows patch_dst as the source camera sees it. It is DepthWarper in one call, with that class’s default mode and padding_mode and the same output bit for bit; see its Convention block. warp_frame_depth() takes a 4x4 pose and one intrinsics matrix instead of two cameras.

Parameters:
  • pinhole_dst (PinholeCamera) – the pinhole model for the destination frame, the one patch_dst is seen from.

  • pinhole_src (PinholeCamera) – the pinhole model for the source frame, the one depth_src belongs to.

  • depth_src (Tensor) – the depth in the source frame, with shape (B, 1, H, W).

  • patch_dst (Tensor) – the image in the destination frame, with shape (B, C, H, W).

  • height (int) – the height of the images, H.

  • width (int) – the width of the images, W.

  • align_corners (bool, optional) – forwarded to grid_sample only; the sampling grid is always normalized for align_corners=True, so False misaligns the warp (#6317). The default True is what DepthWarper uses. Default: True

Return type:

Tensor

Returns:

patch_dst resampled onto the source pixel grid, with shape (B, C, H, W).

See also

Example

>>> # pinholes camera models
>>> pinhole_dst = PinholeCamera(torch.randn(1, 4, 4), torch.randn(1, 4, 4),
... torch.tensor([32]), torch.tensor([32]))
>>> pinhole_src = PinholeCamera(torch.randn(1, 4, 4), torch.randn(1, 4, 4),
... torch.tensor([32]), torch.tensor([32]))
>>> # warp the destination frame to the source frame by depth
>>> depth_src = torch.ones(1, 1, 32, 32)  # Nx1xHxW
>>> image_dst = torch.rand(1, 3, 32, 32)  # NxCxHxW
>>> image_src = depth_warp(pinhole_dst, pinhole_src, depth_src, image_dst, 32, 32)  # NxCxHxW
class kornia.geometry.depth.DepthWarper(pinhole_dst, height, width, mode='bilinear', padding_mode='zeros', align_corners=True)[source]#

Render an image from another viewpoint by depth, given two PinholeCamera models (module form).

Build warper = DepthWarper(pinhole_dst, H, W), call warper.compute_projection_matrix(pinhole_src), then image = warper(depth_src, patch_dst) samples the destination image patch_dst of shape (B, C, H, W) at the pixels where the source-frame depth depth_src projects, so the (B, C, H, W) result shows patch_dst as the source camera sees it. Calling the warper before compute_projection_matrix() raises ValueError. depth_warp() is the one-call form; warp_frame_depth() takes a 4x4 pose and one intrinsics matrix instead of two cameras.

\[ \begin{align}\begin{aligned}P_{src}^{\{dst\}} = K_{dst} * T_{src}^{\{dst\}}\\I_{src} = \omega(I_{dst}, P_{src}^{\{dst\}}, D_{src})\end{aligned}\end{align} \]
Convention:
  • the depth lives in the source frame and the image in the destination frame.

  • until compute_projection_matrix() has been called, warp_grid() raises ValueError and compute_subpixel_step() raises RuntimeError.

  • Half-precision cameras of one dtype compose in float32, and a float16/bfloat16 patch with depth of its dtype builds its grid and samples in float32 on every device and returns the input dtype. warp_grid() casts the grid back to the depth’s dtype.

  • compute_projection_matrix() stores K_dst @ (E_dst @ inverse(E_src)), the matrix above, from the constructor’s pinhole_dst and that method’s pinhole_src (both PinholeCamera, world-to-camera extrinsics). Both cameras use only the top-left 3x3 block of their intrinsics, embedded in homogeneous 4x4 form; the stored fourth row and column are ignored.

  • compute_subpixel_step() is half a pixel divided by how far the image centre, ((W - 1) / 2, (H - 1) / 2), moves in the destination image per unit of inverse depth around inverse depth 1, the smallest over the batch. Two cameras with no baseline have no parallax, so it is inf, or a very large value when rounding leaves a residual translation; such a camera passes no gradient (zero, by convention, since none exists at inf), so it cannot turn the gradients of the rest of the batch into NaN. With focal length f and baseline b, a rectified pair gives 0.5 / (f * b). It measures the image centre only: exact for a rectified pair, where every pixel moves the same, and not a bound for the whole image otherwise. Under forward motion the centre barely moves, so the step can be far coarser than the corners need.

  • the grid attribute is the homogeneous integer-centre pixel grid (pixel (0, 0) is (0, 0, 1)); warp_grid() returns sampling positions in grid_sample’s align_corners=True normalized coordinates of the destination image.

  • align_corners (default True), mode and padding_mode are forwarded to grid_sample, but warp_grid() normalizes for align_corners=True whatever the flag, so False misaligns the warp and an identity warp no longer returns its input (#6317).

  • warp_frame_depth() names the frames the other way round: its dst (the frame holding the depth) is this class’s src, and its image_src is patch_dst.

  • a caller’s autocast region does not change compute_projection_matrix() for cameras of one dtype. Cameras that mix float32 and half dtypes raise outside a region and compose in float32 inside one.

Warning

The two warps agree to rounding for float32/float64 away from z = 0, and split at z = 0, where cam2pixel sends every pixel with a nonzero projected numerator far outside the image: #4267.

Parameters:
  • pinhole_dst (PinholeCamera) – the pinhole model for the destination frame.

  • height (int) – the height of the image to warp.

  • width (int) – the width of the image to warp.

  • mode (str, optional) – interpolation mode to calculate output values 'bilinear' | 'nearest'. Default: "bilinear"

  • padding_mode (str, optional) – padding mode for outside grid values 'zeros' | 'border' | 'reflection'. Default: "zeros"

  • align_corners (bool, optional) – interpolation flag. Default: True

See also

compute_projection_matrix(pinhole_src)[source]#

Compute the projection matrix from the source to destination frame.

See the Convention block on DepthWarper.

Return type:

DepthWarper

compute_subpixel_step()[source]#

Compute the smallest inverse depth step over the batch for sub-pixel sampling of the depth cost volume.

See the Convention block on DepthWarper.

Szeliski, Richard, and Daniel Scharstein. “Symmetric sub-pixel stereo matching.” European Conference on Computer Vision. Springer Berlin Heidelberg, 2002.

Return type:

Tensor

forward(depth_src, patch_dst)[source]#

Warp a tensor from the destination frame to the source frame, given the depth in the source frame.

See the Convention block on DepthWarper.

Parameters:
  • depth_src (Tensor) – the depth in the source frame. The tensor must have a shape (B, 1, H, W).

  • patch_dst (Tensor) – the patch in the destination frame. The tensor must have a shape (B, C, H, W).

Return type:

Tensor

Returns:

the patch warped from the destination frame to the source frame.

Shape:
  • Input: (B, 1, H, W) and (B, C, H, W).

  • Output: (B, C, H, W) where C = number of channels.

Example

>>> # pinholes camera models
>>> pinhole_dst = PinholeCamera(torch.randn(1, 4, 4), torch.randn(1, 4, 4),
... torch.tensor([32]), torch.tensor([32]))
>>> pinhole_src = PinholeCamera(torch.randn(1, 4, 4), torch.randn(1, 4, 4),
... torch.tensor([32]), torch.tensor([32]))
>>> # create the depth warper, compute the projection matrix
>>> warper = DepthWarper(pinhole_dst, 32, 32)
>>> _ = warper.compute_projection_matrix(pinhole_src)
>>> # warp the destination frame to the source frame by depth
>>> depth_src = torch.ones(1, 1, 32, 32)  # Nx1xHxW
>>> image_dst = torch.rand(1, 3, 32, 32)  # NxCxHxW
>>> image_src = warper(depth_src, image_dst)  # NxCxHxW
warp_grid(depth_src)[source]#

Compute where each source pixel samples the destination image, given the depth in the source frame.

See the Convention block on DepthWarper.

The method compute_projection_matrix has to be called beforehand in order to have precomputed the projection matrix, which moves a source-frame point into the destination camera frame and applies the destination intrinsics.

Return type:

Tensor