kornia.sensors.camera#

Warning

kornia.sensors.camera is an experimental API and is subject to change. Once finished, it will subsume kornia.geometry.camera, although the two are kept separate by design – they share the mathematical Pinhole mapping but differ in input types, camera-axis broadcasting and zero-depth handling (see the Convention block and warning on CameraModelBase), which is recorded in #4274.

The objective of kornia.sensors.camera is to express well-known camera models such as Pinhole, Kannala Brandt, and others in terms of distortion and projection types while ensuring differentiability. We also aim to equip the user with tools to define custom camera models.

Defining a Pinhole camera model is as simple as:

import torch
from kornia.image import ImageSize
from kornia.sensors.camera import CameraModel, CameraModelType

params = torch.tensor([328., 328., 320., 240.])  # fx, fy, cx, cy
cam = CameraModel(ImageSize(480, 640), CameraModelType.PINHOLE, params)

To define a custom camera model based on distortion and projection types, one can use the CameraModelBase API:

import torch
from kornia.image import ImageSize
from kornia.sensors.camera import CameraModelBase
from kornia.sensors.camera.distortion_model import AffineTransform
from kornia.sensors.camera.projection_model import Z1Projection

params = torch.tensor([328., 328., 320., 240.])
cam = CameraModelBase(AffineTransform(), Z1Projection(), ImageSize(480, 640), params)

Note

The built-in camera models are Pinhole, Brown-Conrady, Kannala-Brandt K3, and Orthographic.

class kornia.sensors.camera.CameraModelBase(distortion, projection, image_size, params)[source]#

Base class to represent camera models based on distortion and projection types.

Distortion is of 3 types:

Affine, implemented by AffineTransform
Brown Conrady, implemented by BrownConradyTransform
Kannala Brandt K3, implemented by KannalaBrandtK3Transform

Projection is of 2 types:

Z1, implemented by Z1Projection
Orthographic, implemented by OrthographicProjection

Convention:

the API is Vector-typed: project() takes a Vector3 and returns a Vector2, and unproject() the reverse; a raw torch.Tensor is not accepted. This type system is kept separate from kornia.geometry.camera by design (#4274).
params is a flat (N,) or batched $(B, N)$ vector whose length is fixed by the model: [fx, fy, cx, cy] for PINHOLE and ORTHOGRAPHIC, 12 for BROWN_CONRADY and 8 for KANNALA_BRANDT_K3; another length or a rank above 2 raises ValueError.
matrix() and its alias K() return the $(*, 3, 3)$ K, not the $(B, 4, 4)$ intrinsics that PinholeCamera stores.
project() is distortion.distort(params, projection.project(points)) and unproject() the reverse; depth is the camera-frame z, shaped (*,) where unproject_points() takes (*, 1). Pixel centres are integers (see PinholeCamera).

Warning

Batched (B, 4) intrinsics apply along the last point axis, not the camera axis: with (B, N, 3) points and B > 1 they silently pair with the wrong points when B == N, project() broadcasts to a (B, B, 2) result when N == 1, and other N raise, whereas the geometry functions apply them per camera: #4274. project() divides by z with no guard, so z = 0 gives infinities where project_points() returns finite pixels: #4267.

Example

>>> params = torch.Tensor([328., 328., 320., 240.])
>>> cam = CameraModelBase(AffineTransform(), Z1Projection(), ImageSize(480, 640), params)
>>> cam.params
tensor([328., 328., 320., 240.])

K()[source]#

Return the camera matrix.

See the Convention block on CameraModelBase.

Return type:: Tensor

property cx: Tensor#: Returns the principal point in x direction.

property cy: Tensor#: Returns the principal point in y direction.

property fx: Tensor#: Returns the focal length in x direction.

property fy: Tensor#: Returns the focal length in y direction.

property height: int | Tensor#: Returns the height of the image.

property image_size: ImageSize#: Returns the image size of the camera model.

matrix()[source]#

Return the camera matrix.

See the Convention block on CameraModelBase.

Return type:: Tensor

property params: Tensor#: Returns the camera parameters.

project(points)[source]#

Projects 3D points to 2D camera plane.

See the Convention block on CameraModelBase.

Parameters:: points (Vector3) – Vector3 representing 3D points.
Return type:: Vector2
Returns:: Vector2 representing the projected 2D points.

Example

>>> points = Vector3(torch.Tensor([1.0, 1.0, 1.0]))
>>> cam = CameraModel(ImageSize(480, 640), CameraModelType.PINHOLE, torch.Tensor([328., 328., 320., 240.]))
>>> cam.project(points)
x: 648.0
y: 568.0

unproject(points, depth)[source]#

Unprojects 2D points from camera plane to 3D.

See the Convention block on CameraModelBase.

Parameters:

points (Vector2) – Vector2 representing 2D points.
depth (Tensor) – Depth of the points.

Return type:

Vector3

Returns:

Vector3 representing the unprojected 3D points.

Example

>>> points = Vector2(torch.Tensor([1.0, 1.0]))
>>> cam = CameraModel(ImageSize(480, 640), CameraModelType.PINHOLE, torch.Tensor([328., 328., 320., 240.]))
>>> cam.unproject(points, torch.Tensor([1.0]))
x: tensor([-0.9726])
y: tensor([-0.7287])
z: tensor([1.])

property width: int | Tensor#: Returns the width of the image.

class kornia.sensors.camera.CameraModel(image_size, model_type, params)[source]#

Class to represent camera models.

See the Convention block on CameraModelBase. See camera and world conventions for the corresponding camera frame, depth, pixel-centre and intrinsics conventions.

Example

>>> # Pinhole Camera Model
>>> cam = CameraModel(ImageSize(480, 640), CameraModelType.PINHOLE, torch.Tensor([328., 328., 320., 240.]))
>>> # Brown Conrady Camera Model
>>> cam = CameraModel(ImageSize(480, 640), CameraModelType.BROWN_CONRADY, torch.Tensor([1.0, 1.0, 1.0, 1.0,
... 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]))
>>> # Kannala Brandt K3 Camera Model
>>> cam = CameraModel(ImageSize(480, 640), CameraModelType.KANNALA_BRANDT_K3, torch.Tensor([1.0, 1.0, 1.0,
... 1.0, 1.0, 1.0, 1.0, 1.0]))
>>> # Orthographic Camera Model
>>> cam = CameraModel(ImageSize(480, 640), CameraModelType.ORTHOGRAPHIC, torch.Tensor([328., 328., 320., 240.]))
>>> cam.params
tensor([328., 328., 320., 240.])

class kornia.sensors.camera.CameraModelType(value)[source]#

Represent the type of camera projection and distortion model.

Supported types:

PINHOLE: Standard perspective projection with no distortion.
BROWN_CONRADY: Standard radial and tangential distortion model, often used for wide-angle lenses.
KANNALA_BRANDT_K3: Fisheye distortion model using a 9th-order polynomial for equidistant projection.
ORTHOGRAPHIC: Parallel projection where rays are perpendicular to the image plane, with no perspective effect.

class kornia.sensors.camera.PinholeModel(image_size, params)[source]#

Class to represent Pinhole Camera Model.

See the Convention block on CameraModelBase. See camera and world conventions for the corresponding camera frame, depth, pixel-centre and intrinsics conventions.

The pinhole camera model describes the mathematical relationship between the coordinates of a point in three-dimensional space and its projection onto the image plane of an ideal pinhole camera, where the camera aperture is described as a point and no lenses are used to focus light. See more: https://en.wikipedia.org/wiki/Pinhole_camera_model

Example

>>> cam = CameraModel(ImageSize(480, 640), CameraModelType.PINHOLE, torch.Tensor([328., 328., 320., 240.]))
>>> cam
CameraModel(ImageSize(height=480, width=640), PinholeModel, tensor([328., 328., 320., 240.]))

matrix()[source]#

Return the camera matrix.

See the Convention block on CameraModelBase.

The matrix is of the form:

\[\begin{split}\begin{bmatrix} fx & 0 & cx \\ 0 & fy & cy \\ 0 & 0 & 1\end{bmatrix}\end{split}\]

Example

>>> cam = CameraModel(ImageSize(480, 640), CameraModelType.PINHOLE, torch.Tensor([1.0, 2.0, 3.0, 4.0]))
>>> cam.matrix()
tensor([[1., 0., 3.],
        [0., 2., 4.],
        [0., 0., 1.]])

Return type:: Tensor

scale(scale_factor)[source]#

Scales the camera model by a scale factor.

Convention:

returns a new model whose focal lengths, principal point and image size are multiplied by scale_factor: fx' = s * fx and cx' = s * cx, as scale() does.

Warning

cx' = s * cx disagrees with kornia’s integer pixel centres, and a tensor scale_factor rebuilds image_size with 0-dim tensors where the constructor took python integers: #4263.

Parameters:: scale_factor (Tensor) – Scale factor to scale the camera model.
Return type:: PinholeModel
Returns:: Scaled camera model.

Example

>>> cam = CameraModel(ImageSize(480, 640), CameraModelType.PINHOLE, torch.Tensor([328., 328., 320., 240.]))
>>> cam_scaled = cam.scale(2)
>>> cam_scaled.params
tensor([656., 656., 640., 480.])

class kornia.sensors.camera.BrownConradyModel(image_size, params)[source]#

Brown Conrady Camera Model.

See camera and world conventions for shared intrinsics, pixel-centre and camera-frame conventions.

class kornia.sensors.camera.KannalaBrandtK3(image_size, params)[source]#

Kannala Brandt K3 Camera Model.

See camera and world conventions for shared intrinsics, pixel-centre and camera-frame conventions.

class kornia.sensors.camera.Orthographic(image_size, params)[source]#

Orthographic Camera Model.

See camera and world conventions for shared intrinsics, pixel-centre and camera-frame conventions.

Distortions#

class kornia.sensors.camera.distortion_model.AffineTransform[source]#

Apply an affine transformation to a set of 2D points.

This class handles the scaling and shifting of coordinates, typically used to map normalized coordinates to pixel coordinates.

distort(params, points)[source]#

Distort one or more Vector2 points using the affine transform.

Convention:

points is on the normalized $z = 1$ plane and the result is in pixels: u = fx * x + cx and v = fy * y + cy, with params laid out as [fx, fy, cx, cy]; the Vector2 counterpart of distort_points_affine().
undistort() is the closed-form inverse.

Parameters:

params (Tensor) – torch.Tensor representing the affine transform parameters.
points (Vector2) – Vector2 representing the points to distort.

Return type:

Vector2

Returns:

Vector2 representing the distorted points.

Example

>>> params = torch.Tensor([1., 2., 3., 4.])
>>> points = Vector2.from_coords(1., 2.)
>>> AffineTransform().distort(params, points)
x: 4.0
y: 8.0

undistort(params, points)[source]#

Undistort one or more Vector2 points using the affine transform.

Convention:

points is in pixels and the result is on the normalized $z = 1$ plane: x = (u - cx) / fx and y = (v - cy) / fy, the closed-form inverse of distort() with the same params layout – see the Convention block there.

Parameters:

params (Tensor) – torch.Tensor representing the affine transform parameters.
points (Vector2) – Vector2 representing the points to undistort.

Return type:

Vector2

Returns:

Vector2 representing the undistorted points.

Example

>>> params = torch.Tensor([1., 2., 3., 4.])
>>> points = Vector2.from_coords(1., 2.)
>>> AffineTransform().undistort(params, points)
x: -2.0
y: -1.0

class kornia.sensors.camera.distortion_model.BrownConradyTransform[source]#

Implement the Brown-Conrady model for lens distortion and undistortion.

The model accounts for radial distortion (due to lens shape) and tangential distortion (due to lens misalignment). It is commonly used to transform points between ideal pinhole projections and distorted image coordinates.

The transform expects a 12-value parameter vector ordered as $(fx, fy, cx, cy, k1, k2, p1, p2, k3, k4, k5, k6)$. The first four values are camera intrinsics and the trailing eight values are the Brown-Conrady distortion coefficients.

distort() maps normalized $z = 1$ image-plane coordinates to distorted pixel coordinates, while undistort() maps distorted pixel coordinates back to normalized $z = 1$ coordinates.

distort(params, points)[source]#

Apply Brown-Conrady lens distortion to ideal normalized points.

Parameters:

params (Tensor) – Camera and distortion parameters ordered as (fx, fy, cx, cy, k1, k2, p1, p2, k3, k4, k5, k6). The first four values define the camera intrinsics and the trailing eight values follow the Brown-Conrady coefficient layout used by kornia.geometry.calibration.distort_points().
points (Vector2) – Ideal two-dimensional points on the normalized $z = 1$ image plane. Leading dimensions may represent a batch.

Return type:

Vector2

Returns:

Distorted two-dimensional points in pixel coordinates.

undistort(params, points)[source]#

Remove Brown-Conrady lens distortion from observed points.

Parameters:

params (Tensor) – Camera and distortion parameters ordered as (fx, fy, cx, cy, k1, k2, p1, p2, k3, k4, k5, k6). The layout is identical to distort().
points (Vector2) – Distorted two-dimensional points in pixel coordinates. Leading dimensions may represent a batch.

Return type:

Vector2

Returns:

Undistorted two-dimensional points on the normalized $z = 1$ image plane.

class kornia.sensors.camera.distortion_model.KannalaBrandtK3Transform[source]#

Apply the Kannala-Brandt (K3) distortion model.

This model is specifically designed for fisheye lenses with significant radial distortion, using a polynomial approximation for the projection.

distort(params, points)[source]#

Apply Kannala-Brandt K3 fisheye distortion to normalized points.

Parameters:

params (Tensor) – Fisheye distortion coefficients for the K3 polynomial model.
points (Vector2) – Ideal two-dimensional normalized points before fisheye distortion is applied.

Return type:

Vector2

Returns:

Distorted two-dimensional points following the K3 fisheye model.

undistort(params, points)[source]#

Remove Kannala-Brandt K3 fisheye distortion from observed points.

Parameters:

params (Tensor) – Fisheye distortion coefficients matching the K3 model used for distortion.
points (Vector2) – Distorted two-dimensional fisheye points.

Return type:

Vector2

Returns:

Undistorted normalized points that approximate ideal pinhole coordinates.

Projections#

class kornia.sensors.camera.projection_model.Z1Projection[source]#

Project 3D points from the camera frame into the canonical $z=1$ plane.

This performs perspective division by dividing the $x$ and $y$ coordinates by the depth $z$.

project(points)[source]#

Project one or more Vector3 from the camera frame into the canonical z=1 plane through perspective division.

Convention:

points is in the camera frame and the result is on the normalized $z = 1$ plane, not in pixels: the map is xy / z, with no epsilon and no validation.

Warning

A point with $z = 0$ projects to an infinity (nan for a zero numerator) instead of raising: #4267. A point behind the camera projects to a finite coordinate and is not flagged: #4555.

Parameters:: points (Vector3) – Vector3 representing the points to project.
Return type:: Vector2
Returns:: Vector2 representing the projected points.

Example

>>> points = Vector3.from_coords(1., 2., 3.)
>>> Z1Projection().project(points)
x: 0.3333333432674408
y: 0.6666666865348816

unproject(points, depth)[source]#

Unproject one or more Vector2 from the canonical z=1 plane into the camera frame.

Convention:

depth is the camera-frame z: the $z = 1$ point is multiplied by it, so the third coordinate of the result is the depth that was passed in, and not a Euclidean ray length.
a python float or int depth is converted to the device and dtype of points.

Parameters:

points (Vector2) – Vector2 representing the points to unproject.
depth (Tensor | float) – a torch.Tensor of shape (B,), or a python scalar for a single point.

Return type:

Vector3

Returns:

Vector3 representing the unprojected points.

Example

>>> points = Vector2.from_coords(1., 2.)
>>> Z1Projection().unproject(points, 3)
x: tensor([3.])
y: tensor([6.])
z: tensor([3.])

class kornia.sensors.camera.projection_model.OrthographicProjection[source]#

Project 3D points using an orthographic projection model.

This model assumes parallel projection where the $z$ coordinate is discarded and no perspective scaling is applied.

project(points)[source]#

Project 3D camera-frame points with an orthographic camera model.

Orthographic projection keeps the horizontal and vertical coordinates unchanged and discards depth. Unlike perspective projection, objects do not shrink as their z value increases.

Parameters:: points (Vector3) – Three-dimensional point container with coordinates x, y, and z. Leading dimensions may represent a batch of points.
Return type:: Vector2
Returns:: Two-dimensional point container containing the projected x and y coordinates.

unproject(points, depth)[source]#

Lift orthographic image-plane points back into 3D using depth.

Parameters:

points (Vector2) – Two-dimensional point container with image-plane coordinates x and y.
depth (Tensor) – Tensor containing the target z coordinate for each unprojected point.

Return type:

Vector3

Returns:

Three-dimensional point container with x and y copied from points and z supplied by depth.