Local affine frames (LAF)#

A local affine frame (LAF) describes a keypoint by a 2 x 3 matrix [A | c], and a set of them is a (B, N, 2, 3) tensor. The centre c = (x, y) is in pixels, and the shape A maps the canonical patch frame onto the image, in pixels: its scale is \(\sqrt{|\det A|}\) and its orientation atan2(a12, a11), in degrees counter-clockwise on screen for a LAF of positive determinant (clockwise for a mirrored one). Local features gives the frame and the conversion of OpenCV keypoints. kornia’s scale-space and KeyNet detectors (ScaleSpaceDetector, MultiResolutionDetector, KeyNetDetector) return LAFs, LAFDescriptor describes them, and match_fginn(), match_adalam() and LightGlueMatcher read them; laf_from_center_scale_ori() turns the keypoints of the other detectors into LAFs.

Creating and reading LAFs#

kornia.feature.laf_from_center_scale_ori(xy, scale=None, ori=None)[source]#

Build circular LAFs from keypoint centres, scales and orientations.

laf = laf_from_center_scale_ori(xy, scale, ori) takes keypoint centres xy (B, N, 2), scales (B, N, 1, 1) (1 when omitted) and orientations (B, N, 1) in degrees (0 when omitted), and returns the (B, N, 2, 3) LAFs [scale * R(ori) | xy], whose regions are circles of radius scale around xy. This turns the keypoints of a detector that returns no LAFs, such as DISK or XFeat, into the LAFs that LAFDescriptor and the LAF matchers take, e.g. laf_from_center_scale_ori(keypoints[None]). For an elliptical region, use ellipse_to_laf() instead.

Convention:

  • A LAF is a (B, N, 2, 3) tensor [A | c], with the centre c = (x, y) and A in pixels. A maps the patch frame onto the image: its first column is the image of the patch’s x axis and its second column that of the patch’s y axis. Local features gives the pixel frame, the patch that extract_patches_from_pyramid() samples and the conversion of OpenCV keypoints.

  • This function returns A = scale * R, with R = [[cos, sin], [-sin, cos]] of ori in degrees, the matrix of angle_to_rotation_matrix(). A positive ori turns the patch’s x axis counter-clockwise on screen, with image y pointing down.

  • For any LAF, the scale is \(\sqrt{|\det A|}\) (get_laf_scale(), which adds 1e-10 to the determinant) and the orientation is atan2(a12, a11) in degrees (get_laf_orientation()), within [-180, 180]: a LAF made with ori = 181 reads back -179. For \(\det A > 0\), A = U R with U the upright A that make_upright() returns, up to its stabilizers, and R the rotation matrix of that orientation.

  • A negative determinant mirrors the patch frame. Increasing its orientation turns its x axis clockwise on screen. With a nonzero scale, this constructor produces a positive determinant.

Parameters:

  • xy (Tensor) – (B, N, 2).

  • scale (Optional[Tensor], optional) – (B, N, 1, 1). If not provided, scale = 1.0 is assumed Default: None

  • ori (Optional[Tensor], optional) – angle in degrees (B, N, 1). If not provided orientation = 0 is assumed Default: None

Return type:

Tensor

Returns:

LAF (B, N, 2, 3)

See also

Example

>>> xy = torch.tensor([[[5.0, 7.0]]])
>>> laf_from_center_scale_ori(xy, torch.full((1, 1, 1, 1), 2.0), torch.tensor([[[30.0]]]))
tensor([[[[ 1.7321,  1.0000,  5.0000],
          [-1.0000,  1.7321,  7.0000]]]])
kornia.feature.ellipse_to_laf(ells)[source]#

Convert ellipses in the Oxford [x, y, a, b, c] format to upright LAFs.

laf = ellipse_to_laf(ells) takes (B, N, 5) ellipses [x, y, a, b, c], as the Oxford affine-region files write them, and returns the upright (B, N, 2, 3) LAFs of those regions: centred on (x, y), with an orientation of 0. To draw the region of a LAF, use laf_to_boundary_points().

Convention:

  • An ellipse [x, y, a, b, c] is the region \(a u^2 + 2 b u v + c v^2 \le 1\) in the coordinates (u, v) relative to its centre (x, y), as in the Oxford affine-region files. (x, y) becomes the LAF centre unchanged.

  • A is the lower-triangular matrix with a positive diagonal and A @ A.T == inverse([[a, b], [b, c]]): the Cholesky factor of the inverse ellipse matrix, as VLFeat’s vl_frame2oell.m builds it, not the symmetric inverse square root (the two differ by a rotation). Either one maps the unit circle onto the ellipse; the lower-triangular one is what make_upright() returns, up to its stabilizers, for any LAF of that region.

  • The conversion checks only the shape (B, N, 5); it does not check the ellipse values and does not raise on them. An ellipse that is degenerate or indefinite (a, c or a * c - b * b not positive) is not a bounded region; its LAF has inf or nan entries, or huge finite ones where rounding leaves a - b * b / c barely positive. Rounding also works the other way: a nearly singular valid ellipse can give inf or nan. laf_is_valid() screens out the inf and nan results, which an isnan test misses for inf; the huge finite ones pass it, so check the ellipse itself where that matters.

See also vlfeat/vlfeat

Parameters:

ells (Tensor) – torch.Tensor (B, N, 5) of ellipses in Oxford format [x y a b c].

Return type:

Tensor

Returns:

LAF (B, N, 2, 3)

See also

Example

>>> input = torch.ones(1, 10, 5)  # BxNx5
>>> output = ellipse_to_laf(input)  #  BxNx2x3
kornia.feature.laf_from_three_points(threepts)[source]#

Build LAFs from three points each: the tips of the patch’s x and y axes, then the centre.

laf = laf_from_three_points(pts) takes (B, N, 2, 3) tensors whose columns are three points p0, p1 and c and returns the (B, N, 2, 3) LAFs [p0 - c, p1 - c | c], whose patch x and y axes end at p0 and p1. Collinear points give a singular shape A. For LAFs from centres, scales and orientations, use laf_from_center_scale_ori() instead.

Convention:

See the Convention block on laf_to_three_points(), which this inverts: the columns of threepts are the images of the patch points (1, 0), (0, 1) and (0, 0), in that order.

Parameters:

threepts (Tensor) – (B, N, 2, 3).

Return type:

Tensor

Returns:

laf (B, N, 2, 3).

See also

kornia.feature.get_laf_center(LAF)[source]#

Read the centre of each LAF, its keypoint position (x, y).

xy = get_laf_center(laf) takes (B, N, 2, 3) LAFs and returns their centres c = (x, y), the last column of each LAF, as a (B, N, 2) tensor in the LAF’s units and dtype. The result is a view of laf, not a copy. To build LAFs from keypoints, use laf_from_center_scale_ori().

Convention:

The centre is (x, y) in pixels, with pixel centres at integers; see Local features.

Parameters:

LAF (Tensor) – (B, N, 2, 3)

Return type:

Tensor

Returns:

xy (B, N, 2)

See also

Example

>>> input = torch.ones(1, 5, 2, 3)  # BxNx2x3
>>> output = get_laf_center(input)  # BxNx2
kornia.feature.get_laf_scale(LAF)[source]#

Read the scale of each LAF, the square root of the absolute determinant of its shape.

scale = get_laf_scale(laf) takes (B, N, 2, 3) LAFs and returns their scales \(\sqrt{|\det A|}\) as a (B, N, 1, 1) tensor, in pixels for LAFs in pixels: the radius of a circular LAF’s region, and the geometric mean of the two semi-axes of an elliptical one. The result broadcasts against the shape A, so use scale_laf(laf, 1.0 / scale) to normalize an ordinary LAF. For the orientation and centre, use get_laf_orientation() and get_laf_center().

Convention:

  • See the Convention block on laf_from_center_scale_ori(), which defines the scale.

  • The 1e-10 added to the determinant makes frames smaller than about 1e-4 px read too large.

  • The determinant of a float16 LAF is formed in float32, so that a LAF whose entry products leave the float16 range does not overflow; the scale is returned in the LAF dtype.

Parameters:

LAF (Tensor) – (B, N, 2, 3)

Return type:

Tensor

Returns:

scale (B, N, 1, 1)

See also

Example

>>> input = torch.ones(1, 5, 2, 3)  # BxNx2x3
>>> output = get_laf_scale(input)  # BxNx1x1
kornia.feature.get_laf_orientation(LAF)[source]#

Read the orientation of each LAF in degrees, from the first row of its shape.

ori = get_laf_orientation(laf) takes (B, N, 2, 3) LAFs and returns their orientations atan2(a12, a11) in degrees, within [-180, 180], as a (B, N, 1) tensor. It reads only the first row of the shape A, which for A = U R, an upright factor times the rotation matrix of an angle, gives that angle whatever the shear of U. For a circular similarity frame A = s R, OpenCV’s KeyPoint.angle is (-ori) mod 360. For a general affine frame, use atan2(a21, a11) in degrees modulo 360; an OpenCV keypoint cannot retain its affine shape or reflection. To change the orientation, use set_laf_orientation() or rotate_laf().

Convention:

See the Convention block on laf_from_center_scale_ori(), which defines the orientation.

Parameters:

LAF (Tensor) – (B, N, 2, 3)

Return type:

Tensor

Returns:

angle in degrees (B, N, 1)

See also

Example

>>> input = torch.ones(1, 5, 2, 3)  # BxNx2x3
>>> output = get_laf_orientation(input)  # BxNx1
kornia.feature.laf_to_three_points(laf)[source]#

Convert LAFs to three points: the tips of the patch’s x and y axes, then the centre.

pts = laf_to_three_points(laf) takes (B, N, 2, 3) LAFs and returns (B, N, 2, 3) tensors whose columns are the image points c + A[:, 0], c + A[:, 1] and c. Mapping the three columns through a point transformation and converting them back with laf_from_three_points() transforms the LAFs, as perspective_transform_lafs() does for a homography.

Convention:

  • The columns of the result are the points c + A[:, 0], c + A[:, 1] and c, in that order: the images of the patch points (1, 0), (0, 1) and (0, 0), at the frame’s scale rather than of unit length. laf_from_three_points() is the inverse.

Parameters:

laf (Tensor) – (B, N, 2, 3).

Return type:

Tensor

Returns:

threepts (B, N, 2, 3).

See also

kornia.feature.laf_to_boundary_points(LAF, n_pts=50)[source]#

Sample points on the outline of each LAF’s region, after its centre, to draw it.

pts = laf_to_boundary_points(laf, n_pts) takes (B, N, 2, 3) LAFs and returns (B, N, n_pts, 2) points (x, y) in the LAF’s units: the centre, then n_pts - 1 points around the ellipse c + A u over the unit vectors u. Drawn as one line, the points of a LAF show its region and a radius from the centre to the tip of the patch’s y axis. For the centre and the tips of both axes only, use laf_to_three_points().

Convention:

  • The first point is the centre c. The other n_pts - 1 points are c + A @ (sin t, cos t) for t from 0 to \(2 \pi\) in equal steps: they start and end at c + A[:, 1], the tip of the patch’s y axis, and so close the outline of the region.

Parameters:

  • LAF (Tensor) – (B, N, 2, 3)

  • n_pts (int, optional) – number of points to output, the centre included. Default: 50

Returns:

(B, N, n_pts, 2)

Return type:

torch.Tensor of boundary points LAF

See also

Transforming LAFs#

kornia.feature.scale_laf(laf, scale_coef)[source]#

Multiply the shape of each LAF by a coefficient, keeping its centre.

scaled = scale_laf(laf, k) takes (B, N, 2, 3) LAFs and a coefficient k, a float or a tensor such as one coefficient per LAF in (B, N, 1, 1), and returns the (B, N, 2, 3) LAFs [k A | c], whose regions are scaled by |k| about the same centres. A descriptor then sees a larger or smaller area around each keypoint, as with the scaling_coef of LocalFeature. To read the scale, use get_laf_scale().

Convention:

  • The result is [k A | c] for k = scale_coef, a product taken elementwise: the centre is kept. With one coefficient per LAF, the scale is multiplied by |k|, and a negative k also turns the frame by 180 degrees.

  • scale_coef is a float or a tensor that broadcasts against (B, N, 2, 2), such as one coefficient per LAF in a (B, N, 1, 1) tensor; an int is rejected.

Parameters:

  • laf (Tensor) – (B, N, 2, 3)

  • scale_coef (Union[float, Tensor]) – broadcastable torch.Tensor or float.

Return type:

Tensor

Returns:

LAF (B, N, 2, 3)

Raises:

TypeError – if scale_coef is neither a float nor a tensor.

See also

Example

>>> input = torch.ones(1, 5, 2, 3)  # BxNx2x3
>>> scale = 0.5
>>> output = scale_laf(input, scale)  # BxNx2x3
kornia.feature.rotate_laf(LAF, angles_degrees)[source]#

Rotate each LAF’s patch frame by an angle in degrees, adding it to the orientation.

rotated = rotate_laf(laf, angles) takes (B, N, 2, 3) LAFs and angles (B, N, 1) in degrees and returns the (B, N, 2, 3) LAFs whose orientation is the input’s plus angles. Only the patch frame turns: the centre, scale and region, the ellipse c + A u over unit vectors u, stay the same. To set given orientations instead, use set_laf_orientation().

Convention:
Parameters:

  • LAF (Tensor) – (B, N, 2, 3)

  • angles_degrees (Tensor) – (B, N, 1) in degrees.

Return type:

Tensor

Returns:

LAF oriented with angles (B, N, 2, 3)

See also

kornia.feature.set_laf_orientation(LAF, angles_degrees)[source]#

Set the orientation of each LAF to an angle in degrees, keeping its region.

oriented = set_laf_orientation(laf, angles) takes (B, N, 2, 3) LAFs and orientations (B, N, 1) in degrees and returns the (B, N, 2, 3) LAFs rotated to them, as rotate_laf() rotates: the centre, the scale and region are kept. With angles of 0, a LAF of positive determinant becomes the upright LAF of make_upright(), up to its eps. To add angles to the current orientation, use rotate_laf() instead.

Convention:

See the Convention block on rotate_laf(). This rotates by angles_degrees minus the current orientation, so the orientation reads back as angles_degrees wrapped into [-180, 180].

Parameters:

  • LAF (Tensor) – (B, N, 2, 3)

  • angles_degrees (Tensor) – (B, N, 1) in degrees.

Return type:

Tensor

Returns:

LAF oriented with angles (B, N, 2, 3)

See also

kornia.feature.make_upright(laf, eps=1e-9)[source]#

Remove the rotation and reflection of LAFs, keeping their regions, centres and scales.

upright = make_upright(laf) takes (B, N, 2, 3) LAFs and returns the (B, N, 2, 3) LAFs of the same regions and centres with an orientation of 0 and no reflection: the shape A becomes the lower-triangular U of the Convention block. To set another orientation, use set_laf_orientation(); to re-estimate the shape from the image instead, use LAFAffineShapeEstimator.

Convention:

  • The result is [U | c]: the centre is kept, and U is the lower-triangular matrix with a positive diagonal and \(U U^\top = A A^\top\). The region, the ellipse c + A u over the unit vectors u, and the scale are therefore kept, and the orientation becomes 0.

  • For \(\det A > 0\), A = U R with R the rotation matrix of the LAF’s orientation, as in laf_from_center_scale_ori(). For \(\det A < 0\) the reflection is dropped too: U has the determinant \(|\det A|\).

  • The equalities above hold up to two stabilizers: U is built from the norm of the first row of A plus eps, and from the scale of get_laf_scale(), which adds 1e-10 to the determinant. Both are negligible for a LAF in pixels, but a tiny frame such as A = diag(1e-12, 1) keeps neither its region nor its scale. The centre and the zero orientation are exact.

  • The all-zero LAF that pads an unfilled detector slot (see laf_is_filled()) is returned unchanged, so the padding stays recognizable.

  • A float16 LAF is rectified in float32, so that a LAF whose entry products leave the float16 range does not overflow; the result is returned in the LAF dtype.

Parameters:

  • laf (Tensor) – (B, N, 2, 3)

  • eps (float, optional) – for safe division. Default: 1e-9

Returns:

(B, N, 2, 3)

Return type:

laf

See also

  • set_laf_orientation(): with angles of 0, the same LAFs for a positive determinant, but a reflection is kept.

  • LAFAffineShapeEstimator: re-estimates the shape from the image patch; with preserve_orientation=False its output is upright too.

  • ellipse_to_laf(): the same upright LAF, built from the coefficients of the region’s ellipse.

Example

>>> input = torch.ones(1, 5, 2, 3)  # BxNx2x3
>>> output = make_upright(input)  #  BxNx2x3
kornia.feature.perspective_transform_lafs(trans_01, lafs_1)[source]#

Map the LAFs of one image into another through a homography, by mapping three points of each.

lafs_0 = perspective_transform_lafs(trans_01, lafs_1) takes (B, 3, 3) homographies from the pixels of image 1 to those of image 0 and (B, N, 2, 3) LAFs of image 1, and returns the (B, N, 2, 3) LAFs of the same regions in image 0: the centres mapped exactly and the shapes approximated through the mapped tips of the patch axes. To map points, such as the LAF centres alone, use transform_points().

Convention:

  • trans_01 maps pixels of frame 1 to pixels of frame 0, as in transform_points(), and the centre maps through it exactly. The new A is the chord through the mapped tips of the patch axes: the three points of laf_to_three_points() are transformed and turned back into a LAF. For an affine trans_01 this equals the Jacobian of trans_01 at the centre times A; for a perspective one the two differ, the more so the larger the LAF.

Parameters:

  • trans_01 (Tensor) – torch.Tensor for perspective transformations of shape (B, 3, 3).

  • lafs_1 (Tensor) – torch.Tensor of lafs of shape (B, N, 2, 3).

Return type:

Tensor

Returns:

the transformed LAFs in frame 0, of shape (B, N, 2, 3).

Raises:

  • TypeError – if trans_01 is not a tensor or is on another device than lafs_1.

  • ValueError – if the batch sizes differ or trans_01 is not 3 x 3.

See also

Examples

>>> rng = torch.manual_seed(0)
>>> lafs_1 = torch.rand(2, 4, 2, 3)  # BxNx2x3
>>> lafs_1
tensor([[[[0.4963, 0.7682, 0.0885],
          [0.1320, 0.3074, 0.6341]],

         [[0.4901, 0.8964, 0.4556],
          [0.6323, 0.3489, 0.4017]],

         [[0.0223, 0.1689, 0.2939],
          [0.5185, 0.6977, 0.8000]],

         [[0.1610, 0.2823, 0.6816],
          [0.9152, 0.3971, 0.8742]]],


        [[[0.4194, 0.5529, 0.9527],
          [0.0362, 0.1852, 0.3734]],

         [[0.3051, 0.9320, 0.1759],
          [0.2698, 0.1507, 0.0317]],

         [[0.2081, 0.9298, 0.7231],
          [0.7423, 0.5263, 0.2437]],

         [[0.5846, 0.0332, 0.1387],
          [0.2422, 0.8155, 0.7932]]]])
>>> trans_01 = torch.eye(3).repeat(2, 1, 1)  # Bx3x3
>>> trans_01.shape
torch.Size([2, 3, 3])
>>> lafs_0 = perspective_transform_lafs(trans_01, lafs_1)  # BxNx2x3
kornia.feature.normalize_laf(LAF, images)[source]#

Normalize pixel LAFs by the image size: centre to [0, 1], shape by min(W - 1, H - 1).

nlaf = normalize_laf(laf, images) takes (B, N, 2, 3) LAFs in pixels and an image tensor (B, CH, H, W), read for its height and width only, and returns the (B, N, 2, 3) LAFs with the centre divided by (max(W - 1, 1), max(H - 1, 1)) and A by the smaller of the two. A singleton axis (H == 1 or W == 1) has no spatial extent and counts as one pixel, so the conversion stays finite instead of dividing by zero. denormalize_laf() on an image of another size then moves the LAFs to that resolution.

Convention:

  • The centre becomes (x / max(W - 1, 1), y / max(H - 1, 1)), which lies in [0, 1] for 0 <= x <= W - 1 and 0 <= y <= H - 1. A is divided by the smaller of the two denominators as a whole, not per axis, and can exceed 1.

  • Taken to an image of another size with denormalize_laf(), x becomes x max(W' - 1, 1) / max(W - 1, 1), which maps corner to corner when neither width is 1. A resize with align_corners=False, the default of resize() in bilinear mode, maps pixel x to (x + 0.5) W' / W - 0.5 instead, so the two disagree away from the image centre.

Parameters:

  • LAF (Tensor) – (B, N, 2, 3)

  • images (Tensor) – (B, CH, H, W)

Returns:

(B, N, 2, 3)

Return type:

the normalized LAF

See also

kornia.feature.denormalize_laf(LAF, images)[source]#

Convert LAFs from the frame of normalize_laf back to the pixels of an image.

laf = denormalize_laf(nlaf, images) takes (B, N, 2, 3) LAFs in the frame of normalize_laf() and an image tensor (B, CH, H, W), read for its height and width only, and returns the (B, N, 2, 3) LAFs in that image’s pixels: the centre multiplied by (max(W - 1, 1), max(H - 1, 1)) and A by the smaller of the two. A singleton axis (H == 1 or W == 1) has no spatial extent and counts as one pixel, so the conversion stays finite and round-trips with normalize_laf(). An image of another size than the one the LAFs were normalized on moves them to its resolution.

Convention:

See the Convention block on normalize_laf(), which this inverts.

Parameters:

  • LAF (Tensor) – (B, N, 2, 3)

  • images (Tensor) – (B, CH, H, W)

Returns:

(B, N, 2, 3), scale in pixels

Return type:

the denormalized LAF

See also

Checking LAFs#

kornia.feature.laf_is_valid(laf)[source]#

Check that each LAF is finite and has a finite, nonzero determinant.

valid = laf_is_valid(laf) takes (B, N, 2, 3) LAFs and returns a boolean (B, N) mask, true where every entry is finite and the shape A has a finite, nonzero determinant, so that it maps the patch onto a region of nonzero area. It does not check where the LAF lies: for the LAFs whose region lies inside the image, use laf_is_inside_image(), and to drop only the padding of a detector’s unfilled slots, use laf_is_filled() instead.

Convention:

  • A LAF with a negative determinant, a mirrored frame, is valid. The all-zero padding LAF (see laf_is_filled()), a rank-1 A and an A whose determinant overflows are not.

  • The determinant of a float16 LAF is formed in float32, so the LAF is not judged by an intermediate outside the float16 range: one whose entry products overflow float16 (for instance any LAF of scale 256 px or more) is valid, and so is one whose determinant is too small for float16. The determinant of a float32, float64 or bfloat16 LAF is formed in that dtype, where it can overflow.

Parameters:

laf (Tensor) – (B, N, 2, 3).

Return type:

Tensor

Returns:

validity mask (B, N).

See also

Example

>>> laf = torch.eye(2, 3).view(1, 1, 2, 3)
>>> laf_is_valid(laf)
tensor([[True]])
kornia.feature.laf_is_filled(laf)[source]#

Check which slots hold a detection rather than the zero-LAF padding.

filled = laf_is_filled(lafs) takes the (B, N, 2, 3) LAFs of a detector and returns a boolean (B, N) mask, false for all-zero padded frames: index the LAFs, responses and descriptors of image b with laf_is_filled(lafs)[b] to keep its detections. A filled frame need not be usable; to also drop singular and non-finite frames, use laf_is_valid() instead.

Convention:

  • kornia’s LAF detectors pad unused slots with an all-zero LAF and a zero response, so that a batch is one tensor. Drop those slots before matching: the padding frames are identical to one another, so a mutual nearest-neighbour test does not reject them and they match each other at zero distance.

  • Only an all-zero frame is padding. Any other frame is filled: a detection at the image origin, which has a zero centre but a nonzero shape, and also a singular or non-finite frame, which laf_is_valid() screens out.

  • Occupancy cannot be read off the response instead, because a pluggable signed response may have a genuine maximum at exactly zero.

Parameters:

laf (Tensor) – (B, N, 2, 3).

Return type:

Tensor

Returns:

mask of the slots a detection filled (B, N).

See also

Example

>>> laf = torch.tensor([[[[2., 0., 5.], [0., 3., 7.]], [[0., 0., 0.], [0., 0., 0.]]]])
>>> laf_is_filled(laf)
tensor([[ True, False]])
kornia.feature.laf_is_inside_image(laf, images, border=0)[source]#

Check which LAFs have their elliptical region fully inside the image, within a border.

inside = laf_is_inside_image(laf, images, border) takes (B, N, 2, 3) LAFs in pixels and an image tensor (B, CH, H, W), read for its height and width only, and returns a boolean (B, N) mask, true where the elliptical region of the LAF lies at least border pixels inside the outer pixel centres. The patch of extract_patches_from_pyramid() covers the square c + A [-1, 1]^2, which reaches beyond the region, so a LAF that is inside can still have patch cells that read the image border.

Convention:

  • The region of a LAF [A | c] is the ellipse c + A u over the unit vectors u. Along image axis i (0 for x, 1 for y) it spans exactly c_i - r_i to c_i + r_i, where r_i is the Euclidean norm of row i of A. A LAF is inside when both spans lie in border <= x <= w - 1 - border and border <= y <= h - 1 - border, so a region that touches the last valid pixel coordinate is inside.

  • The all-zero padding LAF is a point at the origin, so it counts as inside for border <= 0, the default included; combine the mask with laf_is_filled() to drop the padding.

Parameters:

  • laf (Tensor) – (B, N, 2, 3).

  • images (Tensor) – images, lafs are detected in (B, CH, H, W).

  • border (int, optional) – additional border. Default: 0

Return type:

Tensor

Returns:

mask with shape (B, N).

See also

  • laf_is_filled(): tells detections from the padding, which this counts as inside.

  • laf_is_valid(): checks that the frame is finite and regular, wherever it lies.

kornia.feature.KORNIA_CHECK_LAF(laf, raises=True)[source]#

Check that a tensor has the shape of local affine frames, (B, N, 2, 3).

KORNIA_CHECK_LAF(laf) returns True when laf is (B, N, 2, 3); it is KORNIA_CHECK_SHAPE(laf, ["B", "N", "2", "3"]). On failure it raises ShapeError.

Convention:

Only the shape is checked: any dtype and any values pass, N = 0 included.

Parameters:

  • laf (Tensor) – local affine frame tensor to evaluate.

  • raises (bool, optional) – whether to raise on failure; with False, the check returns False instead. Default: True

Raises:

ShapeError – if laf does not have a shape (B, N, 2, 3) and raises is True.

Return type:

bool

Note

Skipped while checks are disabled; see disable_checks().

See also

Example

>>> lafs = torch.rand(2, 10, 2, 3)
>>> KORNIA_CHECK_LAF(lafs)
True

Extracting patches#

kornia.feature.extract_patches_from_pyramid(img, laf, PS=32, normalize_lafs_before_extraction=True)[source]#

Sample a patch per LAF from the image pyramid level that matches its scale.

patches = extract_patches_from_pyramid(img, laf, PS) takes images (B, CH, H, W) and their LAFs (B, N, 2, 3) in pixels and returns one PS x PS patch per LAF, (B, N, CH, PS, PS) in the image dtype: the square c + A [-1, 1]^2 resampled from the level of a Gaussian pyramid of img that matches the LAF’s scale, which limits aliasing. It is the sampler of get_laf_descriptors(), LAFOrienter and the affine shape estimators. With normalize_lafs_before_extraction=False, laf is read in the frame of normalize_laf(). To sample the full-resolution image only, use extract_patches_simple().

Convention:

  • Cell (i, j) of a PS x PS patch samples the image at c + A @ (u_j, u_i), with u_k = (2 k + 1) / PS - 1 and the LAF [A | c] in pixels, pixel centres at integers. Row 0 is on the patch’s -y side, the top for an upright LAF, the columns run along the patch’s x axis, and the cells tile the square c + A [-1, 1]^2. For an odd PS, a LAF with A = (PS / 2) I centred on a pixel returns the image crop around that pixel.

  • The pyramid level is floor(log2(2 s / PS)) for the scale s = sqrt(|det A|), clamped to the levels that exist: pyrdown() builds them while the shorter side stays at least PS, so a larger LAF is sampled from the coarsest level that can still provide a full patch. Level 0, the image itself, always exists: an image whose shorter side is below PS has only that level, so even a large LAF is sampled from it. The level uses the geometric-mean scale, so a highly anisotropic LAF can still alias along its long axis. A level samples the same image positions as level 0, in its own pixels.

  • A cell outside the image reads the nearest border pixel of the level it samples.

  • A caller’s autocast region does not change the sampling grid or the pyramid, outside TorchScript. Under torch.compile on MPS it still changes the pyramid: a convolution under the guard raises there, so pyrdown() runs in the region’s dtype.

  • The patches have the image’s dtype. The sampling runs in the promoted dtype of the image and the LAF, and in at least float32. A LAF with a non-finite entry gives a zero patch.

Parameters:

  • img (Tensor) – images, LAFs are detected in (B, CH, H, W).

  • laf (Tensor) – (B, N, 2, 3).

  • PS (int, optional) – patch size. Default: 32

  • normalize_lafs_before_extraction (bool, optional) – if True, lafs are normalized to image size; if False, laf must already be in the frame that normalize_laf() gives for img. Default: True

Return type:

Tensor

Returns:

patches with shape (B, N, CH, PS,PS).

See also

kornia.feature.extract_patches_simple(img, laf, PS=32, normalize_lafs_before_extraction=True)[source]#

Sample a patch per LAF from the full-resolution image only, without anti-aliasing.

patches = extract_patches_simple(img, laf, PS) takes images (B, CH, H, W) and their LAFs (B, N, 2, 3) in pixels and returns one PS x PS patch per LAF, (B, N, CH, PS, PS) in the image dtype, sampled bilinearly from img at the positions that extract_patches_from_pyramid() samples. For LAFs of a scale below PS the two return the same patches up to rounding; a larger LAF is not smoothed first, so its patch aliases. For patches to describe, orient or adapt, use extract_patches_from_pyramid() instead, as LAFDescriptor, LAFOrienter and the affine shape estimators do.

Convention:

See the Convention block on extract_patches_from_pyramid(). This function samples the same positions, always from img itself: a LAF larger than the patch is not smoothed first, so it aliases.

Parameters:

  • img (Tensor) – images, LAFs are detected in (B, CH, H, W).

  • laf (Tensor) – (B, N, 2, 3).

  • PS (int, optional) – patch size. Default: 32

  • normalize_lafs_before_extraction (bool, optional) – if True, lafs are normalized to image size; if False, laf must already be in the frame that normalize_laf() gives for img. Default: True

Return type:

Tensor

Returns:

patches with shape (B, N, CH, PS,PS).

See also

Orientation and affine shape estimation#

class kornia.feature.LAFOrienter(patch_size=32, num_angular_bins=36, angle_detector=None)[source]#

Orient LAFs by the angle that a patch orientation estimator finds in each LAF’s patch.

lafs = LAFOrienter(patch_size)(laf, img) takes (B, N, 2, 3) LAFs in pixels and their grayscale images (B, 1, H, W), samples each LAF’s patch_size patch with extract_patches_from_pyramid(), runs angle_detector on it, PatchDominantGradientOrientation with num_angular_bins bins by default, and returns the (B, N, 2, 3) LAFs rotated by the angles found. Passed to a detector as ori_module, it runs after the aff_module. To keep the LAFs upright, pass PassLAF instead; to rotate them by known angles, use rotate_laf().

Convention:

  • Each LAF’s patch (extract_patches_from_pyramid()) gives an angle, measured in the patch, and the output orientation is the input orientation plus that angle, converted to degrees, modulo 360 degrees. get_laf_orientation() reads it back in [-180, 180]. The centre, the scale and the upright shape (make_upright()) of the LAF are kept.

  • For an isotropic LAF with positive determinant, the default angle detector turns the LAF’s x axis onto the dominant gradient, whatever the input orientation: content turned 30 degrees counter-clockwise on screen turns the output orientation by +30, the sign of laf_from_center_scale_ori() (see Local features for the orientation convention).

Parameters:

  • patch_size (int, optional) – side of the square patch sampled for each LAF. Default: 32

  • num_angular_bins (int, optional) – number of histogram bins of the default angle detector; not used when angle_detector is given. Default: 36

  • angle_detector (Optional[Module], optional) – Patch orientation estimator, e.g. PatchDominantGradientOrientation or OriNet. Default: None

See also

  • PatchDominantGradientOrientation: the default angle_detector, the dominant gradient direction.

  • OriNet: a learned angle_detector for 32 x 32 patches.

  • PassLAF: keeps the input orientation, for upright features.

  • rotate_laf(): rotates LAFs by known angles, as this module does with the estimated ones.

  • LAFAffineShapeEstimator: estimates the affine shape, which a detector runs before this module.

  • LAFDescriptor: describes the oriented LAFs, from patches that follow their orientation.

forward(laf, img)[source]#

Run forward.

Parameters:

  • laf (Tensor) – (B, N, 2, 3)

  • img (Tensor) – (B, 1, H, W)

Returns:

(B, N, 2, 3)

Return type:

LAF_out

class kornia.feature.PatchDominantGradientOrientation(patch_size=32, num_angular_bins=36, eps=1e-8)[source]#

Estimate the orientation of patches from the peak of their histogram of gradient directions.

angles = PatchDominantGradientOrientation(patch_size)(patches) takes (B, 1, patch_size, patch_size) patches and returns the direction of each patch’s dominant gradient, (B,) in radians, 0 for a gradient that points to the right: the peak, refined between bins, of a num_angular_bins-bin histogram of the gradient directions, each pixel weighted by its gradient magnitude and a Gaussian centred on the patch, as in SIFT’s orientation assignment. It returns one angle per patch, that of the highest peak. To orient LAFs on an image, run it through LAFOrienter, whose default it is; for a learned estimate, use OriNet instead.

Convention:

  • The angle is that of the dominant gradient, which points toward brighter values, counter-clockwise on screen with image y pointing down: \(-\operatorname{atan2}(g_y, g_x)\) in pixel coordinates. A gradient pointing down the patch gives \(-\pi / 2\). LAFOrienter adds the angle to the LAF orientation.

  • The angle is not wrapped into \((-\pi, \pi]\): it can exceed \(\pi\) by up to pi / num_angular_bins.

  • A patch whose Sobel gradients are all exactly zero, such as an all-zero patch, gets the angle \(\pi\) from its empty histogram, as does any constant patch. A nearly flat patch gets an arbitrary angle set by rounding noise, such as a patch sampled from a flat image region by extract_patches_from_pyramid().

  • A float16 or bfloat16 patch is computed in float32, whatever the module’s dtype and any enclosing autocast, and only the returned angle is cast back: the histogram bins of a textured patch are subnormal in float16. A float32 patch under autocast gets its float32 result.

  • The backward casts the patch gradient to float16 too, so a gradient above 65504 becomes inf for a patch of very low amplitude. Through extract_patches_from_pyramid() it reaches the image gradient as inf and NaN.

Parameters:

  • patch_size (int, optional) – size of the (square) input patch. Default: 32

  • num_angular_bins (int, optional) – number of histogram bins. Default: 36

  • eps (float, optional) – for safe division, and arctan. Default: 1e-8

Note

Each gradient pixel is accumulated into its two neighbouring histogram bins with scatter_add_. On CUDA that accumulation uses atomics, so two calls on identical input can differ at the ulp level, and a patch whose two strongest bins are nearly tied can return the neighbouring peak. torch.use_deterministic_algorithms(True) selects the deterministic kernel and restores run-to-run reproducibility.

See also

  • OriNet: a learned orientation of 32 x 32 patches.

  • LAFOrienter: samples each LAF’s patch, runs this module on it by default and rotates the LAF.

  • PatchAffineShapeEstimator: the affine shape of a patch, from the second moments of the same Sobel gradients.

forward(patch)[source]#

Run forward.

Parameters:

patch (Tensor) – (B, 1, patch_size, patch_size); another size raises TypeError.

Returns:

(B,)

Return type:

angle in radians

class kornia.feature.OriNet(pretrained=True, eps=1e-8)[source]#

Estimate the orientation of 32 x 32 patches with the learned OriNet CNN.

angles = OriNet()(patches) builds the network with its pretrained weights, in eval mode, and takes (B, 1, 32, 32) patches and returns a learned canonical orientation per patch, (B,) in radians, in the sense of the angle of PatchDominantGradientOrientation; its zero is learned, so on a patch it need not match the dominant gradient direction. pretrained=False leaves the weights random. To orient LAFs, pass it to LAFOrienter as angle_detector, as KeyNetHardNet and KeyNetAffNetHardNet do. For an estimate that needs no weights, use PatchDominantGradientOrientation instead. This is based on the original code from the paper “Repeatability Is Not Enough: Learning Discriminative Affine Regions via Discriminability”. See [MRM18] for more details.

Convention:

  • With the pretrained weights, the angle turns with the patch content, in the sense of PatchDominantGradientOrientation: content turned counter-clockwise on screen increases the angle. See LAFOrienter for its use.

  • Each patch is normalised first: its mean is subtracted and the result is divided by its standard deviation plus 1e-6. An offset of its intensity therefore does not change the angle, and a positive gain changes it only through that 1e-6: little on a textured patch, more as the patch flattens, up to pi rad on a nearly flat one. An inverted patch generally gets a different angle.

Parameters:

  • pretrained (bool, optional) – Download and set pretrained weights to the model. Default: True

  • eps (float, optional) – to avoid division by zero in atan2. Default: 1e-8

Returns:

Angle in radians.

Shape:

  • Input: (B, 1, 32, 32)

  • Output: (B,)

See also

Examples

>>> input = torch.rand(16, 1, 32, 32)
>>> orinet = OriNet(pretrained=False).eval()
>>> angle = orinet(input) # 16
forward(patch)[source]#

Run forward.

Parameters:

patch (Tensor) – (B, 1, 32, 32), the patch size the weights were trained on.

Returns:

(B,)

Return type:

angle in radians

class kornia.feature.LAFAffineShapeEstimator(patch_size=32, affine_shape_detector=None, preserve_orientation=True)[source]#

Adapt the affine shape of LAFs to the second-moment matrix of each LAF’s image patch.

lafs = LAFAffineShapeEstimator(patch_size)(laf, img) takes (B, N, 2, 3) LAFs in pixels and their grayscale images (B, 1, H, W), samples each LAF’s patch through its upright frame, estimates the shape with affine_shape_detector (PatchAffineShapeEstimator by default) and returns the (B, N, 2, 3) LAFs with the adapted shape and the input’s centres and scales. For the shape that AffNet predicts, use LAFAffNetShapeEstimator instead; to drop the rotation and the reflection and keep the shape, without reading the image, use make_upright().

Convention:

  • Each patch is sampled through the upright form of its LAF (make_upright()), so the shape is estimated in the coordinates of that frame, and the output region is the frame composed with the estimated shape. The estimated shape’s ellipse \(A A^\top\) is proportional to the inverse of the second moment matrix that affine_shape_detector returns. For an isotropic input LAF (a circle), the output shape is the estimated shape itself. An input whose shape already matches the local structure gives a nearly isotropic patch, so its shape changes little: one call is one step of the iterative shape adaptation of [Baumberg00].

  • The centre and the scale of the input LAF are kept. With preserve_orientation=True (the default) its orientation is kept too; otherwise the output is upright, and an orientation estimator such as LAFOrienter can be run after this module.

  • An adapted LAF has a positive determinant: a mirrored input loses its reflection. A LAF that laf_is_valid() rejects, such as the all-zero padding of a detector’s unfilled slot, comes back unchanged with preserve_orientation=True, and with a zero A and its centre otherwise. A valid LAF whose patch gives no finite, positive-definite shape is not adapted either: it comes back unchanged with preserve_orientation=True, and as make_upright() returns it otherwise.

Parameters:

  • patch_size (int, optional) – the input image patch size. Default: 32

  • affine_shape_detector (Optional[Module], optional) – Patch affine shape estimator, PatchAffineShapeEstimator. Default: None

  • preserve_orientation (bool, optional) – if True, the original orientation is preserved. Default: True

See also

forward(laf, img)[source]#

Run forward.

Parameters:

  • laf (Tensor) – (B, N, 2, 3)

  • img (Tensor) – (B, 1, H, W)

Returns:

(B, N, 2, 3)

Return type:

LAF_out

class kornia.feature.PatchAffineShapeEstimator(patch_size=19, eps=1e-10)[source]#

Estimate the affine shape of patches from the second-moment matrix of their gradients.

abc = PatchAffineShapeEstimator(patch_size)(patches) takes (B, 1, patch_size, patch_size) patches and returns the second-moment matrix of each patch’s Gaussian-weighted gradients as the ellipse coefficients (a, b, c) that ellipse_to_laf() reads, (B, 1, 3), divided by the largest entry, as in [Baumberg00]. It returns coefficients, not LAFs: to adapt LAFs on an image, use LAFAffineShapeEstimator, which runs it on each LAF’s patch.

For float16 and bfloat16 inputs, the gradients, Gaussian weighting, moments, and normalization are computed in float32; for a float64 input, the Gaussian weighting is built in float64. The output keeps the input dtype.

Convention:

  • The output \((a, b, c)\) is \((\langle g_x^2 \rangle, \langle g_x g_y \rangle, \langle g_y^2 \rangle)\), the means over the patch of the products of its Sobel gradients, x right and y down, each gradient weighted by a Gaussian centred on the patch. It is divided by its largest entry, which is therefore 1. When two of the three entries are below eps before that division – a flat or low-contrast patch, or one whose gradient lies along an image axis – the output is \((1, 0, 1)\), a circle.

  • The patch must be patch_size pixels square.

  • Known defects: a negative \(b\) counts as below eps, so a patch and its mirror image can get a thin ellipse and a circle (#5824).

Parameters:

  • patch_size (int, optional) – the input image patch size. Default: 19

  • eps (float, optional) – for safe division. Default: 1e-10

See also

forward(patch)[source]#

Run forward.

Parameters:

patch (Tensor) – (B, 1, patch_size, patch_size)

Returns:

ellipse_shape (B, 1, 3)

Return type:

torch.Tensor

class kornia.feature.LAFAffNetShapeEstimator(pretrained=True, preserve_orientation=True)[source]#

Adapt the affine shape of LAFs to the shape the learned AffNet CNN predicts on each LAF’s patch.

lafs = LAFAffNetShapeEstimator()(laf, img) builds AffNet with its pretrained weights, in eval mode, and takes (B, N, 2, 3) LAFs in pixels and their grayscale images (B, 1, H, W), predicts each LAF’s shape on its 32 x 32 patch, sampled through its upright frame, and returns the (B, N, 2, 3) LAFs with that shape and the input’s centres and scales. GFTTAffNetHardNet, HesAffNetHardNet, DoGAffNetHardNet and KeyNetAffNetHardNet run it with preserve_orientation=False, before their orientation step. For the second-moment estimate, which needs no weights, use LAFAffineShapeEstimator instead. This is based on the original code from the paper “Repeatability Is Not Enough: Learning Discriminative Affine Regions via Discriminability”. See [MRM18] for more details.

Convention:

As LAFAffineShapeEstimator, with AffNet’s prediction on a 32 x 32 patch in place of the second moment estimate; see its Convention block for the frame the patch is sampled through and for what the output keeps of the input LAF.

Parameters:

  • pretrained (bool, optional) – Download and set pretrained weights to the model. Default: True

  • preserve_orientation (bool, optional) – if True, the original orientation is preserved. Default: True

See also

  • LAFAffineShapeEstimator: the same adaptation, with the second-moment matrix and no weights.

  • OriNet: the orientation network of the same paper.

  • KeyNetAffNetHardNet: a local feature that runs it on KeyNet detections, before OriNet and HardNet.

forward(laf, img)[source]#

Run forward.

Parameters:

  • laf (Tensor) – (B, N, 2, 3)

  • img (Tensor) – (B, 1, H, W)

Returns:

(B, N, 2, 3)

Return type:

LAF_out

class kornia.feature.PassLAF(*args, **kwargs)[source]#

Return LAFs unchanged, in place of an orientation or affine shape estimator.

out = PassLAF()(laf, img) takes (B, N, 2, 3) LAFs and an image and returns its laf argument itself, not a copy, without reading img. It is the default ori_module and aff_module of ScaleSpaceDetector, MultiResolutionDetector and KeyNetDetector, and the ori_module that keeps features upright. To estimate the orientation, use LAFOrienter; to estimate the affine shape, LAFAffineShapeEstimator or LAFAffNetShapeEstimator.

See also

forward(laf, img)[source]#

Run forward.

Parameters:
Return type:

Tensor

Returns:

LAF, unchanged (B, N, 2, 3)