Local features (detector and descriptor together)#

LocalFeature combines a detector and a LAF descriptor. Its SIFT and HardNet presets return LAFs (B, N, 2, 3) in pixels, responses (B, N) and descriptors (B, N, D), with N = num_features slots per image. Unused slots contain zero LAFs. Use laf_is_filled() to keep detected features and discard padded slots; apply the same mask to all three outputs. LocalFeatureMatcher filters this padding automatically.

SIFTFeature, KeyNetHardNet and KeyNetAffNetHardNet take one grayscale image (1, 1, H, W) per call. The other presets take a batch (B, 1, H, W). SIFTFeatureScaleSpace(descriptor_backend="pyramid") also accepts an RGB batch and converts it to grayscale. The learned models have separate interfaces and are not LocalFeature subclasses.

Choosing a SIFT pipeline#

For new scale-space SIFT pipelines, start with SIFTFeatureScaleSpace and explicitly select descriptor_backend="pyramid":

import kornia.feature as KF

feature = KF.SIFTFeatureScaleSpace(
    num_features=2048, descriptor_backend="pyramid", rootsift=True
).to(grayscale_image.device)
lafs, responses, descriptors = feature(grayscale_image)

grayscale_image is a floating-point tensor of shape (B, 1, H, W) with values in [0, 1]. Outputs are LAFs (B, N, 2, 3), responses (B, N) and descriptors (B, N, 128). N is the requested feature budget; fewer detections leave padded slots. Use laf_is_filled() to identify real detections before matching. rootsift=True selects RootSIFT normalization. Use descriptor_layout="opencv" when feeding descriptors to LightGlueMatcher with "sift".

Both feature classes default to descriptor_backend="patch". Backend performance depends on device, image size, feature count and batching; the recipe is not a claim that pyramid extraction is always faster.

SIFT pipeline choices#
Pipeline	Detection and extraction	When to use it
`SIFTFeatureScaleSpace(..., descriptor_backend="pyramid")`	Dedicated DoG scale-space detector; orientation and description share the detector’s Gaussian layers.	Primary recipe for new scale-space SIFT use.
`SIFTFeatureScaleSpace(..., descriptor_backend="patch")`	Generic `ScaleSpaceDetector`; orientation and description operate on resampled patches.	Existing pipelines, or workloads where measurements favor patch extraction.
`SIFTFeature(..., descriptor_backend="patch")`	Multiresolution, per-level DoG detection; patch orientation and description.	Existing multiresolution pipelines; this is an approximation to scale-space SIFT.
`SIFTFeature(..., descriptor_backend="pyramid")`	The same multiresolution detector; a separate image pyramid supplies smoothed gradients for orientation and pooled histogram maps for description.	Measured multiresolution workloads; see the validation below.

The two feature classes have different detector algorithms. SIFTFeature applies a single-scale DoG response independently at each image resolution, with 2-D non-maximum suppression. It does not select extrema across scale or refine positions and scales to subpixel precision. SIFTFeatureScaleSpace searches 3-D DoG scale-space extrema and refines their positions and scales. Neither preset implements classical SIFT’s contrast and Hessian edge rejection, and each assigns only one orientation per detection (or preserves upright frames when requested).

descriptor_backend is not an interchangeable descriptor switch. In SIFTFeature it changes orientation and description but keeps detection; in SIFTFeatureScaleSpace it also selects a different detector implementation. Consequently, the scale-space backends can return different detections as well as different descriptors.

The scale-space pyramid path ranks valid refined extrema by absolute response. Precise doubling and integer octave decimation keep pixel coordinates aligned. The converged integer Gaussian layer and refined continuous scale are retained separately. Gradients are computed once per used layer and shared between orientation and description; no pyramid is retained between calls. Sampling and Gaussian support differ from patch extraction, so descriptors are not numerically interchangeable. For this backend, compile_modules accepts scale_pyr and subpix (or True for both); sparse orientation and description run eagerly. The patch backend passes compile_modules to ScaleSpaceDetector.

Matching with LightGlue#

Use descriptor_layout="opencv" with LightGlueMatcher in "sift" mode. Pass the oriented LAFs returned by the feature, filter zero-LAF padding from both LAFs and descriptor rows with laf_is_filled(), and supply the original image sizes as hw1=(height0, width0) and hw2=(height1, width1). The matcher converts the LAF angle and scale internally; do not apply those conversions twice.

The public adapter uses one sixth of the LAF scale. For the scale-space presets’ 6-sigma frames this is the detection sigma. SIFTFeature uses a different measurement-region convention, so its multiresolution frames do not imply the same detection sigma. Descriptor-layout compatibility alone does not guarantee equal matching accuracy across the four pipelines. Evaluate the complete feature/matcher combination at the intended feature budget.

Validation and possible deprecation#

The automatic orientation used by SIFTFeature(descriptor_backend="pyramid") integrates a 19x19 support of smoothed image gradients before angular binning, without extracting image patches. Its acceptance criteria, tracked in #5748, include performance at 1024 features and Oxford accuracy at 2048 features. Promotion of this convenience combination requires accuracy matching the patch backend and a demonstrated runtime or memory advantage on a stated workload. Validation should compare with the patch backend at 1024 and 2048 features on IMC PhotoTourism pose estimation and Oxford homography estimation, reporting matcher settings, device and timing methodology.

If validation does not establish those benefits, the plan is to deprecate this convenience combination and remove it only after the API stability policy window: at least one minor release of warnings and a documented replacement. No deprecation warning or removal version is set. This plan concerns the combination, not the standalone SIFTDescriptorFromPyramid descriptor-only interface; see Choosing a SIFT descriptor interface.

HardNet presets#

The six HardNet presets describe 32 x 32 patches with pretrained HardNet through LAFDescriptor. They load pretrained weights for every network they use and start in evaluation mode. Choose a preset by its detector and whether you need affine shape adaptation. The AffNet presets adapt each frame’s shape before estimating its orientation; upright=True skips orientation estimation.

HardNet presets#
Preset	Detector and frames	Orientation	Input, `num_features`	`LightGlueMatcher` mode
`GFTTAffNetHardNet`	`ScaleSpaceDetector` on `CornerGFTT` corners, AffNet shapes	`LAFOrienter(19)`	`(B, 1, H, W)`, 8000	none
`HesAffNetHardNet`	`ScaleSpaceDetector` on `BlobHessian` blobs, AffNet shapes	`LAFOrienter(19)`	`(B, 1, H, W)`, 2048	none
`DoGHardNet`	The DoG detector of `SIFTFeatureScaleSpace(descriptor_backend="patch")`, frames scaled to 12 sigma	`LAFOrienter(19)`	`(B, 1, H, W)`, 2048	`"doghardnet"`
`DoGAffNetHardNet`	The same, with AffNet shapes	`LAFOrienter(19)`	`(B, 1, H, W)`, 2048	`"dog_affnet_hardnet"`
`KeyNetHardNet`	`KeyNetDetector`	`OriNet`	`(1, 1, H, W)` in `[0, 1]`, 8000	none
`KeyNetAffNetHardNet`	`KeyNetDetector`, AffNet shapes	`OriNet`	`(1, 1, H, W)` in `[0, 1]`, 8000	`"keynet_affnet_hardnet"`

For a preset with a LightGlueMatcher mode, drop the padded slots and pass its LAFs and descriptors to the matcher in that mode. The DoG presets return the 12-sigma frames that the "doghardnet" weights expect.

Learned local features#

ALIKED, DISK, XFeat and DeDoDe detect and describe keypoints. Load them with from_pretrained to get pretrained weights and evaluation mode. Their constructors otherwise create models in training mode with random weights, except that the default DeDoDe() also loads a pretrained DINOv2 backbone. SOLD2_detector detects line segments and loads pretrained weights by default.

The four keypoint models return (x, y) pixel coordinates. To match one image pair with LightGlueMatcher, use laf_from_center_scale_ori() on keypoints[None] to construct LAFs, then pass the LAFs and descriptors to the matcher in the mode listed below. Subtract 0.5 from DeDoDe’s keypoints first to convert its half-pixel centres to integer pixel centres. ALIKED.forward_laf() can return LAFs directly.

XFeat.match_xfeat_star anchors both matched points at bin 3 of their coarse cells, in pixels of the image the network saw, before refining the first point. These coordinates differ from the raw top-left cell positions returned by detectAndComputeDense; see XFeat for the measured accuracy and convention.

XFeat returns inference tensors. Match these outputs under torch.no_grad() or torch.inference_mode() (#6533). The image matching guide compares the speed and accuracy of the four keypoint models with the authors’ code on each device.

Learned local features#
Model	Input	Output	`LightGlueMatcher` mode
`ALIKED`	RGB `(B, 3, H, W)` in `[0, 1]`, or grayscale `(B, 1, H, W)`, of any size	one `ALIKEDFeatures` per image: sub-pixel keypoints and 128-D descriptors (64-D for `"aliked-t16"`); `ALIKED.forward_laf()` pads them into LAF tensors	`"aliked"` (not for `"aliked-t16"`)
`DISK`	RGB `(B, 3, H, W)`, sides multiples of 16 unless `pad_if_not_divisible=True`	one `DISKFeatures` per image: integer-pixel keypoints in raster order and 128-D descriptors	`"disk"`
`XFeat`	`(B, C, H, W)`, any channel count, resized to multiples of 32	`XFeat.detectAndCompute()`: one dict per image, keypoints and 64-D descriptors; `XFeat.match_xfeat()` and `XFeat.match_xfeat_star()` also match image pairs	`"xfeat"`
`DeDoDe`	RGB `(B, 3, H, W)` in `[0, 1]`, padded to multiples of 14	tensors of the `n` best keypoints of each image, `(B, N, 2)` with pixel `i` at `i + 0.5`, their scores and 256-D descriptions	`"dedodeb"`, `"dedodeg"`
`SOLD2_detector`	grayscale `(B, 1, H, W)` in `[0, 1]`, sides multiples of 8	line segments, one `(N, 2, 2)` tensor of `(row, col)` endpoints per image; `SOLD2` also describes and matches them	none

API reference#

class kornia.feature.LocalFeature(detector, descriptor, scaling_coef=1.0)[source]#

Detect and describe LAFs in one call with a given detector and LAF descriptor module.

feature = LocalFeature(detector, descriptor), then lafs, responses, descs = feature(img, mask) runs a detector and descriptor as one pipeline. It takes images (B, C, H, W) and an optional mask. The detector returns LAFs (B, N, 2, 3) in pixels and responses (B, N). The descriptor returns one descriptor per LAF, (B, N, D). scaling_coef changes the LAF shape before description. Use LAFDescriptor to adapt a patch descriptor such as HardNet; SIFTDescriptorFromPyramid already takes images and LAFs. The detector and descriptor define accepted batch sizes, channels and intensity ranges. To match features from two images, use LocalFeatureMatcher. For learned features, use DISK, ALIKED, XFeat or DeDoDe.

Convention:

The returned LAFs use the pixel convention of laf_from_center_scale_ori() (see Local features). scaling_coef multiplies their shape A (scale_laf()) before the description and leaves the centres and the responses unchanged.
Cast the module to the image dtype before calling it, for example feature.to(img.dtype). See each preset for its supported dtypes.
A detector can return zero-LAF padding when fewer than N features are found. laf_is_filled(lafs) is True for detections; index LAFs, responses and descriptors with it to discard padding before matching. LocalFeatureMatcher does this automatically.

Parameters:

detector (Module) – the detection module, called as detector(img, mask) and returning (lafs, responses), such as ScaleSpaceDetector.
descriptor (Module) – the descriptor module, called as descriptor(img, lafs) and returning (B, N, D), such as LAFDescriptor.
scaling_coef (float, optional) – the multiplier of the detector’s LAFs before the description, see the Convention block. An int is converted to float. Must be positive. Default: 1.0

Raises:

ValueError – if scaling_coef is not positive.

See also

ScaleSpaceDetector: a scale-space detector for a batch of images.
MultiResolutionDetector: a detector for one image per call.
laf_is_filled(): returns the detection slots to keep and discard padding.
LocalFeatureMatcher: runs a local feature on two images and matches the descriptors.
SIFTFeatureScaleSpace: the SIFT preset for a batch of images.

Example

>>> feature = LocalFeature(ScaleSpaceDetector(10), LAFDescriptor(SIFTDescriptor(32), patch_size=32))
>>> lafs, responses, descs = feature(torch.rand(2, 1, 64, 64))
>>> lafs.shape, responses.shape, descs.shape
(torch.Size([2, 10, 2, 3]), torch.Size([2, 10]), torch.Size([2, 10, 128]))

forward(img, mask=None)[source]#

Run forward.

Parameters:

img (Tensor) – image to extract features with shape (B,C,H,W).
mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape (B,1,H,W) with the spatial size of the image. It is forwarded to the detector unchanged; see the detector for its semantics. Default: None

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

Detected local affine frames with shape (B,N,2,3).
Response function values for corresponding lafs with shape (B,N).
Local descriptors of shape (B,N,D) where \(D\) is descriptor size.

The Convention block of LocalFeature gives the frame, N and the slots the detector did not fill, which have to be dropped before matching.

class kornia.feature.ALIKED(model_name='aliked-n16', max_num_keypoints=-1, detection_threshold=0.2, nms_radius=2)[source]#

Detect sub-pixel keypoints in images of any size, described by deformable sampling (ALIKED).

aliked = ALIKED.from_pretrained("aliked-n16") loads the trained weights in evaluation mode, and features = aliked(images) takes RGB images (B, 3, H, W) in \([0, 1]\), or grayscale (B, 1, H, W), of any height and width, and returns a list of B ALIKEDFeatures, one per image. Each holds sub-pixel keypoints (N, 2) as (x, y) pixels, their scores (N,) and L2-normalised descriptors (N, D), with D 128, or 64 for "aliked-t16", and N set by the image’s score map. With lafs, responses, descs = aliked.forward_laf(images), the same keypoints, scores and descriptors come as tensors (B, N, 2, 3), (B, N, 1) and (B, N, D), padded to the largest N of the batch. To match two images, pass the 128-D descriptors with laf_from_center_scale_ori(keypoints[None]) to LightGlueMatcher in "aliked" mode; for the same number of keypoints in every image, use DeDoDe instead.

ALIKED (Adaptive Local Image KEypoint Detection) combines a multi-scale ResNet backbone with deformable descriptor sampling (SDDH) and a differentiable keypoint detector (DKD); see [ZWC+23] for details.

Speed against the authors’ Python code (640 px, 2,048 keypoints): on CUDA, compile the model for repeated inference. Eager, it runs at 0.8x the authors’ speed on an RTX 4090, because its PyTorch deformable convolution launches many small kernels; torch.compile(model) runs at 1.6x (1.2x with dynamic=True). On CPUs keep it eager: it runs at 1.7-2.0x the authors’ speed, and compiling it made it slower on an x86 CPU. See the image matching guide.

Convention:

ALIKED() builds random weights in training mode, where BatchNorm normalises with the batch, updates its running statistics on every call, and a single small image raises; call .eval() for inference. from_pretrained() returns the trained model in evaluation mode.
The network applies no normalisation, so pass images in \([0, 1]\), the range of the pretrained weights; a (B, 1, H, W) image is repeated to three channels, and other channel counts raise. Any H and W work: the image is replicate-padded around its centre to multiples of 32 and the maps are cropped back, so the padding does not move the keypoints. The image must have the model’s dtype: a float16, bfloat16 or float64 image raises on a float32 model, so cast the model first, aliked.to(images.dtype).
The keypoints of forward() are sub-pixel (x, y) in the pixel convention of laf_from_center_scale_ori() (see Local features): a strict maximum of the score map in its 2 * nms_radius + 1 window (see the note), at least nms_radius pixels from the border, moved by a soft-argmax over that window. Its score is the score map read bilinearly there, and the descriptors are L2-normalised. The keypoints, and the LAFs of forward_laf(), are float32 for a float16 or bfloat16 model and otherwise in the image’s dtype; scores and descriptors keep the model’s dtype.
With detection_threshold > 0 (threshold mode), the maxima strictly above it are kept, at most max_num_keypoints of them, and at most 20000 for a non-positive max_num_keypoints. An image with no maximum above the threshold keeps the maxima above its own mean score instead, whatever the rest of the batch. With detection_threshold <= 0 and max_num_keypoints > 0 (top-k mode), the max_num_keypoints strongest maxima are kept, or all of them when there are fewer; with both <= 0, the maxima above the image’s mean score.
image_size holds the valid (W, H) of each image of a batch padded at the bottom and right to a common size: maxima closer than nms_radius pixels to that image’s last valid row or column are dropped.
forward_laf() returns padded tensors like LocalFeature, but N is the largest keypoint count in the batch and the responses are (B, N, 1). Padded slots hold zero LAFs, responses and descriptors; laf_is_filled(lafs) is true for detections, so index every output with it to discard the padding. Its mask multiplies the score map, and it takes no image_size.
Known defects: on CPU, float16 and bfloat16 score and feature maps go to torch’s grid_sample without an upcast: the half-precision kernel can return wrong values, NaN or overflow, and where torch has none the call raises (#5810).

Parameters:

model_name (str, optional) – backbone configuration, one of 'aliked-t16', 'aliked-n16', 'aliked-n16rot', 'aliked-n32'. Default: "aliked-n16"
max_num_keypoints (int, optional) – the keypoint budget, a cap on the number of maxima kept in threshold mode and in top-k mode; see the Convention block. Default: -1
detection_threshold (float, optional) – a positive value selects threshold mode and is the score a maximum must exceed; zero or a negative value selects top-k mode when max_num_keypoints is positive. See the Convention block. Default: 0.2
nms_radius (int, optional) – NMS radius (kernel size = 2 * nms_radius + 1). The suppression keeps strict local maxima only; see the note below. Default: 2

Note

Keypoints are the strict local maxima of the score map: nms2d() keeps a position only if its score is greater than every other score in its 2 * nms_radius + 1 window. This departs on purpose from the reference implementations (Shiaoming/ALIKED and LightGlue’s port). Their simple_nms also keeps tied maxima, and in two further rounds it adds the positions that become the maximum of their window once the windows around the maxima already kept are zeroed. Those positions lie on the flank of a stronger peak and are not local maxima of the score map. In threshold mode, when neither the keypoint budget nor the fallback below decides the count, kornia therefore returns a subset of the reference keypoints: with the pretrained aliked-n16 and the default threshold it returns up to 10 % fewer keypoints than the reference on the Oxford img1 images, and each one left out has a higher score in its window. Otherwise the two keypoint sets need not be nested, because the reference ranks those extra positions with the maxima: in top-k mode, where it also fills top_k slots with positions that are not maxima when the score map has fewer maxima, and when the budget cuts the count in threshold mode. The reference falls back to the mean score only when no image of the batch has a maximum above the threshold; kornia returns maxima only and decides the fallback for each image, so an image’s keypoints do not depend on the rest of the batch.

See also

ALIKEDFeatures: the per-image result of forward().
DISK: learned integer-pixel keypoints with 128-D descriptors, for RGB images whose sides are multiples of 16.
XFeat: learned keypoints with 64-D descriptors, and its own mutual-nearest-neighbour and refined matching of image pairs.
DeDoDe: separate learned detector and descriptor networks, with n keypoints in every image.
LocalFeature: a classical detector and LAF descriptor in one call; forward_laf() returns padded LAF tensors like it, with (B, N, 1) responses.
LightGlueMatcher: matches ALIKED’s descriptors and keypoints with the "aliked" weights.

Example

>>> aliked = ALIKED.from_pretrained('aliked-n16')
>>> images = torch.rand(1, 3, 256, 256)
>>> features = aliked(images)

forward(images, image_size=None)[source]#

Detect and describe local features in a batch of images.

Parameters:

images (Tensor) – (B, 3, H, W) float images (can be grayscale (B, 1, H, W) — will be broadcast to 3 channels automatically).
image_size (Optional[Tensor], optional) – optional (B, 2) tensor of valid (W, H) for border masking when images are padded to a common size. Default: None

Return type:

list[ALIKEDFeatures]

Returns:

A list of ALIKEDFeatures of length B, one per image. Keypoints are in pixel coordinates [x, y].

forward_laf(img, mask=None, compute_affine=True)[source]#

Detect and describe local features, returning results in kornia LAF format.

With compute_affine=True the 2x2 block A of each LAF comes from the covariance of the soft-argmax weights in the keypoint’s 2 * nms_radius + 1 detection window, computed by the keypoint detection step (DKD) without any additional network parameters: A @ A.T is that covariance, so the axes of the frame are its principal axes, and det A > 0 (the frame is never mirrored). The frame describes how sharply the score map peaks, not an image structure. Its scale (get_laf_scale()) is about a pixel or less with the pretrained aliked-n16. That is much smaller than the region the descriptor samples, so rescale the frames, for example with scale_laf(), before using them as a region size.

All per-image tensors are zero-padded along the keypoint dimension to the largest keypoint count in the batch, so that the outputs are proper batched tensors.

Parameters:

img (Tensor) – image to extract features with shape (B, 3, H, W), or (B, 1, H, W), repeated to three channels.
mask (Optional[Tensor], optional) – optional spatial mask (B, 1, H, W) with values in [0, 1]; the score map is multiplied by this mask before keypoint detection so that features are suppressed in masked regions. Default: None
compute_affine (bool, optional) – if True (default), estimate the 2x2 affine shape of each LAF using torch.linalg.eigh on the soft-argmax covariance. Set to False to skip the eigendecomposition and return identity affines, which avoids the linalg call entirely (useful when only keypoint positions are needed). Default: True

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

Detected local affine frames with shape (B, N, 2, 3).
Response function values for corresponding LAFs with shape (B, N, 1).
Local descriptors of shape (B, N, D).

classmethod from_pretrained(model_name='aliked-n16', max_num_keypoints=-1, detection_threshold=0.2, nms_radius=2, device=None)[source]#

Build an ALIKED model and load the pretrained checkpoint of model_name, in evaluation mode.

Parameters:

model_name (str, optional) – one of 'aliked-t16', 'aliked-n16', 'aliked-n16rot', 'aliked-n32'. Default: "aliked-n16"
max_num_keypoints (int, optional) – passed to ALIKED constructor. Default: -1
detection_threshold (float, optional) – passed to ALIKED constructor. Default: 0.2
nms_radius (int, optional) – passed to ALIKED constructor. Default: 2
device (Optional[device], optional) – target device; defaults to CPU. Default: None

Return type:

ALIKED

Returns:

Pretrained ALIKED in eval mode.

class kornia.feature.ALIKEDFeatures(keypoints, descriptors, keypoint_scores)[source]#

Hold the sub-pixel keypoints, descriptors and scores that ALIKED detects in one image.

features = aliked(images)[b] is what ALIKED returns for image b: features.keypoints (N, 2), sub-pixel (x, y) pixels, features.descriptors (N, D), L2-normalised, and features.keypoint_scores (N,), with N = features.n. It is not batched, because ALIKED detects a different number of keypoints in each image; features.to(device) moves the three tensors. For the keypoints of a whole batch as padded LAF tensors, use ALIKED.forward_laf() instead.

Convention:: The keypoints, descriptors and scores that ALIKED returns for one image; its Convention block gives their frame and how many there are.

Parameters:

keypoints (Tensor) – pixel coordinates (N, 2) as [x, y].
descriptors (Tensor) – L2-normalised descriptors (N, D).
keypoint_scores (Tensor) – detection confidence scores (N,).

See also

ALIKED: the detector, which returns one of these per image.
DISKFeatures: the same container for DISK, whose keypoints are integer pixels.

property n: int#: Number of detected keypoints.

to(*args, **kwargs)[source]#

Move all tensors to a new device / dtype.

Return type:: ALIKEDFeatures

class kornia.feature.DeDoDe(detector_model='L', descriptor_model='G', amp_dtype=torch.float16)[source]#

Detect the n best keypoints of each RGB image and describe them with two networks (DeDoDe).

dedode = DeDoDe.from_pretrained(detector_weights="L-C4-v2", descriptor_weights="B-upright") loads trained weights in evaluation mode. keypoints, scores, descriptions = dedode(images, n=10_000) takes RGB images (B, 3, H, W) in \([0, 1]\) and returns the best N = min(n, H W) keypoints per image. Outputs are (x, y) pixel coordinates (B, N, 2) with pixel centres at i + 0.5, scores (B, N) and unnormalised descriptors (B, N, 256). Use ALIKED or DISK when the keypoint count should depend on image content.

See [EBWF24] for details.

Convention:

detect() and describe() run the detector and descriptor separately. The default descriptor, G, combines VGG19 with a frozen DINOv2 ViT-L/14 backbone and loads the backbone’s weights in the constructor. Descriptor B uses VGG19 alone.
For LightGlueMatcher, construct LAFs centred on the keypoints minus 0.5 and pass them with the descriptors in "dedodeb" or "dedodeg" mode, matching the descriptor variant.
forward(), detect() and describe() take RGB images in the range \([0, 1]\) and apply the ImageNet normalization themselves; pass apply_imagenet_normalization=False for images that are already normalized. pad_if_not_divisible zero-pads the normalized image at the bottom and right to multiples of 14, so the padding is the ImageNet mean colour, and removes it from the maps again.
forward() returns pixel i as i + 0.5: subtract 0.5 for the pixel convention of laf_from_center_scale_ori() (see Local features). detect() returns it normalized, as (2 i + 1) / (W, H) - 1 in \([-1, 1]\) (align_corners=False), the form describe() takes; (k + 1) * (W, H) / 2 - 0.5 converts such a k to kornia’s frame. A description is read at the centre of pixel i.
The keypoints of an image come in descending order of score, with no threshold and no padding. Keypoints, and the descriptions of forward() and of describe(images, keypoints), are float32. The scoremap softmax and the keypoint sampling run in at least float32, so the scores are float32 for a half-precision model and float64 for a float64 one. The dense map of describe(images) follows the network dtype: amp_dtype on CUDA, the dtype of the model on CPU and MPS.
kornia detects and describes at the size of the input. The authors resize every image to 784 x 784, without keeping its aspect ratio, in detect_from_path and describe_keypoints_from_path, and train the detector and the B descriptor at 512 x 512 (G at 560 x 560). The 51-pixel coverage window of the keypoint sampling and the 14-pixel patches of G do not scale with the image. After resizing to 784 x 784, the normalized keypoints of detect() hold for the original image unchanged; multiply those of forward() by (W / 784, H / 784).
amp_dtype is CUDA mixed precision in every component: on CUDA tensors the networks run under torch.autocast in amp_dtype (and the DINOv2 backbone of the G descriptor is cast to it). On CPU and MPS it has no effect, so the default argument runs there with float32 weights and images.

Parameters:

detector_model (Literal['L'], optional) – The detector model kind. Available options are: "L". Default: "L"
descriptor_model (Literal['G', 'B'], optional) – The descriptor model kind. Available options are: "G" or "B". Default: "G"
amp_dtype (dtype, optional) – The dtype of the automatic mixed precision used on CUDA; it has no effect on other devices. Default: torch.float16

See also

ALIKED: learned sub-pixel keypoints whose count follows each image’s score map, for RGB images of any size.
DISK: learned integer-pixel keypoints with 128-D descriptors from one U-Net, for RGB images whose sides are multiples of 16.
XFeat: learned keypoints with 64-D descriptors, and its own mutual-nearest-neighbour and refined matching of image pairs.
LightGlueMatcher: matches the descriptions and keypoints with the "dedodeb" weights for the B descriptor and the "dedodeg" weights for G.

Example

>>> dedode = DeDoDe.from_pretrained(detector_weights="L-C4-v2", descriptor_weights="B-upright")
>>> images = torch.rand(1, 3, 256, 256)
>>> keypoints, scores = dedode.detect(images)
>>> descriptions = dedode.describe(images, keypoints = keypoints)
>>> keypoints, scores, features = dedode(images) # alternatively do both

describe(images, keypoints=None, apply_imagenet_normalization=True, pad_if_not_divisible=True, crop_h=None, crop_w=None)[source]#

Describe keypoints in the input images. If keypoints are not provided, returns the dense descriptors.

Note

This method unconditionally sets the model to eval mode via self.train(False) so that BatchNorm and Dropout behave deterministically. This is intentional: the descriptor is only used at inference time and its statistics must be frozen.

Parameters:

images (Tensor) – A torch.Tensor of shape (B, 3, H, W) containing the RGB input images in the range \([0, 1]\).
keypoints (Optional[Tensor], optional) – An optional torch.Tensor of shape (B, N, 2) containing the detected keypoints. Default: None
apply_imagenet_normalization (bool, optional) – Whether to apply ImageNet normalization to the input images. Pass False if they are already ImageNet-normalized. Default: True
pad_if_not_divisible (bool, optional) – Zero-pad the (normalised) image on the bottom and right so that H and W are multiples of 14; the pad is the ImageNet mean colour, as in forward() and detect(). Required when using the G descriptor backed by DINOv2 (patch size 14). The padding is removed from the descriptor map, so the map has the size of the input image. Default: True
crop_h (Optional[int], optional) – The height of the crop to be used for description. If None, the full image is used. Default: None
crop_w (Optional[int], optional) – The width of the crop to be used for description. If None, the full image is used. Default: None

Returns:

A torch.Tensor of shape (B, N, DIM) containing the descriptions: of the detected keypoints. If the dense descriptors are requested, the shape is (B, DIM, H, W).

Return type:

descriptions

detect(images, n=10000, apply_imagenet_normalization=True, pad_if_not_divisible=True, crop_h=None, crop_w=None)[source]#

Detect keypoints in the input images.

Note

This method unconditionally sets the model to eval mode via self.train(False) so that BatchNorm and Dropout behave deterministically. This is intentional: the detector is only used at inference time and its statistics must be frozen.

Parameters:

images (Tensor) – A torch.Tensor of shape (B, 3, H, W) containing the RGB input images in the range \([0, 1]\).
n (Optional[int], optional) – The number of keypoints to detect. Default: 10000
apply_imagenet_normalization (bool, optional) – Whether to apply ImageNet normalization to the input images. Pass False if they are already ImageNet-normalized. Default: True
pad_if_not_divisible (bool, optional) – Zero-pad the (normalised) image on the bottom and right so that H and W are multiples of 14; the pad is the ImageNet mean colour, as in forward() and describe(). The padding is removed from the scoremap, so keypoints are never reported in it. Default: True
crop_h (Optional[int], optional) – The height of the crop to be used for detection. If None, the full image is used. Default: None
crop_w (Optional[int], optional) – The width of the crop to be used for detection. If None, the full image is used. Default: None

Returns:

A torch.Tensor of shape (B, N, 2) containing the detected keypoints, normalized to the range \([-1, 1]\). scores: A torch.Tensor of shape (B, N) containing the scores of the detected keypoints.

Return type:

keypoints

forward(images, n=10_000, apply_imagenet_normalization=True, pad_if_not_divisible=True)[source]#

Detect and describe keypoints in the input images.

Parameters:

images (Tensor) – A torch.Tensor of shape (B, 3, H, W) containing the RGB input images in the range \([0, 1]\).
n (Optional[int], optional) – The number of keypoints to detect in each image, at most H W; None keeps every pixel. Default: 10_000
apply_imagenet_normalization (bool, optional) – Whether to apply ImageNet normalization to the input images. Pass False if they are already ImageNet-normalized. Default: True
pad_if_not_divisible (bool, optional) – Zero-pad the (normalised) image on the bottom and right so that H and W are multiples of 14; the pad is the ImageNet mean colour. The padding is removed from the scoremap, so keypoints are never reported in it. Default: True

Returns:

A torch.Tensor of shape (B, N, 2) containing the detected keypoints in pixels, with pixel: i at i + 0.5, unlike detect(), which normalises them.

scores: A torch.Tensor of shape (B, N) containing the scores of the detected keypoints.

descriptions: A torch.Tensor of shape (B, N, DIM) containing the descriptions: of the detected keypoints. DIM is 256 for both B and G.

Return type:

keypoints

classmethod from_pretrained(detector_weights='L-C4-v2', descriptor_weights='G-upright', amp_dtype=torch.float16)[source]#

Load a pretrained model.

Parameters:

detector_weights (str, optional) – The weights to load for the detector. One of ‘L-upright’ (original paper, https://arxiv.org/abs/2308.08479), ‘L-C4’, ‘L-SO2’ (from steerers, better for rotations, https://arxiv.org/abs/2312.02152), ‘L-C4-v2’ (from dedode v2, better at rotations, less clustering, https://arxiv.org/abs/2404.08928). Default is ‘L-C4-v2’. Default: "L-C4-v2"
descriptor_weights (str, optional) – The weights to load for the descriptor. One of ‘B-upright’,’G-upright’ (original paper, https://arxiv.org/abs/2308.08479), ‘B-C4’, ‘B-SO2’, ‘G-C4’, ‘G-SO2’ (from steerers, better for rotations, https://arxiv.org/abs/2312.02152). Default is ‘G-upright’. Default: "G-upright"
amp_dtype (dtype, optional) – the dtype of the automatic mixed precision used on CUDA, one of torch.float16 or torch.float32. It has no effect on CPU or MPS, where every component runs in the dtype of the input and the model. Default is torch.float16. Default: torch.float16

Return type:

Module

Returns:

The pretrained model.

class kornia.feature.DISK(desc_dim=128, unet=None)[source]#

Detect integer-pixel keypoints and describe them with the dense maps of one U-Net (DISK).

disk = DISK.from_pretrained("depth") loads trained weights in evaluation mode. features = disk(images, n=2048) takes RGB images (B, 3, H, W) and returns one DISKFeatures per image: up to n integer-pixel keypoints (N, 2), scores (N,) and L2-normalised descriptors (N, 128). Image dimensions must be multiples of 16 unless pad_if_not_divisible=True. Use ALIKED for subpixel keypoints or DeDoDe for a fixed keypoint budget per image.

See [TFT20] for details.

Convention:

DISK() creates random weights in training mode. For matching, construct LAFs with laf_from_center_scale_ori() on keypoints[None] and pass them with the descriptors to LightGlueMatcher in "disk" mode.
images are RGB, (B, 3, H, W), which the U-Net sees as given, without normalisation. With the default U-Net, a common positive gain of the input, such as \([0, 255]\) instead of \([0, 1]\), changes the outputs only through the instance norms’ eps, which weighs more the lower the image’s contrast. At ordinary contrast the maps barely change, though a keypoint can still swap where scores nearly tie or at the n cut; at low contrast the keypoints and descriptors change substantially. With the depth weights and n=2048, the \([0, 1]\) and \([0, 255]\) versions of Oxford graf, wall and bark img1 differ in 0 to 2 keypoints, in 2 to 6 % of them with the intensities multiplied by 0.1, and in about 30 % multiplied by 0.01. H and W must be multiples of 16, unless pad_if_not_divisible=True zero-pads the bottom and right, in which case the maps are cropped back to the image before the suppression and no keypoint lies in the padding. The image must have the network’s dtype: a float16, bfloat16 or float64 image raises on a float32 model, so cast the model first, disk.to(images.dtype).
Keypoints are the integer (x, y) pixels of the strict heatmap maxima (see the note), using the pixel convention of laf_from_center_scale_ori() (see Local features), listed in raster order (row by row, then column by column) rather than by score, in float32 for a float16 or bfloat16 network and otherwise in the dtype of the network’s output. detection_scores are the heatmap values there, in the network’s dtype, and each descriptor is the dense descriptor at the keypoint’s pixel, L2-normalised; heatmap_and_dense_descriptors() returns the dense descriptors without normalisation.
score_threshold keeps the scores strictly above it. n is an upper bound on the maxima above score_threshold in each image; the n argument of forward() says which of them are kept when there are more.

Parameters:

desc_dim (int, optional) – The dimension of the descriptor. Default: 128
unet (Module | None, optional) – The U-Net to use. If None, a default U-Net is used. Kornia doesn’t provide the training code for DISK so this is only useful when using a custom checkpoint trained using the code released with the paper. The unet should take as input a torch.Tensor of shape (B, C, H, W) and output a torch.Tensor of shape (B, desc_dim + 1, H, W): the descriptors, then the heatmap. Default: None

Note

Keypoints are the strict local maxima of the heatmap: nms2d() keeps a position only if its score is greater than every other score in its full window_size window. Tied maxima, and the positions near the border whose window runs off the image (see nms2d()), are never keypoints. This departs on purpose from the reference implementation (cvlab-epfl/disk). Its max-pool NMS clips the window at the border, so it judges a border position against part of its window. Of several equal maxima it keeps the one the max-pool reports first, which depends on an order PyTorch does not specify. With the pretrained depth weights on Oxford graf and boat img1, every keypoint of the reference that kornia leaves out lies in that border strip. The reference’s n keeps the scores strictly above the (n + 1)-th largest, which drops the weakest maximum, with every maximum tied with it, of an image with n maxima or fewer, and every maximum tied across the cut. kornia’s n is an upper bound, so such an image keeps every maximum, and a tie at the cut is filled in raster order.

See also

DISKFeatures: the per-image result of forward().
ALIKED: learned sub-pixel keypoints for RGB images of any size, also as padded LAF tensors.
XFeat: learned keypoints with 64-D descriptors, and its own mutual-nearest-neighbour and refined matching of image pairs.
DeDoDe: separate learned detector and descriptor networks, with n keypoints in every image.
LocalFeature: a classical detector and LAF descriptor in one call, returning padded LAF tensors instead of a list per image.
laf_from_center_scale_ori(): turns the keypoints of one image into the LAFs that LightGlueMatcher and LAFDescriptor take.
LightGlueMatcher: matches DISK’s descriptors and keypoints with the "disk" weights.

Example

>>> disk = DISK.from_pretrained('depth')
>>> images = torch.rand(1, 3, 256, 256)
>>> features = disk(images)

forward(images, n=None, window_size=5, score_threshold=0.0, pad_if_not_divisible=False)[source]#

Detect features in an image, returning keypoint locations, descriptors and detection scores.

Parameters:

images (Tensor) – The image to detect features in. Shape (B, 3, H, W).
n (int | None, optional) – The maximum number of keypoints per image, counted among the maxima above score_threshold. An image with n such maxima or fewer keeps all of them; with more, its n strongest, where maxima tied at the n-th score are taken in raster order. If None, all keypoints are returned. Default: None
window_size (int, optional) – The size of the non-maxima suppression window used to filter detections. Default: 5
score_threshold (float, optional) – Detections must score strictly above it to be returned. See DISKFeatures for details. Default: 0.0
pad_if_not_divisible (bool, optional) – if True, an input whose height or width is not a multiple of 16 is zero-padded at the bottom and right to the next multiple of 16. Default: False

Return type:

list[DISKFeatures]

Returns:

A list of length \(B\) containing the detected features.

classmethod from_pretrained(checkpoint='depth', device=None)[source]#

Load a pretrained model.

Depth model was trained using depth map supervision and is slightly more precise but biased to detect keypoints only where SfM depth is available. Epipolar model was trained using epipolar geometry supervision and is less precise but detects keypoints everywhere where they are matchable. The difference is especially pronounced on thin structures and on edges of objects.

Parameters:

checkpoint (str, optional) – The checkpoint to load. One of ‘depth’ or ‘epipolar’. Default: "depth"
device (device | None, optional) – The device to load the model to. Default: None

Return type:

DISK

Returns:

The pretrained model.

heatmap_and_dense_descriptors(images)[source]#

Return the heatmap and the dense descriptors.

Parameters:: images (Tensor) – The image to detect features in. Shape (B, 3, H, W).
Return type:: tuple[Tensor, Tensor]
Returns:: A tuple of dense detection scores and descriptors. Shapes are (B, 1, H, W) and (B, D, H, W), where D is the descriptor dimension; the descriptors are not normalised.

class kornia.feature.DISKFeatures(keypoints, descriptors, detection_scores)[source]#

Hold the integer-pixel keypoints, descriptors and detection scores that DISK detects in one image.

features = disk(images)[b] is what DISK returns for image b: features.keypoints (N, 2), integer (x, y) pixels, also read as x and y, features.descriptors (N, D), L2-normalised, and features.detection_scores (N,), with N = features.n. It is not batched, because DISK detects a different number of keypoints in each image; features.to(device) moves the three tensors.

Convention:: The keypoints, scores and descriptors that DISK returns for one image; its Convention block gives their frame and order.

Parameters:

keypoints (Tensor) – torch.Tensor of shape (N, 2), where N is the number of keypoints.
descriptors (Tensor) – torch.Tensor of shape (N, D), where D is the descriptor dimension.
detection_scores (Tensor) – torch.Tensor of shape (N,), the detection heatmap value at each keypoint: a raw logit of the network, not a log-probability.

See also

DISK: the detector, which returns one of these per image.
ALIKEDFeatures: the same container for ALIKED, whose keypoints are sub-pixel.

to(*args, **kwargs)[source]#

Return a copy whose keypoints, descriptors and detection scores went through torch.Tensor.to().

Parameters:

*args (Any) – Arguments passed to torch.Tensor.to().
**kwargs (Any) – Keyword arguments passed to torch.Tensor.to().

Return type:

DISKFeatures

Returns:

A new DISKFeatures object with tensors of appropriate type and location.

property x: Tensor#: Accesses the x coordinates of keypoints (along image width).

property y: Tensor#: Accesses the y coordinates of keypoints (along image height).

class kornia.feature.XFeat(top_k=4096, detection_threshold=0.05)[source]#

Detect and describe keypoints with XFeat, and match image pairs, sparse or refined (XFeat*).

Try in browser

xfeat = XFeat.from_pretrained() loads trained weights in evaluation mode. out = xfeat.detectAndCompute(images) takes images (B, C, H, W) with any channel count and returns one dictionary per image. Each contains up to top_k keypoints (N, 2) in (x, y) pixels, scores (N,) in descending order and L2-normalised descriptors (N, 64). Use match_xfeat() to extract and match sparse features, or match_xfeat_star() for refined matches from semi-dense features. For subpixel keypoint detection, use ALIKED instead.

Reference:: “XFeat: Accelerated Features for Lightweight Image Matching”, CVPR 2024. https://www.verlab.dcc.ufmg.br/descriptors/xfeat_cvpr24/

Convention:

Use the named extraction and matching methods: XFeat has no forward. Its network, xfeat.net, is an XFeatModel that produces dense maps. To match with LighterGlue, construct LAFs with laf_from_center_scale_ori() on keypoints[None] and pass them with the descriptors to LightGlueMatcher in "xfeat" mode, with gradients disabled.
XFeat() builds random weights in training mode. Batch normalisation can raise on a single small image; call .eval() for inference. from_pretrained() returns the trained model in evaluation mode.
detectAndCompute(), detectAndComputeDense(), match_xfeat() and match_xfeat_star() run under torch.inference_mode, so they return inference tensors.
The image is (B, C, H, W) with any channel count. Each extraction resizes the image it gets bilinearly to (H', W'): H and W rounded down to multiples of 32, with a minimum of 32. XFeatModel then averages the channels and instance-normalises the result, so an affine change of the intensities with a positive gain changes its maps only through the norm’s eps, which weighs more the lower the image’s contrast. With the pretrained weights and top_k=2048, the \([0, 1]\) and \([0, 255]\) versions of Oxford graf and wall img1 differ in at most 1 keypoint, in 1 to 2 % of them with the intensities multiplied by 0.1, and in 35 to 40 % multiplied by 0.01.
Keypoints are float32. Scores and descriptors have the dtype of the weights, and the resized image is cast to it, so a float16 or bfloat16 model runs on an image of any float dtype; under autocast, the descriptors of detectAndComputeDense() have the dtype autocast gives the network. With the pretrained weights, top_k=2048 and the 290 HPatches pairs on an RTX 4090 with torch 2.15, match_xfeat() keeps its inliers at 3 px within 0.6 % of float32, and float16 and bfloat16 keep 99.4 % and 95.0 % of its keypoints. On an Apple M1 half precision takes about 15 times as long as float32 on the CPU and about as long on MPS. On torch 2.5.1 the CPU float16 instance norm is wrong: on HPatches a [0, 255] image gave no keypoints and a [0, 1] one 18 % of the inliers. Use float32 or bfloat16 weights there. torch 2.15 and a 2.16 nightly are correct; versions in between were not checked.
detectAndCompute() keypoints are integer pixels of the resized image, strict maxima of the keypoint heatmap in their 5 x 5 window (see the note). Pixel \(i\) of the resized image is reported at its centre in the input, \((i + 0.5) W / W' - 0.5\) in \(x\) and likewise in \(y\), in the pixel convention of laf_from_center_scale_ori() (see Local features). That is exactly \(i\) when H and W are multiples of 32. The reference implementation reports \(i W / W'\) instead, up to nearly half a pixel away.
The heatmap value must be strictly above detection_threshold. scores is that value times the reliability map, and the top_k best keypoints come sorted by it. Each descriptor is 64-D, sampled bicubically from the dense map at the keypoint by InterpolateSparse2d, and L2-normalised.
With multiscale=False, detectAndComputeDense() takes cell (j, i) of the 1/8-resolution map at pixel 8 * (j, i) of the resized image, the top-left pixel of the cell and not its centre, and reports that pixel’s centre in the input as above: exactly 8 * (j, i) when H and W are multiples of 32. It keeps up to top_k cells by reliability; top_k < 1 keeps every cell. With the default multiscale=True, it rescales the image bilinearly by 0.6 and by 1.3, runs that extraction on each with int(0.2 * top_k) and the remaining top_k - int(0.2 * top_k) cells, and maps each keypoint p of the image rescaled by s back through the pixel centre, (p + 0.5) / s - 0.5; scales is 1 / 0.6 or 1 / 1.3 for each keypoint. The reference divides by the factor, p / s, 0.333 px up and to the left of the pixel at 0.6 and 0.115 px down and to the right at 1.3 (#6537). A share of 0 yields no cell from its scale, and a scale with fewer cells than its share returns all of them, so there are never more than top_k keypoints, and exactly top_k when both scales have enough cells. A top_k below 1 keeps every cell of both scales. Its descriptors are not normalised. The reference implementation gives the 1.3 scale int(0.8 * top_k) and reads a share of 0 as “every cell”; kornia does not, because that returns more keypoints than top_k asks for.
Known defects: autograd cannot save the inference tensors of the four methods above for a backward pass, so LightGlueMatcher, whose parameters require gradients, raises on them with gradients enabled, the default grad mode: call it under torch.no_grad() or torch.inference_mode(), or clone() the outputs first (#6533).

Parameters:

top_k (int, optional) – maximum number of keypoints to keep per image. Default: 4096.
detection_threshold (float, optional) – the keypoint heatmap value a keypoint must exceed, see the Convention block. Default: 0.05.

Note

Keypoints are the strict local maxima of the keypoint heatmap: nms2d() keeps a position only if its score is greater than every other score in its full 5x5 window. Tied maxima, and the positions within 2 pixels of the border whose window runs off the image, are never keypoints. This departs on purpose from the reference implementation (verlab/accelerated_features). Its NMS keeps every position equal to the maximum of its window: all positions of a plateau survive, and whether a near-tie survives depends on rounding-level differences of the scores. Its window is also clipped at the border, so it judges a border position against part of its window. With the pretrained weights on Oxford graf and boat img1, every candidate of the reference that kornia leaves out lies in that border strip.

XFeat* (match_xfeat_star()) anchors both matched cells at bin 3: it moves both points three pixels into their cells, then moves the first point by the fine displacement. Both are pixels of the image the network saw, and one such pixel spans r / s input pixels per axis, for extraction scale s (0.6 or 1.3) and the ratio r of the resize to multiples of 32. This differs from the raw top-left coordinates returned by detectAndComputeDense() and from the reference, which starts both points at the top-left pixel and scales the displacement by 1 / s.

On 40 Oxford affine pairs, this match origin raises homography AUC@3 from 0.365 to 0.401 (three correspondence-order seeds, float32, top_k=4096, USAC_MAGSAC at 1 px), and on 813 IMC PhotoTourism validation pairs scene-mean relative-pose mAA@10 from 0.458 to 0.473 (USAC_MAGSAC at 0.5 px, measured before the pixel-centre mapping of the dual-scale rescale below).

The displacement of the first image is the soft-argmax of the fine matcher’s 8 x 8 heatmap times the keypoint’s scale, measured from bin 3 of each axis. This also departs on purpose from the reference implementation, which measures it from bin 4, the window centre: the pretrained fine matcher puts a zero displacement at bin 3. Matching Oxford bikes img1 with itself (pretrained weights, multiscale=False), the reference’s refinement moves the points by (-1.04, -1.25) px on average and leaves 1 % of them within 1 px of their position, against (-0.04, -0.25) px and 94 % from bin 3. On 18 Oxford affine pairs the median homography corner error drops from 2.58 to 2.30 px; IMC 2021 PhotoTourism relative pose mAA@10 is the same within noise (0.441 and 0.443 over 813 pairs). Away from a zero displacement the soft-argmax still moves a point only about half of the way, as in the reference: on Oxford graf 1-2 to 1-5 its response to the true sub-cell offset has a slope of about 0.5, so bin 3 removes the constant part of the error, not that shrinkage.

Reporting keypoints at the pixel centre of the resize to a multiple of 32 also departs on purpose from the reference. With the pretrained weights and 8 Oxford images, each matched with resizes of itself to two sizes whose true map is known exactly, the share of matches within 1 px goes from 24.2 to 24.5 % for match_xfeat(), from 40.1 to 40.7 % for LightGlueMatcher and from 28.7 to 29.1 % for match_xfeat_star(). Homography AUC on Oxford affine and HPatches stays within the noise of the estimator for the first two, and on HPatches at about 320 px AUC@3 of match_xfeat_star() gains 0.0025. The 0.6 and 1.3 rescale of the dual-scale extraction goes through the pixel centre as well. With the bin-3 anchor, Oxford affine AUC@3 of match_xfeat_star() stays within seed noise (0.404 before, 0.401 after), and the share of matches within 1 px of an exactly known map goes from 46.6 to 50.4 % for an image matched with its 0.7 resize, and from 47.6 to 48.0 %, 35.9 to 36.9 % and 21.6 to 21.2 % for its rotations by 20 degrees, by 20 degrees at scale 0.7 and by -35 degrees at scale 1.2 (#6537).

See also

XFeatModel: the network it runs, which returns the dense maps at 1/8 resolution.
InterpolateSparse2d: samples the dense descriptors at the keypoints.
ALIKED: learned sub-pixel keypoints with 128-D descriptors, for RGB images of any size.
DISK: learned integer-pixel keypoints with 128-D descriptors, for RGB images whose sides are multiples of 16.
DeDoDe: separate learned detector and descriptor networks, with n keypoints in every image.
laf_from_center_scale_ori(): turns the keypoints of one image into the LAFs that LightGlueMatcher takes.
LightGlueMatcher: matches the descriptors and keypoints with the LighterGlue "xfeat" weights instead of mutual nearest neighbours.

Example

>>> model = XFeat.from_pretrained()
>>> img = torch.rand(1, 3, 256, 256)
>>> out = model.detectAndCompute(img)
>>> out[0]['keypoints'].shape
torch.Size([..., 2])

detectAndCompute(x, top_k=None, detection_threshold=None)[source]#

Detect sparse keypoints and compute descriptors.

Parameters:

x (Tensor) – image tensor of shape (B, C, H, W).
top_k (Optional[int], optional) – number of keypoints to keep (overrides self.top_k). Default: None
detection_threshold (Optional[float], optional) – the heatmap value to exceed (overrides self.detection_threshold). Default: None

Returns:

'keypoints': (N, 2) keypoints in (x, y) pixel coordinates.
'scores': (N,) keypoint heatmap value times reliability, in descending order.
'descriptors': (N, 64) L2-normalised descriptors.

Return type:

List of length B. Each element is a dict with

detectAndComputeDense(x, top_k=None, multiscale=True)[source]#

Detect keypoints and compute dense coarse descriptors.

Parameters:

x (Tensor) – image tensor of shape (B, C, H, W).
top_k (Optional[int], optional) – number of features to keep (overrides self.top_k). A value below 1 keeps every cell. With multiscale=True the budget is split between the scales, int(0.2 * top_k) for the 0.6x extraction and the rest for the 1.3x one, so the result never exceeds a positive top_k; a scale with a budget of 0 contributes nothing. This departs from the reference implementation, which gives the 1.3x scale int(0.8 * top_k) and reads a budget that rounds down to 0 as “every cell”. Default: None
multiscale (bool, optional) – use dual-scale (0.6x and 1.3x) extraction. Default: True.

Returns:

'keypoints': (B, K, 2) coarse keypoints.
'descriptors': (B, K, 64) coarse descriptors.
'scales': (B, K) extraction scale per keypoint.

Return type:

Dict with

classmethod from_pretrained(top_k=4096, detection_threshold=0.05)[source]#

Instantiate XFeat with pretrained weights downloaded from the official release.

Parameters:

top_k (int, optional) – maximum number of keypoints to keep. Default: 4096.
detection_threshold (float, optional) – the keypoint heatmap value a keypoint must exceed. Default: 0.05.

Return type:

XFeat

Returns:

XFeat model with pretrained weights loaded, set to eval mode.

match_xfeat(img1, img2, top_k=None, min_cossim=-1)[source]#

Detect, describe and mutually match keypoints from two batches of images.

Pair b is image b of img1 with image b of img2. Each batch goes through detectAndCompute() in one call, and then the descriptors of each pair are matched on their own by mutual nearest neighbour. A backend may round a batched convolution differently from a single-image one (MPS, and CUDA with cuDNN’s default TF32 convolutions), so a pair matched in a batch can gain or lose a few near-tied matches compared with matching it alone.

Parameters:

img1 (Tensor) – first images (B, C, H, W).
img2 (Tensor) – second images (B, C, H, W), with the same B as img1.
top_k (Optional[int], optional) – number of top keypoints to use. Default: None
min_cossim (float, optional) – the cosine similarity a match must exceed; a value of 0 or below, such as the default -1, disables the check. Default: -1

Returns:

list of (N, 4) tensors with (x1, y1, x2, y2) matches, one per pair; an empty batch gives an empty list. If B == 1: tuple (mkpts0, mkpts1) of matched keypoints, each (N, 2).

Return type:

If B != 1

See also

match_xfeat_star(): the dense, refined variant, with the same return convention.

match_xfeat_star(im_set1, im_set2, top_k=None)[source]#

Extract coarse features, match pairs and refine matches (XFeat*).

Both matched cells are anchored three pixels into the cell, in pixels of the image the network saw. The first point also receives the fine displacement measured from bin 3. Raw detectAndComputeDense() coordinates stay at the top-left cell origin; see the class note for the coordinate convention.

Parameters:

im_set1 (Tensor) – batch of images (B, C, H, W).
im_set2 (Tensor) – batch of images (B, C, H, W), with the same B as im_set1.
top_k (Optional[int], optional) – number of top features to use. Default: None

Returns:

list of (N, 4) tensors with (x1, y1, x2, y2) matches, one per pair; an empty batch gives an empty list. If B == 1: tuple of (N, 2) matched keypoint tensors.

Return type:

If B != 1

class kornia.feature.XFeatModel[source]#

Compute the dense descriptors, keypoint logits and reliability map of XFeat at 1/8 resolution.

net = XFeatModel().eval() builds the random-weight network for inference (XFeat.from_pretrained().net is trained), and feats, keypoints, heatmap = net(images) takes float images (B, C, H, W) with any channel count and H and W multiples of 8, and returns the dense descriptors (B, 64, H/8, W/8), not normalised, the keypoint logits (B, 65, H/8, W/8) and the reliability map (B, 1, H/8, W/8), a sigmoid. It finds no keypoints itself: for keypoints, descriptors and matches, use XFeat instead. It implements the architecture of “XFeat: Accelerated Features for Lightweight Image Matching”, CVPR 2024.

Convention:

The channels are averaged and the result is instance-normalised, so any channel count is accepted and an affine change a x + b of the intensities with a > 0 changes the outputs only through the norm’s eps (1e-5): negligibly while the variance of the channel mean is much larger than eps, and substantially for a low-contrast image, whose variance is close to it (see XFeat).
A new XFeatModel is in training mode. A single 32 x 32 image reaches BatchNorm with one value per channel and raises; call .eval() for inference.
H and W must be multiples of 8; other sizes raise an error or give maps of mismatched sizes. XFeat resizes its input to multiples of 32.
keypoints holds 65 logits per 8 x 8 cell: channel (y % 8) * 8 + x % 8 of cell (y // 8, x // 8) is pixel (x, y), and channel 64 is the “no keypoint” dustbin.

Note

Image normalisation (InstanceNorm2d) is wrapped in torch.no_grad() following the original design; backpropagating through it is not supported.

See also

XFeat: runs this network and detects, describes and matches keypoints from its maps.
InterpolateSparse2d: samples the descriptor map at sparse keypoints, as XFeat does.

forward(x)[source]#

Run the XFeat backbone.

Parameters:

x (Tensor) – image tensor of shape (B, C, H, W).

Returns:

feats: dense descriptors (B, 64, H/8, W/8).
keypoints: keypoint logits (B, 65, H/8, W/8).
heatmap: reliability map (B, 1, H/8, W/8).

Return type:

Tuple of

class kornia.feature.InterpolateSparse2d(mode='bicubic', align_corners=False)[source]#

Sample a feature map at sparse (x, y) pixels with the XFeat weights’ coordinate scaling.

interp = InterpolateSparse2d("bicubic") and out = interp(x, pos, H, W) take a feature map (B, C, h, w) and positions (B, N, 2) as (x, y) pixels of an image of height H and width W, and return the features (B, N, C) that torch.nn.functional.grid_sample() reads there with mode, in the dtype of x. The positions are divided by W - 1 and H - 1 but sampled with align_corners=False, as the pretrained XFeat weights expect, so H and W must be greater than 1, and a map of the image’s own size is not read at pos itself (see the Convention block). To read a map of the image’s own size at the pixel itself, normalise the positions with normalize_pixel_coordinates() and pass align_corners=True to grid_sample instead.

Convention:

pos holds (x, y) pixels of an image of height H and width W whose feature map has height h and width w. A position (px, py) is read at px * w / (W - 1) - 0.5 and py * h / (H - 1) - 0.5 in the map’s pixel frame: normgrid’s 2 p / (W - 1) - 1 sampled with align_corners=False, the convention the pretrained XFeat weights use.
A float16 or bfloat16 map is sampled in float32 and returned in its dtype, and float16 or bfloat16 positions are normalised in float32: torch’s half-precision grid_sample can return wrong values, NaN or overflow on CPU, and a half-precision grid is off by a pixel or more on a large map.

Parameters:

mode (str, optional) – interpolation mode for torch.nn.functional.grid_sample(). Default: 'bicubic'.
align_corners (bool, optional) – stored and not used: grid_sample is always called with align_corners=False. Default: False.

Shape:

Input x: (B, C, h, w).
Input pos: (B, N, 2) integer or float (x, y) coordinates.
Output: (B, N, C).

See also

XFeat: samples its dense descriptors at its keypoints with this module.
XFeatModel: the network whose descriptor map XFeat samples with this module.
normalize_pixel_coordinates(): the same division by W - 1 and H - 1, meant for grid_sample with align_corners=True, which reads a map of the image’s size at the pixel itself.

forward(x, pos, H, W)[source]#

Sample x at positions pos.

Parameters:

x (Tensor) – feature map (B, C, h, w).
pos (Tensor) – sampling positions (B, N, 2) in pixel coordinates.
H (int) – height used for coordinate normalisation, greater than 1.
W (int) – width used for coordinate normalisation, greater than 1.

Return type:

Tensor

Returns:

Sampled features (B, N, C).

class kornia.feature.SOLD2_detector(pretrained=True, config=None)[source]#

Detect line segments in grayscale images with SOLD2, without describing or matching them.

detector = SOLD2_detector() builds the network with its pretrained weights, in evaluation mode, and outputs = detector(images) takes grayscale images (B, 1, H, W) in \([0, 1]\), with H and W multiples of 8, and returns a dict: "line_segments", one (N, 2, 2) tensor of (row, col) endpoints in pixels per image, and the "junction_heatmap" and "line_heatmap", (B, H, W) each. With the same weights and config, these are the segments and heatmaps of SOLD2; to also describe the segments and match them between images, use SOLD2 instead. This is based on the original code from the paper “SOLD²: Self-supervised Occlusion-aware Line Detector and Descriptor”. See [PautratLinL+21] for more details.

Convention:

Sizes that are not multiples of 8 raise an error or give heatmaps of another size.
line_segments holds one (N, 2, 2) tensor per image: the two endpoints of each segment as (row, col) in the pixel convention of laf_from_center_scale_ori() (see Local features), so flip(-1) gives (x, y). The endpoints of a segment, and the segments by their first endpoint, are in raster order: the smaller row first or, on the same row, the smaller column, whatever the junction scores.
The endpoints are junction coordinates. The default junction refinement (use_junction_refinement of LineDetectorCfg) can produce sub-pixel endpoints in 0.25-pixel steps, up to one pixel per axis.

Parameters:

config (DetectorCfg) – Configuration object containing all parameters, or a dict of its fields. None will load the default parameters, which are tuned for images in the range 400~800 px. A config with use_descriptor=True builds the descriptor head, which the outputs do not use. Default: None
pretrained (bool) – If True, download and set pretrained weights to the model, and return it in eval mode; otherwise the weights are random and the model is in training mode. Default: True

Returns:

The raw junction and line heatmaps, as well as the list of detected line segments (ij coordinates convention).

See also

SOLD2: the same line detection plus the descriptor map and line matching.

Example

>>> img = torch.rand(1, 1, 128, 128)
>>> sold2_detector = SOLD2_detector(pretrained=False).eval()
>>> line_segments = sold2_detector(img)["line_segments"]

forward(img)[source]#

Run forward.

Parameters:: img (Tensor) – batched images with shape (B, 1, H, W).
Returns:: a list of B tensors (N, 2, 2), the (row, col) segment endpoints per image. junction_heatmap: raw junction heatmap of shape (B, H, W). line_heatmap: raw line heatmap of shape (B, H, W).
Return type:: line_segments

class kornia.feature.SIFTFeatureScaleSpace(num_features=8000, upright=False, rootsift=True, device=None, compile_modules=False, descriptor_backend='patch', descriptor_layout='kornia')[source]#

Detect refined DoG scale-space extrema and describe them with SIFT, for a batch.

sift = SIFTFeatureScaleSpace(2048, descriptor_backend="pyramid") builds a DoG scale-space SIFT pipeline. lafs, responses, descs = sift(img) takes grayscale images (B, 1, H, W). It returns six-sigma LAFs (B, N, 2, 3), absolute DoG responses (B, N), strongest first, and RootSIFT descriptors (B, N, 128). Use laf_is_filled() to keep detections and discard padding. The "pyramid" backend reuses Gaussian images and accepts RGB input; "patch" samples patches and requires grayscale input. Use descriptor_layout="opencv" with LightGlueMatcher("sift"); use DoGHardNet for HardNet descriptors on the "patch" detections.

For CPU throughput use OpenCV’s SIFT: on an Apple M1 CPU, a matching pipeline on it ran 8x faster than one on this module with descriptor_backend="pyramid", and 14x faster with "patch" (640 px, 2,048 features). See the image matching guide.

Convention:

A LocalFeature, whose Convention block gives the output contract, of grayscale images (B, 1, H, W); the pyramid backend converts an RGB image to grayscale first. descriptor_backend="patch", the default, runs ScaleSpaceDetector with BlobDoG on ScalePyramid(3, 1.6, 32, double_image=True), keeping minima as well as maxima, then LAFOrienter(19) unless upright, and SIFTDescriptor on 41 x 41 patches. "pyramid" runs a dedicated DoG detector and reuses its Gaussian images and the octave and layer of each detection for orientation and description: it refines only sparse extrema, ranks all valid candidates by absolute response, and computes image gradients once per used layer.
Each backend assigns one orientation per detection unless upright=True. The backend selects a different detector as well as different orientation and description stages, so the backends can return different detections as well as different descriptors; the feature budget is the same.
On both backends the LAF scale is 6 times the detection sigma, and responses is the absolute DoG, the same for a bright blob as for a dark one. The LAF centres are in the frame of laf_from_center_scale_ori() (see Local features) on every octave. For a float16 or bfloat16 image the LAFs are float32 on both backends, and the responses and descriptors keep the image dtype.
Descriptors come in kornia’s SIFT layout; descriptor_layout="opencv" returns them in OpenCV’s, which LightGlueMatcher expects with "sift". convert_sift_descriptor_layout() converts between the two.

Parameters:

num_features (int, optional) – the number of slots per image; the strongest detections fill them, the rest are padding. Default: 8000
upright (bool, optional) – if True, skip the orientation estimation and return upright frames. Default: False
rootsift (bool, optional) – if True, the default, return RootSIFT descriptors, otherwise SIFT. Default: True
device (Union[str, device, None], optional) – the device of the detector and descriptor modules. Default: CPU.
compile_modules (Union[bool, List[str]], optional) – the detector sub-modules to wrap with torch.compile(). With "patch", the compile_modules of ScaleSpaceDetector; with "pyramid", "scale_pyr" and "subpix" only, both for True, while the orientation and description run eagerly. Default: False
descriptor_backend (Literal['patch', 'pyramid'], optional) – "patch" or "pyramid", see the Convention block. Default: "patch"
descriptor_layout (Literal['kornia', 'opencv'], optional) – "kornia" or "opencv", the order of the descriptor values. Default: "kornia"

Raises:

ValueError – for a descriptor_backend or descriptor_layout other than those listed, or a compile_modules name the selected backend does not have.

See also

SIFTFeature: a per-level DoG approximation of SIFT, without selection across scale or sub-pixel refinement, one image per call.
ScaleSpaceDetector: the detector of the "patch" backend, on a BlobDoG response.
BlobDoG: the scale-space DoG response of the "patch" backend.
SIFTDescriptor: the descriptor of the "patch" backend, run on 41 x 41 patches through LAFDescriptor.
convert_sift_descriptor_layout(): the reordering that descriptor_layout="opencv" applies.
LightGlueMatcher: its "sift" weights match these features, in the "opencv" layout.
DoGHardNet: the detections of the "patch" backend described by HardNet, at twice the frame scale.
LocalFeature: its base class, which gives the output contract.

Example

>>> sift = SIFTFeatureScaleSpace(100, descriptor_backend="pyramid")
>>> lafs, responses, descs = sift(torch.rand(2, 1, 64, 64))
>>> lafs.shape, responses.shape, descs.shape
(torch.Size([2, 100, 2, 3]), torch.Size([2, 100]), torch.Size([2, 100, 128]))

forward(img, mask=None)[source]#

Detect and describe SIFT features; the "pyramid" backend reuses the detector’s Gaussian images.

Return type:: Tuple[Tensor, Tensor, Tensor]

class kornia.feature.SIFTFeature(num_features=8000, upright=False, rootsift=True, device=None, config=None, compile_model=False, score_threshold=0.0, descriptor_backend='patch', descriptor_layout='kornia')[source]#

Detect per-level DoG keypoints and describe them with SIFT, one image per call.

sift = SIFTFeature(2048) loads a per-level DoG detector and SIFT descriptor. lafs, responses, descs = sift(img) takes one grayscale image (1, 1, H, W); a batch or color image raises. It returns LAFs (1, N, 2, 3), DoG-response magnitudes (1, N), strongest first, and RootSIFT descriptors (1, N, 128) in kornia’s layout. Use laf_is_filled() to keep detections and discard padding. descriptor_backend chooses patches ("patch", the default) or a gradient pyramid ("pyramid"). For scale-space extrema or batches, use SIFTFeatureScaleSpace with descriptor_backend="pyramid".

For CPU throughput use OpenCV’s SIFT: on an Apple M1 CPU, a matching pipeline on it ran 11x faster than one on this module (640 px, 2,048 features). This module takes one image per call; for batches use SIFTFeatureScaleSpace. See the image matching guide.

Convention:

A LocalFeature, whose Convention block gives the output contract. The detector is a MultiResolutionDetector on the magnitude of a single-scale BlobDoGSingle response. With descriptor_backend="patch", the default, LAFOrienter(19) orients the LAFs unless upright and SIFTDescriptor describes 41 x 41 patches. "pyramid" keeps the detector and orients and describes its LAFs with SIFTDescriptorFromPyramid, which integrates smoothed image gradients without extracting patches.
The detector applies the response independently at each image resolution. It does not apply classical SIFT’s contrast or Hessian edge rejection. One orientation is assigned per detection unless upright=True.
Extrema of both polarities are kept: dark blobs are DoG maxima and bright blobs are DoG minima. On each image-pyramid level, a keypoint is a strict maximum of the response magnitude in its config["nms_size"] window (15 pixels by default), so a stronger extremum of either polarity suppresses a weaker one. responses holds the magnitude.
The frames are those of MultiResolutionDetector: a pixel of a pyramid level, with a scale of config["s_mult"] (22 by default) times that level’s pixel size whatever the size of the blob. Unlike the 6-sigma frames of SIFTFeatureScaleSpace, their scale is not a fixed multiple of the detection sigma: a sixth of it, which LightGlueMatcher("sift") takes as the detection sigma, is not that sigma here (see Choosing a SIFT pipeline).
Descriptors come in kornia’s SIFT layout; descriptor_layout="opencv" returns them in OpenCV’s, which LightGlueMatcher expects with "sift". convert_sift_descriptor_layout() converts between the two.
Known defects: descriptor_backend="pyramid" is less accurate than the patch backend on Oxford homographies at 2048 features, and not faster on CPU at 1024 features (#5748).

Parameters:

num_features (int, optional) – the number of slots per image; the strongest detections fill them, the rest are padding. Default: 8000
upright (bool, optional) – if True, skip the orientation estimation and return upright frames. Default: False
rootsift (bool, optional) – if True, the default, return RootSIFT descriptors, otherwise SIFT. Default: True
device (Union[str, device, None], optional) – the device of the detector and descriptor modules. Default: CPU.
config (Optional[Detector_config], optional) – the pyramid and suppression settings of MultiResolutionDetector, whose config argument lists them with their defaults. Default: None
compile_model (bool, optional) – wrap the DoG response and the non-maxima suppression with torch.compile(), see MultiResolutionDetector. Default: False
score_threshold (float, optional) – the response magnitude a detection has to exceed. Must be non-negative. Default: 0.0
descriptor_backend (Literal['patch', 'pyramid'], optional) – "patch" or "pyramid", see the Convention block. Default: "patch"
descriptor_layout (Literal['kornia', 'opencv'], optional) – "kornia" or "opencv", the order of the descriptor values. Default: "kornia"

Raises:

ValueError – for a descriptor_backend or descriptor_layout other than those listed, or a negative score_threshold.

See also

SIFTFeatureScaleSpace: SIFT on a DoG scale space, with extrema across scale, sub-pixel refinement and 6-sigma frames, for a batch of images.
MultiResolutionDetector: its detector, on the magnitude of the DoG.
BlobDoGSingle: the single-scale DoG response whose magnitude it detects on.
SIFTDescriptor: the descriptor of the "patch" backend, run on 41 x 41 patches through LAFDescriptor.
SIFTDescriptorFromPyramid: the orientation and descriptor of the "pyramid" backend.
convert_sift_descriptor_layout(): the reordering that descriptor_layout="opencv" applies.
LocalFeature: its base class, which gives the output contract.

Example

>>> lafs, responses, descs = SIFTFeature(100)(torch.rand(1, 1, 64, 64))
>>> lafs.shape, responses.shape, descs.shape
(torch.Size([1, 100, 2, 3]), torch.Size([1, 100]), torch.Size([1, 100, 128]))

forward(img, mask=None)[source]#

Return sparse LAFs, detector responses, and one descriptor per LAF.

Return type:: Tuple[Tensor, Tensor, Tensor]

class kornia.feature.GFTTAffNetHardNet(num_features=8000, upright=False, device=None, compile_modules=False)[source]#

Detect GFTT corners over scale with AffNet shapes and HardNet descriptors.

feature = GFTTAffNetHardNet(2048) loads pretrained AffNet and HardNet in evaluation mode. lafs, responses, descs = feature(img) takes grayscale images (B, 1, H, W); convert color images with rgb_to_grayscale() first. It returns affine LAFs (B, N, 2, 3), Shi-Tomasi responses (B, N), strongest first, and unit-norm HardNet descriptors (B, N, 128). Use laf_is_filled() to keep detections and discard padding. For blobs, use HesAffNetHardNet or DoGAffNetHardNet.

Convention:: A LocalFeature, whose Convention block gives the output contract, of grayscale images (B, 1, H, W): a ScaleSpaceDetector with the CornerGFTT response, its maxima only and mr_size=6, whose LAFs get their affine shape from LAFAffNetShapeEstimator, then their orientation from LAFOrienter(19) unless upright, and are described by HardNet on 32 x 32 patches.

Parameters:

num_features (int, optional) – the number of slots per image; the strongest detections fill them, the rest are padding. Default: 8000
upright (bool, optional) – if True, skip the orientation estimation and return upright frames. Default: False
device (Union[str, device, None], optional) – the device of the detector and descriptor modules. Default: CPU.
compile_modules (Union[bool, List[str]], optional) – the detector sub-modules to wrap with torch.compile(), see ScaleSpaceDetector. Default: False

See also

HesAffNetHardNet: the same pipeline on Hessian blobs, 2048 slots by default.
DoGAffNetHardNet: the same shapes and descriptors on DoG blobs, which LightGlueMatcher("dog_affnet_hardnet") matches.
CornerGFTT: the corner response it detects on.
ScaleSpaceDetector: its detector.
LAFAffNetShapeEstimator: the AffNet shape estimation it runs before the orientation.
HardNet: its descriptor, run on 32 x 32 patches through LAFDescriptor.
LocalFeature: its base class, which gives the output contract.

forward(img, mask=None)[source]#

Run forward.

Parameters:

img (Tensor) – image to extract features with shape (B,C,H,W).
mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape (B,1,H,W) with the spatial size of the image. It is forwarded to the detector unchanged; see the detector for its semantics. Default: None

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

Detected local affine frames with shape (B,N,2,3).
Response function values for corresponding lafs with shape (B,N).
Local descriptors of shape (B,N,D) where \(D\) is descriptor size.

The Convention block of LocalFeature gives the frame, N and the slots the detector did not fill, which have to be dropped before matching.

class kornia.feature.HesAffNetHardNet(num_features=2048, upright=False, device=None, compile_modules=False)[source]#

Detect Hessian blobs over scale with AffNet shapes and HardNet descriptors.

feature = HesAffNetHardNet(2048) loads pretrained AffNet and HardNet in evaluation mode. lafs, responses, descs = feature(img) takes grayscale images (B, 1, H, W); convert color images with rgb_to_grayscale() first. It returns affine LAFs (B, N, 2, 3), Hessian responses (B, N), strongest first, and unit-norm HardNet descriptors (B, N, 128). Use laf_is_filled() to keep detections and discard padding. For corners use GFTTAffNetHardNet; for LightGlue-compatible DoG blobs, use DoGAffNetHardNet.

Convention:: A LocalFeature, whose Convention block gives the output contract, of grayscale images (B, 1, H, W): a ScaleSpaceDetector with the BlobHessian response, its maxima only and mr_size=6, whose LAFs get their affine shape from LAFAffNetShapeEstimator, then their orientation from LAFOrienter(19) unless upright, and are described by HardNet on 32 x 32 patches.

Parameters:

num_features (int, optional) – the number of slots per image; the strongest detections fill them, the rest are padding. Default: 2048
upright (bool, optional) – if True, skip the orientation estimation and return upright frames. Default: False
device (Union[str, device, None], optional) – the device of the detector and descriptor modules. Default: CPU.
compile_modules (Union[bool, List[str]], optional) – the detector sub-modules to wrap with torch.compile(), see ScaleSpaceDetector. Default: False

See also

GFTTAffNetHardNet: the same pipeline on Shi-Tomasi corners, 8000 slots by default.
DoGAffNetHardNet: the same shapes and descriptors on DoG blobs, which LightGlueMatcher("dog_affnet_hardnet") matches.
BlobHessian: the blob response it detects on.
ScaleSpaceDetector: its detector, with this response by default.
LAFAffNetShapeEstimator: the AffNet shape estimation it runs before the orientation.
HardNet: its descriptor, run on 32 x 32 patches through LAFDescriptor.
LocalFeature: its base class, which gives the output contract.

forward(img, mask=None)[source]#

Run forward.

Parameters:

img (Tensor) – image to extract features with shape (B,C,H,W).
mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape (B,1,H,W) with the spatial size of the image. It is forwarded to the detector unchanged; see the detector for its semantics. Default: None

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

Detected local affine frames with shape (B,N,2,3).
Response function values for corresponding lafs with shape (B,N).
Local descriptors of shape (B,N,D) where \(D\) is descriptor size.

The Convention block of LocalFeature gives the frame, N and the slots the detector did not fill, which have to be dropped before matching.

class kornia.feature.DoGHardNet(num_features=2048, upright=False, device=None, compile_modules=False)[source]#

Detect DoG blobs over scale and describe them with HardNet, without affine adaptation.

feature = DoGHardNet(2048) loads pretrained HardNet in evaluation mode. lafs, responses, descs = feature(img) takes grayscale images (B, 1, H, W). It returns 12-sigma DoG LAFs (B, N, 2, 3), absolute DoG responses (B, N), strongest first, and unit-norm HardNet descriptors (B, N, 128). Use laf_is_filled() to keep detections and discard padding before calling LightGlueMatcher("doghardnet"). Choose DoGAffNetHardNet for affine shapes or SIFTFeatureScaleSpace for SIFT descriptors.

Convention:

A LocalFeature, whose Convention block gives the output contract. The detector is the DoG ScaleSpaceDetector of SIFTFeatureScaleSpace’s "patch" backend, and the orientation is estimated on its 6-sigma frames. The returned LAFs are those frames scaled by two (scaling_coef=2), to 12 sigma or six times OpenCV’s KeyPoint.size, and HardNet describes them on 32 x 32 patches.
The image must have one channel, (B, 1, H, W): convert a color image with rgb_to_grayscale() first.

Parameters:

num_features (int, optional) – the number of slots per image; the strongest detections fill them, the rest are padding. Default: 2048
upright (bool, optional) – if True, skip the orientation estimation and return upright frames. Default: False
device (Union[str, device, None], optional) – the device of the detector and descriptor modules. Default: CPU.
compile_modules (Union[bool, List[str]], optional) – the detector sub-modules to wrap with torch.compile(), see ScaleSpaceDetector. Default: False

See also

DoGAffNetHardNet: the same detections with AffNet shapes, estimated before the orientation.
SIFTFeatureScaleSpace: the same detections with SIFT descriptors, at 6 sigma, with its "patch" backend.
ScaleSpaceDetector: its detector, on a BlobDoG response.
BlobDoG: the scale-space DoG response it detects on.
HardNet: its descriptor, run on 32 x 32 patches through LAFDescriptor.
LightGlueMatcher: its "doghardnet" weights match these features.
KeyNetHardNet: HardNet descriptors of KeyNet detections, one image per call.
LocalFeature: its base class, which gives the output contract.

forward(img, mask=None)[source]#

Run forward.

Parameters:

img (Tensor) – image to extract features with shape (B,C,H,W).
mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape (B,1,H,W) with the spatial size of the image. It is forwarded to the detector unchanged; see the detector for its semantics. Default: None

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

Detected local affine frames with shape (B,N,2,3).
Response function values for corresponding lafs with shape (B,N).
Local descriptors of shape (B,N,D) where \(D\) is descriptor size.

The Convention block of LocalFeature gives the frame, N and the slots the detector did not fill, which have to be dropped before matching.

class kornia.feature.DoGAffNetHardNet(num_features=2048, upright=False, device=None, compile_modules=False)[source]#

Detect DoG blobs over scale with AffNet shapes and HardNet descriptors.

feature = DoGAffNetHardNet(2048) loads pretrained AffNet and HardNet in evaluation mode. lafs, responses, descs = feature(img) takes grayscale images (B, 1, H, W). It returns the outputs of DoGHardNet, with affine LAF shapes adapted by AffNet before orientation. Use laf_is_filled() to keep detections and discard padding before calling LightGlueMatcher("dog_affnet_hardnet"). Choose GFTTAffNetHardNet or HesAffNetHardNet for corners or Hessian blobs.

Convention:: As DoGHardNet, with the affine shape of each 6-sigma DoG frame estimated by LAFAffNetShapeEstimator before its orientation. The scale of the returned LAFs, the square root of their determinant, is again 12 sigma. The image must have one channel, (B, 1, H, W).

Parameters:

num_features (int, optional) – the number of slots per image; the strongest detections fill them, the rest are padding. Default: 2048
upright (bool, optional) – if True, skip the orientation estimation and return upright frames. Default: False
device (Union[str, device, None], optional) – the device of the detector and descriptor modules. Default: CPU.
compile_modules (Union[bool, List[str]], optional) – the detector sub-modules to wrap with torch.compile(), see ScaleSpaceDetector. Default: False

See also

DoGHardNet: the same detections and descriptors without affine adaptation.
LAFAffNetShapeEstimator: the AffNet shape estimation it runs before the orientation.
ScaleSpaceDetector: its detector, on a BlobDoG response.
HardNet: its descriptor, run on 32 x 32 patches through LAFDescriptor.
LightGlueMatcher: its "dog_affnet_hardnet" mode matches these features.
GFTTAffNetHardNet: the same shapes and descriptors on Shi-Tomasi corners.
HesAffNetHardNet: the same shapes and descriptors on Hessian blobs.
KeyNetAffNetHardNet: the same shapes and descriptors on KeyNet detections, one image per call.
LocalFeature: its base class, which gives the output contract.

forward(img, mask=None)[source]#

Run forward.

Parameters:

img (Tensor) – image to extract features with shape (B,C,H,W).
mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape (B,1,H,W) with the spatial size of the image. It is forwarded to the detector unchanged; see the detector for its semantics. Default: None

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

Detected local affine frames with shape (B,N,2,3).
Response function values for corresponding lafs with shape (B,N).
Local descriptors of shape (B,N,D) where \(D\) is descriptor size.

The Convention block of LocalFeature gives the frame, N and the slots the detector did not fill, which have to be dropped before matching.

class kornia.feature.KeyNetAffNetHardNet(num_features=8000, upright=False, device=None, scale_laf=1.0, compile_model=False, score_threshold=0.0)[source]#

Detect KeyNet keypoints with AffNet shapes and HardNet descriptors.

feature = KeyNetAffNetHardNet(2048) loads pretrained KeyNet, AffNet, OriNet and HardNet in evaluation mode. lafs, responses, descs = feature(img) takes one grayscale image (1, 1, H, W) in [0, 1]. It returns the outputs of KeyNetHardNet, with affine LAF shapes adapted by AffNet before orientation. Use laf_is_filled() to keep detections and discard padding before calling LightGlueMatcher("keynet_affnet_hardnet"). For batches, use DoGAffNetHardNet.

Convention:: As KeyNetHardNet, with the affine shape of each frame estimated by LAFAffNetShapeEstimator before its orientation.

Parameters:

num_features (int, optional) – the number of slots; the strongest detections fill them, the rest are padding. Default: 8000
upright (bool, optional) – if True, skip the orientation estimation and return upright frames. Default: False
device (Union[str, device, None], optional) – the device of the detector and descriptor modules. Default: CPU.
scale_laf (float, optional) – the multiplier of the detector’s LAFs before the description. Must be positive. Default: 1.0
compile_model (bool, optional) – wrap the KeyNet response and the non-maxima suppression with torch.compile(), see KeyNetDetector. Default: False
score_threshold (float, optional) – the KeyNet response a detection has to exceed. Must be non-negative. Default: 0.0

Raises:

ValueError – if scale_laf is not positive or score_threshold is negative.

See also

KeyNetHardNet: the same detections and descriptors without affine adaptation.
KeyNetDetector: its detector.
LAFAffNetShapeEstimator: the AffNet shape estimation it runs before the orientation.
OriNet: the learned orientation, run on each LAF’s patch through LAFOrienter.
HardNet: its descriptor, run on 32 x 32 patches through LAFDescriptor.
LightGlueMatcher: its "keynet_affnet_hardnet" weights match these features.
DoGAffNetHardNet: the same shapes and descriptors on DoG detections, for a batch of images.
LocalFeature: its base class, which gives the output contract.

forward(img, mask=None)[source]#

Run forward.

Parameters:

img (Tensor) – image to extract features with shape (B,C,H,W).
mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape (B,1,H,W) with the spatial size of the image. It is forwarded to the detector unchanged; see the detector for its semantics. Default: None

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

Detected local affine frames with shape (B,N,2,3).
Response function values for corresponding lafs with shape (B,N).
Local descriptors of shape (B,N,D) where \(D\) is descriptor size.

The Convention block of LocalFeature gives the frame, N and the slots the detector did not fill, which have to be dropped before matching.

class kornia.feature.KeyNetHardNet(num_features=8000, upright=False, device=None, scale_laf=1.0, compile_model=False, score_threshold=0.0)[source]#

Detect KeyNet keypoints and describe them with HardNet, without affine adaptation.

feature = KeyNetHardNet(2048) loads pretrained KeyNet, OriNet and HardNet in evaluation mode. lafs, responses, descs = feature(img) takes one grayscale image (1, 1, H, W) in [0, 1]; a batch or color image raises. It returns LAFs (1, N, 2, 3), KeyNet responses (1, N), strongest first, and unit-norm HardNet descriptors (1, N, 128). Use laf_is_filled() to keep detections and discard padding. Choose KeyNetAffNetHardNet for AffNet shapes and its LightGlue weights. For batches, use DoGHardNet.

Convention:: A LocalFeature, whose Convention block gives the output contract, of one grayscale image per call, (1, 1, H, W): a KeyNetDetector, whose LAFs get their orientation from OriNet on 32 x 32 patches unless upright, and are described by HardNet on 32 x 32 patches. scale_laf is the scaling_coef of LocalFeature: it multiplies the detector’s LAFs before the description.

Parameters:

num_features (int, optional) – the number of slots; the strongest detections fill them, the rest are padding. Default: 8000
upright (bool, optional) – if True, skip the orientation estimation and return upright frames. Default: False
device (Union[str, device, None], optional) – the device of the detector and descriptor modules. Default: CPU.
scale_laf (float, optional) – the multiplier of the detector’s LAFs before the description. Must be positive. Default: 1.0
compile_model (bool, optional) – wrap the KeyNet response and the non-maxima suppression with torch.compile(), see KeyNetDetector. Default: False
score_threshold (float, optional) – the KeyNet response a detection has to exceed. Must be non-negative. Default: 0.0

Raises:

ValueError – if scale_laf is not positive or score_threshold is negative.

See also

KeyNetAffNetHardNet: the same detections with AffNet shapes, estimated before the orientation.
KeyNetDetector: its detector.
KeyNet: the learned response it detects on.
OriNet: the learned orientation, run on each LAF’s patch through LAFOrienter.
HardNet: its descriptor, run on 32 x 32 patches through LAFDescriptor.
DoGHardNet: HardNet descriptors of DoG detections, for a batch of images.
LocalFeature: its base class, which gives the output contract.

forward(img, mask=None)[source]#

Run forward.

Parameters:

img (Tensor) – image to extract features with shape (B,C,H,W).
mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape (B,1,H,W) with the spatial size of the image. It is forwarded to the detector unchanged; see the detector for its semantics. Default: None

Return type:

Tuple[Tensor, Tensor, Tensor]

Returns:

Detected local affine frames with shape (B,N,2,3).
Response function values for corresponding lafs with shape (B,N).
Local descriptors of shape (B,N,D) where \(D\) is descriptor size.

The Convention block of LocalFeature gives the frame, N and the slots the detector did not fill, which have to be dropped before matching.