Matching#

Descriptor matching#

kornia.feature.match_nn(desc1, desc2, dm=None)[source]#

Find nearest neighbors in desc2 for each vector in desc1.

If the distance matrix dm is not provided, torch.cdist() is used.

Parameters:

  • desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).

  • desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).

  • dm (Optional[Tensor], optional) – torch.Tensor containing the distances from each descriptor in desc1 to each descriptor in desc2, shape of \((B1, B2)\). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

  • Descriptor distance of matching descriptors, shape of \((B1, 1)\).

  • Long torch.Tensor indexes of matching descriptors in desc1 and desc2, shape of \((B1, 2)\).

kornia.feature.match_mnn(desc1, desc2, dm=None)[source]#

Find mutual nearest neighbors in desc2 for each vector in desc1.

If the distance matrix dm is not provided, torch.cdist() is used.

Parameters:

  • desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).

  • desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).

  • dm (Optional[Tensor], optional) – torch.Tensor containing the distances from each descriptor in desc1 to each descriptor in desc2, shape of \((B1, B2)\). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

  • Descriptor distance of matching descriptors, shape of. \((B3, 1)\).

  • Long torch.Tensor indexes of matching descriptors in desc1 and desc2, shape of \((B3, 2)\), where 0 <= B3 <= min(B1, B2)

kornia.feature.match_snn(desc1, desc2, th=0.8, dm=None)[source]#

Find nearest neighbors in desc2 for each vector in desc1.

The method satisfies first to second nearest neighbor distance <= th.

If the distance matrix dm is not provided, torch.cdist() is used.

Parameters:

  • desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).

  • desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).

  • th (float, optional) – distance ratio threshold. Default: 0.8

  • dm (Optional[Tensor], optional) – torch.Tensor containing the distances from each descriptor in desc1 to each descriptor in desc2, shape of \((B1, B2)\). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

  • Descriptor distance of matching descriptors, shape of \((B3, 1)\).

  • Long torch.Tensor indexes of matching descriptors in desc1 and desc2. Shape: \((B3, 2)\), where 0 <= B3 <= B1.

kornia.feature.match_smnn(desc1, desc2, th=0.95, dm=None)[source]#

Find mutual nearest neighbors in desc2 for each vector in desc1.

the method satisfies first to second nearest neighbor distance <= th.

If the distance matrix dm is not provided, torch.cdist() is used.

Parameters:

  • desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).

  • desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).

  • th (float, optional) – distance ratio threshold. Default: 0.95

  • dm (Optional[Tensor], optional) – torch.Tensor containing the distances from each descriptor in desc1 to each descriptor in desc2, shape of \((B1, B2)\). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

  • Descriptor distance of matching descriptors, shape of. \((B3, 1)\).

  • Long torch.Tensor indexes of matching descriptors in desc1 and desc2, shape of \((B3, 2)\) where 0 <= B3 <= B1.

kornia.feature.match_fginn(desc1, desc2, lafs1, lafs2, th=0.8, spatial_th=10.0, mutual=False, dm=None)[source]#

Find nearest neighbors in desc2 for each vector in desc1.

The method satisfies first to second nearest neighbor distance <= th, and assures 2nd nearest neighbor is geometrically inconsistent with the 1st one (see [MMP15] for more details)

If the distance matrix dm is not provided, torch.cdist() is used.

Note

The geometric check looks at the min(10, B2) nearest candidates and penalizes every one of them that lies within spatial_th pixels of the query’s own 1st nearest neighbor. When all of them do – a dense cluster of detections on one structure – the effective 2nd nearest neighbor distance saturates, the ratio collapses towards zero and the match is accepted. That is the intended reading: no distinct competing structure among the candidates means the 1st nearest neighbor is unambiguous.

Parameters:

  • desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).

  • desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).

  • lafs1 (Tensor) – LAFs of a shape \((1, B1, 2, 3)\). Accepted for API symmetry with match_adalam() but not read by this function – only lafs2 feeds the geometric check.

  • lafs2 (Tensor) – LAFs of a shape \((1, B2, 2, 3)\).

  • th (float, optional) – distance ratio threshold. Default: 0.8

  • spatial_th (float, optional) – minimal distance in pixels to 2nd nearest neighbor. Default: 10.0

  • mutual (bool, optional) – also perform mutual nearest neighbor check. Default: False

  • dm (Optional[Tensor], optional) – torch.Tensor containing the distances from each descriptor in desc1 to each descriptor in desc2, shape of \((B1, B2)\). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

  • Descriptor distance of matching descriptors, shape of \((B3, 1)\).

  • Long torch.Tensor indexes of matching descriptors in desc1 and desc2. Shape: \((B3, 2)\), where 0 <= B3 <= B1.

kornia.feature.match_adalam(desc1, desc2, lafs1, lafs2, config=None, hw1=None, hw2=None, dm=None)[source]#

Perform descriptor matching, followed by AdaLAM filtering.

See [CLO+20] for more details.

If the distance matrix dm is not provided, torch.cdist() is used.

Parameters:

  • desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).

  • desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).

  • lafs1 (Tensor) – LAFs of a shape \((1, B1, 2, 3)\).

  • lafs2 (Tensor) – LAFs of a shape \((1, B2, 2, 3)\).

  • config (Optional[AdalamConfig], optional) – dict with AdaLAM config Default: None

  • dm (Optional[Tensor], optional) – torch.Tensor containing the distances from each descriptor in desc1 to each descriptor in desc2, shape of \((B1, B2)\). Default: None

  • hw1 (Optional[Tuple[int, int]], optional) – Height/width of image. Default: None

  • hw2 (Optional[Tuple[int, int]], optional) – Height/width of image. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

  • Descriptor distance of matching descriptors, shape of \((B3, 1)\).

  • Long torch.Tensor indexes of matching descriptors in desc1 and desc2. Shape: \((B3, 2)\), where 0 <= B3 <= B1.

class kornia.feature.DescriptorMatcher(match_mode='snn', th=0.8)[source]#

nn.Module version of descriptor-only matching functions.

This matcher only requires descriptors (no LAFs). For geometry-aware matching that uses LAFs, see GeometryAwareDescriptorMatcher.

See match_nn(), match_snn(),

match_mnn() or match_smnn() for more details.

Parameters:

  • match_mode (str, optional) – type of matching, can be nn, snn, mnn, smnn. Default: "snn"

  • th (float, optional) – threshold on distance ratio, or other quality measure. Default: 0.8

forward(desc1, desc2)[source]#

Run forward.

Parameters:

  • desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).

  • desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).

Return type:

Tuple[Tensor, Tensor]

Returns:

  • Descriptor distance of matching descriptors, shape of \((B3, 1)\).

  • Long torch.Tensor indexes of matching descriptors in desc1 and desc2,

    shape of \((B3, 2)\) where \(0 <= B3 <= B1\).

class kornia.feature.GeometryAwareDescriptorMatcher(match_mode='fginn', params=None)[source]#

nn.Module version of geometry-aware matching functions that use LAFs (Local Affine Frames).

Unlike DescriptorMatcher, this matcher requires both descriptors and LAFs. See match_fginn() or match_adalam() for more details.

Parameters:

  • match_mode (str, optional) – type of matching, can be fginn or adalam. Default: "fginn"

  • params (Optional[Dict[str, Tensor]], optional) – dictionary of parameters for the matching function. Default: None

forward(desc1, desc2, lafs1, lafs2)[source]#

Run forward.

Parameters:

  • desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).

  • desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).

  • lafs1 (Tensor) – LAFs of a shape \((1, B1, 2, 3)\).

  • lafs2 (Tensor) – LAFs of a shape \((1, B2, 2, 3)\).

Return type:

Tuple[Tensor, Tensor]

Returns:

  • Descriptor distance of matching descriptors, shape of \((B3, 1)\).

  • Long torch.Tensor indexes of matching descriptors in desc1 and desc2,

    shape of \((B3, 2)\) where \(0 <= B3 <= B1\).

class kornia.feature.LocalFeatureMatcher(local_feature, matcher)[source]#

nn.Module, which finds correspondences between two images based on local features.

Parameters:
Returns:

Dictionary with image correspondences and confidence scores.

Return type:

Dict[str, torch.Tensor]

Example

>>> img1 = torch.rand(1, 1, 320, 200)
>>> img2 = torch.rand(1, 1, 128, 128)
>>> input = {"image0": img1, "image1": img2}
>>> gftt_hardnet_matcher = LocalFeatureMatcher(
...     GFTTAffNetHardNet(10), kornia.feature.DescriptorMatcher('snn', 0.8)
... )
>>> out = gftt_hardnet_matcher(input)
forward(data)[source]#

Run forward.

Parameters:

data (Dict[str, Tensor]) – dictionary containing the input data in the following format:

Keyword Arguments:

  • image0 – left image with shape \((N, 1, H1, W1)\).

  • image1 – right image with shape \((N, 1, H2, W2)\).

  • mask0 (optional) – left image mask. ‘0’ suppresses detection, with shape \((N, H1, W1)\) or \((N, 1, H1, W1)\).

  • mask1 (optional) – right image mask. ‘0’ suppresses detection, with shape \((N, H2, W2)\) or \((N, 1, H2, W2)\).

Return type:

Dict[str, Tensor]

Returns:

  • keypoints0, matching keypoints from image0 \((NC, 2)\).

  • keypoints1, matching keypoints from image1 \((NC, 2)\).

  • confidence, 1 - descriptor distance \((NC)\). This lies in \([0, 1]\) only for the ratio-based DescriptorMatcher modes (snn, smnn); nn and mnn return raw distances, which are unbounded for an arbitrary descriptor, so the confidence has no lower bound in general – it goes at least down to -1, the value reached by unit-norm descriptors, whose distance tops out at 2.0.

  • lafs0, matching LAFs from image0 \((1, NC, 2, 3)\).

  • lafs1, matching LAFs from image1 \((1, NC, 2, 3)\).

  • batch_indexes, batch indexes for the keypoints and lafs \((NC)\).

Learned matchers#

class kornia.feature.LightGlueMatcher(feature_name='disk', params=None)[source]#

LightGlue-based matcher in kornia API.

This is based on the original code from paper “LightGlue: Local Feature Matching at Light Speed”. See [LSP23] for more details.

Parameters:

  • feature_name (str, optional) – type of feature for matching, can be disk or superpoint. Default: "disk"

  • params (Optional[Dict], optional) – LightGlue params. Default: None

forward(desc1, desc2, lafs1, lafs2, hw1=None, hw2=None)[source]#

Run forward.

Parameters:

  • desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).

  • desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).

  • lafs1 (Tensor) – LAFs of a shape \((1, B1, 2, 3)\).

  • lafs2 (Tensor) – LAFs of a shape \((1, B2, 2, 3)\).

  • hw1 (Optional[Tuple[int, int]], optional) – Height/width of image. Default: None

  • hw2 (Optional[Tuple[int, int]], optional) – Height/width of image. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

  • Descriptor distance of matching descriptors, shape of \((B3, 1)\).

  • Long torch.Tensor indexes of matching descriptors in desc1 and desc2,

    shape of \((B3, 2)\) where \(0 <= B3 <= B1\).

class kornia.feature.LightGlue(features='superpoint', **conf_)[source]#

Implement the LightGlue matcher for sparse local features.

LightGlue is a deep network that matches local features across image pairs using a series of transformer layers and an adaptive pruning mechanism.

Parameters:

  • features (str, optional) – The type of local features to match (e.g., ‘superpoint’, ‘disk’). Default: "superpoint"

  • conf – A configuration dictionary to override default parameters.

forward(data)[source]#

Match keypoints and descriptors between two images.

Return type:

dict

Input (dict):
image0: dict

keypoints: [B x M x 2] descriptors: [B x M x D] image: [B x C x H x W] or image_size: [B x 2]

image1: dict

keypoints: [B x N x 2] descriptors: [B x N x D] image: [B x C x H x W] or image_size: [B x 2]

Output (dict):

log_assignment: [B x M+1 x N+1] matches0: [B x M] matching_scores0: [B x M] matches1: [B x N] matching_scores1: [B x N] matches: List[[Si x 2]], scores: List[[Si]]

class kornia.feature.OnnxLightGlue(weights=None, device='cpu')[source]#

Wrapper for loading LightGlue-ONNX models and running inference via ONNXRuntime.

LightGlue [LSP23] performs fast descriptor-based deep keypoint matching. This module requires onnxruntime to be installed.

If you have trained your own LightGlue model, see fabio-sim/LightGlue-ONNX for how to export the model to ONNX and optimize it.

Parameters:

  • weights (str | None, optional) – Pretrained weights, or a path to your own exported ONNX model. Available pretrained weights are 'disk', 'superpoint', 'disk_fp16', and 'superpoint_fp16'. Note that FP16 requires CUDA. Defaults to 'disk_fp16' if device is CUDA, and 'disk' if CPU. Default: None

  • device (Union[str, device, None], optional) – Union[str, torch.device, None] to run inference on. Default: "cpu"

forward(data)[source]#

Match keypoints and descriptors between two images.

The output contains the matches (the indices of the matching keypoint pairs between the first and second image) and the corresponding confidence scores. Only a batch size of 1 is supported.

Parameters:

data (dict[str, dict[str, Tensor]]) – Dictionary containing both images and the keypoints and descriptors thereof.

Return type:

dict[str, Tensor]

Returns:

Dictionary containing the matches and scores.

data (dict):
image0 (dict):

keypoints (float32): \((1, M, 2)\)

descriptors (float32): \((1, M, D)\)

image: \((1, C, H, W)\) or image_size: \((1, 2)\)

image1 (dict):

keypoints (float32): \((1, N, 2)\)

descriptors (float32): \((1, N, D)\)

image: \((1, C, H, W)\) or image_size: \((1, 2)\)

output (dict):

matches (int64): \((S, 2)\)

scores (float32): \((S)\)

class kornia.feature.LoFTR(pretrained='outdoor', config=default_cfg)[source]#

nn.Module, which finds correspondences between two images.

This is based on the original code from paper “LoFTR: Detector-Free Local Feature Matching with Transformers”. See [SSW+21] for more details.

If the distance matrix dm is not provided, torch.cdist() is used.

Parameters:

  • config (dict[str, Any], optional) – Dict with initialization parameters. Do not pass it, unless you know what you are doing`. Default: default_cfg

  • pretrained (Optional[str], optional) – Download and set pretrained weights to the model. Options: ‘outdoor’, ‘indoor’. ‘outdoor’ is trained on the MegaDepth dataset and ‘indoor’ on the ScanNet. Default: "outdoor"

Returns:

Dictionary with image correspondences and confidence scores.

Example

>>> img1 = torch.rand(1, 1, 320, 200)
>>> img2 = torch.rand(1, 1, 128, 128)
>>> input = {"image0": img1, "image1": img2}
>>> loftr = LoFTR('outdoor')
>>> out = loftr(input)
forward(data)[source]#

Run forward.

Parameters:

data (dict[str, Tensor]) – dictionary containing the input data in the following format:

Keyword Arguments:

  • image0 – left image with shape \((N, 1, H1, W1)\).

  • image1 – right image with shape \((N, 1, H2, W2)\).

  • mask0 (optional) – left image mask. ‘0’ indicates a padded position \((N, H1, W1)\).

  • mask1 (optional) – right image mask. ‘0’ indicates a padded position \((N, H2, W2)\).

Return type:

dict[str, Tensor]

Returns:

  • keypoints0, matching keypoints from image0 \((NC, 2)\).

  • keypoints1, matching keypoints from image1 \((NC, 2)\).

  • confidence, confidence score [0, 1] \((NC)\).

  • batch_indexes, batch indexes for the keypoints and lafs \((NC)\).