Matching#

Descriptor matching#

kornia.feature.match_nn(desc1, desc2, dm=None)[source]#

Find nearest neighbors in desc2 for each vector in desc1.

Convention:

Every match_* function returns (dists, idxs), one row per match. idxs is an int64 \((M, 2)\) tensor of (index in desc1, index in desc2) rows (match_smnn_batched() adds the batch index as a first column). dists is \((M, 1)\), in the dtype of dm when it is given and in the descriptor dtype otherwise, except that match_adalam() returns float32. Lower values are better, and what they are depends on the function:
- match_nn(), match_mnn(): the distance, the entry of dm when it is given.
- match_snn(): the ratio of the nearest to the second-nearest distance, not a distance.
- match_smnn(), match_smnn_batched(): the larger of the two ratios, desc1 to desc2 and back.
- match_fginn(): the ratio of the nearest distance to that of a second neighbour, taken among the 10 nearest, whose lafs2 centre lies at least spatial_th from the nearest one’s, and 0 without one; see its Convention block.
- match_adalam(): the inverse of the local RANSAC confidence; see its Convention block.
LightGlueMatcher returns LightGlue’s matching score in the place of dists, which is higher for better matches, and LocalFeatureMatcher reports 1 - dists as confidence.
The ratio tests of match_snn(), match_smnn(), match_smnn_batched() and match_fginn() keep ratio <= th: a ratio equal to th passes.
Without dm, the L2 distances between the descriptors are used. A given dm is read as it is: match_nn() and match_mnn() accept any matrix in which smaller is better and return its entries, while the ratio tests divide its entries, so they need non-negative values, and squared distances need th ** 2.
An empty desc1 or desc2 gives a \((0, 1)\) and a \((0, 2)\) tensor (\((0, 3)\) for match_smnn_batched()). The ratio tests need a second neighbour: match_snn() and match_fginn() return no match unless desc2 has two rows, and match_smnn(), match_smnn_batched() and match_adalam(), which scores its putative matches by the same ratio, need two rows in both.

Parameters:

desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).
desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).
dm (Optional[Tensor], optional) – torch.Tensor containing the distances from each descriptor in desc1 to each descriptor in desc2, shape of \((B1, B2)\). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Descriptor distance of matching descriptors, shape of \((B1, 1)\).
Long torch.Tensor indexes of matching descriptors in desc1 and desc2, shape of \((B1, 2)\).

kornia.feature.match_mnn(desc1, desc2, dm=None)[source]#

Find mutual nearest neighbors in desc2 for each vector in desc1.

Convention:: See the Convention block on match_nn(). A pair is kept only when each of the two descriptors is the nearest neighbour of the other.

Parameters:

desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).
desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).
dm (Optional[Tensor], optional) – torch.Tensor containing the distances from each descriptor in desc1 to each descriptor in desc2, shape of \((B1, B2)\). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Descriptor distance of matching descriptors, shape of \((B3, 1)\).
Long torch.Tensor indexes of matching descriptors in desc1 and desc2, shape of \((B3, 2)\), where 0 <= B3 <= min(B1, B2)

kornia.feature.match_snn(desc1, desc2, th=0.8, dm=None)[source]#

Find nearest neighbors in desc2 for each vector in desc1.

The method satisfies first to second nearest neighbor distance <= th.

Convention:: See the Convention block on match_nn(); dists is the ratio of the nearest to the second-nearest distance.

Parameters:

desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).
desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).
th (float, optional) – distance ratio threshold. Default: 0.8
dm (Optional[Tensor], optional) – torch.Tensor containing the distances from each descriptor in desc1 to each descriptor in desc2, shape of \((B1, B2)\). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Ratio of the nearest to the second-nearest distance of each match, shape of \((B3, 1)\).
Long torch.Tensor indexes of matching descriptors in desc1 and desc2. Shape: \((B3, 2)\), where 0 <= B3 <= B1.

kornia.feature.match_smnn(desc1, desc2, th=0.95, dm=None)[source]#

Find mutual nearest neighbors in desc2 for each vector in desc1.

the method satisfies first to second nearest neighbor distance <= th.

Convention:

See the Convention block on match_nn(). A pair is kept when the two descriptors are mutual nearest neighbours and pass the ratio test, ratio <= th, in both directions; dists is the larger of the two ratios.
th defaults to 0.95, where match_snn() and match_fginn() default to 0.8.

Parameters:

desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).
desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).
th (float, optional) – distance ratio threshold. Default: 0.95
dm (Optional[Tensor], optional) – torch.Tensor containing the distances from each descriptor in desc1 to each descriptor in desc2, shape of \((B1, B2)\). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

The larger of the two distance ratios of each match, shape of \((B3, 1)\).
Long torch.Tensor indexes of matching descriptors in desc1 and desc2, shape of \((B3, 2)\) where 0 <= B3 <= B1.

kornia.feature.match_smnn_batched(desc1, desc2, th=0.95, dm=None, mask1=None, mask2=None)[source]#

Match independent descriptor pairs with a batched symmetric nearest-neighbor ratio test.

Like match_smnn(), each match must pass Lowe’s first/second L2 distance ratio test in both directions and have mutual nearest neighbors. The returned quality is the larger of the two ratios. Pairs are processed together with batched distance computation and top-k reductions, without a per-pair loop.

Convention:: See the Convention blocks on match_nn() and match_smnn(). For each batch item it returns the pairs and ratios that match_smnn() returns for the valid descriptors of that item, with the batch index as the first column of the indices. For float16 and bfloat16 descriptors these are the results of match_smnn() on float32 copies, with the ratios rounded back, and a few pairs and ratios can differ slightly from those of match_smnn() on the half-precision descriptors.

Parameters:

desc1 (Tensor) – First descriptors, shape \((B, N, D)\).
desc2 (Tensor) – Second descriptors, shape \((B, M, D)\).
th (float, optional) – Inclusive distance-ratio threshold. Default: 0.95
dm (Optional[Tensor], optional) – Optional precomputed L2 distances, shape \((B, N, M)\). Default: None
mask1 (Optional[Tensor], optional) – Optional boolean mask of valid first descriptors, shape \((B, N)\). Default: None
mask2 (Optional[Tensor], optional) – Optional boolean mask of valid second descriptors, shape \((B, M)\). Masks allow padding variable-length pairs; padding never becomes a neighbor. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Ratios, shape \((K, 1)\), with the input device and dtype.
Long indices, shape \((K, 3)\): (batch_index, index_in_desc1, index_in_desc2). Indices refer to the original padded tensors, sorted by batch and first-descriptor index. A pair with fewer than two valid descriptors on either side contributes no matches.

Note

Memory for the distance matrix scales as \(B N M\). Bucket pairs by descriptor counts and limit batch size when matching large collections. Half-precision descriptors use float32 distance computation before ratios are converted back to the input dtype. Masked inputs compute the distances directly, without the matrix-multiplication form, which avoids its cancellation but can be slower for large masked descriptor sets. As in match_smnn(), a zero second-neighbor distance produces an undefined ratio and that ambiguous match is rejected. Ties follow torch.topk() and do not have a guaranteed cross-device ordering.

Example

>>> a = torch.tensor([[[0., 0.], [1., 1.], [2., 2.]]])
>>> ratios, indices = match_smnn_batched(a, a.flip(1))
>>> indices
tensor([[0, 0, 2],
        [0, 1, 1],
        [0, 2, 0]])

kornia.feature.match_fginn(desc1, desc2, lafs1, lafs2, th=0.8, spatial_th=10.0, mutual=False, dm=None)[source]#

Find nearest neighbors in desc2 for each vector in desc1.

The method satisfies first to second nearest neighbor distance <= th, and assures 2nd nearest neighbor is geometrically inconsistent with the 1st one (see [MMP15] for more details)

Convention:

See the Convention block on match_nn(). dists is a ratio: the nearest distance over the distance of the second neighbour, which is the nearest of the min(10, B2) nearest candidates, the first one excluded, whose lafs2 centre lies at least spatial_th from the centre of the first one. Candidates closer than that count as the same structure, not as competitors, whatever their descriptor distance. When all of them are – a dense cluster of detections on one structure – the ratio is 0 and the match is accepted: no distinct competing structure among the candidates means the nearest neighbour is unambiguous. A descriptor beyond the min(10, B2) nearest is never the second neighbour, even when every candidate is excluded.
Only the centres of lafs2 are read, and spatial_th is in their units: pixels for LAFs in pixels.

Parameters:

desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).
desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).
lafs1 (Tensor) – LAFs of a shape \((1, B1, 2, 3)\). Accepted for API symmetry with match_adalam() but not read by this function – only lafs2 feeds the geometric check.
lafs2 (Tensor) – LAFs of a shape \((1, B2, 2, 3)\).
th (float, optional) – distance ratio threshold. Default: 0.8
spatial_th (float, optional) – minimal distance between the lafs2 centres of the 1st and the 2nd nearest neighbor, in the units of those centres. Default: 10.0
mutual (bool, optional) – also perform mutual nearest neighbor check. Default: False
dm (Optional[Tensor], optional) – torch.Tensor containing the distances from each descriptor in desc1 to each descriptor in desc2, shape of \((B1, B2)\). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Ratio of the nearest to the second-neighbour distance of each match, see the Convention block, shape of \((B3, 1)\).
Long torch.Tensor indexes of matching descriptors in desc1 and desc2. Shape: \((B3, 2)\), where 0 <= B3 <= B1.

kornia.feature.match_adalam(desc1, desc2, lafs1, lafs2, config=None, hw1=None, hw2=None, dm=None)[source]#

Perform descriptor matching, followed by AdaLAM filtering.

See [CLO+20] for more details.

Convention:

See the Convention block on match_nn(). dists is the inverse of the confidence of the local RANSAC that accepted the match, in float32 whatever the descriptor dtype, so at most 1 / min_confidence (1 / 200 by default). If no seed survives, for example with very few keypoints, the matches of a ratio test on squared distances, ratio <= 0.95, are returned with that ratio instead.
The putative matches are the nearest neighbours in the distance matrix dm, scored by the ratio of the two smallest distances; with force_seed_mnn only mutual nearest neighbours can be seeds. dm holds plain distances, like the dm of match_nn() and the other matchers: AdaLAM squares it internally (its thresholds are tuned for squared distances), so dm=torch.cdist(desc1, desc2) gives the matches of dm=None, where the squared L2 distances between the descriptors are used.
The LAFs give each keypoint’s centre, orientation in degrees and scale. Orientations and scales are read only through how their change between the two images varies from match to match: adding an angle to the orientations of one image or both, negating both, or scaling the LAFs of one image or both alike changes nothing.
hw1 and hw2 enter only through the area of their image, which sets the neighbourhood radius in that image, so (H, W) and (W, H) give the same matches. Without them, the extent of each image’s keypoints is used.
config is merged into the defaults of kornia.feature.adalam.get_adalam_default_config(); an unknown key warns and is ignored. The computation runs on the device of desc1 unless config sets "device".
Exact local affine correspondences are retained when enough distinct keypoints support them beyond the seed and minimal fitting pair. Unlike the reference implementation, equal tiny residuals alone do not imply duplicate keypoints. Their confidence uses a conservative residual-area floor of 2**-24 in normalized coordinates, so inverse confidence stays finite even for an exact fit.

Parameters:

desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).
desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).
lafs1 (Tensor) – LAFs of a shape \((1, B1, 2, 3)\).
lafs2 (Tensor) – LAFs of a shape \((1, B2, 2, 3)\).
config (Optional[AdalamConfig], optional) – dict with AdaLAM config Default: None
dm (Optional[Tensor], optional) – torch.Tensor containing the non-negative distances (not squared) from each descriptor in desc1 to each descriptor in desc2, shape of \((B1, B2)\). Smaller is better. Default: None
hw1 (Optional[Sequence[int]], optional) – size of the first image, (H, W) or (W, H). Default: None
hw2 (Optional[Sequence[int]], optional) – size of the second image, (H, W) or (W, H). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Inverse local RANSAC confidence of each match, float32, shape of \((B3, 1)\); see the Convention block.
Long torch.Tensor indexes of matching descriptors in desc1 and desc2. Shape: \((B3, 2)\), where 0 <= B3 <= B1.

class kornia.feature.DescriptorMatcher(match_mode='snn', th=0.8)[source]#

nn.Module version of descriptor-only matching functions.

This matcher only requires descriptors (no LAFs). For geometry-aware matching that uses LAFs, see GeometryAwareDescriptorMatcher.

See match_nn(), match_snn(),: match_mnn() or match_smnn() for more details.
Convention:: See the Convention block on match_nn(). forward returns what the function of match_mode (case-insensitive) returns for the two descriptor sets, with the module’s th for snn and smnn. th defaults to 0.8 for both, while match_smnn() defaults to 0.95, and nn and mnn do not read it. There is no dm.

Parameters:

match_mode (str, optional) – type of matching, can be nn, snn, mnn, smnn. Default: "snn"
th (float, optional) – distance ratio threshold of the snn and smnn modes. Default: 0.8

forward(desc1, desc2)[source]#

Run forward.

Parameters:

desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).
desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).

Return type:

Tuple[Tensor, Tensor]

Returns:

dists of the matching function, see match_nn(), shape of \((B3, 1)\).
Long torch.Tensor indexes of matching descriptors in desc1 and desc2,
shape of \((B3, 2)\) where \(0 <= B3 <= B1\).

class kornia.feature.GeometryAwareDescriptorMatcher(match_mode='fginn', params=None)[source]#

nn.Module version of geometry-aware matching functions that use LAFs (Local Affine Frames).

Unlike DescriptorMatcher, this matcher requires both descriptors and LAFs. See match_fginn() or match_adalam() for more details.

Convention:

See the Convention block on match_nn(). forward returns what the function of match_mode (case-insensitive) returns:

"fginn" reads the keys th, spatial_th and mutual of params and ignores any other key. th defaults to 0.85, while match_fginn() defaults to 0.8.
"adalam" passes params as the AdaLAM config, and no hw1 or hw2, so AdaLAM takes the extent of each image’s keypoints as the image size.

Parameters:

match_mode (str, optional) – type of matching, can be fginn or adalam. Default: "fginn"
params (Optional[Dict[str, Tensor]], optional) – dictionary of parameters for the matching function. Default: None

forward(desc1, desc2, lafs1, lafs2)[source]#

Run forward.

Parameters:

desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).
desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).
lafs1 (Tensor) – LAFs of a shape \((1, B1, 2, 3)\).
lafs2 (Tensor) – LAFs of a shape \((1, B2, 2, 3)\).

Return type:

Tuple[Tensor, Tensor]

Returns:

dists of the matching function, see match_nn(), shape of \((B3, 1)\).
Long torch.Tensor indexes of matching descriptors in desc1 and desc2,
shape of \((B3, 2)\) where \(0 <= B3 <= B1\).

class kornia.feature.LocalFeatureMatcher(local_feature, matcher)[source]#

nn.Module, which finds correspondences between two images based on local features.

Convention:

keypoints0 and keypoints1 are the centres (x, y) of the returned LAF tensors lafs0 and lafs1, respectively; see laf_from_center_scale_ori() and Local features for the LAF frame. confidence is 1 - dists of the matcher (see match_nn()).
All-zero LAFs, the padding slots of kornia’s detectors (see laf_is_filled()), are dropped before matching.
matcher is called as matcher(descriptors0, descriptors1) for each image pair, so it has to be a descriptor-only matcher such as DescriptorMatcher; GeometryAwareDescriptorMatcher and LightGlueMatcher need LAFs and do not fit.

Parameters:

local_feature (Module) – Local feature detector. See GFTTAffNetHardNet.
matcher (Module) – Descriptor matcher, see DescriptorMatcher.

Returns:

Dictionary with image correspondences and confidence scores.

Return type:

Dict[str, torch.Tensor]

Example

>>> img1 = torch.rand(1, 1, 320, 200)
>>> img2 = torch.rand(1, 1, 128, 128)
>>> input = {"image0": img1, "image1": img2}
>>> gftt_hardnet_matcher = LocalFeatureMatcher(
...     GFTTAffNetHardNet(10), kornia.feature.DescriptorMatcher('snn', 0.8)
... )
>>> out = gftt_hardnet_matcher(input)

forward(data)[source]#

Run forward.

Parameters:

data (Dict[str, Tensor]) – dictionary containing the input data in the following format:

Keyword Arguments:

image0 – left image with shape \((N, 1, H1, W1)\).
image1 – right image with shape \((N, 1, H2, W2)\).
mask0 (optional) – left image mask. ‘0’ suppresses detection, with shape \((N, H1, W1)\) or \((N, 1, H1, W1)\).
mask1 (optional) – right image mask. ‘0’ suppresses detection, with shape \((N, H2, W2)\) or \((N, 1, H2, W2)\).
lafs0 (optional) – pre-extracted LAFs of image0, \((N, M0, 2, 3)\). When lafs0 and descriptors0 are both given, they are matched and local_feature does not run on image0.
descriptors0 (optional) – pre-extracted descriptors of image0, \((N, M0, D)\).
lafs1 (optional) – pre-extracted LAFs of image1, \((N, M1, 2, 3)\); with descriptors1, as for image0.
descriptors1 (optional) – pre-extracted descriptors of image1, \((N, M1, D)\).

Return type:

Dict[str, Tensor]

Returns:

keypoints0, matching keypoints from image0 \((NC, 2)\).
keypoints1, matching keypoints from image1 \((NC, 2)\).
confidence, 1 - descriptor distance \((NC)\). This lies in \([0, 1]\) only for the ratio-based DescriptorMatcher modes (snn, smnn); nn and mnn return raw distances, which are unbounded for an arbitrary descriptor, so the confidence has no lower bound in general – it goes at least down to -1, the value reached by unit-norm descriptors, whose distance tops out at 2.0.
lafs0, matching LAFs from image0 \((1, NC, 2, 3)\).
lafs1, matching LAFs from image1 \((1, NC, 2, 3)\).
batch_indexes, batch indexes for the keypoints and lafs \((NC)\).

Learned matchers#

class kornia.feature.LightGlueMatcher(feature_name='disk', params=None)[source]#

LightGlue-based matcher in kornia API.

This is based on the original code from paper “LightGlue: Local Feature Matching at Light Speed”. See [LSP23] for more details.

Convention:

lafs1 and lafs2 are LAFs in the pixel frame of laf_from_center_scale_ori() (see Local features), and their centres go to LightGlue unchanged as the keypoints. hw1 and hw2 are the (H, W) of each image, which become LightGlue’s image_size (W, H); without them, the image size is taken as the largest keypoint coordinates of that image, (max x, max y).
What the network gets besides the keypoints depends on feature_name (case-insensitive):
- "aliked", "dedodeb", "dedodeg", "disk", "superpoint": nothing else.
- "sift": the scale and orientation of the COLMAP keypoints the weights were trained on: the negated LAF orientation in \((-\pi, \pi]\), and a sixth of the LAF scale, which is the detection sigma for 6-sigma DoG frames such as SIFTFeatureScaleSpace’s. The descriptors must be in OpenCV’s SIFT layout, which these weights expect, while kornia’s SIFT descriptors use another order of the same values: create the feature with descriptor_layout="opencv", or convert the descriptors with convert_sift_descriptor_layout(descs, "kornia", "opencv") (convert_sift_descriptor_layout()).
- "doghardnet", "dog_affnet_hardnet": pass the LAFs the HardNet descriptors were computed on, at the scale the weights were trained with: 12 sigma, six times OpenCV’s KeyPoint.size and twice the 6-sigma DoG frames of SIFTFeatureScaleSpace. DoGHardNet and DoGAffNetHardNet return such LAFs and their descriptors; with another 6-sigma DoG detector, apply scale_laf() with 2.0 to its LAFs before describing them. The network takes a sixth of the LAF scale, which is KeyPoint.size for these frames, and the LAF orientation in \([0, 2\pi)\).
- "keynet_affnet_hardnet": the ends of the two LAF axes, c + A[:, 0] and c + A[:, 1], normalised like the keypoints.
The first returned value is LightGlue’s matching score, not a distance: higher is better, and only matches scoring above filter_threshold (0.1 by default) are returned. The index rows are (index in desc1, index in desc2). Fewer than two descriptors on either side give no match.
All-zero padding LAFs are not filtered out: such a slot reaches the network as a keypoint at (0, 0). Drop them with laf_is_filled() first.

Parameters:

feature_name (str, optional) – type of feature for matching, one of known_modes. Default: "disk"
params (Optional[Dict], optional) – keyword arguments of LightGlue, such as filter_threshold. Default: None

forward(desc1, desc2, lafs1, lafs2, hw1=None, hw2=None)[source]#

Run forward.

Parameters:

desc1 (Tensor) – Batch of descriptors of a shape \((B1, D)\).
desc2 (Tensor) – Batch of descriptors of a shape \((B2, D)\).
lafs1 (Tensor) – LAFs of a shape \((1, B1, 2, 3)\).
lafs2 (Tensor) – LAFs of a shape \((1, B2, 2, 3)\).
hw1 (Optional[Tuple[int, int]], optional) – (H, W) of the first image; see the Convention block. Default: None
hw2 (Optional[Tuple[int, int]], optional) – (H, W) of the second image; see the Convention block. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

LightGlue’s matching score of each match, higher is better, shape of \((B3, 1)\).
Long torch.Tensor indexes of matching descriptors in desc1 and desc2,
shape of \((B3, 2)\) where \(0 <= B3 <= B1\).

class kornia.feature.LightGlue(features='superpoint', **conf_)[source]#

Implement the LightGlue matcher for sparse local features.

LightGlue is a deep network that matches local features across image pairs using a series of transformer layers and an adaptive pruning mechanism.

Convention:

The keypoints of each image are (x, y) pixels and image_size is (W, H); an image tensor gives the size from its last two dimensions instead. forward normalises the keypoints to (k - size / 2) / (max(size) / 2), as the reference implementation does, and raises for a keypoint outside \([-1, 1]\) after that. The keypoints can be wider than the model, like the float32 keypoints of kornia’s detectors with a float16 or bfloat16 model; the positional encoding runs in the model’s dtype.
The "sift", "doghardnet" and "dog_affnet_hardnet" configurations, which set add_scale_ori, also read scales and oris (radians) of each image; the last two multiply scales by their scale_coef of 1/6 first, so they take the LAF scale. "keynet_affnet_hardnet", which sets add_laf, reads lafs. LightGlueMatcher builds them from LAFs.
The matching scores, matching_scores0, matching_scores1 and scores, are in the dtype of the descriptors with mp=False. With mp=True, autocast sets it, for every configuration and also when point pruning removes every point. On CPU, autocast runs in float16 for a float16 model and in bfloat16, CPU autocast’s default, for any other: a float16 model gives float16 scores for float16 and float32 descriptors, a float32 or bfloat16 model gives bfloat16 scores for float32 and bfloat16 descriptors, and a float64 model gives float64 scores for float64 descriptors. Float16 descriptors with any other model, and bfloat16 descriptors with a float16 model, are not supported there (#5929). On MPS, float32, float16 and bfloat16 descriptors give float32 scores; on CUDA, float32 descriptors give float32 scores. An image without descriptors runs no layer, and its zero scores keep the descriptor dtype, also with mp=True.
Each pair of a batch gives the matches and scores it gives on its own when point pruning can run, that is with width_confidence > 0, no padding to the static lengths of compile(), and an image with more keypoints than pruning_min_kpts() of the device. Pruning keeps a different number of points in each pair, so then the pairs run one at a time: matches0, matches1, matching_scores0, matching_scores1, prune0 and prune1 stack the rows of the pairs, matches and scores list them, stop is the largest number of layers that a pair ran, and there is no log_assignment. Otherwise a batch runs as one, and early stopping (depth_confidence) ends it at a layer only when every pair meets its own criterion. A pair can therefore run more layers in a batch than on its own, and its matches can differ.
Known defects: on the static-length path of compile(), early stopping also counts the unconfident padded positions, against the number of real keypoints, so a padded pair can stop later than without padding (#5930).

Parameters:

features (str, optional) – The type of local features to match (e.g., ‘superpoint’, ‘disk’), whose configuration and pretrained weights are loaded. None builds the model from default_conf and the keyword arguments and downloads no weights. Default: "superpoint"
**conf – keyword arguments that override entries of default_conf, such as filter_threshold; the configuration of features is applied after them and wins where both set a key.

forward(data)[source]#

Match keypoints and descriptors between two images.

Return type:: dict

Input (dict):

image0: dict: keypoints: [B x M x 2] descriptors: [B x M x D] image: [B x C x H x W] or image_size: [B x 2] scales, oris: [B x M], with add_scale_ori lafs: [B x M x 2 x 3], with add_laf
image1: dict: keypoints: [B x N x 2] descriptors: [B x N x D] image: [B x C x H x W] or image_size: [B x 2] scales, oris: [B x N], with add_scale_ori lafs: [B x N x 2 x 3], with add_laf

Output (dict):

log_assignment: [B x M+1 x N+1], not for a batch that prunes points (see the Convention block) matches0: [B x M] matching_scores0: [B x M] matches1: [B x N] matching_scores1: [B x N] matches: List[[Si x 2]], scores: List[[Si]] stop, prune0, prune1: diagnostics of early stopping and point pruning

class kornia.feature.OnnxLightGlue(weights=None, device='cpu')[source]#

Wrapper for loading LightGlue-ONNX models and running inference via ONNXRuntime.

LightGlue [LSP23] performs fast descriptor-based deep keypoint matching. This module requires onnxruntime to be installed.

If you have trained your own LightGlue model, see fabio-sim/LightGlue-ONNX for how to export the model to ONNX and optimize it.

Convention:

See the Convention block on LightGlue: the keypoints are normalised the same way, with image_size in the order (W, H), and the same range check applies. Unlike LightGlue, it takes float32 inputs with a batch of one and no scales, orientations or LAFs, and it runs on CPU or CUDA only.

Parameters:

weights (str | None, optional) – Pretrained weights, or a path to your own exported ONNX model. Available pretrained weights are 'disk', 'superpoint', 'disk_fp16', and 'superpoint_fp16'. Note that FP16 requires CUDA. Defaults to 'disk_fp16' if device is CUDA, and 'disk' if CPU. Default: None
device (Union[str, device, None], optional) – 'cpu' or 'cuda' to run inference on; another device raises ValueError. None means 'cpu', as in the other kornia feature constructors, not torch’s current default device. Default: "cpu"

forward(data)[source]#

Match keypoints and descriptors between two images.

The output contains the matches (the indices of the matching keypoint pairs between the first and second image) and the corresponding confidence scores. Only a batch size of 1 is supported.

Parameters:: data (dict[str, dict[str, Tensor]]) – Dictionary containing both images and the keypoints and descriptors thereof.
Return type:: dict[str, Tensor]
Returns:: Dictionary containing the matches and scores.

data (dict):

image0 (dict):

keypoints (float32): \((1, M, 2)\)

descriptors (float32): \((1, M, D)\)

image: \((1, C, H, W)\) or image_size: \((1, 2)\)

image1 (dict):

keypoints (float32): \((1, N, 2)\)

descriptors (float32): \((1, N, D)\)

image: \((1, C, H, W)\) or image_size: \((1, 2)\)

output (dict):

matches (int64): \((S, 2)\)

scores (float32): \((S)\)

class kornia.feature.LoFTR(pretrained='outdoor', config=default_cfg)[source]#

nn.Module, which finds correspondences between two images.

Try in browser

This is based on the original code from paper “LoFTR: Detector-Free Local Feature Matching with Transformers”. See [SSW+21] for more details.

Convention:

image0 and image1 are grayscale \((N, 1, H, W)\) images in \([0, 1]\), the input of the pretrained models. With the default config, whose resolution (8, 2) every pretrained model uses, the two images can differ in size, and their heights and widths do not need to be multiples of 8 (the reference implementation expects multiples of 8).
With that resolution, keypoints0 lie on the grid of the 1/8-resolution coarse feature map, at 8 * (column, row) of a coarse cell, and keypoints1 at 8 * (column, row) of the matching cell of image1 plus the offset found by the fine stage, at most 4 pixels along each axis. Both are \((x, y)\) in the frame of the reference implementation, in which the pretrained models put the centre of pixel \((x, y)\) at \((x + 0.5, y + 0.5)\): subtract 0.5 to get kornia’s frame (Local features), in which pixel centres have integer coordinates. The keypoints are float32 for a float16 or bfloat16 model and float64 for a float64 one; confidence keeps the model dtype.
confidence is the coarse matching score of the match, a dual-softmax score with the default match_type; only matches above max(thr, 0) are returned, with the thr of config["match_coarse"], 0.2 by default.

Parameters:

config (dict[str, Any], optional) – Dict with initialization parameters. Do not pass it, unless you know what you are doing. The model works on its own deep copy, so the dict you pass is never modified. Default: default_cfg
pretrained (Optional[str], optional) – Download and set pretrained weights to the model. Options: ‘outdoor’, ‘indoor’, ‘indoor_new’. ‘outdoor’ is trained on the MegaDepth dataset and ‘indoor’ and ‘indoor_new’ on the ScanNet. ‘indoor_new’ also switches the positional encoding of the copy to temp_bug_fix=True, which is what its checkpoint was trained with. None loads no weights. Default: "outdoor"

Returns:

Dictionary with image correspondences and confidence scores.

The backbone features of an image do not depend on the image it is matched with. To match one image against many, compute them once with extract_features() and pass them as features0 / features1.

Example

>>> img1 = torch.rand(1, 1, 320, 200)
>>> img2 = torch.rand(1, 1, 128, 128)
>>> input = {"image0": img1, "image1": img2}
>>> loftr = LoFTR('outdoor')
>>> out = loftr(input)

extract_features(image)[source]#

Compute the features of images that do not depend on the image they are matched with.

These are the backbone’s coarse and fine feature maps, the coarse one with its positional encoding. Pass the returned dict to forward() as features0 or features1 instead of image0 or image1 to match an image against many others without running the backbone again for every pair.

Parameters:

image (Tensor) – grayscale images with shape \((N, 1, H, W)\).

Return type:

dict[str, Tensor]

Returns:

coarse, position-encoded 1/8-resolution features \((N, C_c, \lceil H/8 \rceil, \lceil W/8 \rceil)\).
fine, 1/2-resolution features \((N, C_f, \lceil H/2 \rceil, \lceil W/2 \rceil)\). With the default config this is 128 channels, about 96 MB in float32 for a 1024 x 736 image.

Example

>>> loftr = LoFTR(None)
>>> images = [torch.rand(1, 1, 64, 80), torch.rand(1, 1, 72, 64), torch.rand(1, 1, 64, 64)]
>>> features = [loftr.extract_features(image) for image in images]
>>> out = loftr({"features0": features[0], "features1": features[1]})
>>> out = loftr({"features0": features[0], "image1": images[2]})

forward(data)[source]#

Run forward.

Parameters:

data (dict[str, Any]) – dictionary containing the input data in the following format:

Keyword Arguments:

image0 – left image with shape \((N, 1, H1, W1)\).
image1 – right image with shape \((N, 1, H2, W2)\).
features0 (optional) – instead of image0, its extract_features() output.
features1 (optional) – instead of image1, its extract_features() output.
mask0 (optional) – left image mask. ‘0’ indicates a padded position \((N, H1, W1)\).
mask1 (optional) – right image mask. ‘0’ indicates a padded position \((N, H2, W2)\).

Return type:

dict[str, Tensor]

Returns:

keypoints0, matching keypoints from image0 \((NC, 2)\).
keypoints1, matching keypoints from image1 \((NC, 2)\).
confidence, the coarse matching score, see the Convention block \((NC)\).
batch_indexes, batch index of each match \((NC)\).