Descriptors#

Choosing a SIFT descriptor interface#

These interfaces serve different inputs and outputs; none detects keypoints. For detection and description together, use the SIFT pipeline guide.

SIFT descriptor contracts#
Interface	Input	Output and purpose
`SIFTDescriptor`	Canonically sampled grayscale patches `(N, 1, patch_size, patch_size)`.	One descriptor per patch, `(N, 128)` with default bins. Use when patch extraction, scale normalization and orientation have already been handled.
`DenseSIFTDescriptor`	A grayscale image `(B, 1, H, W)`.	A dense descriptor map `(B, 128, H_out, W_out)` with default bins. Spatial size depends on bin size, stride and padding.
`SIFTDescriptorFromPyramid`	A grayscale image and supplied, already oriented LAFs `(B, N, 2, 3)`.	Sparse descriptors `(B, N, 128)` with default bins, sharing image-gradient histograms between frames. `forward(image, lafs)` preserves their orientation.

All three support RootSIFT normalization. They return kornia’s angle-major layout; convert_sift_descriptor_layout() reorders bins to OpenCV’s layout. Layout conversion does not make differently sampled descriptors numerically equivalent. The converter expects the descriptor dimension last. For a dense map, move its channels last before conversion and restore them afterwards:

opencv_dense = convert_sift_descriptor_layout(
    dense.movedim(1, -1), "kornia", "opencv"
).movedim(-1, 1)

SIFTDescriptorFromPyramid builds its own pyramid rather than reusing a detector’s Gaussian layers. It constructs histogram maps only for octaves used by valid supplied frames, which can reduce work and memory for frames concentrated in a few octaves. See the recorded octave-selection measurements. General affine frames use an approximate angular remapping and spatial pooling; this is not numerically identical to patch extraction.

Its orient_and_describe(image, lafs) method additionally estimates orientation, integrating smoothed gradients before angular binning; this is the orientation used by SIFTFeature(descriptor_backend="pyramid"). Its performance and accuracy acceptance criteria are tracked in #5748; they do not concern the descriptor-only forward use on supplied oriented LAFs. The validation and possible deprecation plan retains all three standalone descriptor interfaces.

API reference#

class kornia.feature.DenseSIFTDescriptor(num_ang_bins=8, num_spatial_bins=4, spatial_bin_size=4, rootsift=True, clipval=0.2, stride=1, padding=1)[source]#

nn.Module, which computes SIFT descriptor densely over the image.

Convention:

Every output pixel is a SIFT descriptor of the image around the input position given below. Its num_spatial_bins x num_spatial_bins cells are spatial_bin_size pixels apart, each cell pooling a triangular window of spatial_bin_size pixels (see get_sift_pooling_kernel), so the cells tile an extent of num_spatial_bins * spatial_bin_size pixels. The cell spacing is that of SIFTDescriptor and of the VLFeat vl_dsift grid.
Output pixel (i, j) describes the neighbourhood centred on the input position x = j * stride - padding + (num_spatial_bins - 1) * spatial_bin_size / 2 - e / 2 (y likewise from i), in pixel-centre coordinates where the first input pixel is at 0, with e 1 for an even spatial_bin_size and 0 for an odd one. The centre lies on a pixel when (num_spatial_bins - 1) * spatial_bin_size and e have the same parity, and half way between two pixels otherwise.
The channels are in kornia’s angle-major SIFT layout, channel a * num_spatial_bins**2 + cy * num_spatial_bins + cx for angle bin a and cell (cy, cx), the entry order of SIFTDescriptor; move them last before convert_sift_descriptor_layout(). Clipping, RootSIFT (the default) and the output for a flat neighbourhood are those of SIFTDescriptor (see its Convention block), but the cells are not weighted by a Gaussian window.

Parameters:

num_ang_bins (int, optional) – Number of angular bins. (8 is default) Default: 8
num_spatial_bins (int, optional) – Number of spatial bins per descriptor (4 is default). With an odd num_spatial_bins and an odd spatial_bin_size, padding = (num_spatial_bins - 1) * spatial_bin_size / 2 keeps the feature map size at stride=1. Default: 4
spatial_bin_size (int, optional) – Size of a spatial bin in pixels, and the distance between neighbouring bins (4 is default) Default: 4
clipval (float, optional) – clipping value to reduce single-bin dominance Default: 0.2
rootsift (bool, optional) – (bool) if True, RootSIFT (Arandjelović et. al, 2012) is computed Default: True
stride (int, optional) – step between neighbouring outputs in pixels (1 is default) Default: 1
padding (int, optional) – zero padding of the pooled bin maps in pixels, on every side (1 is default) Default: 1

Returns:

DenseSIFT descriptor of the image

Return type:

torch.Tensor

Shape:

Input: (B, 1, H, W)
Output: (B, num_ang_bins * num_spatial_bins ** 2, H_out, W_out), with H_out = (H + e + 2 * padding - (num_spatial_bins - 1) * spatial_bin_size - 1) // stride + 1 and W_out likewise, where e is 1 for an even spatial_bin_size and 0 for an odd one.

Examples::

>>> input =  torch.rand(2, 1, 200, 300)
>>> SIFT = DenseSIFTDescriptor()
>>> descs = SIFT(input) # 2x128x191x291

get_gradient_histograms(input)[source]#

Return unnormalised angular histogram maps before spatial pooling.

The result has shape (B, num_ang_bins, H, W). It is useful when a caller shares an image pyramid between orientation assignment and sparse descriptor extraction.

Return type:: Tensor

class kornia.feature.SIFTDescriptorFromPyramid(num_ang_bins=8, num_spatial_bins=4, spatial_bin_size=8, rootsift=True, clipval=0.2, orientation_bins=36)[source]#

Extract sparse SIFT descriptors from a shared image-gradient pyramid.

This module does not detect keypoints or return a dense descriptor map. forward(image, lafs) describes the supplied, already oriented LAFs. orient_and_describe() additionally assigns their orientation while sharing one pyramid between orientation assignment and description. The pyramid belongs to this descriptor; it is not reused from the detector.

The performance and accuracy acceptance criteria of its automatic orientation are tracked in #5748. See Choosing a SIFT descriptor interface for the distinct patch, dense-map and supplied-frame descriptor contracts.

The module combines the shared histogram maps of VLFeat DSIFT with orientation-relative spatial and angular binning as in OpenCV’s SIFT implementation: descriptor gradients are binned once at each pyramid octave. Orientation uses a separate gradient field, smoothed with a one-pixel Gaussian at each octave, and integrates 19x19 gradient samples with a Gaussian of sigma one third of the LAF scale. Gradients are transformed into the frame before orientation histogramming, avoiding interpolation of already quantized angular bins. Unlike patch-wise SIFT, no image patch is resampled before either stage.

For a general affine LAF the global angular bins are remapped through the LAF’s A.T gradient pullback, including its direction-dependent magnitude. This makes affine frames useful without extracting patches, but is an approximation: a histogram bin represents every gradient in its angular interval by its centre direction, and spatial pooling is performed in the image pyramid before the affine footprint is sampled.

Convention:

lafs are in the pixel frame of image, with the layout of laf_from_center_scale_ori() (see Local features). Cell (i, j) of the num_spatial_bins x num_spatial_bins grid is centred at c + A (u_j, u_i) with u_k = (2 k + 1) / num_spatial_bins - 1, at whichever octave the LAF selects: the cell centres of a num_spatial_bins-pixel patch from extract_patches_from_pyramid(), row 0 on the frame’s \(-y\) side.
orient_and_describe(), unless upright=True, rotates each valid frame so that its dominant gradient, measured in the frame’s own coordinates, points along the frame’s \(+x\) axis, whatever orientation the frame had. This is the orientation convention of LAFOrienter, and it puts that gradient direction in the descriptor’s angle bin 0.
A frame that laf_is_valid() rejects, such as the all-zero padding LAF, gets an all-zero descriptor, with or without RootSIFT, and orient_and_describe() returns the frame unchanged. SIFTDescriptor instead describes the flat patch that a zero LAF extracts by 1e-5 in every entry with RootSIFT.
Clipping, RootSIFT (the default) and kornia’s angle-major layout are those of SIFTDescriptor, whose Convention block gives them; convert_sift_descriptor_layout() reorders the descriptors to OpenCV’s.

Parameters:

num_ang_bins (int, optional) – Number of descriptor angular bins. Default: 8
num_spatial_bins (int, optional) – Number of descriptor spatial bins per axis. Default: 4
spatial_bin_size (int, optional) – Side of the DenseSIFT spatial pooling window, in pixels. The default 8 approximately matches the 19-pixel octave support to the patch SIFT descriptor’s 17-pixel pooling bins at its default 41-pixel patch. Default: 8
rootsift (bool, optional) – If True, apply RootSIFT after clipping and normalization. Default: True
clipval (float, optional) – Descriptor clipping threshold. Default: 0.2
orientation_bins (int, optional) – Number of bins for dominant-orientation assignment. Independent of the descriptor’s global histogram resolution (at least 36 bins), which is remapped to num_ang_bins in each local frame. Default: 36

Shape:

image: \((B, 1, H, W)\)
lafs: \((B, N, 2, 3)\) in image pixel coordinates
descriptors: \((B, N, num_ang_bins * num_spatial_bins^2)\)

forward(image, lafs)[source]#

Describe supplied oriented frames without detecting or reorienting them.

Parameters:

image (Tensor) – Grayscale floating-point image of shape (B, 1, H, W).
lafs (Tensor) – Already oriented pixel-coordinate frames of shape (B, N, 2, 3).

Return type:

Tensor

Returns:

Sparse descriptors of shape (B, N, num_ang_bins * num_spatial_bins**2), with the image dtype and device. Invalid frames receive zero descriptors.

orient_and_describe(image, lafs, upright=False)[source]#

Orient and describe sparse frames, building the extraction pyramid once.

Parameters:

image (Tensor) – Grayscale floating-point image of shape (B, 1, H, W).
lafs (Tensor) – Pixel-coordinate frames of shape (B, N, 2, 3).
upright (bool, optional) – Preserve the supplied frame orientations rather than estimating them. Default: False

Return type:

Tuple[Tensor, Tensor]

Returns:

Oriented frames in the input LAF dtype and descriptors in the image dtype, both on the image device. Invalid frames are preserved and receive zero descriptors. Empty frame sets are supported.

class kornia.feature.SIFTDescriptor(patch_size=41, num_ang_bins=8, num_spatial_bins=4, rootsift=True, clipval=0.2)[source]#

nn.Module which computes SIFT descriptors of given patches.

Convention:

The input is a stack of grayscale patches, patch_size pixels square (41 by default; another size raises), with rows along \(+y\) and columns along \(+x\), as extract_patches_from_pyramid() samples them from LAFs (see Local features).
The output is in kornia’s angle-major SIFT layout, which convert_sift_descriptor_layout() defines and reorders to OpenCV’s. In the patch of a LAF oriented by LAFOrienter, the dominant gradient direction it found points along \(+x\), the centre of angle bin 0.
The num_spatial_bins x num_spatial_bins grid of spatial cells and the Gaussian weighting window are both centred on the patch centre \((\text{patch\_size} - 1) / 2\); get_sift_bin_ksize_stride_pad gives the cell spacing and pooling kernel. A patch mirrored upside down therefore has a descriptor that is a permutation of the original’s; so does a patch mirrored left to right when num_ang_bins is even, and a torch.rot90-rotated one when num_ang_bins is a multiple of four, as the default 8 is.
The descriptor is L2-normalised, clipped at clipval and L2-normalised again, so an output entry can exceed clipval. With rootsift=True, the default, it is then L1-normalised and square-rooted (RootSIFT). Either way a row has unit L2 norm, except for a flat patch: it has no gradient and gives an all-zero descriptor, or 1e-5 in every entry with RootSIFT. OpenCV’s cv2.SIFT returns the rootsift=False descriptor scaled to an L2 norm of about 512 and rounded to integers.
An intensity change \(a x + b\) with \(a > 0\) gives the same descriptor unless it leaves the patch nearly flat, so a patch in [0, 1] and the same patch in [0, 255] give one descriptor.

Parameters:

patch_size (int, optional) – Input patch size in pixels. Default: 41
num_ang_bins (int, optional) – Number of angular bins. Default: 8
num_spatial_bins (int, optional) – Number of spatial bins. Default: 4
clipval (float, optional) – clipping value to reduce single-bin dominance Default: 0.2
rootsift (bool, optional) – if True, RootSIFT (Arandjelović et. al, 2012) is computed. Default: True

Returns:

SIFT descriptor of the patches.

Shape:

Input: \((B, 1, \text{patch_size}, \text{patch_size})\)
Output: \((B, \text{num_ang_bins * num_spatial_bins ** 2})\)

Example

>>> input = torch.rand(23, 1, 32, 32)
>>> SIFT = SIFTDescriptor(32, 8, 4)
>>> descs = SIFT(input) # 23x128

kornia.feature.convert_sift_descriptor_layout(descriptors, source, target, num_ang_bins=8, num_spatial_bins=4)[source]#

Reorder SIFT descriptors between kornia’s layout and OpenCV’s.

Both layouts hold the same histogram values, one per angle bin \(a\) and spatial cell \((y, x)\) of the frame, with \(A\) = num_ang_bins and \(S\) = num_spatial_bins:

"kornia", the layout of every kornia SIFT descriptor (SIFTDescriptor, SIFTDescriptorFromPyramid, the channels of DenseSIFTDescriptor, and SIFTFeature and SIFTFeatureScaleSpace with either descriptor_backend): angle-major, value \((a, y, x)\) at index \(a S^2 + y S + x\). Bin \(a\) is centred on the direction \(2 \pi a / A\), measured from \(+x\) towards \(+y\), with \(y\) growing downwards as image rows do.
"opencv", the layout of OpenCV’s cv2.SIFT: spatial-major, index \((y S + x) A + a'\), with angles measured from \(+x\) towards \(-y\). A gradient in kornia’s bin \(a\) falls in OpenCV’s bin \(a' = -a \bmod A\).

LightGlue’s "sift" weights expect the "opencv" layout: convert kornia descriptors before passing them to LightGlueMatcher with "sift". The conversion only reorders values, so it applies equally to SIFT and RootSIFT descriptors and keeps their scale; converting back restores the input exactly.

Parameters:

descriptors (Tensor) – SIFT descriptors of shape \((*, A S^2)\).
source (str) – layout of descriptors, "kornia" or "opencv".
target (str) – layout to return, "kornia" or "opencv".
num_ang_bins (int, optional) – number of angle bins \(A\). Default: 8
num_spatial_bins (int, optional) – number of spatial bins \(S\) per side. Default: 4

Return type:

Tensor

Returns:

The descriptors in the target layout, with the input’s shape, dtype and device.

Example

>>> descs = SIFTDescriptor(32)(torch.rand(4, 1, 32, 32))
>>> opencv_descs = convert_sift_descriptor_layout(descs, "kornia", "opencv")
>>> torch.equal(convert_sift_descriptor_layout(opencv_descs, "opencv", "kornia"), descs)
True

class kornia.feature.MKDDescriptor(patch_size=32, kernel_type='concat', whitening='pcawt', training_set='liberty', output_dims=128)[source]#

nn.Module that computes Multiple Kernel local descriptors.

This is based on the paper “Understanding and Improving Kernel Local Descriptors”. See [MTB+19] for more details.

Convention:

Patches are (B, 1, patch_size, patch_size): patch_size must be the input’s size, and another size raises ShapeError. The kernels are centred on the patch centre \((\text{patch\_size} - 1) / 2\).
The descriptor has unit L2 norm, and an intensity change \(a x + b\) with \(a > 0\) gives the same descriptor unless it leaves the patch nearly flat. The exception is a flat float16 patch with whitening=None: the 1e-8 added to the gradient magnitudes underflows in float16, and the descriptor is all zero.
With whitening the descriptor is \(\min(\text{output\_dims}, E)\) wide, where \(E\) is the width of the kernel embedding: 238 for 'concat', 175 for 'polar' and 63 for 'cart'. With whitening=None it is that embedding, \(E\) wide whatever output_dims says. Constructing a whitened descriptor loads its whitening model, and downloads it then if it is not cached yet; whitening=None needs no download.

Parameters:

patch_size (int, optional) – Input patch size in pixels. Default: 32
kernel_type (str, optional) – Parametrization of kernel 'concat', 'cart', 'polar'. Default: "concat"
whitening (str, optional) – Whitening transform to apply None, 'lw', 'pca', 'pcawt', 'pcaws'. Default: "pcawt"
training_set (str, optional) – Set that model was trained on 'liberty', 'notredame', 'yosemite'. Default: "liberty"
output_dims (int, optional) – Width of the whitened descriptor, at most the embedding width; ignored with whitening=None. Default: 128

Returns:

Descriptors of the patches, of unit norm except as stated in the Convention block.

Shape:

Input: \((B, 1, \text{patch\_size}, \text{patch\_size})\).
Output: \((B, D)\), with \(D\) as in the Convention block.

Examples

>>> patches = torch.rand(23, 1, 32, 32)
>>> mkd = MKDDescriptor(patch_size=32,
...                     kernel_type='concat',
...                     whitening='pcawt',
...                     training_set='liberty',
...                     output_dims=128)
>>> desc = mkd(patches) # 23x128

class kornia.feature.HardNet(pretrained=False)[source]#

Module, which computes HardNet descriptors of given grayscale patches of 32x32.

This is based on the original code from paper “Working hard to know your neighbor’s margins: Local descriptor learning loss”. See [MMRM17] for more details.

Convention:

Patches are (B, 1, 32, 32), one grayscale channel, and another shape raises; patch_size is a class attribute. get_laf_descriptors() extracts such patches from LAFs at its default patch_size, sampled as extract_patches_from_pyramid() describes (see Local features). HardNet8, SOSNet, TFeat and HyNet take the same patches.
The intensity range does not matter: HardNet and HardNet8 standardise each patch to zero mean and unit standard deviation, and SOSNet and TFeat start with an InstanceNorm2d, so \(a x + b\) with \(a > 0\) gives the same descriptor unless it leaves the patch nearly flat. That InstanceNorm adds 1e-5 to the variance, so for SOSNet and TFeat a low-contrast patch in [0, 1] can already give a different descriptor than the same patch in [0, 255]. HyNet is invariant to a scale only; see its Convention block.
The descriptors have unit L2 norm, except TFeat’s (see its Convention block); with pretrained=False, HardNet and HardNet8 map a flat patch to an all-zero descriptor.
pretrained=False, the default, leaves the weights randomly initialised.

Parameters:: pretrained (bool, optional) – Download and set pretrained weights to the model. Default: False
Returns:: HardNet descriptor of the patches.
Return type:: torch.Tensor

Shape:

Input: \((B, 1, 32, 32)\)
Output: \((B, 128)\)

Examples

>>> input = torch.rand(16, 1, 32, 32)
>>> hardnet = HardNet()
>>> descs = hardnet(input) # 16x128

class kornia.feature.HardNet8(pretrained=False)[source]#

Module, which computes HardNet8 descriptors of given grayscale patches of 32x32.

This is based on the original code from paper “Improving the HardNet Descriptor”. See [Pul20] for more details.

Convention:: See the Convention block on HardNet.

Parameters:: pretrained (bool, optional) – Download and set pretrained weights to the model. The 512-d features are projected to 128-d by a PCA stored with the weights; without them, the projection keeps the first 128 features. Default: False
Returns:: HardNet8 descriptor of the patches.
Return type:: torch.Tensor

Shape:

Input: \((B, 1, 32, 32)\)
Output: \((B, 128)\)

Examples

>>> input = torch.rand(16, 1, 32, 32)
>>> hardnet = HardNet8()
>>> descs = hardnet(input) # 16x128

class kornia.feature.HyNet(pretrained=False, is_bias=True, is_bias_FRN=True, dim_desc=128, drop_rate=0.3, eps_l2_norm=1e-10)[source]#

nn.Module, which computes HyNet descriptors of given grayscale patches of 32x32.

This is based on the original code from paper “HyNet: Learning Local Descriptor with Hybrid Similarity Measure and Triplet Loss”. See [TBLN+20] for more details.

Convention:

It takes the (B, 1, 32, 32) patches of HardNet and returns unit-norm rows, but its first layer is a FilterResponseNorm2d on the raw patch, which divides by the root mean square and subtracts no mean, as in the reference implementation. A scale \(a x\) with \(a > 0\) therefore gives the same descriptor as long as the mean squares of the patch and of its scaled copy are both large compared with that layer’s eps, while an offset \(x + b\) changes it: feed non-negative intensities such as [0, 1] or [0, 255], not mean-subtracted or standardised patches.

Parameters:

pretrained (bool, optional) – Download and set pretrained weights to the model. Default: False
is_bias (bool, optional) – use bias in the convolution layers (the final 8 x 8 convolution has none either way) Default: True
is_bias_FRN (bool, optional) – use bias in FRN layers Default: True
dim_desc (int, optional) – descriptor dimensionality, Default: 128
drop_rate (float, optional) – dropout rate, Default: 0.3
eps_l2_norm (float, optional) – to avoid div by zero Default: 1e-10

Returns:

HyNet descriptor of the patches.

Shape:

Input: \((B, 1, 32, 32)\)
Output: \((B, \text{dim\_desc})\)

Note

With float16 weights, the final BatchNorm keeps its running statistics in float32. Convert the model to float16 before calling torch.jit.script.

Examples

>>> input = torch.rand(16, 1, 32, 32)
>>> hynet = HyNet()
>>> descs = hynet(input) # 16x128

class kornia.feature.TFeat(pretrained=False)[source]#

Module, which computes TFeat descriptors of given grayscale patches of 32x32.

This is based on the original code from paper “Learning local feature descriptors with triplets and shallow convolutional neural networks”. See [BRPM16] for more details

Convention:

It takes the patches of HardNet, with the intensity rule of HardNet’s Convention block, but returns the final tanh layer of the reference implementation without L2 normalisation: each entry lies in [-1, 1] and the rows are not unit vectors. Normalise them where a distance threshold assumes unit descriptors.

Parameters:: pretrained (bool, optional) – Download and set pretrained weights to the model. Default: False
Returns:: TFeat descriptor of the patches.
Return type:: torch.Tensor

Shape:

Input: \((B, 1, 32, 32)\)
Output: \((B, 128)\)

Examples

>>> input = torch.rand(16, 1, 32, 32)
>>> tfeat = TFeat()
>>> descs = tfeat(input) # 16x128

class kornia.feature.SOSNet(pretrained=False)[source]#

128-dimensional SOSNet model definition for 32x32 patches.

This is based on the original code from paper “SOSNet:Second Order Similarity Regularization for Local Descriptor Learning”.

Convention:

See the Convention block on HardNet.

Parameters:: pretrained (bool, optional) – Download and set pretrained weights to the model. Default: False

Shape:

Input: \((B, 1, 32, 32)\)
Output: \((B, 128)\)

Note

With float16 weights, the final BatchNorm keeps its running statistics in float32. Convert the model to float16 before calling torch.jit.script.

Examples

>>> input = torch.rand(8, 1, 32, 32)
>>> sosnet = SOSNet()
>>> descs = sosnet(input) # 8x128

class kornia.feature.LAFDescriptor(patch_descriptor_module=None, patch_size=32, grayscale_descriptor=True)[source]#

nn.Module to get local descriptors, corresponding to LAFs (keypoints).

Convention:: See the Convention block on get_laf_descriptors(), which forward calls with this module’s patch_size and grayscale_descriptor. The default descriptor is a pretrained HardNet on 32 x 32 grayscale patches; SIFTDescriptor defaults to 41-pixel patches, so pass patch_size=41 with it.

Parameters:

patch_descriptor_module (Optional[Module], optional) – patch descriptor module, e.g. SIFTDescriptor or HardNet. Default: HardNet with pretrained weights.
patch_size (int, optional) – patch size in pixels, which descriptor expects. Default: 32
grayscale_descriptor (bool, optional) – True if patch_descriptor expects single-channel image; a 3-channel image is then converted to grayscale. Default: True

forward(img, lafs)[source]#

Describe the given LAFs with get_laf_descriptors().

Parameters:

img (Tensor) – image features with shape \((B,C,H,W)\).
lafs (Tensor) – local affine frames \((B,N,2,3)\).

Return type:

Tensor

Returns:

Local descriptors of shape \((B,N,D)\) where \(D\) is descriptor size.

class kornia.feature.SOLD2(pretrained=True, config=None)[source]#

nn.Module, which detects and describe line segments in an image.

This is based on the original code from the paper “SOLD²: Self-supervised Occlusion-aware Line Detector and Descriptor”. See [PautratLinL+21] for more details.

Convention:

forward() takes the input and returns the heatmaps and the (row, col) line segments of SOLD2_detector, whose Convention block gives their frame and order, and also dense_desc, \((B, 128, H / 4, W / 4)\), which is not normalized.
match takes segments as forward() returns them and the descriptor map of one image each. It returns an int64 tensor \((N_1,)\): for each segment of line_seg1, the index of its match in line_seg2, or -1. It is int64 also when either set of segments is empty: an empty tensor when line_seg1 is, -1 for every segment of line_seg1 when line_seg2 is.
Known defects: on CPU, match passes float16 and bfloat16 descriptor maps to torch’s grid_sample without an upcast: the half-precision kernel can return wrong values, NaN or overflow, and where torch has none the call raises (#5810).

Parameters:

config (Optional[DetectorCfg], optional) – DetectorCfg, or a dict of its fields, specifying parameters. None will load the default parameters, which are tuned for images in the range 400~800 px. The model works on a copy with use_descriptor set to True; the object passed in is not modified. Default: None
pretrained (bool, optional) – If True, download and set pretrained weights to the model. Default: True

Returns:

The raw junction and line heatmaps, the semi-dense descriptor map, as well as the list of detected line segments (ij coordinates convention).

Example

>>> images = torch.rand(2, 1, 64, 64)
>>> sold2 = SOLD2()
>>> outputs = sold2(images)
>>> line_seg1 = outputs["line_segments"][0]
>>> line_seg2 = outputs["line_segments"][1]
>>> desc1 = outputs["dense_desc"][0]
>>> desc2 = outputs["dense_desc"][1]
>>> matches = sold2.match(line_seg1, line_seg2, desc1[None], desc2[None])

forward(img)[source]#

Run forward.

Parameters:: img (Tensor) – batched images with shape \((B, 1, H, W)\).
Returns:: list of N line segments in each of the B images \(List[(N, 2, 2)]\). junction_heatmap: raw junction heatmap of shape \((B, H, W)\). line_heatmap: raw line heatmap of shape \((B, H, W)\). dense_desc: the semi-dense descriptor map of shape \((B, 128, H/4, W/4)\).
Return type:: line_segments

kornia.feature.get_laf_descriptors(img, lafs, patch_descriptor, patch_size=32, grayscale_descriptor=True)[source]#

Get local descriptors, corresponding to LAFs (keypoints).

Convention:

The descriptor sees the patches of extract_patches_from_pyramid() at patch_size; see its Convention block for how they are sampled. patch_size is this argument, 32 by default, and is not read from the descriptor, so it has to be a size the descriptor accepts.
With grayscale_descriptor=True, a 3-channel image is converted to grayscale first.
The descriptor runs in eval mode, with its floating-point parameters and buffers cast to the device and dtype of img for the call (a TorchScript module is called on a cast copy). SOSNet and HyNet retain their final BatchNorm statistics in float32 for float16 images. The module keeps its device, dtype and parameters, every submodule is switched to eval for the call and gets its train/eval mode back afterwards, and gradients reach the module’s parameters.
An empty LAF set gives a \((B, 0, D)\) result, D as wide as a non-empty result, and a warning. That result does not require grad, so no gradient reaches img or the descriptor’s parameters through it.

Parameters:

img (Tensor) – image features with shape \((B,C,H,W)\).
lafs (Tensor) – local affine frames \((B,N,2,3)\).
patch_descriptor (Module) – patch descriptor module, e.g. SIFTDescriptor or HardNet.
patch_size (int, optional) – patch size in pixels, which descriptor expects. Default: 32
grayscale_descriptor (bool, optional) – True if patch_descriptor expects single-channel image; a 3-channel img is then converted to grayscale. Default: True

Return type:

Tensor

Returns:

Local descriptors of shape \((B,N,D)\) where \(D\) is descriptor size.