Descriptors#

Patch descriptors and LAFs#

SIFTDescriptor, MKDDescriptor and the learned HardNet, HardNet8, HyNet, SOSNet and TFeat compute one descriptor per grayscale patch (B, 1, PS, PS). The learned networks require PS = 32 and load pretrained weights by default. SIFT and MKD accept a configurable patch_size, with defaults of 41 and 32 respectively. All except TFeat return L2-normalised descriptors. See each descriptor’s conventions for its response to changes in image brightness and contrast.

To describe LAFs in an image, use LAFDescriptor to reuse a descriptor module, or get_laf_descriptors() to supply one on each call. Both extract a patch for each LAF and compute its descriptor. Set patch_size to the size expected by the descriptor; the default is 32. LAFDescriptor uses pretrained HardNet by default. Patches follow the orientation of their LAFs, so orient the LAFs first, for example with LAFOrienter.

For SIFT descriptors of LAFs without extracting patches, use SIFTDescriptorFromPyramid (see below). The local-feature pipelines combine a detector with a descriptor. SOLD2 detects, describes and matches line segments in whole images.

Choosing a SIFT descriptor interface#

These interfaces serve different inputs and outputs; none detects keypoints. For detection and description together, use the SIFT pipeline guide.

SIFT descriptor contracts#

Interface

Input

Output and purpose

SIFTDescriptor

Canonically sampled grayscale patches (N, 1, patch_size, patch_size).

One descriptor per patch, (N, 128) with default bins. Use when patch extraction, scale normalization and orientation have already been handled.

DenseSIFTDescriptor

A grayscale image (B, 1, H, W).

A dense descriptor map (B, 128, H_out, W_out) with default bins. Spatial size depends on bin size, stride and padding.

SIFTDescriptorFromPyramid

A grayscale image and supplied, already oriented LAFs (B, N, 2, 3).

Sparse descriptors (B, N, 128) with default bins, sharing image-gradient histograms between frames. forward(image, lafs) preserves their orientation.

All three support RootSIFT normalization. They return kornia’s angle-major layout; convert_sift_descriptor_layout() reorders bins to OpenCV’s layout. Layout conversion does not make differently sampled descriptors numerically equivalent. The converter expects the descriptor dimension last. For a dense map, move its channels last before conversion and restore them afterwards:

opencv_dense = convert_sift_descriptor_layout(
    dense.movedim(1, -1), "kornia", "opencv"
).movedim(-1, 1)

SIFTDescriptorFromPyramid builds its own pyramid rather than reusing a detector’s Gaussian layers. It constructs histogram maps only for octaves used by valid supplied frames, which can reduce work and memory for frames concentrated in a few octaves. See the recorded octave-selection measurements. General affine frames use an approximate angular remapping and spatial pooling; this is not numerically identical to patch extraction.

Its orient_and_describe(image, lafs) method additionally estimates orientation, integrating smoothed gradients before angular binning; this is the orientation used by SIFTFeature(descriptor_backend="pyramid"). Its performance and accuracy acceptance criteria are tracked in #5748; they do not concern the descriptor-only forward use on supplied oriented LAFs. The validation and possible deprecation plan retains all three standalone descriptor interfaces.

API reference#

class kornia.feature.DenseSIFTDescriptor(num_ang_bins=8, num_spatial_bins=4, spatial_bin_size=4, rootsift=True, clipval=0.2, stride=1, padding=1)[source]#

Compute a SIFT descriptor at every position of a grayscale image, as a dense map.

dsift = DenseSIFTDescriptor(), then descs = dsift(image) takes grayscale images (B, 1, H, W) and returns a descriptor map (B, num_ang_bins * num_spatial_bins ** 2, H_out, W_out), (B, 128, H_out, W_out) with the default bins: descs[b, :, i, j] is the SIFT descriptor of an upright square of num_spatial_bins * spatial_bin_size pixels (16 by default) placed as the Convention block gives, so every output describes one fixed scale and orientation, and stride sets the spacing of the output grid. To describe patches you have already sampled, use SIFTDescriptor instead, and to describe LAFs at their own scale and orientation, SIFTDescriptorFromPyramid.

Convention:

  • Every output pixel is a SIFT descriptor of the image around the input position given below. Its num_spatial_bins x num_spatial_bins cells are spatial_bin_size pixels apart, each cell pooling a triangular window of spatial_bin_size pixels (see get_sift_pooling_kernel), so the cells tile an extent of num_spatial_bins * spatial_bin_size pixels. The cell spacing is that of SIFTDescriptor and of the VLFeat vl_dsift grid.

  • Output pixel (i, j) describes the neighbourhood centred on the input position x = j * stride - padding + (num_spatial_bins - 1) * spatial_bin_size / 2 - e / 2 (y likewise from i), in pixel-centre coordinates where the first input pixel is at 0, with e 1 for an even spatial_bin_size and 0 for an odd one. The centre lies on a pixel when (num_spatial_bins - 1) * spatial_bin_size and e have the same parity, and half way between two pixels otherwise.

  • The channels are in kornia’s angle-major SIFT layout, channel a * num_spatial_bins**2 + cy * num_spatial_bins + cx for angle bin a and cell (cy, cx), the entry order of SIFTDescriptor; move them last before convert_sift_descriptor_layout(). Clipping, RootSIFT (the default) and the output for a flat neighbourhood are those of SIFTDescriptor (see its Convention block), but the cells are not weighted by a Gaussian window.

Parameters:

  • num_ang_bins (int, optional) – Number of angular bins. (8 is default) Default: 8

  • num_spatial_bins (int, optional) – Number of spatial bins per descriptor (4 is default). With an odd num_spatial_bins and an odd spatial_bin_size, padding = (num_spatial_bins - 1) * spatial_bin_size / 2 keeps the feature map size at stride=1. Default: 4

  • spatial_bin_size (int, optional) – Size of a spatial bin in pixels, and the distance between neighbouring bins (4 is default) Default: 4

  • clipval (float, optional) – clipping value to reduce single-bin dominance Default: 0.2

  • rootsift (bool, optional) – (bool) if True, RootSIFT (Arandjelović et. al, 2012) is computed Default: True

  • stride (int, optional) – step between neighbouring outputs in pixels (1 is default) Default: 1

  • padding (int, optional) – zero padding of the pooled bin maps in pixels, on every side (1 is default) Default: 1

Returns:

DenseSIFT descriptor of the image

Return type:

torch.Tensor

Shape:

  • Input: (B, 1, H, W)

  • Output: (B, num_ang_bins * num_spatial_bins ** 2, H_out, W_out), with H_out = (H + e + 2 * padding - (num_spatial_bins - 1) * spatial_bin_size - 1) // stride + 1 and W_out likewise, where e is 1 for an even spatial_bin_size and 0 for an odd one. An image for which either would be below 1 raises RuntimeError.

See also

Example

>>> input =  torch.rand(2, 1, 200, 300)
>>> SIFT = DenseSIFTDescriptor()
>>> descs = SIFT(input) # 2x128x191x291
get_gradient_histograms(input)[source]#

Return unnormalised angular histogram maps before spatial pooling.

The result has shape (B, num_ang_bins, H, W). It is useful when a caller shares an image pyramid between orientation assignment and sparse descriptor extraction.

Return type:

Tensor

class kornia.feature.SIFTDescriptorFromPyramid(num_ang_bins=8, num_spatial_bins=4, spatial_bin_size=8, rootsift=True, clipval=0.2, orientation_bins=36)[source]#

Describe given LAFs with SIFT from a shared gradient pyramid, without sampling patches.

sift = SIFTDescriptorFromPyramid(), then descs = sift(image, lafs) takes grayscale images (B, 1, H, W) and their already oriented LAFs (B, N, 2, 3) in pixels and returns one descriptor per LAF, (B, N, num_ang_bins * num_spatial_bins ** 2): (B, N, 128) by default, in the image dtype, RootSIFT by default, in kornia’s angle-major layout. lafs, descs = sift.orient_and_describe(image, lafs) also gives each LAF the orientation of its dominant gradient first, sharing one pyramid between the two steps. Each LAF reads histograms pooled at one octave of a pyramid that this module builds from image, not reused from a detector: it describes the square c + A [-1, 1]^2 that SIFTDescriptor sees in the patch of extract_patches_from_pyramid(), without sampling that patch, so its values approximate those of SIFTDescriptor without equalling them. For the patch descriptors themselves, use LAFDescriptor(SIFTDescriptor(41), patch_size=41) (LAFDescriptor) instead. SIFTFeature’s "patch" backend uses that descriptor, and its "pyramid" backend uses this module.

The performance and accuracy acceptance criteria of its automatic orientation are tracked in #5748. See Choosing a SIFT descriptor interface for the distinct patch, dense-map and supplied-frame descriptor contracts.

The module combines the shared histogram maps of VLFeat DSIFT with orientation-relative spatial and angular binning as in OpenCV’s SIFT implementation: descriptor gradients are binned once at each pyramid octave. Orientation uses a separate gradient field, smoothed with a one-pixel Gaussian at each octave, and integrates 19x19 gradient samples with a Gaussian of sigma one third of the LAF scale. Gradients are transformed into the frame before orientation histogramming, avoiding interpolation of already quantized angular bins. Unlike patch-wise SIFT, no image patch is resampled before either stage.

For a general affine LAF the global angular bins are remapped through the LAF’s A.T gradient pullback, including its direction-dependent magnitude. This makes affine frames useful without extracting patches, but is an approximation: a histogram bin represents every gradient in its angular interval by its centre direction, and spatial pooling is performed in the image pyramid before the affine footprint is sampled.

Convention:

  • lafs use the pixel convention of image, with the layout of laf_from_center_scale_ori() (see Local features). Cell (i, j) of the num_spatial_bins x num_spatial_bins grid is centred at c + A (u_j, u_i) with u_k = (2 k + 1) / num_spatial_bins - 1, at whichever octave the LAF selects: the cell centres of a num_spatial_bins-pixel patch from extract_patches_from_pyramid(), row 0 on the frame’s \(-y\) side.

  • orient_and_describe(), unless upright=True, rotates each valid frame so that its dominant gradient, measured in the frame’s own coordinates, points along the frame’s \(+x\) axis, whatever orientation the frame had. This is the orientation convention of LAFOrienter, and it puts that gradient direction in the descriptor’s angle bin 0.

  • A frame that laf_is_valid() rejects, such as the all-zero padding LAF, gets an all-zero descriptor, with or without RootSIFT, and orient_and_describe() returns the frame unchanged. SIFTDescriptor instead describes the flat patch that a zero LAF extracts by 1e-5 in every entry with RootSIFT.

  • Clipping, RootSIFT (the default) and kornia’s angle-major layout are those of SIFTDescriptor, whose Convention block gives them; convert_sift_descriptor_layout() reorders the descriptors to OpenCV’s.

Parameters:

  • num_ang_bins (int, optional) – Number of descriptor angular bins. Default: 8

  • num_spatial_bins (int, optional) – Number of descriptor spatial bins per axis. Default: 4

  • spatial_bin_size (int, optional) – Side of the DenseSIFT spatial pooling window, in pixels. The default 8 approximately matches the 19-pixel octave support to the patch SIFT descriptor’s 17-pixel pooling bins at its default 41-pixel patch. Default: 8

  • rootsift (bool, optional) – If True, apply RootSIFT after clipping and normalization. Default: True

  • clipval (float, optional) – Descriptor clipping threshold. Default: 0.2

  • orientation_bins (int, optional) – Number of bins for dominant-orientation assignment. Independent of the descriptor’s global histogram resolution (at least 36 bins), which is remapped to num_ang_bins in each local frame. Default: 36

Shape:

  • image: (B, 1, H, W)

  • lafs: (B, N, 2, 3) in image pixel coordinates

  • descriptors: (B, N, num_ang_bins * num_spatial_bins ** 2)

See also

Example

>>> image = torch.rand(1, 1, 64, 64)
>>> xy, scale = torch.tensor([[[32.0, 32.0]]]), torch.full((1, 1, 1, 1), 8.0)
>>> lafs = kornia.feature.laf_from_center_scale_ori(xy, scale)
>>> SIFTDescriptorFromPyramid()(image, lafs).shape
torch.Size([1, 1, 128])
forward(image, lafs)[source]#

Describe supplied oriented frames without detecting or reorienting them.

Parameters:

  • image (Tensor) – Grayscale floating-point image of shape (B, 1, H, W).

  • lafs (Tensor) – Already oriented pixel-coordinate frames of shape (B, N, 2, 3).

Return type:

Tensor

Returns:

Sparse descriptors of shape (B, N, num_ang_bins * num_spatial_bins**2), with the image dtype and device. Invalid frames receive zero descriptors.

orient_and_describe(image, lafs, upright=False)[source]#

Orient and describe sparse frames, building the extraction pyramid once.

Parameters:

  • image (Tensor) – Grayscale floating-point image of shape (B, 1, H, W).

  • lafs (Tensor) – Pixel-coordinate frames of shape (B, N, 2, 3).

  • upright (bool, optional) – Preserve the supplied frame orientations rather than estimating them. Default: False

Return type:

Tuple[Tensor, Tensor]

Returns:

Oriented frames in the input LAF dtype and descriptors in the image dtype, both on the image device. Invalid frames are preserved and receive zero descriptors. Empty frame sets are supported.

class kornia.feature.SIFTDescriptor(patch_size=41, num_ang_bins=8, num_spatial_bins=4, rootsift=True, clipval=0.2)[source]#

Describe square grayscale patches with SIFT histograms of gradient orientation.

sift = SIFTDescriptor(patch_size), then descs = sift(patches) takes grayscale patches (B, 1, patch_size, patch_size), 41 pixels square by default, and returns one descriptor per patch, (B, num_ang_bins * num_spatial_bins ** 2): (B, 128) with the default 8 angle bins and 4 x 4 spatial cells, RootSIFT by default, in kornia’s angle-major layout. The patches must already be sampled, scaled and oriented; to describe LAFs on an image, wrap it in LAFDescriptor with the same patch_size. For a descriptor at every position of an image, use DenseSIFTDescriptor instead, and to describe or orient LAFs without extracting patches, SIFTDescriptorFromPyramid.

Convention:

  • The input is a stack of grayscale patches, patch_size pixels square (41 by default; another size raises), with rows along \(+y\) and columns along \(+x\), as extract_patches_from_pyramid() samples them from LAFs (see Local features).

  • The output is in kornia’s angle-major SIFT layout, which convert_sift_descriptor_layout() defines and reorders to OpenCV’s. In the patch of a LAF oriented by LAFOrienter, the dominant gradient direction it found points along \(+x\), the centre of angle bin 0.

  • The num_spatial_bins x num_spatial_bins grid of spatial cells and the Gaussian weighting window are both centred on the patch centre \((\text{patch\_size} - 1) / 2\); get_sift_bin_ksize_stride_pad gives the cell spacing and pooling kernel. A patch mirrored upside down therefore has a descriptor that is a permutation of the original’s; so does a patch mirrored left to right when num_ang_bins is even, and a torch.rot90-rotated one when num_ang_bins is a multiple of four, as the default 8 is.

  • The descriptor is L2-normalised, clipped at clipval and L2-normalised again, so an output entry can exceed clipval. With rootsift=True, the default, it is then L1-normalised and square-rooted (RootSIFT). Either way a row has unit L2 norm, except for a flat patch: it has no gradient and gives an all-zero descriptor, or 1e-5 in every entry with RootSIFT. OpenCV’s cv2.SIFT returns the rootsift=False descriptor scaled to an L2 norm of about 512 and rounded to integers.

  • An intensity change \(a x + b\) with \(a > 0\) gives the same descriptor unless it leaves the patch nearly flat, so a patch in [0, 1] and the same patch in [0, 255] give one descriptor.

Parameters:

  • patch_size (int, optional) – Input patch size in pixels. Some sizes cannot hold a centred grid of num_spatial_bins cells, such as 7 with 4 bins, and raise ValueError when the module is built. Default: 41

  • num_ang_bins (int, optional) – Number of angular bins. Default: 8

  • num_spatial_bins (int, optional) – Number of spatial bins per side. Default: 4

  • clipval (float, optional) – clipping value to reduce single-bin dominance Default: 0.2

  • rootsift (bool, optional) – if True, RootSIFT (Arandjelović et. al, 2012) is computed. Default: True

Returns:

SIFT descriptor of the patches.

Shape:

  • Input: (B, 1, patch_size, patch_size)

  • Output: (B, num_ang_bins * num_spatial_bins ** 2)

See also

  • DenseSIFTDescriptor: a descriptor at every position of an image, as a map, with no Gaussian weighting of the cells.

  • SIFTDescriptorFromPyramid: describes, and can orient, LAFs on an image without extracting patches.

  • convert_sift_descriptor_layout(): reorders the descriptors into OpenCV’s layout.

  • LAFDescriptor: runs this module on a patch per LAF of an image.

  • SIFTFeature: detects DoG keypoints and, with the default "patch" backend, describes them with this module.

  • MKDDescriptor: another handcrafted patch descriptor, from kernel embeddings of the gradients.

  • HardNet: a learned descriptor of 32 x 32 patches.

Example

>>> input = torch.rand(23, 1, 32, 32)
>>> SIFT = SIFTDescriptor(32, 8, 4)
>>> descs = SIFT(input) # 23x128
kornia.feature.convert_sift_descriptor_layout(descriptors, source, target, num_ang_bins=8, num_spatial_bins=4)[source]#

Reorder SIFT descriptors between kornia’s layout and OpenCV’s.

opencv_descs = convert_sift_descriptor_layout(descs, "kornia", "opencv") takes SIFT descriptors with the descriptor dimension last, (*, A * S ** 2), and returns the same values reordered into the target layout, with the input’s shape, dtype and device. For the (B, A * S ** 2, H_out, W_out) map of DenseSIFTDescriptor, move the channels last first and back afterwards: convert_sift_descriptor_layout(dense.movedim(1, -1), "kornia", "opencv").movedim(-1, 1). A reordering does not make descriptors sampled by different methods equal.

Both layouts hold the same histogram values, one per angle bin \(a\) and spatial cell \((y, x)\) of the frame, with \(A\) = num_ang_bins and \(S\) = num_spatial_bins:

  • "kornia", the layout of every kornia SIFT descriptor (SIFTDescriptor, SIFTDescriptorFromPyramid, the channels of DenseSIFTDescriptor, and SIFTFeature and SIFTFeatureScaleSpace with either descriptor_backend): angle-major, value \((a, y, x)\) at index \(a S^2 + y S + x\). Bin \(a\) is centred on the direction \(2 \pi a / A\), measured from \(+x\) towards \(+y\), with \(y\) growing downwards as image rows do.

  • "opencv", the layout of OpenCV’s cv2.SIFT: spatial-major, index \((y S + x) A + a'\), with angles measured from \(+x\) towards \(-y\). A gradient in kornia’s bin \(a\) falls in OpenCV’s bin \(a' = -a \bmod A\).

LightGlue’s "sift" weights expect the "opencv" layout: convert kornia descriptors before passing them to LightGlueMatcher with "sift". The conversion only reorders values, so it applies equally to SIFT and RootSIFT descriptors and keeps their scale; converting back restores the input exactly.

Parameters:

  • descriptors (Tensor) – SIFT descriptors of shape (*, A * S ** 2).

  • source (str) – layout of descriptors, "kornia" or "opencv".

  • target (str) – layout to return, "kornia" or "opencv".

  • num_ang_bins (int, optional) – number of angle bins \(A\). Default: 8

  • num_spatial_bins (int, optional) – number of spatial bins \(S\) per side. Default: 4

Return type:

Tensor

Returns:

The descriptors in the target layout, with the input’s shape, dtype and device.

Raises:

  • ValueError – for a layout other than "kornia" and "opencv", or a bin count below 1.

  • BaseError – if the last dimension of descriptors is not A * S ** 2.

See also

Example

>>> descs = SIFTDescriptor(32)(torch.rand(4, 1, 32, 32))
>>> opencv_descs = convert_sift_descriptor_layout(descs, "kornia", "opencv")
>>> torch.equal(convert_sift_descriptor_layout(opencv_descs, "opencv", "kornia"), descs)
True
class kornia.feature.MKDDescriptor(patch_size=32, kernel_type='concat', whitening='pcawt', training_set='liberty', output_dims=128)[source]#

Describe square grayscale patches with handcrafted Multiple Kernel Descriptors (MKD).

mkd = MKDDescriptor(), then descs = mkd(patches) takes grayscale patches (B, 1, patch_size, patch_size), 32 pixels square by default, and returns unit-norm descriptors (B, D), (B, 128) with the defaults: the gradient magnitudes and orientations of each patch embedded in polar and Cartesian spatial kernels (kernel_type="concat"), then whitened by whitening="pcawt" and reduced to output_dims. It has no CNN; only the whitening is learned, on training_set, and the module is in eval mode when built. To describe LAFs on an image, wrap it in LAFDescriptor with the same patch_size; for SIFT histograms, use SIFTDescriptor, and for a learned descriptor, HardNet. This is based on the paper “Understanding and Improving Kernel Local Descriptors”. See [MTB+19] for more details.

Convention:

  • Patches are (B, 1, patch_size, patch_size): patch_size must be the input’s size, and another size raises ShapeError. The kernels are centred on the patch centre \((\text{patch\_size} - 1) / 2\).

  • The descriptor has unit L2 norm, and an intensity change \(a x + b\) with \(a > 0\) gives the same descriptor unless it leaves the patch nearly flat. The exception is a flat float16 patch with whitening=None: the 1e-8 added to the gradient magnitudes underflows in float16, and the descriptor is all zero.

  • With whitening the descriptor is \(\min(\text{output\_dims}, E)\) wide, where \(E\) is the width of the kernel embedding: 238 for 'concat', 175 for 'polar' and 63 for 'cart'. With whitening=None it is that embedding, \(E\) wide whatever output_dims says. Constructing a whitened descriptor loads its whitening model, and downloads it then if it is not cached yet; whitening=None needs no download.

Parameters:

  • patch_size (int, optional) – Input patch size in pixels. Default: 32

  • kernel_type (str, optional) – Parametrization of kernel 'concat', 'cart', 'polar'. Default: "concat"

  • whitening (str, optional) – Whitening transform to apply None, 'lw', 'pca', 'pcawt', 'pcaws'. Default: "pcawt"

  • training_set (str, optional) – Set that model was trained on 'liberty', 'notredame', 'yosemite'. Default: "liberty"

  • output_dims (int, optional) – Width of the whitened descriptor, at most the embedding width; ignored with whitening=None. Default: 128

Returns:

Descriptors of the patches, of unit norm except as stated in the Convention block.

Shape:

  • Input: (B, 1, patch_size, patch_size).

  • Output: (B, D), with \(D\) as in the Convention block.

See also

  • SIFTDescriptor: the handcrafted SIFT histograms of gradient orientation of a patch.

  • HardNet: a learned descriptor of 32 x 32 patches.

  • LAFDescriptor: runs this module on a patch per LAF of an image.

Examples

>>> patches = torch.rand(23, 1, 32, 32)
>>> mkd = MKDDescriptor(patch_size=32,
...                     kernel_type='concat',
...                     whitening='pcawt',
...                     training_set='liberty',
...                     output_dims=128)
>>> desc = mkd(patches) # 23x128
class kornia.feature.HardNet(pretrained=True)[source]#

Describe 32 x 32 grayscale patches with the learned 128-D HardNet CNN.

hardnet = HardNet() builds the network with its pretrained weights, in eval mode, and descs = hardnet(patches) takes grayscale patches (B, 1, 32, 32) and returns unit-norm descriptors (B, 128). Each patch is standardised first, so its intensity range does not matter. To describe LAFs on an image, use LAFDescriptor, which holds a pretrained HardNet by default; the HardNet local features such as DoGHardNet and KeyNetHardNet use it that way. HardNet8 is the wider follow-up network, with the same patches and output size. This is based on the original code from paper “Working hard to know your neighbor’s margins: Local descriptor learning loss”. See [MMRM17] for more details.

Convention:

  • Patches are (B, 1, 32, 32), one grayscale channel, and another shape raises; patch_size is a class attribute. get_laf_descriptors() extracts such patches from LAFs at its default patch_size, sampled as extract_patches_from_pyramid() describes (see Local features). HardNet8, SOSNet, TFeat and HyNet take the same patches.

  • The intensity range does not matter: HardNet and HardNet8 standardise each patch to zero mean and unit standard deviation, and SOSNet and TFeat start with an InstanceNorm2d, so \(a x + b\) with \(a > 0\) gives the same descriptor unless it leaves the patch nearly flat. That InstanceNorm adds 1e-5 to the variance, so for SOSNet and TFeat a low-contrast patch in [0, 1] can already give a different descriptor than the same patch in [0, 255]. HyNet is invariant to a scale only; see its Convention block.

  • The descriptors have unit L2 norm, except TFeat’s (see its Convention block); with pretrained=False and .eval() (the initial BatchNorm statistics), HardNet and HardNet8 map a flat patch to an all-zero descriptor.

  • pretrained=True, the default, downloads the trained weights (Liberty with augmentation for HardNet, hardnet8v2 for HardNet8) and returns the model in evaluation mode. pretrained=False gives random weights in training mode; call .eval() before inference.

Parameters:

pretrained (bool, optional) – Download and set pretrained weights to the model. Default: True

Returns:

HardNet descriptor of the patches.

Return type:

torch.Tensor

Shape:

  • Input: (B, 1, 32, 32)

  • Output: (B, 128)

See also

  • HardNet8: the follow-up network, wider, with a PCA from 512 to 128 dimensions.

  • SOSNet: the same convolution stack, trained with second-order similarity regularisation.

  • HyNet: a descriptor of the same patches, with FRN layers that subtract no mean.

  • TFeat: a shallow network for the same patches, without L2 normalisation.

  • SIFTDescriptor: handcrafted SIFT histograms of patches, with no weights.

  • LAFDescriptor: runs this network, by default, on a patch per LAF of an image.

Examples

>>> input = torch.rand(16, 1, 32, 32)
>>> hardnet = HardNet(pretrained=False).eval()
>>> descs = hardnet(input) # 16x128
class kornia.feature.HardNet8(pretrained=True)[source]#

Describe 32 x 32 grayscale patches with HardNet8, a wider HardNet with a PCA to 128-D.

hardnet8 = HardNet8() builds the network with its pretrained hardnet8v2 weights, in eval mode, and descs = hardnet8(patches) takes grayscale patches (B, 1, 32, 32) and returns unit-norm descriptors (B, 128): the 512 L2-normalised features of the network, projected by the PCA stored with the weights and normalised again. It takes the patches of HardNet and returns descriptors of the same size, but they are not interchangeable with HardNet’s; pass it to LAFDescriptor to describe LAFs on an image. This is based on the original code from paper “Improving the HardNet Descriptor”. See [Pul20] for more details.

Convention:

See the Convention block on HardNet.

Parameters:

pretrained (bool, optional) – Download and set pretrained weights to the model. The 512-d features are projected to 128-d by a PCA stored with the weights; without them, the projection keeps the first 128 features. Default: True

Returns:

HardNet8 descriptor of the patches.

Return type:

torch.Tensor

Shape:

  • Input: (B, 1, 32, 32)

  • Output: (B, 128)

See also

  • HardNet: the original, narrower network, the default descriptor of LAFDescriptor.

  • SOSNet: HardNet’s convolution stack, trained with second-order similarity regularisation.

  • HyNet: a descriptor of the same patches, with FRN layers that subtract no mean.

  • LAFDescriptor: runs a patch descriptor such as this one on a patch per LAF of an image.

Examples

>>> input = torch.rand(16, 1, 32, 32)
>>> hardnet = HardNet8(pretrained=False).eval()
>>> descs = hardnet(input) # 16x128
class kornia.feature.HyNet(pretrained=True, is_bias=True, is_bias_FRN=True, dim_desc=128, drop_rate=0.3, eps_l2_norm=1e-10)[source]#

Describe 32 x 32 grayscale patches with the learned HyNet CNN, built on FRN layers.

hynet = HyNet() builds the network with its pretrained weights (trained on Liberty), in eval mode, and descs = hynet(patches) takes grayscale patches (B, 1, 32, 32) and returns unit-norm descriptors (B, dim_desc), (B, 128) by default. Its FilterResponseNorm2d layers, each followed by TLU, subtract no mean, and the first runs on the raw patch, so unlike HardNet its descriptor changes with an intensity offset (see the Convention block). To describe LAFs on an image, pass it to LAFDescriptor. This is based on the original code from paper “HyNet: Learning Local Descriptor with Hybrid Similarity Measure and Triplet Loss”. See [TBLN+20] for more details.

Convention:

  • It takes the (B, 1, 32, 32) patches of HardNet and returns unit-norm rows, but its first layer is a FilterResponseNorm2d on the raw patch, which divides by the root mean square and subtracts no mean, as in the reference implementation. A scale \(a x\) with \(a > 0\) therefore gives the same descriptor as long as the mean squares of the patch and of its scaled copy are both large compared with that layer’s eps, while an offset \(x + b\) changes it: feed non-negative intensities such as [0, 1] or [0, 255], not mean-subtracted or standardised patches.

Parameters:

  • pretrained (bool, optional) – Download and set pretrained weights to the model. The weights fit the default is_bias, is_bias_FRN and dim_desc only, so pass pretrained=False with another value: the strict load raises RuntimeError for another is_bias or dim_desc, and is_bias_FRN=False raises ValueError before downloading, since the weights would load but leave their trained FRN biases unused. Default: True

  • is_bias (bool, optional) – use bias in the convolution layers (the final 8 x 8 convolution has none either way) Default: True

  • is_bias_FRN (bool, optional) – use bias in FRN layers Default: True

  • dim_desc (int, optional) – descriptor dimensionality, Default: 128

  • drop_rate (float, optional) – dropout rate, Default: 0.3

  • eps_l2_norm (float, optional) – to avoid div by zero Default: 1e-10

Returns:

HyNet descriptor of the patches.

Shape:

  • Input: (B, 1, 32, 32)

  • Output: (B, dim_desc)

See also

  • HardNet: a descriptor of the same patches with BatchNorm, which standardises each patch first.

  • SOSNet: a descriptor of the same patches, trained with second-order similarity regularisation.

  • HardNet8: the wider follow-up of HardNet, with a PCA to 128 dimensions.

  • FilterResponseNorm2d: the normalisation layer of this network.

  • TLU: the learned-threshold activation that follows each of its FRN layers.

  • LAFDescriptor: runs a patch descriptor such as this one on a patch per LAF of an image.

Note

With float16 weights, the final BatchNorm keeps its running statistics in float32. Convert the model to float16 before calling torch.jit.script.

Examples

>>> input = torch.rand(16, 1, 32, 32)
>>> hynet = HyNet(pretrained=False).eval()
>>> descs = hynet(input) # 16x128
class kornia.feature.TFeat(pretrained=True)[source]#

Describe 32 x 32 grayscale patches with the shallow TFeat CNN, without L2 normalisation.

tfeat = TFeat() builds the network with its pretrained weights (trained on Liberty), in eval mode, and descs = tfeat(patches) takes grayscale patches (B, 1, 32, 32) and returns (B, 128) descriptors with entries in [-1, 1], not unit vectors (see the Convention block). It has two convolutions and one linear layer. For unit-norm descriptors of the same patches, use HardNet instead; to describe LAFs on an image, pass it to LAFDescriptor. This is based on the original code from paper “Learning local feature descriptors with triplets and shallow convolutional neural networks”. See [BRPM16] for more details.

Convention:

  • It takes the patches of HardNet, with the intensity rule of HardNet’s Convention block, but returns the final tanh layer of the reference implementation without L2 normalisation: each entry lies in [-1, 1] and the rows are not unit vectors. Normalise them where a distance threshold assumes unit descriptors.

Parameters:

pretrained (bool, optional) – Download and set pretrained weights to the model. Default: True

Returns:

TFeat descriptor of the patches.

Return type:

torch.Tensor

Shape:

  • Input: (B, 1, 32, 32)

  • Output: (B, 128)

See also

  • HardNet: a deeper network for the same patches, with unit-norm descriptors.

  • SOSNet: a deeper network for the same patches that also starts with an InstanceNorm2d.

  • LAFDescriptor: runs a patch descriptor such as this one on a patch per LAF of an image.

Examples

>>> input = torch.rand(16, 1, 32, 32)
>>> tfeat = TFeat(pretrained=False).eval()
>>> descs = tfeat(input) # 16x128
class kornia.feature.SOSNet(pretrained=True)[source]#

Describe 32 x 32 grayscale patches with SOSNet, trained with second-order similarity.

sosnet = SOSNet() builds the network with its pretrained weights (trained on Liberty), in eval mode, and descs = sosnet(patches) takes grayscale patches (B, 1, 32, 32) and returns unit-norm descriptors (B, 128). It has the convolution stack of HardNet, preceded by an InstanceNorm2d, and was trained with a second-order similarity regularisation. To describe LAFs on an image, pass it to LAFDescriptor. This is based on the original code from paper “SOSNet: Second Order Similarity Regularization for Local Descriptor Learning”.

Convention:

  • See the Convention block on HardNet.

Parameters:

pretrained (bool, optional) – Download and set pretrained weights to the model. Default: True

Shape:

  • Input: (B, 1, 32, 32)

  • Output: (B, 128)

See also

  • HardNet: the same convolution stack, trained without the second-order regularisation.

  • HyNet: a descriptor of the same patches with FRN layers, which subtract no mean.

  • HardNet8: the wider follow-up of HardNet, with a PCA to 128 dimensions.

  • TFeat: a shallow network for the same patches that also starts with an InstanceNorm2d, without L2 normalisation.

  • LAFDescriptor: runs a patch descriptor such as this one on a patch per LAF of an image.

Note

With float16 weights, the final BatchNorm keeps its running statistics in float32. Convert the model to float16 before calling torch.jit.script.

Examples

>>> input = torch.rand(8, 1, 32, 32)
>>> sosnet = SOSNet(pretrained=False).eval()
>>> descs = sosnet(input) # 8x128
class kornia.feature.LAFDescriptor(patch_descriptor_module=None, patch_size=32, grayscale_descriptor=True)[source]#

Describe LAFs with a held patch descriptor, as a module (HardNet by default).

describer = LAFDescriptor(), then descs = describer(img, lafs) takes images (B, C, H, W) and their LAFs (B, N, 2, 3) in pixels and returns one descriptor per LAF, (B, N, D): forward calls get_laf_descriptors() with the held descriptor, patch_size and grayscale_descriptor, which samples a patch per LAF with extract_patches_from_pyramid(). By default it holds a pretrained HardNet on 32 x 32 grayscale patches, converts an RGB image to grayscale and returns (B, N, 128); with another descriptor, pass the patch size it takes, e.g. LAFDescriptor(SIFTDescriptor(41), patch_size=41). It is the descriptor of the HardNet local features, such as DoGHardNet, and of the "patch" backend of SIFTFeature and SIFTFeatureScaleSpace. For SIFT descriptors of LAFs without sampling patches, use SIFTDescriptorFromPyramid instead.

Convention:

See the Convention block on get_laf_descriptors(), which forward calls with this module’s patch_size and grayscale_descriptor. The default descriptor is a pretrained HardNet on 32 x 32 grayscale patches; SIFTDescriptor defaults to 41-pixel patches, so pass patch_size=41 with it.

Parameters:

  • patch_descriptor_module (Optional[Module], optional) – patch descriptor module, e.g. SIFTDescriptor or HardNet. Default: HardNet with pretrained weights.

  • patch_size (int, optional) – patch size in pixels, which descriptor expects. Default: 32

  • grayscale_descriptor (bool, optional) – True if patch_descriptor expects single-channel image; a 3-channel image is then converted to grayscale. Default: True

See also

forward(img, lafs)[source]#

Describe the given LAFs with get_laf_descriptors().

Parameters:

  • img (Tensor) – image features with shape (B,C,H,W).

  • lafs (Tensor) – local affine frames (B,N,2,3).

Return type:

Tensor

Returns:

Local descriptors of shape (B,N,D) where \(D\) is descriptor size.

class kornia.feature.SOLD2(pretrained=True, config=None)[source]#

Detect, describe and match line segments in grayscale images with SOLD2.

sold2 = SOLD2() builds the network with its pretrained weights, in eval mode, and outputs = sold2(images) takes grayscale images (B, 1, H, W) in \([0, 1]\), with H and W multiples of 8, and returns a dict: "line_segments", one (N, 2, 2) tensor of (row, col) endpoints in pixels per image, the "junction_heatmap" and "line_heatmap", (B, H, W) each, and the "dense_desc" map (B, 128, H / 4, W / 4). matches = sold2.match(line_seg1, line_seg2, desc1, desc2) then matches the segments of two images by their descriptors sampled along each segment, with desc1 and desc2 the (1, 128, H / 4, W / 4) maps of one image each, and returns for each segment of line_seg1 the index of its match in line_seg2, or -1. To detect segments only, use SOLD2_detector instead, which builds no descriptor head. This is based on the original code from the paper “SOLD²: Self-supervised Occlusion-aware Line Detector and Descriptor”. See [PautratLinL+21] for more details.

Convention:

  • forward() takes the input and returns the heatmaps and the (row, col) line segments of SOLD2_detector, whose Convention block gives their frame and order, and also dense_desc, (B, 128, H / 4, W / 4), which is not normalized.

  • match takes segments as forward() returns them and the descriptor map of one image each. It returns an int64 tensor (N_1,): for each segment of line_seg1, the index of its match in line_seg2, or -1. It is int64 also when either set of segments is empty: an empty tensor when line_seg1 is, -1 for every segment of line_seg1 when line_seg2 is.

  • Known defects: on CPU, match passes float16 and bfloat16 descriptor maps to torch’s grid_sample without an upcast: the half-precision kernel can return wrong values, NaN or overflow, and where torch has none the call raises (#5810).

Parameters:

  • pretrained (bool, optional) – If True, download and set pretrained weights to the model, and return it in eval mode; otherwise the weights are random and the model is in training mode. Default: True

  • config (Union[DetectorCfg, Dict[str, Any], None], optional) – a kornia.feature.sold2.structures.DetectorCfg, or a dict of its fields, specifying parameters. None will load the default parameters, which are tuned for images in the range 400~800 px. The model works on a copy with use_descriptor set to True; the object passed in is not modified. Default: None

Returns:

The raw junction and line heatmaps, the semi-dense descriptor map, as well as the list of detected line segments (ij coordinates convention).

See also

  • SOLD2_detector: the same line detection, without the descriptor map or matching.

Example

>>> images = torch.rand(2, 1, 64, 64)
>>> sold2 = SOLD2()
>>> outputs = sold2(images)
>>> line_seg1 = outputs["line_segments"][0]
>>> line_seg2 = outputs["line_segments"][1]
>>> desc1 = outputs["dense_desc"][0]
>>> desc2 = outputs["dense_desc"][1]
>>> matches = sold2.match(line_seg1, line_seg2, desc1[None], desc2[None])
forward(img)[source]#

Run forward.

Parameters:

img (Tensor) – batched images with shape (B, 1, H, W).

Returns:

a list of B tensors (N, 2, 2), the N line segments of each image. junction_heatmap: raw junction heatmap of shape (B, H, W). line_heatmap: raw line heatmap of shape (B, H, W). dense_desc: the semi-dense descriptor map of shape (B, 128, H/4, W/4).

Return type:

line_segments

kornia.feature.get_laf_descriptors(img, lafs, patch_descriptor, patch_size=32, grayscale_descriptor=True)[source]#

Describe LAFs on an image by running a patch descriptor on one patch sampled per LAF.

descs = get_laf_descriptors(img, lafs, patch_descriptor, patch_size) takes images (B, C, H, W), their LAFs (B, N, 2, 3) in pixels and a patch descriptor module, and returns one descriptor per LAF, (B, N, D). The descriptor determines the output dtype. The function converts a 3-channel image to grayscale when grayscale_descriptor is set, samples a patch_size x patch_size patch per LAF with extract_patches_from_pyramid(), and runs the descriptor on the stack of the B * N patches. The patch follows the LAF’s scale and orientation, so orient the LAFs first, e.g. with LAFOrienter, unless upright descriptors are wanted. patch_size must be what the descriptor takes: 32 for HardNet and the other learned descriptors, 41 for a default SIFTDescriptor. For the module form, use LAFDescriptor; for SIFT descriptors of LAFs without sampling patches, SIFTDescriptorFromPyramid.

Convention:

  • The descriptor sees the patches of extract_patches_from_pyramid() at patch_size; see its Convention block for how they are sampled. patch_size is this argument, 32 by default, and is not read from the descriptor, so it has to be a size the descriptor accepts.

  • The descriptor runs in eval mode, with its floating-point parameters and buffers cast to the device and dtype of img for the call (a TorchScript module is called on a cast copy). SOSNet and HyNet retain their final BatchNorm statistics in float32 for float16 images. The module keeps its device, dtype and parameters, every submodule is switched to eval for the call and gets its train/eval mode back afterwards, and gradients reach the module’s parameters.

  • An empty LAF set gives a (B, 0, D) result, D as wide as a non-empty result, and a warning. That result does not require grad, so no gradient reaches img or the descriptor’s parameters through it.

Parameters:

  • img (Tensor) – image features with shape (B,C,H,W).

  • lafs (Tensor) – local affine frames (B,N,2,3) in the pixels of img, with its batch size.

  • patch_descriptor (Module) – patch descriptor module, e.g. SIFTDescriptor or HardNet.

  • patch_size (int, optional) – patch size in pixels, which descriptor expects. Default: 32

  • grayscale_descriptor (bool, optional) – True if patch_descriptor expects single-channel image; a 3-channel img is then converted to grayscale. An image with another channel count is passed on unchanged. Default: True

Return type:

Tensor

Returns:

Local descriptors of shape (B,N,D) where \(D\) is descriptor size.

See also