Detectors#

Response functions#

kornia.feature.gftt_response(input, grads_mode='sobel', sigmas=None)[source]#

Compute the Shi-Tomasi cornerness function.

Try in browser

_images/gftt_response.png

The function does no non-maxima suppression. The response map is computed according to the following formulation:

\[R = min(eig(M))\]

where \(M\) is the windowed structure tensor of harris_response(), with the same gradients and window.

Convention:

  • \(R\) is not negative, up to roundoff. It is large at a corner and near zero along an edge, where the Harris response is negative.

  • Multiplying the image by c multiplies the response by c ** 2, and an offset does not change it.

  • sigmas multiplies the response by sigmas ** 4, the same power used by harris_response(). This is the function’s scale-weighting convention, not a guarantee of scale invariance. The integration window \(W\) stays fixed as the image blur changes; the response’s decay with blur depends on the image structure, including the corner geometry.

Parameters:

  • input (Tensor) – input image with shape \((B, C, H, W)\).

  • grads_mode (str, optional) – can be 'sobel' for standalone use or 'diff' for use on Gaussian pyramid. Default: "sobel"

  • sigmas (Optional[Tensor], optional) – per-image scales with shape \((B)\). The response of image b is multiplied by sigmas[b] ** 4, so that non-maxima suppression can compare the levels of a scale pyramid. Default: None

Return type:

Tensor

Returns:

the response map per channel with shape \((B, C, H, W)\).

Example

>>> input = torch.tensor([[[
...    [0., 0., 0., 0., 0., 0., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 0., 0., 0., 0., 0., 0.],
... ]]])  # 1x1x7x7
>>> # compute the response map; rounded for a platform-stable display
>>> gftt_response(input).round(decimals=4) + 0.0
tensor([[[[0.0379, 0.0456, 0.0283, 0.0121, 0.0283, 0.0456, 0.0379],
          [0.0456, 0.0598, 0.0402, 0.0168, 0.0402, 0.0598, 0.0456],
          [0.0283, 0.0402, 0.0545, 0.0245, 0.0545, 0.0402, 0.0283],
          [0.0121, 0.0168, 0.0245, 0.0276, 0.0245, 0.0168, 0.0121],
          [0.0283, 0.0402, 0.0545, 0.0245, 0.0545, 0.0402, 0.0283],
          [0.0456, 0.0598, 0.0402, 0.0168, 0.0402, 0.0598, 0.0456],
          [0.0379, 0.0456, 0.0283, 0.0121, 0.0283, 0.0456, 0.0379]]]])
kornia.feature.harris_response(input, k=0.04, grads_mode='sobel', sigmas=None)[source]#

Compute the Harris cornerness function.

Try in browser

_images/harris_response.png

The function does no non-maxima suppression. The response map is computed according to the following formulation:

\[R = det(M) - k \cdot trace(M)^2\]

where \(w\) are the weights of a window \(W\):

\[\begin{split}M = \sum_{(x,y) \in W} w(x, y) \begin{bmatrix} I^{2}_x & I_x I_y \\ I_x I_y & I^{2}_y \\ \end{bmatrix}\end{split}\]

and \(k\) is an empirically determined constant, usually in \([0.04, 0.06]\).

Convention:

  • \(R\) is signed and not clamped: positive at a corner, negative along an edge.

  • \(I_x\) and \(I_y\) come from spatial_gradient() with mode=grads_mode, in intensity per pixel for both modes, and \(W\) is a \(7 \times 7\) Gaussian window of \(\sigma = 1\) whose weights sum to one. A linear ramp of gradient \(g\) therefore gives \(R = -k \|g\|^4\) away from the border.

  • Apart from sigmas, the response depends on the image’s intensity scale: multiplying the image by c multiplies the response by c ** 4, and an offset does not change it, so a threshold tuned on images in [0, 1] does not carry over to [0, 255].

  • k is not validated.

Parameters:

  • input (Tensor) – input image with shape \((B, C, H, W)\).

  • k (Union[Tensor, float], optional) – the Harris detector free parameter. Default: 0.04

  • grads_mode (str, optional) – can be 'sobel' for standalone use or 'diff' for use on Gaussian pyramid. Default: "sobel"

  • sigmas (Optional[Tensor], optional) – per-image scales with shape \((B)\). The response of image b is multiplied by sigmas[b] ** 4, so that non-maxima suppression can compare the levels of a scale pyramid, as in vlfeat. Default: None

Return type:

Tensor

Returns:

the response map per channel with shape \((B, C, H, W)\).

Example

>>> input = torch.tensor([[[
...    [0., 0., 0., 0., 0., 0., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 0., 0., 0., 0., 0., 0.],
... ]]])  # 1x1x7x7
>>> # compute the response map; rounded for a platform-stable display
>>> harris_response(input, 0.04).round(decimals=4) + 0.0
tensor([[[[0.0042, 0.0054, 0.0035, 0.0006, 0.0035, 0.0054, 0.0042],
          [0.0054, 0.0068, 0.0046, 0.0014, 0.0046, 0.0068, 0.0054],
          [0.0035, 0.0046, 0.0034, 0.0014, 0.0034, 0.0046, 0.0035],
          [0.0006, 0.0014, 0.0014, 0.0006, 0.0014, 0.0014, 0.0006],
          [0.0035, 0.0046, 0.0034, 0.0014, 0.0034, 0.0046, 0.0035],
          [0.0054, 0.0068, 0.0046, 0.0014, 0.0046, 0.0068, 0.0054],
          [0.0042, 0.0054, 0.0035, 0.0006, 0.0035, 0.0054, 0.0042]]]])
kornia.feature.hessian_response(input, grads_mode='sobel', sigmas=None)[source]#

Compute the determinant of the Hessian matrix.

Try in browser

_images/hessian_response.png

The function does no non-maxima suppression. The response map is computed according to the following formulation:

\[R = det(H)\]

where:

\[\begin{split}H = \begin{bmatrix} I_{xx} & I_{xy} \\ I_{xy} & I_{yy} \\ \end{bmatrix}\end{split}\]
Convention:

  • \(R = I_{xx} I_{yy} - I_{xy}^2\) is signed and pointwise, without a window: positive at bright and dark blobs alike, negative at a saddle. The second derivatives come from spatial_gradient() with mode=grads_mode and order=2.

  • Multiplying the image by c multiplies the response by c ** 2, and an offset does not change it.

Parameters:

  • input (Tensor) – input image with shape \((B, C, H, W)\).

  • grads_mode (str, optional) – can be 'sobel' for standalone use or 'diff' for use on Gaussian pyramid. Default: "sobel"

  • sigmas (Optional[Tensor], optional) – per-image scales with shape \((B)\). The response of image b is multiplied by sigmas[b] ** 4, so that non-maxima suppression can compare the levels of a scale pyramid. Default: None

Return type:

Tensor

Returns:

the response map per channel with shape \((B, C, H, W)\).

Shape:

  • Input: \((B, C, H, W)\)

  • Output: \((B, C, H, W)\)

Examples

>>> input = torch.tensor([[[
...    [0., 0., 0., 0., 0., 0., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 0., 0., 0., 0., 0., 0.],
... ]]])  # 1x1x7x7
>>> # compute the response map; rounded for a platform-stable display
>>> hessian_response(input).round(decimals=4) + 0.0
tensor([[[[-0.0137, -0.0332, -0.0205,  0.0000, -0.0205, -0.0332, -0.0137],
          [-0.0332,  0.0098,  0.0381,  0.0000,  0.0381,  0.0098, -0.0332],
          [-0.0205,  0.0381,  0.0547,  0.0000,  0.0547,  0.0381, -0.0205],
          [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
          [-0.0205,  0.0381,  0.0547,  0.0000,  0.0547,  0.0381, -0.0205],
          [-0.0332,  0.0098,  0.0381,  0.0000,  0.0381,  0.0098, -0.0332],
          [-0.0137, -0.0332, -0.0205,  0.0000, -0.0205, -0.0332, -0.0137]]]])
kornia.feature.dog_response(input)[source]#

Compute the Difference-of-Gaussian response.

Convention:

  • Output level i is input[:, :, i + 1] - input[:, :, i] along the level axis D (dim 2), so D levels give D - 1. For a stack blurred from fine to coarse, this is Lowe’s \(L(k\sigma) - L(\sigma)\): negative at a bright blob, positive at a dark one.

Parameters:

input (Tensor) – a given the gaussian 5d torch.Tensor \((B, C, D, H, W)\).

Return type:

Tensor

Returns:

the response map per channel with shape \((B, C, D-1, H, W)\).

kornia.feature.dog_response_single(input, sigma1=1.0, sigma2=1.6)[source]#

Compute the Difference-of-Gaussian response.

Try in browser

_images/dog_response_single.png
Convention:

  • Returns the Gaussian blur of sigma2 minus that of sigma1, the order of dog_response(); see its Convention block. Each kernel has the size int(8 * sigma + 1), rounded up to an odd number. The order of sigma1 and sigma2 is not checked: swapping them flips the sign.

Parameters:

  • input (Tensor) – a given the gaussian 4d torch.Tensor \((B, C, H, W)\).

  • sigma1 (float, optional) – lower gaussian sigma Default: 1.0

  • sigma2 (float, optional) – bigger gaussian sigma Default: 1.6

Return type:

Tensor

Returns:

the response map per channel with shape \((B, C, H, W)\).

class kornia.feature.BlobHessian(grads_mode='sobel')[source]#

nn.Module that calculates Hessian blobs.

_images/hessian_response.png
Convention:

Returns hessian_response() with the constructor’s grads_mode; see its Convention block.

class kornia.feature.CornerGFTT(grads_mode='sobel')[source]#

nn.Module that calculates Shi-Tomasi corners.

_images/gftt_response.png
Convention:

Returns gftt_response() with the constructor’s grads_mode; see its Convention block.

class kornia.feature.CornerHarris(k, grads_mode='sobel')[source]#

nn.Module that calculates Harris corners.

_images/harris_response.png
Convention:

Returns harris_response() with the constructor’s k and grads_mode; see its Convention block. Unlike the function’s, this k has no default.

class kornia.feature.BlobDoG[source]#

nn.Module that calculates Difference-of-Gaussians blobs.

Convention:

Returns dog_response() of the \((B, C, D, H, W)\) scale stack; see its Convention block. sigmas is ignored.

class kornia.feature.BlobDoGSingle(sigma1=1.0, sigma2=1.6)[source]#

nn.Module that calculates Difference-of-Gaussians blobs.

_images/dog_response_single.png
Convention:

Returns dog_response_single() with the constructor’s sigma1 and sigma2; see its Convention block. sigmas is ignored.

Detectors#

class kornia.feature.KeyNet(pretrained=True, keynet_conf=None)[source]#

Key.Net model definition – local feature detector (response function).

This is based on the original code from paper “Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters”. See [BLRPM19] for more details.

_images/KeyNet.png
Convention:

  • The input is a grayscale image, and the pretrained weights expect it in [0, 1], as the Key.Net reference scales its images. The response is not proportional to the intensity, so an image in [0, 255] gives other detections.

  • The response is a non-negative map of the input’s size.

Parameters:

  • pretrained (bool, optional) – Download and set pretrained weights to the model. Default: True

  • keynet_conf (Optional[KeyNet_conf], optional) – Dict with initialization parameters. Do not pass it, unless you know what you are doing. A num_filters, num_levels or kernel_size other than the default changes the architecture: pass pretrained=False with it, since the strict load of the weights raises RuntimeError. Default: None

Returns:

KeyNet response score.

Shape:

  • Input: \((B, 1, H, W)\)

  • Output: \((B, 1, H, W)\)

class kornia.feature.MultiResolutionDetector(model, num_features=2048, config=None, ori_module=None, aff_module=None, compile_model=False, score_threshold=0.0)[source]#

Multi-scale feature detector, based on KeyNet, for a response function that returns one map of its input’s size.

This is based on the original code from paper “Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters”. See [BLRPM19] for more details.

The Key.Net reference code zeroes a 15 px strip along each side of a level before the non-maxima suppression, which reports a “maximum” beside the strip wherever the response rises into it. This detector suppresses on the whole response and removes the strip from the suppression output instead; see detect_features_on_single_level().

Convention:

  • Detections are maxima of the response above score_threshold. For a signed response whose minima are features as well, such as a difference of Gaussians, pass its magnitude, as SIFTFeature does.

  • A detection is an integer pixel \((u, v)\) of a pyramid level of size \((H_l, W_l)\), without a sub-pixel step. It is reported at \((u W / W_l, v H / H_l)\) of the \((H, W)\) input, in the frame of laf_from_center_scale_ori() (see Local features), and not at the level pixel’s centre \(((u + 0.5) W / W_l - 0.5, (v + 0.5) H / H_l - 0.5)\); the Key.Net reference maps it the same way.

  • Before aff_module and ori_module run, the frame is upright and isotropic, with the scale s_mult * (W / W_l + H / H_l) / 2 whatever the response; s_mult of config is 22 by default.

Parameters:

  • model (Module) – response function, such as KeyNet or BlobHessian

  • num_features (int, optional) – Number of features to detect. Every pyramid level is searched for this many candidates and the whole set is ranked together, so the result is the num_features highest responses in the image, and the sorted response vector for one value is a prefix of the vector for any larger one. Which detection carries a given response is not pinned when responses tie: topk breaks a tie by position, and the positions it ranks over depend on num_features. Default: 2048

  • config (Optional[Detector_config], optional) – Dict with initialization parameters. Do not pass it, unless you know what you are doing. Default: None

  • ori_module (Optional[Module], optional) – for local feature orientation estimation. Default: PassLAF, which does nothing. See LAFOrienter for details.

  • aff_module (Optional[Module], optional) – for local feature affine shape estimation. Default: PassLAF, which does nothing. See LAFAffineShapeEstimator for details.

  • compile_model (bool, optional) – wrap the response function and the non-maxima suppression with torch.compile(). Default: False

  • score_threshold (float, optional) – minimum response for a position to count as a detection. Must be non-negative: non-maxima suppression writes an exact zero at every suppressed position, so a negative threshold would admit all of them. Default: 0.0

detect(img, mask=None)[source]#

Detect local features in an image batch.

Parameters:

  • img (Tensor) – Input image tensor with shape (1, C, H, W).

  • mask (Optional[Tensor], optional) – Optional mask with shape (1, 1, H, W) saying where a detection may be. A boolean or integer mask is binary (any non-zero value keeps a position), a floating-point mask is used as weights on the detection scores: a weight in (0, 1] scales a score toward the worst, so a down-weighted maximum never outranks a full-weight one, and a zero or negative weight suppresses. Weights above one are clamped to one, so a float 0/255 mask means what the integer one means. The weights are cast to the image dtype. The mask is resampled onto every pyramid level conservatively, so a zero region suppresses every maximum within one level pixel of it. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple containing detection scores and local affine frames, shaped (1, num_features) and (1, num_features, 2, 3), holding the num_features highest responses in the image and sorted by response, descending. The shape holds even when the image yields fewer above-threshold maxima than requested: those slots carry a zero response and a zero LAF, and sort after every real detection. The LAFs are float32 for a float16 or bfloat16 image and float64 for a float64 one; the responses keep the response function’s dtype.

detect_features_on_single_level(level_img, num_kp, factor, *, mask=None)[source]#

Detect keypoints on one image-pyramid level.

The response function may consume a multi-channel image – for example, a learned color detector – but must return one response map. A LAF has no response-channel identity, so independent per-channel detections are ambiguous and rejected. ScaleSpaceDetector follows the same contract.

A detection is a strict maximum of the response over its nms_size window, above score_threshold, that lies outside the 15 px strip along the level’s sides. The suppression reads the whole response, the strip included, and remove_borders() then zeroes the strip in its output: a position beside the strip is reported only when no response at least as strong, inside the strip or not, lies within its window.

Parameters:

  • level_img (Tensor) – Image tensor for a single pyramid level.

  • num_kp (int) – Number of keypoints requested from this pyramid level.

  • factor (Tuple[float, float]) – Per-axis factor (W / W_l, H / H_l) from the current pyramid level to the original image: level pixel \((u, v)\) is reported at (u * factor[0], v * factor[1]).

  • mask (Optional[Tensor], optional) – Optional mask with shape \((1, 1, H, W)\) at the original image resolution, saying where a detection may be. It is resampled onto this level conservatively and applied to the non-maxima suppression output, so a zero region drops every maximum within one level pixel of it and a floating-point weight scales the score. Keyword-only. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple containing scores and local affine frames detected at the requested pyramid level, with min(num_kp, H * W) slots. When the level holds fewer above-threshold maxima than that, the remaining slots are padded with a zero response and a zero LAF.

forward(img, mask=None)[source]#

Three stage local feature detection.

First the location and scale of interest points are determined by detect function. Then affine shape and orientation.

Parameters:

  • img (Tensor) – image to extract features with shape [1xCxHxW]. Only a batch of one image is accepted.

  • mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape [1x1xHxW]. A boolean or integer mask is binary, a floating-point mask weights the detection scores; see detect(). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple of lafs with shape [1xNx2x3], the detected local affine frames, and responses with shape [1xN], the response function values for the corresponding lafs. When the image yields fewer above-threshold maxima than num_features, the remaining slots hold a zero response and a zero LAF, whichever affine-shape and orientation modules are configured.

remove_borders(score_map, borders=15)[source]#

Zero the strip of borders pixels along each side of a level, so that no detection is made there.

detect_features_on_single_level() applies it to the non-maxima suppression output, not to the response, so an override receives the suppressed map. The strip is set to an exact zero whatever the score there, an infinite one included.

Return type:

Tensor

class kornia.feature.ScaleSpaceDetector(num_features=500, mr_size=6.0, scale_pyr_module=None, resp_module=None, subpix_module=None, ori_module=None, aff_module=None, minima_are_also_good=False, scale_space_response=False, compile_modules=False)[source]#

nn.Module for differentiable local feature detection.

It has 5 modules inside: scale pyramid generator, response (“cornerness”) function, sub-pixel localization, affine shape estimator and patch orientation estimator. Each of those modules could be replaced with a learned custom one, as long as they respect output shape.

Convention:

  • The candidates are the strict \(3 \times 3 \times 3\) maxima of the response over level, row and column (nms3d_minmax()) on the levels given under scale_pyr_module, whatever their sign. With minima_are_also_good a strict minimum is a candidate too when its negated response after sub-pixel refinement exceeds the response the maxima pass reports at that voxel, which the built-in refiners leave at its sampled value: in effect every minimum below zero, and a minimum above zero only when the refinement extrapolates it below minus its sampled value. There is no response threshold. The num_features strongest candidates of all octaves are kept; a minimum is ranked and returned with its refined response negated.

  • The default Hessian response is a maximum at bright and dark blobs alike. A DoG response (BlobDoG) is a maximum at a dark blob and a minimum at a bright one, which only minima_are_also_good=True detects.

  • A LAF centre is the refined sub-pixel position, mapped to input pixels as given under scale_pyr_module, in the frame of laf_from_center_scale_ori() (see Local features). Before aff_module and ori_module run, the frame is diagonal: mr_size times the detection scale, mapped to input pixels along each axis like the centre.

  • A candidate is kept when half its frame lies at least 5 octave pixels inside the octave’s outer pixel centres; the whole frame can reach outside the image.

  • The output has num_features slots: the detections by decreasing response, then the unfilled slots, which hold a zero response and a zero LAF.

  • The LAFs are float32 for a float16 or bfloat16 image and float64 for a float64 one; the responses keep the response function’s dtype. Non-maximum suppression and sub-pixel refinement run on the response promoted to at least float32.

  • A batch of zero images returns empty outputs of the documented shapes, after the checks a non-empty batch runs.

  • Known defects: an octave passes only its num_features strongest candidates to the border test and to the mask check of their refined centres, so the slot of a candidate rejected there stays empty even when a weaker valid candidate exists (#5741).

Parameters:

  • num_features (int, optional) – Number of features to detect. In order to keep everything batchable, output would always have num_features output, even for completely homogeneous images. Default: 500

  • mr_size (float, optional) – multiplier from the detection scale to the LAF scale. The default 6.0 gives a frame of about six times the \(\sigma\) of a detected Gaussian blob. Default: 6.0

  • scale_pyr_module (Optional[Module], optional) – generates scale pyramid. See ScalePyramid for details. Default: ScalePyramid(3, 1.6, 16, extra_levels=3) with a scale-space response, extra_levels=2 without. A detection at octave pixel \((u, v)\) is reported at \((u (W - 1) / (W_o - 1), v (H - 1) / (H_o - 1))\) in the input, the map of ScalePyramid’s align_corners=True resizes; a custom pyramid has to resize the same way. Each octave searches n_levels response levels, levels 1..n_levels; the levels below and above them are NMS neighbours only, and any further level is ignored. A per-level response therefore needs extra_levels >= 2, and DoG, which has one level fewer than its pyramid, extra_levels >= 3.

  • resp_module (Optional[Module], optional) – calculates 'cornerness' of the pixel. Default: BlobHessian.

  • subpix_module (Optional[Module], optional) – refines the position and scale of each candidate to sub-pixel / sub-scale accuracy. The candidates are those of the Convention block, whatever the module. Default: AdaptiveQuadInterp3d with allow_scale_steps=True.

  • ori_module (Optional[Module], optional) – for local feature orientation estimation. Default: PassLAF, which does nothing. See LAFOrienter for details.

  • aff_module (Optional[Module], optional) – for local feature affine shape estimation. Default: PassLAF, which does nothing. See LAFAffineShapeEstimator for details.

  • minima_are_also_good (bool, optional) – if True, then both response function minima and maxima are detected. A DoG response needs it to detect bright blobs. Default is False. With a built-in subpix_module both signs are refined in one stacked call, which roughly doubles the peak memory of the refinement step: the response volume and its three coordinate maps are materialised for 2B images at once. Default: False

  • scale_space_response (bool, optional) – if True, resp_module gets each whole octave \((B, C, L, H, W)\) with its sigmas \((B \cdot L)\) and returns a \((B, 1, L', H, W)\) scale-space response, as BlobDoG does. If False (the default), it gets the octave’s levels 0 .. n_levels + 1 as \((B \cdot L, C, H, W)\) images, \(L = n_{levels} + 2\), with their sigmas and returns \((B \cdot L, 1, H, W)\). Default: False

  • compile_modules (Union[bool, List[str]], optional) – selects which sub-modules to wrap with torch.compile(). Pass True to compile every sub-module, False (default) for none, or a list containing any subset of ["scale_pyr", "resp", "subpix", "ori", "aff"]. Compiling subpix fuses the iteration loop of conv_quad_interp3d(), which the default AdaptiveQuadInterp3d runs on CUDA. The first call incurs a one-time compilation cost; subsequent calls are fast. Default: False

detect(img, num_feats, mask=None)[source]#

Detect local features in an image batch.

Parameters:

  • img (Tensor) – Input image tensor with shape (B, C, H, W).

  • num_feats (int) – Number of features requested from the detector.

  • mask (Optional[Tensor], optional) – Optional mask with shape (1 or B, 1, H, W) saying where a detection may be. A boolean or integer mask is binary (any non-zero value keeps a position), a floating-point mask is used as weights on the detection scores: a weight in (0, 1] scales a score toward the worst (a non-negative score is multiplied by it, a negative one divided), so a down-weighted candidate never outranks a full-weight one, and a zero or negative weight suppresses. Weights above one are clamped to one, so a float 0/255 mask means what the integer one means. The weights are cast to the image dtype. The mask is resampled onto every octave conservatively, so a zero region suppresses every candidate within one octave pixel of it. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple containing detection scores and local affine frames, shaped (B, num_feats) and (B, num_feats, 2, 3). A slot that no detection filled – there were fewer candidates than requested, or the border or mask test rejected a candidate – carries a zero response and a zero LAF, and sorts after every real detection. The converse does not hold: a signed response function can peak at exactly zero, and that slot keeps its frame. A batch of zero images returns empty outputs of these shapes.

forward(img, mask=None)[source]#

Three stage local feature detection.

First the location and scale of interest points are determined by detect function. Then affine shape and orientation.

Parameters:

  • img (Tensor) – image to extract features with shape [BxCxHxW]

  • mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape [1x1xHxW] or [Bx1xHxW]. A boolean or integer mask is binary, a floating-point mask weights the detection scores; see detect(). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple of lafs with shape [BxNx2x3], the detected local affine frames, and responses with shape [BxN], the response function values for the corresponding lafs. When an image yields fewer maxima than num_features, the remaining slots hold a zero response and a zero LAF, whichever affine-shape and orientation modules are configured.

class kornia.feature.KeyNetDetector(pretrained=True, num_features=2048, keynet_conf=None, ori_module=None, aff_module=None, compile_model=False, score_threshold=0.0)[source]#

Multi-scale feature detector based on KeyNet.

This is based on the original code from paper “Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters”. See [BLRPM19] for more details.

_images/keynet.jpg
Convention:

A MultiResolutionDetector with KeyNet as its response; see their Convention blocks for the frame of the detections and for the input range.

Parameters:

  • pretrained (bool, optional) – Download and set pretrained weights to the model. Default: True

  • num_features (int, optional) – Number of features to detect. Default: 2048

  • keynet_conf (Optional[KeyNet_conf], optional) – Dict with initialization parameters. Do not pass it, unless you know what you are doing. A num_filters, num_levels or kernel_size other than the default changes the architecture: pass pretrained=False with it, since the strict load of the weights raises RuntimeError. Default: None

  • ori_module (Optional[Module], optional) – for local feature orientation estimation. Default: PassLAF, which does nothing. See LAFOrienter for details.

  • aff_module (Optional[Module], optional) – for local feature affine shape estimation. Default: PassLAF, which does nothing. See LAFAffineShapeEstimator for details.

  • compile_model (bool, optional) – wrap the response function and the non-maxima suppression with torch.compile(). Default: False

  • score_threshold (float, optional) – minimum response for a position to count as a detection. Must be non-negative: non-maxima suppression writes an exact zero at every suppressed position, so a negative threshold would admit all of them. Default: 0.0

forward(img, mask=None)[source]#

Three stage local feature detection.

First the location and scale of interest points are determined by detect function. Then affine shape and orientation.

Parameters:

  • img (Tensor) – image to extract features with shape [1xCxHxW]. Only a batch of one image is accepted.

  • mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape [1x1xHxW]. A boolean or integer mask is binary, a floating-point mask weights the detection scores; see detect(). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple of lafs with shape [1xNx2x3], the detected local affine frames, and responses with shape [1xN], the response function values for the corresponding lafs. When the image yields fewer above-threshold maxima than num_features, the remaining slots hold a zero response and a zero LAF, whichever affine-shape and orientation modules are configured.