Detectors#

Response functions score image locations; detectors select local maxima and return feature frames. Use a response function when you need a dense score map, or a detector when you need sparse keypoints.

For an image (B, C, H, W), harris_response() and gftt_response() score corners; hessian_response() and dog_response_single() score blobs. These functions return maps of the same shape as the image, without non-maximum suppression. dog_response() instead takes a Gaussian scale stack (B, C, D, H, W) and returns differences between adjacent levels, (B, C, D - 1, H, W).

To use a response in a detector, choose its module form: CornerHarris, CornerGFTT, BlobHessian, BlobDoGSingle or BlobDoG. Pass it as resp_module to ScaleSpaceDetector, or as model to MultiResolutionDetector. BlobDoG requires a scale stack, so use it only with ScaleSpaceDetector and scale_space_response=True.

The detectors return LAFs (B, N, 2, 3) in input pixels and responses (B, N), strongest first. Unused slots contain zero LAFs and zero responses. Use laf_is_filled() to keep detected features and discard padded slots; apply the same mask to LAFs and responses.

ScaleSpaceDetector takes a batch and refines maxima over a Gaussian scale space. Its default response is the Hessian determinant.
MultiResolutionDetector takes one image and finds maxima at integer pixels of each pyramid level, without scale selection or subpixel refinement.
KeyNetDetector uses MultiResolutionDetector with the learned KeyNet response.

For detection and description together, see Local features (detector and descriptor together).

Response functions#

kornia.feature.gftt_response(input, grads_mode='sobel', sigmas=None)[source]#

Compute the Shi-Tomasi corner response, the smaller eigenvalue of the windowed structure tensor.

Try in browser

R = gftt_response(img) takes images (B, C, H, W) and returns the Shi-Tomasi (“good features to track”) response of every pixel of every channel, (B, C, H, W), without non-maxima suppression: large at a corner, near zero along an edge and zero on a constant image. Unlike harris_response(), from the same structure tensor, it has no k to tune. To pass it to a detector as resp_module, use its module form CornerGFTT; for blobs, use hessian_response().

The response map is computed according to the following formulation:

\[R = min(eig(M))\]

where \(M\) is the windowed structure tensor of harris_response(), with the same gradients and window.

Convention:

\(R\) is not negative, up to roundoff. Along an edge, where it is near zero, the Harris response is negative.
Multiplying the image by c multiplies the response by c ** 2, and an offset does not change it.
sigmas multiplies the response by sigmas ** 4, the same power used by harris_response(). This is the function’s scale-weighting convention, not a guarantee of scale invariance. The integration window \(W\) stays fixed as the image blur changes; the response’s decay with blur depends on the image structure, including the corner geometry.
Each channel is processed independently. The window extends its reflected border across short axes, including singleton pixels.

Parameters:

input (Tensor) – input image with shape (B, C, H, W).
grads_mode (str, optional) – can be 'sobel' for standalone use or 'diff' for use on Gaussian pyramid. Default: "sobel"
sigmas (Optional[Tensor], optional) – per-image scales with shape (B,). The response of image b is multiplied by sigmas[b] ** 4, so that non-maxima suppression can compare the levels of a scale pyramid. Another type raises TypeError and another shape ValueError. Default: None

Return type:

Tensor

Returns:

the response map per channel with shape (B, C, H, W).

See also

CornerGFTT: the module form, with grads_mode fixed at construction.
harris_response(): the signed Harris measure of the same structure tensor, with a free k.
hessian_response(): a blob response from second derivatives, without a window.
GFTTAffNetHardNet: a ScaleSpaceDetector on this response, with AffNet shapes and HardNet descriptors.

Example

>>> input = torch.tensor([[[
...    [0., 0., 0., 0., 0., 0., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 0., 0., 0., 0., 0., 0.],
... ]]])  # 1x1x7x7
>>> # compute the response map; rounded for a platform-stable display
>>> gftt_response(input).round(decimals=4) + 0.0
tensor([[[[0.0379, 0.0456, 0.0283, 0.0121, 0.0283, 0.0456, 0.0379],
          [0.0456, 0.0598, 0.0402, 0.0168, 0.0402, 0.0598, 0.0456],
          [0.0283, 0.0402, 0.0545, 0.0245, 0.0545, 0.0402, 0.0283],
          [0.0121, 0.0168, 0.0245, 0.0276, 0.0245, 0.0168, 0.0121],
          [0.0283, 0.0402, 0.0545, 0.0245, 0.0545, 0.0402, 0.0283],
          [0.0456, 0.0598, 0.0402, 0.0168, 0.0402, 0.0598, 0.0456],
          [0.0379, 0.0456, 0.0283, 0.0121, 0.0283, 0.0456, 0.0379]]]])

kornia.feature.harris_response(input, k=0.04, grads_mode='sobel', sigmas=None)[source]#

Compute the Harris corner response det(M) - k trace(M)^2 of the windowed structure tensor M.

Try in browser

R = harris_response(img, k) takes images (B, C, H, W) and returns the Harris response of every pixel of every channel, (B, C, H, W), without non-maxima suppression: positive at a corner, negative along an edge and zero on a constant image. k defaults to 0.04. To pass it to a detector as resp_module, use its module form CornerHarris; for a corner response with no k to tune and no negative values up to roundoff, use gftt_response(); for blobs, hessian_response().

The response map is computed according to the following formulation:

\[R = det(M) - k \cdot trace(M)^2\]

where \(w\) are the weights of a window \(W\):

\[\begin{split}M = \sum_{(x,y) \in W} w(x, y) \begin{bmatrix} I^{2}_x & I_x I_y \\ I_x I_y & I^{2}_y \\ \end{bmatrix}\end{split}\]

and \(k\) is an empirically determined constant, usually in \([0.04, 0.06]\).

Convention:

\(R\) is signed and not clamped: positive at a corner, negative along an edge.
\(I_x\) and \(I_y\) come from spatial_gradient() with mode=grads_mode, in intensity per pixel for both modes, and \(W\) is a \(7 \times 7\) Gaussian window of \(\sigma = 1\) whose weights sum to one. A linear ramp of gradient \(g\) therefore gives \(R = -k \|g\|^4\) away from the border.
Apart from sigmas, the response depends on the image’s intensity scale: multiplying the image by c multiplies the response by c ** 4, and an offset does not change it, so a threshold tuned on images in [0, 1] does not carry over to [0, 255].
k is not validated.
Each channel is processed independently. The window extends its reflected border across short axes, including singleton pixels.

Parameters:

input (Tensor) – input image with shape (B, C, H, W).
k (Union[Tensor, float], optional) – the Harris detector free parameter, a number or a tensor that broadcasts against (B, C, H, W), such as (B, 1, 1, 1) for one value per image. Default: 0.04
grads_mode (str, optional) – can be 'sobel' for standalone use or 'diff' for use on Gaussian pyramid. Default: "sobel"
sigmas (Optional[Tensor], optional) – per-image scales with shape (B,). The response of image b is multiplied by sigmas[b] ** 4, so that non-maxima suppression can compare the levels of a scale pyramid, as in vlfeat. Another type raises TypeError and another shape ValueError. Default: None

Return type:

Tensor

Returns:

the response map per channel with shape (B, C, H, W).

See also

CornerHarris: the module form, with k and grads_mode fixed at construction.
gftt_response(): the smaller eigenvalue of the same structure tensor, with no k and not negative up to roundoff.
hessian_response(): a blob response from second derivatives, positive at blobs of both polarities.
ScaleSpaceDetector: detects LAFs at the scale-space maxima of a response module such as CornerHarris.

Example

>>> input = torch.tensor([[[
...    [0., 0., 0., 0., 0., 0., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 0., 0., 0., 0., 0., 0.],
... ]]])  # 1x1x7x7
>>> # compute the response map; rounded for a platform-stable display
>>> harris_response(input, 0.04).round(decimals=4) + 0.0
tensor([[[[0.0042, 0.0054, 0.0035, 0.0006, 0.0035, 0.0054, 0.0042],
          [0.0054, 0.0068, 0.0046, 0.0014, 0.0046, 0.0068, 0.0054],
          [0.0035, 0.0046, 0.0034, 0.0014, 0.0034, 0.0046, 0.0035],
          [0.0006, 0.0014, 0.0014, 0.0006, 0.0014, 0.0014, 0.0006],
          [0.0035, 0.0046, 0.0034, 0.0014, 0.0034, 0.0046, 0.0035],
          [0.0054, 0.0068, 0.0046, 0.0014, 0.0046, 0.0068, 0.0054],
          [0.0042, 0.0054, 0.0035, 0.0006, 0.0035, 0.0054, 0.0042]]]])

kornia.feature.hessian_response(input, grads_mode='sobel', sigmas=None)[source]#

Compute the blob response det(H) of the per-pixel Hessian, positive at bright and dark blobs alike.

Try in browser

R = hessian_response(img) takes images (B, C, H, W) and returns the determinant of the Hessian of every pixel of every channel, (B, C, H, W), without non-maxima suppression. Its module form BlobHessian is the default resp_module of ScaleSpaceDetector. For a Difference-of-Gaussian response, which tells a bright blob (a minimum) from a dark one (a maximum), use dog_response(); for corners, harris_response() or gftt_response().

The response map is computed according to the following formulation:

\[R = det(H)\]

where:

\[\begin{split}H = \begin{bmatrix} I_{xx} & I_{xy} \\ I_{xy} & I_{yy} \\ \end{bmatrix}\end{split}\]

Convention:

\(R = I_{xx} I_{yy} - I_{xy}^2\) is signed and pointwise, without a window: positive at bright and dark blobs alike, negative at a saddle. The second derivatives come from spatial_gradient() with mode=grads_mode and order=2.
Multiplying the image by c multiplies the response by c ** 2, and an offset does not change it.
Each channel is processed independently.

Parameters:

input (Tensor) – input image with shape (B, C, H, W).
grads_mode (str, optional) – can be 'sobel' for standalone use or 'diff' for use on Gaussian pyramid. Default: "sobel"
sigmas (Optional[Tensor], optional) – per-image scales with shape (B,). The response of image b is multiplied by sigmas[b] ** 4, so that non-maxima suppression can compare the levels of a scale pyramid. Another type raises TypeError and another shape ValueError. Default: None

Return type:

Tensor

Returns:

the response map per channel with shape (B, C, H, W).

Shape:

Input: (B, C, H, W)
Output: (B, C, H, W)

See also

BlobHessian: the module form, with grads_mode fixed at construction.
dog_response(): a signed blob response of a scale stack, a minimum at a bright blob.
harris_response(): a corner response from first derivatives in a window.
ScaleSpaceDetector: detects LAFs at the scale-space maxima of this response, its default.

Examples

>>> input = torch.tensor([[[
...    [0., 0., 0., 0., 0., 0., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 0., 0., 0., 0., 0., 0.],
... ]]])  # 1x1x7x7
>>> # compute the response map; rounded for a platform-stable display
>>> hessian_response(input).round(decimals=4) + 0.0
tensor([[[[-0.0137, -0.0332, -0.0205,  0.0000, -0.0205, -0.0332, -0.0137],
          [-0.0332,  0.0098,  0.0381,  0.0000,  0.0381,  0.0098, -0.0332],
          [-0.0205,  0.0381,  0.0547,  0.0000,  0.0547,  0.0381, -0.0205],
          [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
          [-0.0205,  0.0381,  0.0547,  0.0000,  0.0547,  0.0381, -0.0205],
          [-0.0332,  0.0098,  0.0381,  0.0000,  0.0381,  0.0098, -0.0332],
          [-0.0137, -0.0332, -0.0205,  0.0000, -0.0205, -0.0332, -0.0137]]]])

kornia.feature.dog_response(input)[source]#

Compute Difference-of-Gaussian responses between adjacent levels of a (B, C, D, H, W) scale stack.

R = dog_response(stack) takes a Gaussian scale stack (B, C, D, H, W), with its D levels along dim 2, and returns the differences of adjacent levels, (B, C, D - 1, H, W). An octave of ScalePyramid is such a stack; to detect at the extrema, pass the module form BlobDoG to ScaleSpaceDetector with scale_space_response=True and minima_are_also_good=True. For the DoG of one image at two given blurs, use dog_response_single().

Convention:

Output level i is input[:, :, i + 1] - input[:, :, i] along the level axis D (dim 2), so D levels give D - 1. For a stack blurred from fine to coarse, this is Lowe’s \(L(k\sigma) - L(\sigma)\): negative at a bright blob, positive at a dark one.

Parameters:: input (Tensor) – a Gaussian scale stack (B, C, D, H, W), its levels along D.
Return type:: Tensor
Returns:: the response map per channel with shape (B, C, D - 1, H, W).

See also

BlobDoG: the module form, to pass to ScaleSpaceDetector as resp_module.
dog_response_single(): the DoG of a (B, C, H, W) image, blurred at two given sigmas.
hessian_response(): a blob response of one image, positive at blobs of both polarities.
ScalePyramid: builds the scale stacks, one (B, C, L, H_o, W_o) per octave.

kornia.feature.dog_response_single(input, sigma1=1.0, sigma2=1.6)[source]#

Compute the Difference-of-Gaussian response of a (B, C, H, W) image, its sigma2 blur minus its sigma1 blur.

Try in browser

R = dog_response_single(img, sigma1, sigma2) takes images (B, C, H, W) and returns, per channel, their Gaussian blur of sigma2 minus their blur of sigma1, (B, C, H, W): for sigma2 > sigma1, negative at a bright blob and positive at a dark one. It blurs the image itself, so it needs no scale stack; for the differences of adjacent levels of a precomputed stack, use dog_response(). Its module form is BlobDoGSingle.

Convention:

Returns the Gaussian blur of sigma2 minus that of sigma1, the order of dog_response(); see its Convention block. Each kernel has the size int(8 * sigma + 1), rounded up to an odd number. The order of sigma1 and sigma2 is not checked: swapping them flips the sign.
Both blurs extend reflected borders across short axes, including singleton pixels.

Parameters:

input (Tensor) – images (B, C, H, W).
sigma1 (float, optional) – lower gaussian sigma Default: 1.0
sigma2 (float, optional) – bigger gaussian sigma Default: 1.6

Return type:

Tensor

Returns:

the response map per channel with shape (B, C, H, W).

See also

BlobDoGSingle: the module form, with sigma1 and sigma2 fixed at construction.
dog_response(): the differences of adjacent levels of a (B, C, D, H, W) scale stack.
hessian_response(): a blob response that is positive at blobs of both polarities.
SIFTFeature: detects at the maxima of this response’s magnitude on each level of an image pyramid.

class kornia.feature.BlobHessian(grads_mode='sobel')[source]#

Compute the Hessian-determinant blob response; the module form of hessian_response.

R = BlobHessian()(img, sigmas) returns hessian_response(img, grads_mode, sigmas), the (B, C, H, W) Hessian determinant of (B, C, H, W) images, positive at bright and dark blobs alike. It is the default resp_module of ScaleSpaceDetector, which passes each level’s sigma so that the levels compare. For a DoG response over a scale stack, use BlobDoG; for corners, CornerGFTT or CornerHarris.

Convention:: Returns hessian_response() with the constructor’s grads_mode; see its Convention block.

See also

hessian_response(): the function form.
BlobDoG: the DoG blob response of a scale stack, a minimum at a bright blob.
CornerGFTT: a corner response, the smaller structure-tensor eigenvalue.
ScaleSpaceDetector: detects LAFs at the scale-space maxima of this response, its default.
HesAffNetHardNet: that detector on this response, with AffNet shapes and HardNet descriptors.

class kornia.feature.CornerGFTT(grads_mode='sobel')[source]#

Compute the Shi-Tomasi corner response; the module form of gftt_response.

R = CornerGFTT()(img, sigmas) returns gftt_response(img, grads_mode, sigmas), the (B, C, H, W) smaller structure-tensor eigenvalue of (B, C, H, W) images, large at a corner. Pass it to ScaleSpaceDetector as resp_module to detect corners over scale, as GFTTAffNetHardNet does; for the signed Harris measure, use CornerHarris instead.

Convention:: Returns gftt_response() with the constructor’s grads_mode; see its Convention block.

See also

gftt_response(): the function form.
CornerHarris: the Harris corner response of the same structure tensor, with a free k.
BlobHessian: the Hessian-determinant blob response.
GFTTAffNetHardNet: a ScaleSpaceDetector on this response, with AffNet shapes and HardNet descriptors.

class kornia.feature.CornerHarris(k, grads_mode='sobel')[source]#

Compute the Harris corner response with a fixed k; the module form of harris_response.

R = CornerHarris(k)(img, sigmas) returns harris_response(img, k, grads_mode, sigmas), the (B, C, H, W) Harris response of (B, C, H, W) images, positive at a corner. k is stored as a buffer, so it moves with torch.nn.Module.to(). Pass it to ScaleSpaceDetector as resp_module to detect corners over scale; for a response with no k to choose, use CornerGFTT instead.

Convention:: Returns harris_response() with the constructor’s k and grads_mode; see its Convention block. Unlike the function’s, this k has no default. A float, int or NumPy scalar k becomes a 0-dimensional float32 tensor, and a float64 image uses the number itself while the buffer still holds it; a tensor is stored as it is.

See also

harris_response(): the function form, with k defaulting to 0.04.
CornerGFTT: the Shi-Tomasi corner response of the same structure tensor, without k.
BlobHessian: the Hessian-determinant blob response.
ScaleSpaceDetector: detects LAFs at the scale-space maxima of a response module such as this one.

class kornia.feature.BlobDoG[source]#

Compute Difference-of-Gaussian responses of a (B, C, D, H, W) scale stack; the module form of dog_response.

R = BlobDoG()(stack, sigmas) returns dog_response(stack), the (B, C, D - 1, H, W) differences of adjacent levels, and ignores sigmas. It is the resp_module that ScaleSpaceDetector takes with scale_space_response=True; since a bright blob is a DoG minimum, pass minima_are_also_good=True too, as DoGHardNet and the "patch" backend of SIFTFeatureScaleSpace do. For the DoG of one image at two given blurs, use BlobDoGSingle instead.

Convention:: Returns dog_response() of the (B, C, D, H, W) scale stack; see its Convention block.

See also

dog_response(): the function form.
BlobDoGSingle: the DoG of a (B, C, H, W) image at two sigmas fixed at construction.
BlobHessian: a per-level blob response, positive at blobs of both polarities.
ScaleSpaceDetector: detects LAFs at the extrema of this response over its scale pyramid.
SIFTFeatureScaleSpace: a SIFT preset whose "patch" backend is that detector on this response.

class kornia.feature.BlobDoGSingle(sigma1=1.0, sigma2=1.6)[source]#

Compute the Difference-of-Gaussian response of a (B, C, H, W) image; the module form of dog_response_single.

R = BlobDoGSingle(sigma1, sigma2)(img, sigmas) returns dog_response_single(img, sigma1, sigma2), the (B, C, H, W) blur of sigma2 minus the blur of sigma1, and ignores sigmas. Passed to MultiResolutionDetector, which keeps maxima only, it misses the centres of bright blobs, the DoG minima: SIFTFeature passes its magnitude instead. For the DoG of a whole scale stack, use BlobDoG.

Convention:: Returns dog_response_single() with the constructor’s sigma1 and sigma2; see its Convention block.

See also

dog_response_single(): the function form, with the sigmas passed on each call.
BlobDoG: the DoG between adjacent levels of a (B, C, D, H, W) scale stack.
MultiResolutionDetector: detects LAFs at the maxima of a per-image response on an image pyramid.

Detectors#

class kornia.feature.KeyNet(pretrained=True, keynet_conf=None)[source]#

Compute the learned Key.Net keypoint response map of a grayscale image, at the input’s size.

response = KeyNet()(img) takes grayscale images (B, 1, H, W) in [0, 1] and returns their Key.Net response, a non-negative (B, 1, H, W) map, without non-maxima suppression: handcrafted first- and second-order derivative filters and a small CNN run on the image and on num_levels - 1 successive 1.2x downscales of it (two by default), and a last convolution merges their features, resized back to the input size. The pretrained weights, the default, come back in evaluation mode; a random model (pretrained=False) stays in training mode, so call .eval() before inference (see Pretrained models). For keypoints, use KeyNetDetector, which finds the maxima of this response on an image pyramid.

This is based on the original code from the paper “Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters”. See [BLRPM19] for more details.

Convention:

The input is a grayscale image, and the pretrained weights expect it in [0, 1], as the Key.Net reference scales its images. The response is not proportional to the intensity, so an image in [0, 255] gives other detections. A multi-channel image raises RuntimeError.
The response is a non-negative map of the input’s size when kernel_size is odd. With pretrained=False and an even kernel_size, the final convolution returns a map one pixel larger on each axis.

Parameters:

pretrained (bool, optional) – Download and set pretrained weights to the model. Default: True
keynet_conf (Optional[KeyNet_conf], optional) – Dict with initialization parameters. Do not pass it, unless you know what you are doing. A num_filters, num_levels or an odd kernel_size other than the default changes the architecture. Pass pretrained=False with it, since the strict load of the weights raises RuntimeError. Default: None

Returns:

KeyNet response score.

Shape:

Input: (B, 1, H, W)
Output: (B, 1, H, W)

See also

KeyNetDetector: detects LAFs at the maxima of this response on an image pyramid.
BlobHessian: a handcrafted blob response, with no weights to load.
KeyNetHardNet: KeyNet detection with OriNet orientation and HardNet descriptors.

class kornia.feature.MultiResolutionDetector(model, num_features=2048, config=None, ori_module=None, aff_module=None, compile_model=False, score_threshold=0.0)[source]#

Detect LAFs at the 2D maxima of a response on each level of an image pyramid, one image per call.

detector = MultiResolutionDetector(model, num_features) builds an image pyramid detector from a response model. lafs, responses = detector(img) takes one image (1, C, H, W) and returns LAFs (1, num_features, 2, 3) in input pixels and responses (1, num_features), strongest first. Use laf_is_filled() to keep detected features and discard padded slots in both outputs. Detections lie at integer pixels of each pyramid level, without scale selection or subpixel refinement. For a batch or refined centres and scales, use ScaleSpaceDetector; for KeyNet responses, use KeyNetDetector.

This is based on the original code from the paper “Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters”. See [BLRPM19] for more details.

The Key.Net reference code zeroes a 15 px strip along each side of a level before the non-maxima suppression, which reports a “maximum” beside the strip wherever the response rises into it. This detector suppresses on the whole response and removes the strip from the suppression output instead; see detect_features_on_single_level().

Convention:

model maps each pyramid level (1, C, H_l, W_l) to a response (1, 1, H_l, W_l). Only responses carry gradients; the selected centres and scales do not. aff_module and then ori_module modify the upright frames; both default to PassLAF. detector.detect(img) returns (responses, lafs) before those two modules run, reversing the output order of detector(img).
Detections are maxima of the response above score_threshold. For a signed response whose minima are features as well, such as a difference of Gaussians, pass its magnitude, as SIFTFeature does.
A detection is an integer pixel \((u, v)\) of a pyramid level of size (H_l, W_l), without a sub-pixel step. It is reported at the centre of that level pixel, \(((u + 0.5) W / W_l - 0.5, (v + 0.5) H / H_l - 0.5)\) of the (H, W) input, in the frame of laf_from_center_scale_ori() (see Local features). The Key.Net reference reports \((u W / W_l, v H / H_l)\) instead. For \(f = W / W_l > 1\) (likewise along y) that is \(0.5 (f - 1)\) px up and to the left of the centre, about 1.5 px on the default coarsest level; on the default up level, where \(f < 1\), it is about 0.15 px down and to the right.
Before aff_module and ori_module run, the frame is upright and isotropic, with the scale s_mult * (W / W_l + H / H_l) / 2 whatever the response; s_mult of config is 22 by default.

Parameters:

model (Module) – response function, such as KeyNet or BlobHessian
num_features (int, optional) – Number of features to detect. Every pyramid level is searched for this many candidates and the whole set is ranked together, so the result is the num_features highest responses in the image, and the sorted response vector for one value is a prefix of the vector for any larger one. Which detection carries a given response is not pinned when responses tie: topk breaks a tie by position, and the positions it ranks over depend on num_features. Default: 2048
config (Optional[Detector_config], optional) – Dict with initialization parameters. Do not pass it, unless you know what you are doing. The pyramid has up_levels levels (1 by default) upscaled by successive powers of scale_factor_levels (sqrt(2)), the input, and pyramid_levels levels (4) below it, each the previous one downscaled by scale_factor_levels with pyrdown(); nms_size (15) is the side of the suppression window in level pixels, and s_mult (22) the frame scale, in level pixels. Default: None
ori_module (Optional[Module], optional) – for local feature orientation estimation. Default: PassLAF, which does nothing. See LAFOrienter for details.
aff_module (Optional[Module], optional) – for local feature affine shape estimation. Default: PassLAF, which does nothing. See LAFAffineShapeEstimator for details.
compile_model (bool, optional) – wrap the response function and the non-maxima suppression with torch.compile(). Default: False
score_threshold (float, optional) – minimum response for a position to count as a detection. Must be non-negative: non-maxima suppression writes an exact zero at every suppressed position, so a negative threshold would admit all of them; it raises ValueError. Default: 0.0

See also

ScaleSpaceDetector: a batch of images, the sub-pixel maxima of a response over a Gaussian scale space, with scale selection across levels.
KeyNetDetector: this detector on the learned KeyNet response.
SIFTFeature: this detector on the magnitude of a DoG response, with SIFT descriptors.

Example

>>> detector = MultiResolutionDetector(BlobHessian(), num_features=50)
>>> lafs, responses = detector(torch.rand(1, 1, 64, 64))
>>> lafs.shape, responses.shape
(torch.Size([1, 50, 2, 3]), torch.Size([1, 50]))

detect(img, mask=None)[source]#

Detect local features in one image, a batch of one.

Parameters:

img (Tensor) – Input image tensor with shape (1, C, H, W).
mask (Optional[Tensor], optional) – Optional mask with shape (1, 1, H, W) saying where a detection may be. A boolean or integer mask is binary (any non-zero value keeps a position), a floating-point mask is used as weights on the detection scores: a weight in (0, 1] scales a score toward the worst, so a down-weighted maximum never outranks a full-weight one, and a zero or negative weight suppresses. Weights above one are clamped to one, so a float 0/255 mask means what the integer one means. The weights are cast to the response’s dtype. The mask is resampled onto every pyramid level conservatively, so a zero region suppresses every maximum within one level pixel of it. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple containing detection scores and local affine frames, shaped (1, num_features) and (1, num_features, 2, 3), in that order, the reverse of forward’s, holding the num_features highest responses in the image and sorted by response, descending. The shape holds even when the image yields fewer above-threshold maxima than requested: those slots carry a zero response and a zero LAF, and sort after every real detection. The LAFs are float32 for a float16 or bfloat16 image and float64 for a float64 one; the responses keep the response function’s dtype.

detect_features_on_single_level(level_img, num_kp, factor, *, mask=None)[source]#

Detect keypoints on one image-pyramid level.

The response function may consume a multi-channel image – for example, a learned color detector – but must return one response map. A LAF has no response-channel identity, so independent per-channel detections are ambiguous and rejected. ScaleSpaceDetector follows the same contract.

A detection is a strict maximum of the response over its nms_size window, above score_threshold, that lies outside the 15 px strip along the level’s sides. The suppression reads the whole response, the strip included, and remove_borders() then zeroes the strip in its output: a position beside the strip is reported only when no response at least as strong, inside the strip or not, lies within its window.

Parameters:

level_img (Tensor) – Image tensor for a single pyramid level.
num_kp (int) – Number of keypoints requested from this pyramid level.
factor (Tuple[float, float]) – Per-axis factor (W / W_l, H / H_l) from the current pyramid level to the original image: level pixel \((u, v)\) is reported at its centre, ((u + 0.5) * factor[0] - 0.5, (v + 0.5) * factor[1] - 0.5).
mask (Optional[Tensor], optional) – Optional mask with shape (1, 1, H, W) at the original image resolution, saying where a detection may be. It is resampled onto this level conservatively and applied to the non-maxima suppression output, so a zero region drops every maximum within one level pixel of it and a floating-point weight scales the score. Keyword-only. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple containing scores and local affine frames detected at the requested pyramid level, with min(num_kp, H_l * W_l) slots for a level of size (H_l, W_l). When the level holds fewer above-threshold maxima than that, the remaining slots are padded with a zero response and a zero LAF.

forward(img, mask=None)[source]#

Three stage local feature detection.

First the location and scale of interest points are determined by detect function. Then affine shape and orientation.

Parameters:

img (Tensor) – image to extract features with shape (1, C, H, W). Only a batch of one image is accepted.
mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape (1, 1, H, W). A boolean or integer mask is binary, a floating-point mask weights the detection scores; see detect(). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple of lafs with shape (1, N, 2, 3), the detected local affine frames, and responses with shape (1, N), the response function values for the corresponding lafs, with N = num_features. When the image yields fewer above-threshold maxima than num_features, the remaining slots hold a zero response and a zero LAF, whichever affine-shape and orientation modules are configured.

remove_borders(score_map, borders=15)[source]#

Zero the strip of borders pixels along each side of a level, so that no detection is made there.

detect_features_on_single_level() applies it to the non-maxima suppression output, not to the response, so an override receives the suppressed map. The strip is set to an exact zero whatever the score there, an infinite one included.

Return type:: Tensor

ScaleSpaceDetector applies border and refined-centre mask checks before each octave’s feature budget. Rejected candidates do not consume slots while weaker valid detections remain; any slots left after all valid candidates are exhausted contain zero responses and zero LAFs.

class kornia.feature.ScaleSpaceDetector(num_features=500, mr_size=6.0, scale_pyr_module=None, resp_module=None, subpix_module=None, ori_module=None, aff_module=None, minima_are_also_good=False, scale_space_response=False, compile_modules=False)[source]#

Detect LAFs at the sub-pixel scale-space maxima of a pluggable response, for a batch of images.

detector = ScaleSpaceDetector(num_features) builds a detector with a Hessian response by default. lafs, responses = detector(img) takes grayscale images (B, 1, H, W) and returns LAFs (B, num_features, 2, 3) in input pixels and responses (B, num_features), strongest first. Use laf_is_filled() to keep detected features and discard padded slots in both outputs. For per-level 2D maxima, use MultiResolutionDetector; for SIFT detection and description, use SIFTFeatureScaleSpace.

Convention:

The modules run in this order: scale_pyr_module, resp_module, subpix_module, aff_module, then ori_module. The last two default to PassLAF, which leaves frames unchanged. detector.detect(img, num_feats) returns (responses, lafs) before shape and orientation estimation; note the reversed output order compared with detector(img).
The candidates are the strict \(3 \times 3 \times 3\) maxima of the response over level, row and column (nms3d_minmax()) on the levels given under scale_pyr_module, whatever their sign. With minima_are_also_good a strict minimum is a candidate too when its negated response after sub-pixel refinement exceeds the response the maxima pass reports at that voxel, which the built-in refiners leave at its sampled value: in effect every minimum below zero, and a minimum above zero only when the refinement extrapolates it below minus its sampled value. There is no response threshold. The num_features strongest candidates of all octaves are kept; a minimum is ranked and returned with its refined response negated.
The default Hessian response is a maximum at bright and dark blobs alike. It keeps the channels, so a multi-channel image needs a resp_module that returns one map. A DoG response (BlobDoG) is a maximum at a dark blob and a minimum at a bright one, which only minima_are_also_good=True detects.
A LAF centre is the refined sub-pixel position, mapped to input pixels as given under scale_pyr_module, using the pixel convention of laf_from_center_scale_ori() (see Local features). Before aff_module and ori_module run, the frame is diagonal: mr_size times the detection scale, mapped to input pixels along each axis like the centre.
A candidate is kept when half its frame lies at least 5 octave pixels inside the octave’s outer pixel centres; the whole frame can reach outside the image.
The output has num_features slots: the detections by decreasing response, then the unfilled slots, which hold a zero response and a zero LAF.
The LAFs are float32 for a float16 or bfloat16 image and float64 for a float64 one; the responses keep the response function’s dtype. Non-maximum suppression and sub-pixel refinement run on the response promoted to at least float32. With ScalePyramid, a BlobHessian, BlobDoG, CornerHarris or CornerGFTT response, and a response and refiner holding only scalar parameters and buffers, a float16 image is detected in float32 with autocast off and its scores are returned in float16, because the gradient overflows float16 on the way through the scale space. A candidate whose score is zero in float16 is dropped. With any other pyramid or response function, a subclass or a CornerHarris with a non-scalar k included, or a refiner with non-scalar state, the detector passes the float16 image on, in the caller’s autocast region.
With the default modules the responses, centres and scales carry gradients to the image.
A batch of zero images returns empty outputs of the documented shapes, after the checks a non-empty batch runs.
Border and refined-centre mask checks run before the per-octave feature budget, so a rejected candidate cannot displace a weaker valid candidate.

Parameters:

num_features (int, optional) – Number of features to detect. In order to keep everything batchable, output would always have num_features output, even for completely homogeneous images. Default: 500
mr_size (float, optional) – multiplier from the detection scale to the LAF scale. The default 6.0 gives a frame of about six times the \(\sigma\) of a detected Gaussian blob. Default: 6.0
scale_pyr_module (Optional[Module], optional) – generates scale pyramid. See ScalePyramid for details. Default: ScalePyramid(3, 1.6, 16, extra_levels=3) with a scale-space response, extra_levels=2 without. A detection at octave pixel \((u, v)\) is reported at \((u (W - 1) / (W_o - 1), v (H - 1) / (H_o - 1))\) in the input, the map of ScalePyramid’s align_corners=True resizes; a custom pyramid has to resize the same way. Each octave searches n_levels response levels, levels 1..n_levels; the levels below and above them are NMS neighbours only, and any further level is ignored. A per-level response therefore needs extra_levels >= 2, and DoG, which has one level fewer than its pyramid, extra_levels >= 3.
resp_module (Optional[Module], optional) – calculates 'cornerness' of the pixel. Default: BlobHessian.
subpix_module (Optional[Module], optional) – refines the position and scale of each candidate to sub-pixel / sub-scale accuracy. The candidates are those of the Convention block, whatever the module. Default: AdaptiveQuadInterp3d with allow_scale_steps=True.
ori_module (Optional[Module], optional) – for local feature orientation estimation. Default: PassLAF, which does nothing. See LAFOrienter for details.
aff_module (Optional[Module], optional) – for local feature affine shape estimation. Default: PassLAF, which does nothing. See LAFAffineShapeEstimator for details.
minima_are_also_good (bool, optional) – if True, then both response function minima and maxima are detected. A DoG response needs it to detect bright blobs. Default is False. Refining both signs can use roughly twice the peak memory of refining maxima alone. Default: False
scale_space_response (bool, optional) – if True, resp_module gets each whole octave (B, C, L, H, W) with its sigmas (B * L,) and returns a (B, 1, L', H, W) scale-space response, as BlobDoG does. If False (the default), it gets the octave’s levels 0 .. n_levels + 1 as (B * L, C, H, W) images, L = n_levels + 2, with their sigmas and returns (B * L, 1, H, W). Default: False
compile_modules (Union[bool, List[str]], optional) – selects which sub-modules to wrap with torch.compile(). Pass True to compile every sub-module, False (default) for none, or a list containing any subset of ["scale_pyr", "resp", "subpix", "ori", "aff"]. The first call compiles, and a new input shape, such as another batch size, can compile again. Default: False

See also

MultiResolutionDetector: one image per call, the 2D maxima of a response on each level of an image pyramid, at integer level pixels and without scale selection.
KeyNetDetector: that detector on the learned KeyNet response.
SIFTFeatureScaleSpace: SIFT, whose "patch" backend is this detector on a DoG response.
DoGHardNet: this detector on a DoG response, with HardNet descriptors.
ConvSoftArgmax3d: a subpix_module that takes the soft-argmax of a window instead of a quadratic fit.

Example

>>> lafs, responses = ScaleSpaceDetector(num_features=100)(torch.rand(2, 1, 64, 64))
>>> lafs.shape, responses.shape
(torch.Size([2, 100, 2, 3]), torch.Size([2, 100]))

detect(img, num_feats, mask=None)[source]#

Detect local features in an image batch.

Parameters:

img (Tensor) – Input image tensor with shape (B, C, H, W).
num_feats (int) – Number of features requested from the detector.
mask (Optional[Tensor], optional) – Optional mask with shape (1 or B, 1, H, W) saying where a detection may be. A boolean or integer mask is binary (any non-zero value keeps a position), a floating-point mask is used as weights on the detection scores: a weight in (0, 1] scales a score toward the worst (a non-negative score is multiplied by it, a negative one divided), so a down-weighted candidate never outranks a full-weight one, and a zero or negative weight suppresses. Weights above one are clamped to one, so a float 0/255 mask means what the integer one means. The weights are cast to the response’s dtype. The mask is resampled onto every octave conservatively, so a zero region suppresses every candidate within one octave pixel of it. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple containing detection scores and local affine frames, shaped (B, num_feats) and (B, num_feats, 2, 3), in that order, the reverse of forward’s. A slot that no detection filled – there were fewer candidates inside the border and the mask than requested – carries a zero response and a zero LAF, and sorts after every real detection. The converse does not hold: a signed response function can peak at exactly zero, and that slot keeps its frame. A batch of zero images returns empty outputs of these shapes.

forward(img, mask=None)[source]#

Three stage local feature detection.

First the location and scale of interest points are determined by detect function. Then affine shape and orientation.

Parameters:

img (Tensor) – image to extract features with shape (B, C, H, W).
mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape (1, 1, H, W) or (B, 1, H, W). A boolean or integer mask is binary, a floating-point mask weights the detection scores; see detect(). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple of lafs with shape (B, N, 2, 3), the detected local affine frames, and responses with shape (B, N), the response function values for the corresponding lafs, with N = num_features. When fewer detections than num_features survive for an image, the remaining slots hold a zero response and a zero LAF, whichever affine-shape and orientation modules are configured.

class kornia.feature.KeyNetDetector(pretrained=True, num_features=2048, keynet_conf=None, ori_module=None, aff_module=None, compile_model=False, score_threshold=0.0)[source]#

Detect LAFs at the maxima of the learned Key.Net response on an image pyramid, one image per call.

detector = KeyNetDetector(num_features=2048), then lafs, responses = detector(img) takes one grayscale image (1, 1, H, W) in [0, 1] and returns (1, num_features, 2, 3) LAFs in input pixels and their (1, num_features) KeyNet responses, strongest first. laf_is_filled(lafs) is true for detections; index every output with it to discard padded slots. It is a MultiResolutionDetector on the KeyNet response with the default config: maxima at the integer pixels of six pyramid levels, in upright frames unless ori_module or aff_module change them. For KeyNet detections with orientation and HardNet descriptors, use KeyNetAffNetHardNet instead.

This is based on the original code from the paper “Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters”. See [BLRPM19] for more details.

Convention:: A MultiResolutionDetector with KeyNet as its response; see their Convention blocks for the pixel convention and input range. Only KeyNet is in evaluation mode after construction, so this detector reports training=True; calling .train() changes its detections.

Parameters:

pretrained (bool, optional) – Download and set pretrained weights to the model. Default: True
num_features (int, optional) – Number of features to detect. Default: 2048
keynet_conf (Optional[KeyNet_conf], optional) – Dict with initialization parameters. Do not pass it, unless you know what you are doing. A num_filters, num_levels or an odd kernel_size other than the default changes the architecture. Pass pretrained=False with it, since the strict load of the weights raises RuntimeError. Default: None
ori_module (Optional[Module], optional) – for local feature orientation estimation. Default: PassLAF, which does nothing. See LAFOrienter for details.
aff_module (Optional[Module], optional) – for local feature affine shape estimation. Default: PassLAF, which does nothing. See LAFAffineShapeEstimator for details.
compile_model (bool, optional) – wrap the response function and the non-maxima suppression with torch.compile(). Default: False
score_threshold (float, optional) – minimum response for a position to count as a detection. Must be non-negative: non-maxima suppression writes an exact zero at every suppressed position, so a negative threshold would admit all of them; it raises ValueError. Default: 0.0

See also

KeyNet: the response network alone, a dense (B, 1, H, W) map.
MultiResolutionDetector: the same detector on another response function.
ScaleSpaceDetector: batched detection at the sub-pixel scale-space extrema of a pluggable response, the Hessian determinant by default.
KeyNetHardNet: this detector with OriNet orientation and HardNet descriptors.
KeyNetAffNetHardNet: the same, with AffNet affine shapes before the orientation.
LAFOrienter: an ori_module that orients each frame from its image patch.
LAFAffineShapeEstimator: an aff_module that adapts each frame to its patch’s second moments.
laf_is_filled(): is true for detections; use it to discard padded slots.

forward(img, mask=None)[source]#

Three stage local feature detection.

First the location and scale of interest points are determined by detect function. Then affine shape and orientation.

Parameters:

img (Tensor) – image to extract features with shape (1, C, H, W). Only a batch of one image is accepted.
mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape (1, 1, H, W). A boolean or integer mask is binary, a floating-point mask weights the detection scores; see detect(). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple of lafs with shape (1, N, 2, 3), the detected local affine frames, and responses with shape (1, N), the response function values for the corresponding lafs, with N = num_features. When the image yields fewer above-threshold maxima than num_features, the remaining slots hold a zero response and a zero LAF, whichever affine-shape and orientation modules are configured.