Detectors#

Response functions#

kornia.feature.gftt_response(input, grads_mode='sobel', sigmas=None)[source]#

Compute the Shi-Tomasi cornerness function.

_images/gftt_response.png

Function does not do any normalization or nms. The response map is computed according the following formulation:

\[R = min(eig(M))\]

torch.where:

\[\begin{split}M = \sum_{(x,y) \in W} \begin{bmatrix} I^{2}_x & I_x I_y \\ I_x I_y & I^{2}_y \\ \end{bmatrix}\end{split}\]
Parameters:

  • input (Tensor) – input image with shape \((B, C, H, W)\).

  • grads_mode (str, optional) – can be 'sobel' for standalone use or 'diff' for use on Gaussian pyramid. Default: "sobel"

  • sigmas (Optional[Tensor], optional) – coefficients to be multiplied by multichannel response. Should be shape of \((B)\) It is necessary for performing non-maxima-suppression across different scale pyramid levels. See vlfeat. Default: None

Return type:

Tensor

Returns:

the response map per channel with shape \((B, C, H, W)\).

Example

>>> input = torch.tensor([[[
...    [0., 0., 0., 0., 0., 0., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 0., 0., 0., 0., 0., 0.],
... ]]])  # 1x1x7x7
>>> # compute the response map; rounded for a platform-stable display
>>> gftt_response(input).round(decimals=4) + 0.0
tensor([[[[0.0379, 0.0456, 0.0283, 0.0121, 0.0283, 0.0456, 0.0379],
          [0.0456, 0.0598, 0.0402, 0.0168, 0.0402, 0.0598, 0.0456],
          [0.0283, 0.0402, 0.0545, 0.0245, 0.0545, 0.0402, 0.0283],
          [0.0121, 0.0168, 0.0245, 0.0276, 0.0245, 0.0168, 0.0121],
          [0.0283, 0.0402, 0.0545, 0.0245, 0.0545, 0.0402, 0.0283],
          [0.0456, 0.0598, 0.0402, 0.0168, 0.0402, 0.0598, 0.0456],
          [0.0379, 0.0456, 0.0283, 0.0121, 0.0283, 0.0456, 0.0379]]]])
kornia.feature.harris_response(input, k=0.04, grads_mode='sobel', sigmas=None)[source]#

Compute the Harris cornerness function.

_images/harris_response.png

Function does not do any normalization or nms. The response map is computed according the following formulation:

\[R = max(0, det(M) - k \cdot trace(M)^2)\]

torch.where:

\[\begin{split}M = \sum_{(x,y) \in W} \begin{bmatrix} I^{2}_x & I_x I_y \\ I_x I_y & I^{2}_y \\ \end{bmatrix}\end{split}\]

and \(k\) is an empirically determined constant \(k ∈ [ 0.04 , 0.06 ]\)

Parameters:

  • input (Tensor) – input image with shape \((B, C, H, W)\).

  • k (Union[Tensor, float], optional) – the Harris detector free parameter. Default: 0.04

  • grads_mode (str, optional) – can be 'sobel' for standalone use or 'diff' for use on Gaussian pyramid. Default: "sobel"

  • sigmas (Optional[Tensor], optional) –

    coefficients to be multiplied by multichannel response. Should be shape of \((B)\) It is necessary for performing non-maxima-suppression across different scale pyramid levels. See vlfeat. Default: None

Return type:

Tensor

Returns:

the response map per channel with shape \((B, C, H, W)\).

Example

>>> input = torch.tensor([[[
...    [0., 0., 0., 0., 0., 0., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 0., 0., 0., 0., 0., 0.],
... ]]])  # 1x1x7x7
>>> # compute the response map; rounded for a platform-stable display
>>> harris_response(input, 0.04).round(decimals=4) + 0.0
tensor([[[[0.0042, 0.0054, 0.0035, 0.0006, 0.0035, 0.0054, 0.0042],
          [0.0054, 0.0068, 0.0046, 0.0014, 0.0046, 0.0068, 0.0054],
          [0.0035, 0.0046, 0.0034, 0.0014, 0.0034, 0.0046, 0.0035],
          [0.0006, 0.0014, 0.0014, 0.0006, 0.0014, 0.0014, 0.0006],
          [0.0035, 0.0046, 0.0034, 0.0014, 0.0034, 0.0046, 0.0035],
          [0.0054, 0.0068, 0.0046, 0.0014, 0.0046, 0.0068, 0.0054],
          [0.0042, 0.0054, 0.0035, 0.0006, 0.0035, 0.0054, 0.0042]]]])
kornia.feature.hessian_response(input, grads_mode='sobel', sigmas=None)[source]#

Compute the absolute of determinant of the Hessian matrix.

_images/hessian_response.png

Function does not do any normalization or nms. The response map is computed according the following formulation:

\[R = det(H)\]

torch.where:

\[\begin{split}M = \sum_{(x,y) \in W} \begin{bmatrix} I_{xx} & I_{xy} \\ I_{xy} & I_{yy} \\ \end{bmatrix}\end{split}\]
Parameters:

  • input (Tensor) – input image with shape \((B, C, H, W)\).

  • grads_mode (str, optional) – can be 'sobel' for standalone use or 'diff' for use on Gaussian pyramid. Default: "sobel"

  • sigmas (Optional[Tensor], optional) –

    coefficients to be multiplied by multichannel response. Should be shape of \((B)\) It is necessary for performing non-maxima-suppression across different scale pyramid levels. See vlfeat. Default: None

Return type:

Tensor

Returns:

the response map per channel with shape \((B, C, H, W)\).

Shape:

  • Input: \((B, C, H, W)\)

  • Output: \((B, C, H, W)\)

Examples

>>> input = torch.tensor([[[
...    [0., 0., 0., 0., 0., 0., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 1., 1., 1., 1., 1., 0.],
...    [0., 0., 0., 0., 0., 0., 0.],
... ]]])  # 1x1x7x7
>>> # compute the response map; rounded for a platform-stable display
>>> hessian_response(input).round(decimals=4) + 0.0
tensor([[[[-0.0564, -0.0759, -0.0253,  0.0000, -0.0253, -0.0759, -0.0564],
          [-0.0759, -0.0330,  0.0333,  0.0000,  0.0333, -0.0330, -0.0759],
          [-0.0253,  0.0333,  0.0542,  0.0000,  0.0542,  0.0333, -0.0253],
          [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
          [-0.0253,  0.0333,  0.0542,  0.0000,  0.0542,  0.0333, -0.0253],
          [-0.0759, -0.0330,  0.0333,  0.0000,  0.0333, -0.0330, -0.0759],
          [-0.0564, -0.0759, -0.0253,  0.0000, -0.0253, -0.0759, -0.0564]]]])
kornia.feature.dog_response(input)[source]#

Compute the Difference-of-Gaussian response.

Parameters:

input (Tensor) – a given the gaussian 5d torch.Tensor \((B, C, D, H, W)\).

Return type:

Tensor

Returns:

the response map per channel with shape \((B, C, D-1, H, W)\).

kornia.feature.dog_response_single(input, sigma1=1.0, sigma2=1.6)[source]#

Compute the Difference-of-Gaussian response.

_images/dog_response_single.png
Parameters:

  • input (Tensor) – a given the gaussian 4d torch.Tensor \((B, C, H, W)\).

  • sigma1 (float, optional) – lower gaussian sigma Default: 1.0

  • sigma2 (float, optional) – bigger gaussian sigma Default: 1.6

Return type:

Tensor

Returns:

the response map per channel with shape \((B, C, H, W)\).

class kornia.feature.BlobHessian(grads_mode='sobel')[source]#

nn.Module that calculates Hessian blobs.

_images/hessian_response.png

See hessian_response() for details.

class kornia.feature.CornerGFTT(grads_mode='sobel')[source]#

nn.Module that calculates Shi-Tomasi corners.

_images/gftt_response.png

See gftt_response() for details.

class kornia.feature.CornerHarris(k, grads_mode='sobel')[source]#

nn.Module that calculates Harris corners.

_images/harris_response.png

See harris_response() for details.

class kornia.feature.BlobDoG[source]#

nn.Module that calculates Difference-of-Gaussians blobs.

See :func: ~kornia.feature.dog_response for details.

class kornia.feature.BlobDoGSingle(sigma1=1.0, sigma2=1.6)[source]#

nn.Module that calculates Difference-of-Gaussians blobs.

_images/dog_response_single.png

See dog_response_single() for details.

Detectors#

class kornia.feature.KeyNet(pretrained=False, keynet_conf=None)[source]#

Key.Net model definition – local feature detector (response function).

This is based on the original code from paper “Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters”. See [BLRPM19] for more details.

_images/KeyNet.png
Parameters:

  • pretrained (bool, optional) – Download and set pretrained weights to the model. Default: False

  • keynet_conf (Optional[KeyNet_conf], optional) – Dict with initialization parameters. Do not pass it, unless you know what you are doing`. Default: None

Returns:

KeyNet response score.

Shape:

  • Input: \((B, 1, H, W)\)

  • Output: \((B, 1, H, W)\)

class kornia.feature.MultiResolutionDetector(model, num_features=2048, config=None, ori_module=None, aff_module=None, compile_model=False, score_threshold=0.0)[source]#

Multi-scale feature detector, based on code from KeyNet. Can be used with any response function.

This is based on the original code from paper “Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters”. See [BLRPM19] for more details.

Parameters:

  • model (Module) – response function, such as KeyNet or BlobHessian

  • num_features (int, optional) – Number of features to detect. Default: 2048

  • conf – Dict with initialization parameters. Do not pass it, unless you know what you are doing`.

  • ori_module (Optional[Module], optional) – for local feature orientation estimation. Default: PassLAF, which does nothing. See LAFOrienter for details.

  • aff_module (Optional[Module], optional) – for local feature affine shape estimation. Default: PassLAF, which does nothing. See LAFAffineShapeEstimator for details.

  • compile_model (bool, optional) – wrap the response function and the non-maxima suppression with torch.compile(). Default: False

  • score_threshold (float, optional) – minimum response for a position to count as a detection. Must be non-negative: non-maxima suppression writes an exact zero at every suppressed position, so a negative threshold would admit all of them. Default: 0.0

detect(img, mask=None)[source]#

Detect local features in an image batch.

Parameters:

  • img (Tensor) – Input image tensor with shape (1, C, H, W).

  • mask (Optional[Tensor], optional) – Optional mask with shape (1, 1, H, W) saying where a detection may be. A boolean or integer mask is binary (any non-zero value keeps a position), a floating-point mask is used as weights on the detection scores: a weight in (0, 1] scales a score toward the worst, so a down-weighted maximum never outranks a full-weight one, and a zero or negative weight suppresses. Weights above one are clamped to one, so a float 0/255 mask means what the integer one means. The weights are cast to the image dtype. The mask is resampled onto every pyramid level conservatively, so a zero region suppresses every maximum within one level pixel of it. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple containing detection scores and local affine frames, shaped (1, num_features) and (1, num_features, 2, 3). The shape holds even when the image yields fewer above-threshold maxima than requested: those slots carry a zero response and a zero LAF, and sort after every real detection. LAF centres are pixel coordinates cast to the image dtype, so a half-precision image gives centres at that dtype’s integer resolution: exact up to 256 in bfloat16 and up to 2048 in float16, and coarser beyond.

detect_features_on_single_level(level_img, num_kp, factor, *, mask=None)[source]#

Detect keypoints on one image-pyramid level.

The response function may consume a multi-channel image – for example, a learned color detector – but must return one response map. A LAF has no response-channel identity, so independent per-channel detections are ambiguous and rejected. ScaleSpaceDetector follows the same contract.

Parameters:

  • level_img (Tensor) – Image tensor for a single pyramid level.

  • num_kp (int) – Number of keypoints requested from this pyramid level.

  • factor (Tuple[float, float]) – Scale factor mapping coordinates from the current pyramid level back to the original image resolution.

  • mask (Optional[Tensor], optional) – Optional mask with shape \((1, 1, H, W)\) at the original image resolution, saying where a detection may be. It is resampled onto this level conservatively and applied to the non-maxima suppression output, so a zero region drops every maximum within one level pixel of it and a floating-point weight scales the score. Keyword-only. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple containing scores and local affine frames detected at the requested pyramid level, with min(num_kp, H * W) slots. When the level holds fewer above-threshold maxima than that, the remaining slots are padded with a zero response and a zero LAF.

forward(img, mask=None)[source]#

Three stage local feature detection.

First the location and scale of interest points are determined by detect function. Then affine shape and orientation.

Parameters:

  • img (Tensor) – image to extract features with shape [1xCxHxW]. KeyNetDetector does not support batch processing, because the number of detections is different on each image.

  • mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape [1x1xHxW]. A boolean or integer mask is binary, a floating-point mask weights the detection scores; see detect(). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple of lafs with shape [1xNx2x3], the detected local affine frames, and responses with shape [1xN], the response function values for the corresponding lafs. When the image yields fewer above-threshold maxima than num_features, the remaining slots hold a zero response and a zero LAF, whichever affine-shape and orientation modules are configured.

remove_borders(score_map, borders=15)[source]#

Remove the borders of the image to avoid detections on the corners.

Return type:

Tensor

class kornia.feature.ScaleSpaceDetector(num_features=500, mr_size=6.0, scale_pyr_module=None, resp_module=None, subpix_module=None, ori_module=None, aff_module=None, minima_are_also_good=False, scale_space_response=False, compile_modules=False)[source]#

nn.Module for differentiable local feature detection.

As close as possible to classical local feature detectors like Harris, Hessian-Affine or SIFT (DoG).

It has 5 modules inside: scale pyramid generator, response (“cornerness”) function, sub-pixel localization, affine shape estimator and patch orientation estimator. Each of those modules could be replaced with a learned custom one, as long as they respect output shape.

Parameters:

  • num_features (int, optional) – Number of features to detect. In order to keep everything batchable, output would always have num_features output, even for completely homogeneous images. Default: 500

  • mr_size (float, optional) – multiplier for local feature scale compared to the detection scale. 6.0 is matching OpenCV 12.0 convention for SIFT. Default: 6.0

  • scale_pyr_module (Optional[Module], optional) – generates scale pyramid. See ScalePyramid for details. Default: ScalePyramid(3, 1.6, 15).

  • resp_module (Optional[Module], optional) – calculates 'cornerness' of the pixel. Default: None

  • subpix_module (Optional[Module], optional) – performs non-maximum suppression and refines keypoint location to sub-pixel / sub-scale accuracy. See ConvQuadInterp3d for details. Default: None

  • ori_module (Optional[Module], optional) – for local feature orientation estimation. Default:class:~kornia.feature.PassLAF, which does nothing. See LAFOrienter for details. Default: None

  • aff_module (Optional[Module], optional) – for local feature affine shape estimation. Default: PassLAF, which does nothing. See LAFAffineShapeEstimator for details.

  • minima_are_also_good (bool, optional) – if True, then both response function minima and maxima are detected. Useful for symmetric response functions like DoG or Hessian. Default is False. Default: False

  • compile_modules (Union[bool, List[str]], optional) – selects which sub-modules to wrap with torch.compile(). Pass True to compile every sub-module, False (default) for none, or a list containing any subset of ["scale_pyr", "resp", "subpix", "ori", "aff"]. Compiling subpix gives ~5x GPU speedup for the default ConvQuadInterp3d backend by fusing its iteration loop. The first call incurs a one-time compilation cost; subsequent calls are fast. Default: False

detect(img, num_feats, mask=None)[source]#

Detect local features in an image batch.

Parameters:

  • img (Tensor) – Input image tensor with shape (B, C, H, W).

  • num_feats (int) – Number of features requested from the detector.

  • mask (Optional[Tensor], optional) – Optional mask with shape (1 or B, 1, H, W) saying where a detection may be. A boolean or integer mask is binary (any non-zero value keeps a position), a floating-point mask is used as weights on the detection scores: a weight in (0, 1] scales a score toward the worst (a non-negative score is multiplied by it, a negative one divided), so a down-weighted candidate never outranks a full-weight one, and a zero or negative weight suppresses. Weights above one are clamped to one, so a float 0/255 mask means what the integer one means. The weights are cast to the image dtype. The mask is resampled onto every octave conservatively, so a zero region suppresses every candidate within one octave pixel of it. Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple containing detection scores and local affine frames, shaped (B, num_feats) and (B, num_feats, 2, 3). A slot that no detection filled – there were fewer candidates than requested, or a candidate’s frame reached outside the image – carries a zero response and a zero LAF, and sorts after every real detection. The converse does not hold: a signed response function can peak at exactly zero, and that slot keeps its frame.

forward(img, mask=None)[source]#

Three stage local feature detection.

First the location and scale of interest points are determined by detect function. Then affine shape and orientation.

Parameters:

  • img (Tensor) – image to extract features with shape [BxCxHxW]

  • mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape [1x1xHxW] or [Bx1xHxW]. A boolean or integer mask is binary, a floating-point mask weights the detection scores; see detect(). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple of lafs with shape [BxNx2x3], the detected local affine frames, and responses with shape [BxN], the response function values for the corresponding lafs. When an image yields fewer maxima than num_features, the remaining slots hold a zero response and a zero LAF, whichever affine-shape and orientation modules are configured.

class kornia.feature.KeyNetDetector(pretrained=False, num_features=2048, keynet_conf=None, ori_module=None, aff_module=None, compile_model=False, score_threshold=0.0)[source]#

Multi-scale feature detector based on KeyNet.

This is based on the original code from paper “Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters”. See [BLRPM19] for more details.

_images/keynet.jpg
Parameters:

  • pretrained (bool, optional) – Download and set pretrained weights to the model. Default: False

  • num_features (int, optional) – Number of features to detect. Default: 2048

  • keynet_conf (Optional[KeyNet_conf], optional) – Dict with initialization parameters. Do not pass it, unless you know what you are doing`. Default: None

  • ori_module (Optional[Module], optional) – for local feature orientation estimation. Default: PassLAF, which does nothing. See LAFOrienter for details.

  • aff_module (Optional[Module], optional) – for local feature affine shape estimation. Default: PassLAF, which does nothing. See LAFAffineShapeEstimator for details.

  • compile_model (bool, optional) – wrap the response function and the non-maxima suppression with torch.compile(). Default: False

  • score_threshold (float, optional) – minimum response for a position to count as a detection. Must be non-negative: non-maxima suppression writes an exact zero at every suppressed position, so a negative threshold would admit all of them. Default: 0.0

forward(img, mask=None)[source]#

Three stage local feature detection.

First the location and scale of interest points are determined by detect function. Then affine shape and orientation.

Parameters:

  • img (Tensor) – image to extract features with shape [1xCxHxW]. KeyNetDetector does not support batch processing, because the number of detections is different on each image.

  • mask (Optional[Tensor], optional) – a mask saying where a detection may be, shape [1x1xHxW]. A boolean or integer mask is binary, a floating-point mask weights the detection scores; see detect(). Default: None

Return type:

Tuple[Tensor, Tensor]

Returns:

Tuple of lafs with shape [1xNx2x3], the detected local affine frames, and responses with shape [1xN], the response function values for the corresponding lafs. When the image yields fewer above-threshold maxima than num_features, the remaining slots hold a zero response and a zero LAF, whichever affine-shape and orientation modules are configured.