Layers and other models

Layers

FilterResponseNorm2d and TLU are the normalisation and activation layers of HyNet, from the Filter Response Normalization paper. They take a (B, num_features, H, W) feature map and use no batch statistics.

class kornia.feature.FilterResponseNorm2d(num_features, eps=1e-6, is_bias=True, is_scale=True, is_eps_leanable=False)

Normalise each channel map by its root mean square, without batch statistics (FRN).

frn = FilterResponseNorm2d(C), then y = frn(x) takes feature maps (B, C, H, W) and returns them in the same shape, each map divided by the root of its mean square plus |eps|, then, by default, scaled and shifted per channel. It subtracts no mean and uses no batch statistics, so it behaves the same in training and eval mode and for any batch size. HyNet applies it, followed by TLU, after every convolution but the last, where HardNet uses BatchNorm and ReLU, and on the raw patch, which HardNet standardises instead. A float16 or bfloat16 input is normalised in float32, and the output dtype is the promotion of the input’s and the layer’s, so a float16 input to a float32 layer returns float32. This is the Filter Response Normalization layer from ‘Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks’, see [SK20] for more details.

\[y = \gamma \frac{x}{\sqrt{\nu^2 + |\epsilon|}} + \beta\]

Convention:

• \(\nu^2\) is the mean of \(x^2\) over the H x W map of each sample and channel, so the output does not depend on the rest of the batch; no mean is subtracted. \(\gamma\) (weight) and \(\beta\) (bias) are per-channel parameters initialised to 1 and 0, applied only when is_scale and is_bias are set.

Parameters:

• num_features (int) – number of channels

• eps (float, optional) – normalization constant Default: 1e-6

• is_bias (bool, optional) – use bias Default: True

• is_scale (bool, optional) – use scale Default: True

• is_eps_leanable (bool, optional) – if eps is learnable Default: False

Returns:

Normalized features

Return type:

torch.Tensor

Shape:

• Input: (B, num_features, H, W)

• Output: (B, num_features, H, W)

See also

• TLU: the learned threshold that follows each of these layers in HyNet.

• HyNet: the patch descriptor built from these layers.

class kornia.feature.TLU(num_features)

Floor each channel at a learned threshold, the TLU activation that follows FRN in HyNet.

tlu = TLU(C), then y = tlu(x) takes feature maps (B, C, H, W) and returns torch.max(x, tau), the input floored at a learned threshold tau per channel, in the same shape. It is the activation that the FRN paper pairs with FilterResponseNorm2d, which subtracts no mean, in place of a ReLU. The output dtype is the promotion of the input’s and the layer’s. This is the TLU layer from ‘Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks’, see [SK20] for more details.

\[y = \max(x, {\tau})\]

Convention:

• \(\tau\) (tau) is a (1, num_features, 1, 1) parameter, one threshold per channel, initialised to -1.

Parameters:

num_features (int) – number of channels

Returns:

torch.Tensor

Shape:

• Input: (B, num_features, H, W)

• Output: (B, num_features, H, W)

See also

• FilterResponseNorm2d: the normalisation that each of these layers follows in HyNet.

• HyNet: the patch descriptor built from these layers.

Fast moving objects

DeFMO deblurs a fast-moving object, given the blurred image and an estimate of its background, into 24 RGBA sub-frames of the object alone.

class kornia.feature.DeFMO(pretrained=True)

Deblur a fast-moving object into 24 RGBA sub-frames of the object alone (DeFMO).

defmo = DeFMO() builds the network with its pretrained weights, in eval mode, and frames = defmo(x) takes (B, 6, H, W) inputs, each a blurred RGB image followed by an RGB estimate of its background, and returns (B, 24, 4, H, W), with H and W rounded up to multiples of 16: the object at 24 instants in time order, each an RGBA image whose alpha channel separates it from the background, which is not rendered. It is an image model, not a local feature: it detects and describes nothing. This is based on the original code from paper “DeFMO: Deblurring and Shape Recovery of Fast Moving Objects”. See [ROF+21] for more details.

Convention:

• The input is (B, 6, H, W): the RGB channels of the blurred image, then those of the background estimate, in \([0, 1]\) for the pretrained weights. H and W that are multiples of 16 give an output of the same size; other sizes give an output rounded up to the next multiples of 16.

• The output is (B, 24, 4, H, W): 24 sub-frames in time order, sub-frame k rendered at the time k / 23, each with red, green, blue and alpha channels in \([0, 1]\) after a sigmoid.

Parameters:

pretrained (bool, optional) – Download and set pretrained weights to the model, and return it in eval mode. Default: True. With False the weights are random and the model is in training mode.

Returns:

Temporal super-resolution without background.

Shape:

• Input: (B, 6, H, W)

• Output: (B, 24, 4, H, W), with H and W rounded up to multiples of 16

Examples

>>> import kornia
>>> input = torch.rand(2, 6, 240, 320)
>>> defmo = kornia.feature.DeFMO(pretrained=False).eval()
>>> tsr_nobgr = defmo(input) # 2x24x4x240x320

forward(input_data)

Deblur a fast-moving object into a sequence of RGBA sub-frames.

Parameters:

input_data (Tensor) – Tensor with shape (B, 6, H, W) containing the blurred RGB image concatenated with an RGB background estimate.

Return type:

Tensor

Returns:

Tensor with shape (B, T, 4, H, W), where T is the number of temporal sub-frames, 24, and 4 stores red, green, blue, and alpha channels.