Losses

ivy.hinge_embedding_loss(input, target, *, margin=1.0, reduction='mean')

Measures loss from input x and label y with values 1 or -1. It evaluates if two inputs are similar or not, often used for embedding or semi-supervised learning.

Loss for the n-th sample:

\[\begin{split}l_n = \begin{cases} x_n, & \text{if}\; y_n = 1,\\ \max \{0, margin - x_n\}, & \text{if}\; y_n = -1, \end{cases}\end{split}\]

Total loss:

\[\begin{split}\ell(x, y) = \begin{cases} \operatorname{mean}(L), & \text{if reduction} = \text{`mean';}\\ \operatorname{sum}(L), & \text{if reduction} = \text{`sum'.} \end{cases}\end{split}\]

where \(L = \{l_1,\dots,l_N\}^\top\) .

Parameters:

• input (Union[Array, NativeArray]) – Input tensor with dtype float. The shape is [N, *], where N is batch size and * represents any number of additional dimensions.

• label – Label tensor containing 1 or -1 with dtype float32 or float64. Its shape matches that of the input.

• margin (float, default: 1.0) – Sets the hyperparameter margin. Determines the necessary input size for hinge_embedding_loss calculations when label is -1. Inputs smaller than the margin are minimized with hinge_embedding_loss. Default is 1.0.

• reduction (str, default: 'mean') – Specifies how to aggregate the loss across the batch. Options are: - 'none': Returns the unreduced loss. - 'mean': Returns the mean loss. - 'sum': Returns the summed loss. Default is 'mean'.

• Shape –

• ----- –

  • Input: \((*)\) where \(*\) means, any number of dimensions.

  The sum operation operates over all the elements. - Target: \((*)\), same shape as the input - Output: scalar. If reduction is 'none', then same shape as the input

Return type:

Array

Returns:

ret – Hinge embedding loss calculated from the input and label, shaped based on the reduction method.

Examples

>>> input_tensor = ivy.array([1, 2, 3, 4], dtype=ivy.float64)
>>> target_tensor = ivy.array([1, 1, 1, 1], dtype=ivy.float64)
>>> loss = ivy.hinge_embedding_loss(input_tensor, target_tensor, reduction="none")
>>> loss
ivy.array([1., 2., 3., 4.])

>>> input_tensor = ivy.array([21, 22], dtype=ivy.float32)
>>> target_tensor = ivy.array([-1, 1], dtype=ivy.float32)
>>> loss = ivy.hinge_embedding_loss(input_tensor,target_tensor,
...                                 margin=2.0, reduction="sum")
>>> loss
ivy.array(22.)

ivy.huber_loss(true, pred, /, *, delta=1.0, reduction='mean', out=None)

Compute the Huber loss (smooth L1 loss) between true and predicted values.

Parameters:

• true (array_like) – The true (ground truth) values.

• pred (array_like) – The predicted values by the model.

• delta (float, optional) – The threshold parameter that determines the point where the loss transitions fro -m squared error to absolute error. Default is 1.0.

• reduction (str, optional) – The type of reduction to apply to the loss. Possible values are “mean” (default) and “sum”.

• out (array_like, optional) – Optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret (array_like) – The Huber loss between the true and predicted values.

Examples

>>> true = ivy.array([2, 4, 7, 1])
>>> pred = ivy.array([2.5, 3.5, 8, 0.8])
>>> huber_loss(true, pred, delta=1.0)
ivy.array([0.125, 0.125, 0.5  , 0.125])

>>> huber_loss(true, pred, delta=2.0)
ivy.array([0.125, 0.125, 0.5  , 0.2  ])

>>> huber_loss(true, pred, delta=0.5)
ivy.array([0.25 , 0.25 , 0.   , 0.125])

ivy.kl_div(input, target, /, *, reduction='mean', log_target=False, out=None)

Compute the Kullback-Leibler divergence loss between two input tensors (conventionally, probability distributions).

Parameters:

• input (array_like) – Tensor of arbitrary shape in log-probabilities

• target (array_like) – Tensor of the same shape as input. See log_target for the target’s interpretation

• reduction ({'mean', 'sum', 'batchmean', 'none'}, optional) – Type of reduction to apply to the output. Default is ‘mean’.

• log_target (bool) – A flag indicating whether target is passed in the log space. It is recommended to pass certain distributions (like softmax) in the log space to avoid numerical issues caused by explicit log. Default: False

Return type:

Array

Returns:

ret (array) – The Kullback-Leibler divergence loss between the two input tensors.

Examples

>>> input = ivy.array([[0.2, 0.8], [0.5, 0.5]])
>>> target = ivy.array([[0.6, 0.4], [0.3, 0.7]])
>>> ivy.kl_div(input, target)
ivy.array(-0.555969)

>>> input = ivy.array([[0.2, 0.8], [0.5, 0.5]])
>>> target = ivy.array([[0.6, 0.4], [0.3, 0.7]])
>>> ivy.kl_div(input, target, reduction='sum')
ivy.array(-2.223876)

>>> input = ivy.array([[0.2, 0.8], [0.5, 0.5]])
>>> target = ivy.array([[0.6, 0.4], [0.3, 0.7]])
>>> ivy.kl_div(input, target, reduction='batchmean')
ivy.array(-1.111938)

>>> input = ivy.array([0.2, 0.8], [0.5, 0.5])
>>> target = ivy.array([0.6, 0.4], [0.3, 0.7])
>>> ivy.kl_div(input, target, reduction='none')
ivy.array([[-0.42649534, -0.68651628],
            [-0.51119184, -0.59967244]])

ivy.l1_loss(input, target, /, *, reduction='mean', out=None)

Compute L1 loss (Mean Absolute Error - MAE) between targeticted and input values.

Parameters:

• input (Union[ivy.Array, ivy.NativeArray]) – Input array containing input values.

• target (Union[ivy.Array, ivy.NativeArray]) – Input array containing targeted values.

• reduction (str, optional) – Reduction method for the output loss. Options: “none” (no reduction), “mean” (mean of losses), “sum” (sum of losses). Default: “mean”.

• out (Optional[ivy.Array], optional) – Optional output array for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ivy.Array – The L1 loss (MAE) between the given input and targeticted values.

Examples

>>> x = ivy.array([1.0, 2.0, 3.0])
>>> y = ivy.array([0.5, 2.5, 2.0])
>>> ivy.l1_loss(x, y)
ivy.array(0.666)
>>> a = ivy.array([[1.0, 2.0], [3.0, 4.0]])
>>> b = ivy.array([[0.5, 1.5], [2.5, 3.5]])
>>> ivy.l1_loss(a, b)
ivy.array(0.5)

ivy.log_poisson_loss(true, pred, /, *, compute_full_loss=False, axis=-1, reduction='none', out=None)

Compute the log-likelihood loss between the prediction and the target under the assumption that the target has a Poisson distribution. Caveat: By default, this is not the exact loss, but the loss minus a constant term [log(z!)]. That has no effect for optimization, but does not play well with relative loss comparisons. To compute an approximation of the log factorial term, specify compute_full_loss=True to enable Stirling’s Approximation.

Parameters:

• true (Union[Array, NativeArray]) – input array containing true labels.

• pred (Union[Array, NativeArray]) – input array containing Predicted labels.

• compute_full_loss (bool, default: False) – whether to compute the full loss. If false, a constant term is dropped in favor of more efficient optimization. Default: False.

• axis (int, default: -1) – the axis along which to compute the log-likelihood loss. If axis is -1, the log-likelihood loss will be computed along the last dimension. Default: -1.

• reduction (str, default: 'none') – 'none': No reduction will be applied to the output. 'mean': The output will be averaged. 'sum': The output will be summed. Default: 'none'.

• out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The binary log-likelihood loss between the given distributions.

Examples

>>> x = ivy.array([0, 0, 1, 0])
>>> y = ivy.array([0.25, 0.25, 0.25, 0.25])
>>> print(ivy.log_poisson_loss(x, y))
ivy.array([1.28402555, 1.28402555, 1.03402555, 1.28402555])

>>> z = ivy.array([0.1, 0.1, 0.7, 0.1])
>>> print(ivy.log_poisson_loss(x, z, reduction='mean'))
ivy.array(1.1573164)

ivy.poisson_nll_loss(input, target, *, log_input=True, full=False, eps=1e-08, reduction='mean')

Compute the Poisson Negative Log Likelihood Loss.

This function calculates the negative log likelihood loss between the input and target`under the assumption that the target follows a Poisson distribution. By default, the loss is not the exact loss, but the loss minus a constant term [log(z!)]. This omission does not affect optimization but can be significant for relative loss comparisons. The Stirling’s Approximation is used to approximate the log factorial term when `full is set to True.

Parameters:

• input (Union[Array, NativeArray]) – Expectation of the underlying Poisson distribution.

• target (Union[Array, NativeArray]) – Random sample from the Poisson distribution described by the input.

• log_input (bool, default: True) – If True, the loss is computed as \(exp(input) - target * input\). If False, the loss is computed as \(input - target * log(input + eps)\). Default is True.

• full (bool, default: False) – Whether to compute the full loss, i.e., to add the Stirling approximation term \(target * log(target) - target + 0.5 * log(2 * pi * target)\). Default is False.

• eps (float, default: 1e-08) – Small value to prevent evaluation of log(0) when log_input is False. Default is 1e-8.

• reduction (str, default: 'mean') – Specifies the reduction applied to the output. Options are ‘none’, ‘mean’, or ‘sum’. ‘none’: no reduction will be applied. ‘mean’: the output will be averaged. ‘sum’: the output will be summed. Default is ‘mean’.

Return type:

Array

Returns:

ret – An array of the same shape as input representing the Poisson Negative Log Likelihood Loss.

Raises:

ValueError – If the input and target tensors do not have the same shape.

Examples

>>> input_tensor = ivy.array([1, 2, 3, 4], dtype=ivy.float64)
>>> target_tensor = ivy.array([2, 2, 2, 2], dtype=ivy.float64)
>>> loss = ivy.poisson_nll_loss(input_tensor, target_tensor, log_input=False)
>>> print(loss)
ivy.array(0.91097307)

ivy.smooth_l1_loss(input, target, /, *, beta=1.0, reduction='mean', out=None)

Compute the smooth L1 loss between two input tensors.

Parameters:

• input (array_like) – First input tensor.

• target (array_like) – Second input tensor.

• beta (float, optional) – The smooth parameter. Default is 1.0.

• reduction (str, optional) – Specifies the type of reduction to apply to the output. Should be one of ‘none’, ‘sum’, or ‘mean’. Default is ‘mean’.

• out (array, optional) – Optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret (array) – The smooth_l1_loss between the two input tensors.

Examples

>>> x = ivy.array([1.0, 2.0, 3.0])
>>> y = ivy.array([2.5, 1.8, 3.2])
>>> ivy.smooth_l1_loss(x, y, beta=1.0)
ivy.array(0.3467)
>>> x = ivy.array([1.0, 2.0, 3.0])
>>> y = ivy.array([6.0, 2.0, 3.0])
>>> ivy.smooth_l1_loss(x, y, beta=1.0)
ivy.array(1.5)
>>> input = ivy.array([2.0, 3.0, 5.0, 7.0])
>>> target = ivy.array([2.5, 3.5, 5.5, 6.5])
>>> loss = ivy.smooth_l1_loss(input, target, beta=1.5, reduction='sum')
ivy.array(0.5)
>>> input = ivy.array([0.8, 1.2, 2.5, 3.7])
>>> target = ivy.array([0.9, 1.0, 2.3, 3.6])
>>> loss = ivy.smooth_l1_loss(input, target, beta=0.5, reduction='none')
ivy.array([0.0133, 0.0250, 0.0056, 0.0025])
>>> input = ivy.array([2.0, 3.0, 5.0, 7.0])
>>> target = ivy.array([2.5, 3.5, 5.5, 6.5])
>>> loss = ivy.smooth_l1_loss(input, target, beta=0.2, reduction='mean')
ivy.array(0.025)

With ivy.NativeArray input:

>>> x = ivy.native_array([1.5, 2.2, 3.7])
>>> y = ivy.native_array([2.1, 1.9, 3.5])
>>> print(ivy.smooth_l1_loss(x, y, beta=0.5))
ivy.array(0.0675)

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([1.0, 2.0, 3.0]))
>>> y = ivy.Container(a=ivy.array([2.5, 1.8, 3.2]))
>>> print(ivy.smooth_l1_loss(x, y, beta=1.0))
{
    a: ivy.array(0.3467)
}

With a mix of ivy.Array and ivy.NativeArray inputs:

>>> x = ivy.array([1.0, 2.0, 3.0])
>>> y = ivy.native_array([6.0, 2.0, 3.0])
>>> print(ivy.smooth_l1_loss(x, y, beta=0.5))
ivy.array(1.5)

With a mix of ivy.Array and ivy.Container inputs:

>>> x = ivy.array([1.0, 2.0, 3.0])
>>> y = ivy.Container(a=ivy.array([6.0, 2.0, 3.0]))
>>> print(ivy.smooth_l1_loss(x, y, beta=1.0))
{
    a: ivy.array(1.5)
}

Instance Method Examples

With ivy.Array input:

>>> x = ivy.array([1.0, 2.0, 3.0])
>>> y = ivy.array([2.5, 1.8, 3.2])
>>> print(x.smooth_l1_loss(y, beta=1.0))
ivy.array(0.3467)

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([1.0, 2.0, 3.0]))
>>> y = ivy.Container(a=ivy.array([2.5, 1.8, 3.2]))
>>> print(x.smooth_l1_loss(y, beta=1.0))
{
    a: ivy.array(0.3467)
}

ivy.soft_margin_loss(input, target, /, *, reduction='mean', out=None)

Compute the soft-margin hinge loss between predicted scores and true binary labels.

Parameters:

• input (array_like) – True binary labels, of shape (batch_size,).

• target (array_like) – Predicted scores, of shape (batch_size,).

• reduction ({'mean', 'sum', 'none'}, optional) – Type of reduction to apply to the output. Default is ‘mean’.

• out (array_like, optional) – Optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret (array) – The soft-margin hinge loss between the predicted scores and true binary labels.

Examples

>>> input = ivy.array([1, 0, 1, 0])
>>> target = ivy.array([0.8, 0.2, -0.6, 1.5])
>>> ivy.soft_margin_loss(input, target)
ivy.array(0.6987)

>>> input = ivy.array([1, 1, 0, 0])
>>> target = ivy.array([0.8, 0.7, 0.2, 0.1])
>>> ivy.soft_margin_loss(input, target, reduction='sum')
ivy.array(2.1606)

>>> input = ivy.array([1, 1, 0, 0])
>>> target = ivy.array([0.8, 0.7, 0.2, 0.1])
>>> ivy.soft_margin_loss(input, target, reduction='none')
ivy.array([0.3711, 0.4032, 0.6931, 0.6931])

This should have hopefully given you an overview of the losses submodule, if you have any questions, please feel free to reach out on our discord!