Ctrl+K

Layers#

Collection of Ivy neural network layers in functional form.

ivy.conv(x, filters, strides, padding, /, *, transpose=False, dims=2, output_shape=None, data_format='channel_last', filter_format='channel_last', feature_group_count=1, x_dilations=1, dilations=1, bias=None, out=None)[source]#

Compute a 1-D, 2-D, and 3-D transpose or dilated convolution given 3-D, 4-D and 5-D input x respectively and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,d,h,w,d_in] or [batch_size,d_in,d,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fd,fh,fw,d_in/feature_group_count,d_out].

  • strides (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (Union[str, Sequence[Tuple[int, int]]]) – either the string ‘SAME’ (padding with zeros evenly), the string ‘VALID’ (no padding), or a sequence of n (low, high) integer pairs that give the padding to apply before and after each spatial dimension.

  • transpose (bool, default: False) – True for computing transpose convolution, and False for dilated convolution. When True, x_dilations must be 1 (the default).

  • dims (int, default: 2) – Either 1, 2, or 3 corresponding to 1-D, 2-D, and 3-D convolution.

  • output_shape (Optional[Union[Shape, NativeShape]], default: None) – Shape of the output (Default value = None)

  • data_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “NCW”, “NCHW”, “NCDHW” input data formatS for 1-D, 2-D, 3-D convolution respectively, while “channel_last” corresponds to “NWC”, “NHWC”, “NDHWC” respectively.

  • filter_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “OIW”, “OIHW”, “OIDHW” input data formats for 1-D, 2-D, 3-D convolution respectively, while “channel_last” corresponds to “WIO”, “HWIO”, “DHWIO” respectively.

  • feature_group_count (int, default: 1) – split input into groups, d_in should be divisible by the number of groups. (Default value = 1)

  • x_dilations (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • dilations (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Union[Array, NativeArray]], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The result of the transpose or dilated convolution operation.

ivy.conv1d(x, filters, strides, padding, /, *, data_format='NWC', filter_format='channel_last', x_dilations=1, dilations=1, bias=None, out=None)[source]#

Compute a 1-D convolution given 3-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,w,d_in] or [batch_size,d_in,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fw,d_in,d_out].

  • strides (Union[int, Tuple[int]]) – The stride of the sliding window for each dimension of input.

  • padding (Union[str, int, Sequence[Tuple[int, int]]]) – either the string ‘SAME’ (padding with zeros evenly), the string ‘VALID’ (no padding), or a sequence of n (low, high) integer pairs that give the padding to apply before and after each spatial dimension.

  • data_format (str, default: 'NWC') – The ordering of the dimensions in the input, one of “NWC” or “NCW”. “NWC” corresponds to input with shape (batch_size, width, channels), while “NCW” corresponds to input with shape (batch_size, channels, width).

  • filter_format (str, default: 'channel_last') –

    Either “channel_first” or “channel_last”. “channel_first” corresponds to “OIW”,

    input data formats, while “channel_last” corresponds to “WIO”, “HWIO”, “DHWIO”.

    x_dilations

    The dilation factor for each dimension of input. (Default value = 1)

  • dilations (Union[int, Tuple[int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Array], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The result of the convolution operation.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.asarray([[[0.], [3.], [0.]]]) #NWC
>>> filters = ivy.array([[[0.]], [[1.]], [[0.]]]) #WIO
>>> result = ivy.conv1d(x, filters, (1,), 'SAME', data_format='NWC',dilations= (1,))
>>> print(result)
ivy.array([[[0.], [3.], [0.]]])

With ivy.NativeArray input:

>>> x = ivy.native_array([[[1., 3.], [2., 4.], [5., 7]]])
>>> filters = ivy.native_array([[[0., 1.], [1., 0.]]])
>>> result = ivy.conv1d(x, filters, (2,),'VALID')
>>> print(result)
ivy.array([[[3., 1.],
...         [7., 5.]]])

With a mix of ivy.Array and ivy.Container inputs:

>>> x = ivy.Container(a=ivy.array([[[1.2, 3.1, 4.8], [5.9, 2.2, 3.3],
...                                 [10.8, 7.6, 4.9], [6.1, 2.2, 9.5]]]),
...                   b=ivy.array([[[8.8, 7.7, 6.6], [1.1, 2.2, 3.5]]]))
>>> filters = ivy.array([[[1., 0., 1.], [0., 1., 0.], [1., 1., 0.]]])
>>> result  = ivy.conv1d(x, filters, 3, 'VALID')
>>> print(result)
{
        a: ivy.array([[[6., 7.9, 1.2],
...                    [15.6, 11.7, 6.1]]]),
...     b: ivy.array([[[15.4, 14.3, 8.8]]])
}
ivy.conv1d_transpose(x, filters, strides, padding, /, *, output_shape=None, filter_format='channel_last', data_format='NWC', dilations=1, bias=None, out=None)[source]#

Compute a 1-D transpose convolution given 3-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,w,d_in] or [batch_size,d_in,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fw,d_out,d_in].

  • strides (Union[int, Tuple[int]]) – The stride of the sliding window for each dimension of input.

  • padding (str) – Either ‘SAME’ (padding so that the output’s shape is the same as the input’s), or ‘VALID’ (padding so that the output’s shape is output_shape).

  • output_shape (Optional[Union[Shape, NativeShape]], default: None) – Shape of the output (Default value = None)

  • filter_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “IOW”,input data formats, while “channel_last” corresponds to “WOI”.

  • data_format (str, default: 'NWC') – The ordering of the dimensions in the input, one of “NWC” or “NCW”. “NWC” corresponds to input with shape (batch_size, width, channels), while “NCW” corresponds to input with shape (batch_size, channels, width).

  • dilations (Union[int, Tuple[int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Array], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The result of the transpose convolution operation.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 28, 3])
>>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 6, 3])
>>> y = ivy.conv1d_transpose(x, filters, 2, 'SAME')
>>> print(y.shape)
ivy.Shape(1, 56, 6)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 128, 64])
>>> filters = ivy.random_normal(mean=0, std=1, shape=[1, 64, 64])
>>> ivy.conv1d_transpose(x, filters, 1, 'VALID', out=x)
>>> print(x.shape)
ivy.Shape(1, 128, 64)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 256, 64])
>>> y = ivy.zeros((1, 258, 32))
>>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 32, 64])
>>> ivy.conv1d_transpose(x, filters, 1, 'VALID', out=y)
>>> print(y.shape)
ivy.Shape(1, 258, 32)

With ivy.NativeArray input:

>>> x = ivy.native_array(
...         ivy.random_normal(mean=0, std=1, shape=[1, 256, 128]))
>>> filters = ivy.native_array(
...         ivy.random_normal(mean=0, std=1, shape=[3, 32, 128]))
>>> y = ivy.conv1d_transpose(x, filters, 2, 'SAME')
>>> print(y.shape)
ivy.Shape(1, 512, 32)

With one ivy.Container input:

>>> x = ivy.full((1, 6, 1), 2.7)
>>> a = ivy.random_normal(mean=0, std=1, shape=[3, 1, 1])
>>> b = ivy.random_normal(mean=0, std=1, shape=[3, 1, 1])
>>> filters = ivy.Container(a=a, b=b)
>>> y = ivy.conv1d_transpose(x, filters, 1, 'VALID', dilations=2)
>>> print(y.shape)
{
    a: ivy.Shape(1, 10, 1),
    b: ivy.Shape(1, 10, 1)
}

With multiple ivy.Container inputs:

>>> a = ivy.random_normal(mean=0, std=1, shape=[1, 14, 3])
>>> b = ivy.random_normal(mean=0, std=1, shape=[1, 28, 3])
>>> c = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3])
>>> d = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3])
>>> x = ivy.Container(a=a, b=b)
>>> filters = ivy.Container(c=c, d=d)
>>> y = ivy.conv1d_transpose(x, filters, 2, 'SAME')
>>> print(y.shape)
{
    a: {
        c: ivy.Shape(1, 28, 3),
        d: ivy.Shape(1, 28, 3)
    },
    b: {
        c: ivy.Shape(1, 56, 3),
        d: ivy.Shape(1, 56, 3)
    },
    c: {
        c: ivy.Shape(6, 6, 3),
        d: ivy.Shape(6, 6, 3)
    },
    d: {
        c: ivy.Shape(6, 6, 3),
        d: ivy.Shape(6, 6, 3)
    }
}
ivy.conv2d(x, filters, strides, padding, /, *, data_format='NHWC', filter_format='channel_last', x_dilations=1, dilations=1, bias=None, out=None)[source]#

Compute a 2-D convolution given 4-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,h,w,d_in] or [batch_size,d_in,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fh,fw,d_in,d_out].

  • strides (Union[int, Tuple[int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (Union[str, int, Sequence[Tuple[int, int]]]) – either the string ‘SAME’ (padding with zeros evenly), the string ‘VALID’ (no padding), or a sequence of n (low, high) integer pairs that give the padding to apply before and after each spatial dimension.

  • data_format (str, default: 'NHWC') – The ordering of the dimensions in the input, one of “NHWC” or “NCHW”. “NHWC” corresponds to inputs with shape (batch_size, height, width, channels), while “NCHW” corresponds to input with shape (batch_size, channels, height, width).

  • filter_format (str, default: 'channel_last') –

    Either “channel_first” or “channel_last”. “channel_first” corresponds to “OIHW”,

    input data formats, while “channel_last” corresponds to “HWIO”.

    x_dilations

    The dilation factor for each dimension of input. (Default value = 1)

  • dilations (Union[int, Tuple[int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Array], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The result of the convolution operation.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.array([[[[1.], [2.0],[3.]],
...                 [[1.], [2.0],[3.]],
...                 [[1.], [2.0],[3.]]]])
>>> filters = ivy.array([[[[0.]],[[1.]],[[0.]]],
...                      [[[0.]],[[1.]], [[0.]]],
...                      [[[0.]],[[1.]], [[0.]]]])
>>> result = ivy.conv2d(x, filters, 1, 'SAME', data_format='NHWC', dilations=1)
>>> print(result)
ivy.array([[
          [[2.],[4.],[6.]],
          [[3.],[6.],[9.]],
          [[2.],[4.],[6.]]
          ]])

With one ivy.Container input:

>>> x = ivy.Container(a=ivy.array([[[[1.], [2.0],[3.]],
...                                 [[1.], [2.0],[3.]],
...                                 [[1.], [2.0],[3.]]]]))
>>> filters = ivy.eye(3, 3).reshape((3, 3, 1, 1)).astype(ivy.float32)
>>> result = ivy.conv2d(x, filters, 2, 'SAME', data_format='NHWC', dilations= 1)
>>> print(result)
{
    a:ivy.array([[[[3.], [3.]], [[1.], [5.]]]])
}

With multiple ivy.Container inputs:

>>> x = ivy.Container(a = ivy.eye(3, 3).reshape((1, 3, 3, 1)),
...                   b = ivy.eye(4, 4).reshape((1, 4, 4, 1)),
...                   c = ivy.eye(5, 5).reshape((1, 5, 5, 1)))
>>> filters = ivy.array([[1, 1, 1],
...                      [0, 1, 1],
...                      [0, 0, 1]], dtype = ivy.float32).reshape((3, 3, 1, 1))
>>> result = ivy.conv2d(x, filters, 2, 'SAME')
>>> print(result)
{
    a:ivy.array([[[[2.], [0.]], [[1.], [2.]]]]),
    b:ivy.array([[[[3.], [0.]], [[1.], [2.]]]]),
    c:ivy.array([[[[2.], [0.], [0.]],
                  [[1.], [3.], [0.]],
                  [[0.], [1.], [2.]]
                ]])
}

With a mix of ivy.Array and ivy.Container inputs:

>>> x = ivy.Container(a = ivy.eye(3, 3).reshape((1, 3, 3, 1)),
...                   b = ivy.eye(5, 5).reshape((1, 5, 5, 1)))
>>> filters = ivy.array([[2, 0, 1],
...                      [1, 3, 1],
...                      [0, 1, 1]], dtype = ivy.float32).reshape((3, 3, 1, 1))
>>> result = ivy.conv2d(x, filters, 2, 'SAME')
>>> print(result)
{
    a:ivy.array([[[[4.],[0.]],[[1.],[5.]]]]),
    b:ivy.array([[[[4.],[0.],[0.]],[[1.],[6.],[0.]],[[0.],[1.],[5.]]]])
}
ivy.conv2d_transpose(x, filters, strides, padding, /, *, output_shape=None, filter_format='channel_last', data_format='NHWC', dilations=1, bias=None, out=None)[source]#

Compute a 2-D transpose convolution given 4-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,h,w,d_in] or [batch_size,d_in,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fh,fw,d_out,d_in].

  • strides (Union[int, Tuple[int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (str) – Either ‘SAME’ (padding so that the output’s shape is the same as the input’s), or ‘VALID’ (padding so that the output’s shape is output_shape).

  • output_shape (Optional[Union[Shape, NativeShape]], default: None) – Shape of the output (Default value = None)

  • data_format (str, default: 'NHWC') – The ordering of the dimensions in the input, one of “NHWC” or “NCHW”. “NHWC” corresponds to inputs with shape (batch_size, height, width, channels), while “NCHW” corresponds to input with shape (batch_size, channels, height, width).

  • filter_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “IOHW”,input data formats, while “channel_last” corresponds to “HWOI”.

  • x_dilations – The dilation factor for each dimension of input. (Default value = 1)

  • dilations (Union[int, Tuple[int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Array], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The result of the transpose convolution operation.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input: >>> x = ivy.random_normal(mean=0, std=1, shape=[1, 28, 28, 3]) >>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 6, 3]) >>> y = ivy.conv2d_transpose(x,filters,2,’SAME’) >>> print(y.shape) ivy.Shape(1, 56, 56, 6)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 128, 128, 64])
>>> filters = ivy.random_normal(mean=0, std=1, shape=[1, 1, 64, 64])
>>> ivy.conv2d_transpose(x,filters,1,'VALID',out=x)
>>> print(x.shape)
ivy.Shape(1, 128, 128, 64)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 256, 256, 64])
>>> y = ivy.zeros((1, 258, 258, 32))
>>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 32, 64])
>>> ivy.conv2d_transpose(x,filters,[1, 1, 1],'VALID',out=y)
>>> print(y.shape)
ivy.Shape(1, 258, 258, 32)

With one ivy.Container inputs: >>> x = ivy.full((1, 6, 6, 1), 2.7) >>> a = ivy.random_normal(mean=0, std=1, shape=[3, 3, 1, 1]) >>> b = ivy.random_normal(mean=0, std=1, shape=[3, 3, 1, 1]) >>> filters = ivy.Container(a=a, b=b) >>> y = ivy.conv2d_transpose(x,filters,1,’VALID’,dilations=2) >>> print(y.shape) {

a: ivy.Shape(1, 10, 10, 1), b: ivy.Shape(1, 10, 10, 1)

}

With multiple ivy.Container inputs: >>> a = ivy.random_normal(mean=0, std=1, shape=[1, 14, 14, 3]) >>> b = ivy.random_normal(mean=0, std=1, shape=[1, 28, 28, 3]) >>> c = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3, 3]) >>> d = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3, 3]) >>> x = ivy.Container(a=a, b=b) >>> filters = ivy.Container(c=c, d=d) >>> y = ivy.conv2d_transpose(x,filters,2,’SAME’) >>> print(y.shape) {

a: {

c: ivy.Shape(1, 28, 28, 3), d: ivy.Shape(1, 28, 28, 3)

}, b: {

c: ivy.Shape(1, 56, 56, 3), d: ivy.Shape(1, 56, 56, 3)

}, c: {

c: ivy.Shape(6, 6, 6, 3), d: ivy.Shape(6, 6, 6, 3)

}, d: {

c: ivy.Shape(6, 6, 6, 3), d: ivy.Shape(6, 6, 6, 3)

}

}

ivy.conv3d(x, filters, strides, padding, /, *, data_format='NDHWC', filter_format='channel_last', x_dilations=1, dilations=1, bias=None, out=None)[source]#

Compute a 3-D convolution given 5-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray, Container]) – Input volume [batch_size,d,h,w,d_in] or [batch_size,d_in,d,h,w].

  • filters (Union[Array, NativeArray, Container]) – Convolution filters [fd,fh,fw,d_in,d_out].

  • strides (Union[int, Tuple[int, int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (Union[str, int, Sequence[Tuple[int, int]]]) – either the string ‘SAME’ (padding with zeros evenly), the string ‘VALID’ (no padding), or a sequence of n (low, high) integer pairs that give the padding to apply before and after each spatial dimension.

  • data_format (str, default: 'NDHWC') – The ordering of the dimensions in the input, one of “NDHWC” or “NCDHW”. “NDHWC” corresponds to inputs with shape (batch_size, depth, height, width, channels), while “NCDHW” corresponds to input with shape (batch_size, channels, depth, height, width).

  • filter_format (str, default: 'channel_last') –

    Either “channel_first” or “channel_last”. “channel_first” corresponds

    to “OIDHW”,input data formats, while “channel_last” corresponds to “DHWIO”.

    x_dilations

    The dilation factor for each dimension of input. (Default value = 1)

  • dilations (Union[int, Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Array], default: None) – Bias array of shape [d_out]

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The result of the convolution operation.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.array([[[1., 2. ,1.], [1., 2. ,1.], [1., 2. ,1.]],
...         [[1., 2. ,1.], [1., 2. ,1.], [1., 2. ,1.]],
...         [[1., 2. ,1.], [1., 2. ,1.], [1., 2. ,1.]]]).reshape((1, 3, 3, 3, 1))
>>> filters = ivy.array([[[0.,1.,0.],
...                       [0.,1.,0.],
...                       [0.,1.,0.]]]).reshape((1,3,3,1,1))
>>> result = ivy.conv3d(x, filters, 1, 'SAME', data_format='NDHWC', dilations=1)
>>> print(result)
ivy.array([[[[[2.],[4.],[2.]],[[3.],[6.],[3.]],[[2.],[4.],[2.]]],
            [[[2.],[4.],[2.]],[[3.],[6.],[3.]],[[2.],[4.],[2.]]],
            [[[2.],[4.],[2.]],[[3.],[6.],[3.]],[[2.],[4.],[2.]]]]])

With one ivy.Container input:

>>> x = ivy.Container(a = ivy.ones((1, 3, 3, 3, 1)).astype(ivy.float32))
>>> filters = ivy.ones((3, 3, 3, 1, 1)).astype(ivy.float32)
>>> result = ivy.conv3d(x, filters, 2, 'SAME')
>>> print(result)
{
    a: ivy.array([[[[[8.],[8.]],[[8.],[8.]]],[[[8.],[8.]],[[8.],[8.]]]]])
}

With multiple ivy.Container input:

>>> x = ivy.Container( a = ivy.random_normal(mean = 0, std = 1,
...                        shape = [1, 3, 5, 5, 1]),
...                    b = ivy.random_normal(mean = 0, std = 1,
...                        shape = [1, 5, 32 ,32, 1]),
...                    c = ivy.random_normal(mean = 0, std = 1,
...                        shape = [1, 32, 32, 32, 1]))
>>> filters = ivy.ones((3, 5, 5, 1, 3)).astype(ivy.float32)
>>> result = ivy.conv3d(x, filters, 1, 'SAME')
>>> print(result.cont_shapes)
{
    a: ivy.Shape(1, 3, 5, 5, 3),
    b: ivy.Shape(1, 5, 32, 32, 3),
    c: ivy.Shape(1, 32, 32, 32, 3)
}
ivy.conv3d_transpose(x, filters, strides, padding, /, *, output_shape=None, filter_format='channel_last', data_format='NDHWC', dilations=1, bias=None, out=None)[source]#

Compute a 3-D transpose convolution given 5-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input volume [batch_size,d,h,w,d_in] or [batch_size,d_in,d,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fd,fh,fw,d_out,d_in].

  • strides (Union[int, Tuple[int, int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (str) – Either ‘SAME’ (padding so that the output’s shape is the same as the input’s), or ‘VALID’ (padding so that the output’s shape is output_shape).

  • output_shape (Optional[Union[Shape, NativeShape]], default: None) – Shape of the output (Default value = None)

  • filter_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “IODHW”,input data formats, while “channel_last” corresponds to “DHWOI”.

  • data_format (str, default: 'NDHWC') – The ordering of the dimensions in the input, one of “NDHWC” or “NCDHW”. “NDHWC” corresponds to inputs with shape (batch_size, depth, height, width, channels), while “NCDHW” corresponds to input with shape (batch_size, channels, depth, height, width).

  • dilations (Union[int, Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • bias (Optional[Array], default: None) – Bias array of shape [d_out]

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The result of the transpose convolution operation.

Examples

With ivy.Array input:

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 3, 28, 28, 3])
>>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 6, 3])
>>> y = ivy.conv3d_transpose(x, filters, [2, 2, 2], 'SAME')
>>> print(y.shape)
ivy.Shape(1, 6, 56, 56, 6)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 3, 64, 64, 3])
>>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 6, 3])
>>> y = ivy.conv3d_transpose(x, filters, [2, 2, 2], 'VALID', dilations=[1, 1, 1])
>>> print(y.shape)
ivy.Shape(1, 7, 129, 129, 6)

With ivy.Container inputs:

>>> a = ivy.random_normal(mean=0, std=1, shape=[1, 3, 14, 14, 3])
>>> b = ivy.random_normal(mean=0, std=1, shape=[1, 3, 28, 28, 3])
>>> c = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3, 3, 3])
>>> d = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3, 3, 3])
>>> x = ivy.Container(a=a, b=b)
>>> filters = ivy.Container(c=c, d=d)
>>> y = ivy.conv3d_transpose(x, filters, [2, 2, 2], 'SAME')
>>> print(y.shape)
{
    a: {
        c: ivy.Shape(1, 6, 28, 28, 3),
        d: ivy.Shape(1, 6, 28, 28, 3)
    },
    b: {
        c: ivy.Shape(1, 6, 56, 56, 3),
        d: ivy.Shape(1, 6, 56, 56, 3)
    },
    c: {
        c: ivy.Shape(6, 6, 6, 6, 3),
        d: ivy.Shape(6, 6, 6, 6, 3)
    },
    d: {
        c: ivy.Shape(6, 6, 6, 6, 3),
        d: ivy.Shape(6, 6, 6, 6, 3)
    }
}

With a mix of ivy.Array and ivy.Container inputs:

>>> x = ivy.full((1, 6, 6, 6, 1), 2.7)
>>> a = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1])
>>> b = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1])
>>> filters = ivy.Container(a=a, b=b)
>>> y = ivy.conv3d_transpose(x, filters, [1, 1, 1], 'VALID', dilations=[1, 1, 1])
>>> print(y.shape)
{
    a: ivy.Shape(1, 8, 8, 8, 1),
    b: ivy.Shape(1, 8, 8, 8, 1)
}

>>> x = ivy.full((1, 6, 6, 6, 1), 1.23)
>>> a = ivy.array(ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1]))
>>> b = ivy.array(ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1]))
>>> filters = ivy.Container(a=a, b=b)
>>> y = ivy.conv3d_transpose(x, filters, [1, 1, 1], 'VALID', dilations=[1, 1, 1])
>>> print(y.shape)
{
    a: ivy.Shape(1, 8, 8, 8, 1),
    b: ivy.Shape(1, 8, 8, 8, 1)
}
ivy.conv_general_dilated(x, filters, strides, padding, /, *, dims=2, data_format='channel_last', filter_format='channel_last', feature_group_count=1, x_dilations=1, dilations=1, bias=None, out=None)[source]#

Compute a 1-D, 2-D, and 3-D convolution given 3-D, 4-D and 5-D input x respectively and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,d,h,w,d_in] or [batch_size,d_in,d,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fd,fh,fw,d_in/feature_group_count,d_out].

  • strides (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (Union[str, int, Sequence[Tuple[int, int]]]) – either the string ‘SAME’ (padding with zeros evenly), the string ‘VALID’ (no padding), or a sequence of n (low, high) integer pairs that give the padding to apply before and after each spatial dimension.

  • dims (int, default: 2) – Either 1, 2, or 3 corresponding to 1-D, 2-D, and 3-D convolution.

  • data_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “NCW”, “NCHW”, “NCDHW” input data formatS for 1-D, 2-D, 3-D convolution respectively, while “channel_last” corresponds to “NWC”, “NHWC”, “NDHWC” respectively.

  • filter_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “OIW”, “OIHW”, “OIDHW” input data formats for 1-D, 2-D, 3-D convolution respectively, while “channel_last” corresponds to “WIO”, “HWIO”, “DHWIO” respectively.

  • feature_group_count (int, default: 1) – split input into groups, d_in should be divisible by the number of groups. (Default value = 1)

  • x_dilations (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • dilations (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of filter. (Default value = 1)

  • bias (Optional[Union[Array, NativeArray]], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The result of the transpose convolution operation.

ivy.conv_general_transpose(x, filters, strides, padding, /, *, dims=2, output_shape=None, filter_format='channel_last', data_format='channel_last', dilations=1, feature_group_count=1, bias=None, out=None)[source]#

Compute a 1-D, 2-D, and 3-D transpose convolution given 3-D, 4-D and 5-D input x respectively and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,d,h,w,d_in] or [batch_size,d_in,d,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fd,fh,fw,d_out,d_in].

  • strides (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (str) – Either ‘SAME’ (padding so that the output’s shape is the same as the input’s), or ‘VALID’ (padding so that the output’s shape is output_shape).

  • dims (int, default: 2) – Either 1, 2, or 3 corresponding to 1-D, 2-D, and 3-D convolution.

  • output_shape (Optional[Union[Shape, NativeShape]], default: None) – Shape of the output.

  • filter_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “IODHW”,input data formats, while “channel_last” corresponds to “DHWOI”.

  • data_format (str, default: 'channel_last') – Either “channel_first” or “channel_last”. “channel_first” corresponds to “NCW”, “NCHW”, “NCDHW” input data formatS for 1-D, 2-D, 3-D convolution respectively, while “channel_last” corresponds to “NWC”, “NHWC”, “NDHWC” respectively.

  • dilations (Union[int, Tuple[int], Tuple[int, int], Tuple[int, int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • feature_group_count (int, default: 1) – split input into groups, d_in should be divisible by the number of groups.

  • bias (Optional[Union[Array, NativeArray]], default: None) – Bias array of shape [d_out].

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

ret – The result of the transpose convolution operation.

Examples

With ivy.Array input: >>> x = ivy.random_normal(mean=0, std=1, shape=[1, 3, 28, 28, 3]) >>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 6, 3]) >>> y = ivy.conv3d_transpose(x, filters, [2, 2, 2], ‘SAME’) >>> print(y.shape) ivy.Shape(1, 6, 56, 56, 6) >>> x = ivy.random_normal(mean=0, std=1, shape=[1, 3, 64, 64, 3]) >>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 6, 3]) >>> y = ivy.conv3d_transpose(x, filters, [2, 2, 2], ‘VALID’, dilations=[1, 1, 1]) >>> print(y.shape) ivy.Shape(1, 7, 129, 129, 6) With :class: ‘ivy.Container’ inputs: >>> a = ivy.random_normal(mean=0, std=1, shape=[1, 3, 14, 14, 3]) >>> b = ivy.random_normal(mean=0, std=1, shape=[1, 3, 28, 28, 3]) >>> c = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3, 3, 3]) >>> d = ivy.random_normal(mean=0, std=1, shape=[6, 3, 3, 3, 3]) >>> x = ivy.Container(a=a, b=b) >>> filters = ivy.Container(c=c, d=d) >>> y = ivy.conv3d_transpose(x, filters, [2, 2, 2], ‘SAME’) >>> print(y.shape) {

a: {

c: ivy.Shape(1, 6, 28, 28, 3), d: ivy.Shape(1, 6, 28, 28, 3)

}, b: {

c: ivy.Shape(1, 6, 56, 56, 3), d: ivy.Shape(1, 6, 56, 56, 3)

}, c: {

c: ivy.Shape(6, 6, 6, 6, 3), d: ivy.Shape(6, 6, 6, 6, 3)

}, d: {

c: ivy.Shape(6, 6, 6, 6, 3), d: ivy.Shape(6, 6, 6, 6, 3)

}

} With a mix of ivy.Array and ivy.Container inputs: >>> x = ivy.full((1, 6, 6, 6, 1), 2.7) >>> a = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1]) >>> b = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1]) >>> filters = ivy.Container(a=a, b=b) >>> y = ivy.conv3d_transpose(x, filters, [1, 1, 1], ‘VALID’, dilations=[1, 1, 1]) >>> print(y.shape) {

a: ivy.Shape(1, 8, 8, 8, 1), b: ivy.Shape(1, 8, 8, 8, 1)

} >>> x = ivy.full((1, 6, 6, 6, 1), 1.23) >>> a = ivy.array(ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1])) >>> b = ivy.array(ivy.random_normal(mean=0, std=1, shape=[3, 3, 3, 1, 1])) >>> filters = ivy.Container(a=a, b=b) >>> y = ivy.conv3d_transpose(x, filters, [1, 1, 1], ‘VALID’, dilations=[1, 1, 1]) >>> print(y.shape) {

a: ivy.Shape(1, 8, 8, 8, 1), b: ivy.Shape(1, 8, 8, 8, 1)

}

ivy.depthwise_conv2d(x, filters, strides, padding, /, *, data_format='NHWC', dilations=1, out=None)[source]#

Compute a 2-D depthwise convolution given 4-D input x and filters arrays.

Parameters:

  • x (Union[Array, NativeArray]) – Input image [batch_size,h,w,d_in] or [batch_size,d_in,h,w].

  • filters (Union[Array, NativeArray]) – Convolution filters [fh,fw,d_in]. (d_in must be the same as d from x)

  • strides (Union[int, Tuple[int, int]]) – The stride of the sliding window for each dimension of input.

  • padding (Union[str, Sequence[Tuple[int, int]]]) – either the string ‘SAME’ (padding with zeros evenly), the string ‘VALID’ (no padding), or a sequence of n (low, high) integer pairs that give the padding to apply before and after each spatial dimension.

  • data_format (str, default: 'NHWC') – The ordering of the dimensions in the input, one of “NHWC” or “NCHW”. “NHWC” corresponds to inputs with shape (batch_size, height, width, channels), while “NCHW” corresponds to input with shape (batch_size, channels, height, width).

  • dilations (Union[int, Tuple[int, int]], default: 1) – The dilation factor for each dimension of input. (Default value = 1)

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The result of the convolution operation.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 28, 28, 3])
>>> filters = ivy.random_normal(mean=0, std=1, shape=[3, 3, 3])
>>> y = ivy.depthwise_conv2d(x, filters, (1, 1), 'VALID')
>>> print(y.shape)
ivy.Shape(1, 26, 26, 3)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 32, 32, 3])
>>> y = ivy.zeros((1, 16, 16, 3))
>>> filters = ivy.random_normal(mean=0, std=1, shape=[5, 5, 3])
>>> ivy.depthwise_conv2d(x, filters, [2, 2], 'SAME', out=y)
>>> print(y.shape)
ivy.Shape(1, 16, 16, 3)

>>> x = ivy.random_normal(mean=0, std=1, shape=[1, 64, 64, 32])
>>> y = ivy.zeros((1, 61, 61, 32))
>>> filters = ivy.random_normal(mean=0, std=1, shape=[4, 4, 32])
>>> ivy.depthwise_conv2d(x, filters, [1, 1], 'VALID', out=y)
>>> print(x.shape)
ivy.Shape(1, 64, 64, 32)

With ivy.NativeArray input:

>>> x = ivy.native_array(ivy.random_normal(mean=0, std=1, shape=[1, 7, 7, 64]))
>>> filters = ivy.native_array(ivy.random_normal(mean=0, std=1, shape=[3, 3, 64]))
>>> y = ivy.depthwise_conv2d(x, filters, [1, 1], 'SAME')
>>> print(y.shape)
ivy.Shape(1, 7, 7, 64)

With a mix of ivy.Array and ivy.Container inputs:

>>> x = ivy.eye(6, 6).reshape((1, 6, 6, 1)) #NHWC
>>> a = ivy.array([[1., 1., 1.], [1., -8., 1.], [1., 1., 1.]]).expand_dims(axis=-1)
>>> b = ivy.array([[1., 1., 1.],
...                [1., 1., 1.],
...                [1., 1., 1.]]).expand_dims(axis=-1) / 9.0
>>> filters = ivy.Container(a = a, b = b)
>>> y = ivy.depthwise_conv2d(x, filters, 1, 'VALID', dilations=2)
>>> print(y)
{
    a: ivy.array([[[[-6.],
                    [0.]],
                   [[0.],
                    [-6.]]]]),
    b: ivy.array([[[[0.33333334],
                    [0.]],
                   [[0.],
                    [0.33333334]]]])
}

With a mix of ivy.Array, code:ivy.NativeArray and ivy.Container inputs:

>>> x = ivy.eye(6, 6).reshape((1, 6, 6, 1)) #NHWC
>>> y = ivy.native_array(ivy.eye(6, 6).reshape((1, 6, 6, 1)))
>>> inp = ivy.Container(x = x, y = y)
>>> filter = ivy.array([[1., 1., 1.],
...                     [1., -8., 1.],
...                     [1., 1., 1.]]).expand_dims(axis=-1)
>>> y = ivy.depthwise_conv2d(inp, filter, 1, 'VALID', dilations=2)
>>> print(y)
{
    x: ivy.array([[[[-6.],
                    [0.]],
                   [[0.],
                    [-6.]]]]),
    y: ivy.array([[[[-6.],
                    [0.]],
                   [[0.],
                    [-6.]]]])
}
ivy.dropout(x, prob, /, *, scale=True, dtype=None, training=True, seed=None, noise_shape=None, out=None)[source]#

Randomly setting a fraction of input tensor to zeroes with probability.

prob at each update during training time to prevent possible overfitting. The inputs not set to 0 are scaled up 1 / (1 - prob) by default, so that overall sum is unchanged at training time and inference time.

Parameters:

  • x (Union[Array, NativeArray]) – The input array x to perform dropout on.

  • prob (float) – The probability of zeroing out each array element, float between 0 and 1.

  • scale (bool, default: True) – Whether to scale the output by 1/(1-prob). Default is True.

  • dtype (Optional[Union[Dtype, NativeDtype]], default: None) – output array data type. If dtype is None, the output array data type must be inferred from x. Default is None.

  • training (bool, default: True) – Turn on dropout if training, turn off otherwise. Default is True.

  • seed (Optional[int], default: None) – Set a default seed for random number generating (for reproducibility). Default is None.

  • noise_shape (Optional[Sequence[int]], default: None) – a sequence representing the shape of the binary dropout mask that will be multiplied with the input. A shape dimension set to None means that a different mask value will be applied to each element of the input across that dimension. A dimension set to 1 means the same mask value will be applied to all elements of the input across that dimension.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – Result array after dropout is performed.

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.array([[1., 2., 3.],
...                [4., 5., 6.],
...                [7., 8., 9.],
...                [10., 11., 12.]])
>>> y = ivy.dropout(x,0.3)
>>> print(y)
ivy.array([[ 1.42857146,  2.85714293,  4.28571415],
       [ 0.        ,  7.14285755,  8.5714283 ],
       [10.        , 11.4285717 ,  0.        ],
       [14.2857151 ,  0.        , 17.1428566 ]])

>>> x = ivy.array([[1.5, 2.6],
...                [4.9, 6.6],
...                [7.2, 8.7]])
>>> y = ivy.dropout(x,0.5)
>>> print(y)
ivy.array([[ 0.        ,  5.19999981],
           [ 0.        ,  0.        ],
           [ 0.        , 17.39999962]])

>>> x = ivy.array([[1., 2., 3.],
...                [4., 5., 6.],
...                [7., 8., 9.],
...                [10., 11., 12.]])
>>> y = ivy.dropout(x,0.3,scale=False)
>>> print(y)
ivy.array([[ 1.,  2., 3.],
           [ 4.,  5., 0.],
           [ 7.,  0., 9.],
           [10., 11., 0.]])

>>> x = ivy.array([[1.5, 2.6],
...                [4.9, 6.6],
...                [7.2, 8.7]])
>>> y = ivy.dropout(x,0.5,scale=False)
>>> print(y)
ivy.array([[0., 2.6],
           [0., 0. ],
           [0., 8.7]])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([[1., 2., 3.], [4., 5., 6.]]),
...                   b=ivy.array([7., 8., 9.]))
>>> y = ivy.dropout(x,0.3)
>>> print(y)
{
a: ivy.array([[0., 0., 4.28571415],
              [5.71428585, 7.14285755, 0.]]),
b: ivy.array([0., 11.4285717, 12.8571434])
}

>>> x = ivy.Container(a=ivy.array([[1.1, 2.2, 3.3], [11., 22., 33.]]),
...                   b=ivy.array([[1.245, 0.278, 4.105], [7., 13., 17.]]))
>>> y = ivy.dropout(x,0.5)
>>> print(y)
{
    a: ivy.array([[0., 4.4000001, 6.5999999],
                  [22., 44., 0.]]),
    b: ivy.array([[2.49000001, 0.55599999, 8.21000004],
                  [14., 0., 0.]])
}

>>> x = ivy.Container(a=ivy.array([[1., 2., 3.], [4., 5., 6.]]),
...                   b=ivy.array([7., 8., 9.]))
>>> y = ivy.dropout(x,0.3)
>>> print(y)
{
    a: ivy.array([[0., 0., 3.],
                  [4., 5., 0.]]),
    b: ivy.array([0., 8., 9.])
}

>>> x = ivy.Container(a=ivy.array([[1.1, 2.2, 3.3], [11., 22., 33.]]),
...                   b=ivy.array([[1.245, 0.278, 4.105], [7., 13., 17.]]))
>>> y = ivy.dropout(x,0.5)
>>> print(y)
{
    a: ivy.array([[0., 2.2, 3.3],
                  [11., 22., 0.]]),
    b: ivy.array([[1.245, 0.278, 4.105],
                  [7., 0., 0.]])
}
ivy.linear(x, weight, /, *, bias=None, out=None)[source]#

Apply a linear transformation to the incoming data: y = x * t(weight) + bias. The operation also supports batching of the weight matrices. This is useful if a batch of different network parameters are to be represented.

Parameters:

  • x (Union[Array, NativeArray]) – The input x to compute linear transformation on. [outer_batch_shape,inner_batch_shape,in_features]

  • weight (Union[Array, NativeArray]) – The weight matrix. [outer_batch_shape,out_features,in_features]

  • bias (Optional[Union[Array, NativeArray]], default: None) – The bias vector, default is None. [outer_batch_shape,out_features]

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – Result array of the linear transformation. [outer_batch_shape,inner_batch_shape,out_features]

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> x = ivy.array([1., 2., 3.])
>>> w = ivy.array([[1., 0., 0.]])
>>> y = ivy.linear(x, w)
>>> print(y)
ivy.array([1.])

>>> x = ivy.array([[0.666, -0.4269, 1.911]])
>>> w = ivy.array([[1., 0., 0.], [0., 0., 1.]])
>>> y = ivy.zeros((1, 2))
>>> ivy.linear(x, w, out=y)
>>> print(y)
ivy.array([[0.666, 1.91 ]])

>>> x = ivy.array([[1.546, 5.234, 6.487],
...                [0.157, 5.753, 4.52],
...                [5.165, 3.159, 7.101]])
>>> w = ivy.array([[1.545, 2.547, 3.124],
...                [5.852, 8.753, 6.963]])
>>> b = ivy.array([-1., 1.])
>>> y = ivy.zeros((3, 2))
>>> ivy.linear(x, w, bias=b, out=y)
>>> print(y)
ivy.array([[ 34.98495483, 101.0293808 ],
       [ 28.0159359 ,  83.74752808],
       [ 37.20942307, 108.3205719 ]])

With ivy.Container input:

>>> x = ivy.Container(a=ivy.array([[1., 2., 3.],
...                                [4., 5., 6.]]),
...                   b=ivy.array([1.1, 2.2, 3.3]))
>>> w = ivy.Container(a=ivy.array([[1., 2., 3.],
...                                [-1., 1., 2.]]),
...                   b=ivy.array([[0., -1., 1.],
...                                [0., 1., 1.]]))
>>> b = ivy.Container(a=ivy.array([1., -1.]), b=ivy.array([1., 1.]))
>>> y = ivy.linear(x, w, bias=b)
>>> print(y)
{
    a: ivy.array([[15., 6.],
                  [33., 12.]]),
    b: ivy.array([2.1, 6.5])
}

With a mix of ivy.Array and ivy.Container inputs:

>>> x = ivy.Container(a=ivy.array([[1.1, 2.2, 3.3],
...                                [11., 22., 33.]]),
...                   b=ivy.array([[1.245, 0.278, 4.105],
...                                [7., 13., 17.]]))
>>> w = ivy.array([[1., 2., 3.],
...                [4., 5., 6.],
...                [7., 8., 9.]])
>>> b = ivy.Container(a=ivy.array([1., 0., -1.]),
...                   b=ivy.array([1., 1., 0.]))
>>> ivy.linear(x, w, bias=b, out=x)
>>> print(x)
{
    a: ivy.array([[16.4, 35.2, 54.],
                  [155., 352., 549.]]),
    b: ivy.array([[15.1, 32., 47.9],
                  [85., 196., 306.]])
}
ivy.lstm(input, initial_states, all_weights, num_layers, dropout, train, bidirectional, batch_first=False, batch_sizes=None, weights_transposed=False, has_ih_bias=True, has_hh_bias=True)[source]#

Applies a multi-layer long-short term memory to an input sequence.

Parameters:

  • input (Array) – input array of shape (seq_len, batch, input_size) when batch_first is False or (batch, seq_len, input_size) when batch_first is True

  • initial_states (Tuple[Array]) –

    tuple of two arrays (h_0, c_0) where h_0 is the initial hidden state of shape (num_layers * num_directions, batch, hidden_size) and c_0 is the initial cell state of shape (num_layers * num_directions, batch, hidden_size)

    (num_directions being 2 when bidirectional, otherwise 1)

  • all_weights (Tuple[Array]) –

    tuple of arrays representing the learnable weights of the lstm, with each layer having up to four arrays (w_ih, w_hh, b_ih, b_hh) representing the weights and biases (if biases are being used)

    w_ih: weight of shape (4 * hidden_size, input_size) w_hh: weight of shape (4 * hidden_size, hidden_size) b_ih: bias of shape (4 * hidden_size,) b_hh: bias of shape (4 * hidden_size,)

  • num_layers (int) – number of layers for the lstm to use

  • dropout (float) – dropout rate

  • train (bool) – whether to run the lstm in train mode or eval mode

  • bidirectional (bool) – whether the lstm is bidirectional or unidirectional

  • batch_first (bool, default: False) – defines the data format of the input and output arrays

  • batch_sizes (Optional[Sequence], default: None) – specifies the batch size at each timestep, when the input is a packed sequence

  • weights_transposed (bool, default: False) – whether the weights are transposed compared to the format in which they are expected (input_size, 4 * hidden_size) rather than (4 * hidden_size, input_size)

  • has_ih_bias (bool, default: True) – whether the all_weights argument includes a input-hidden bias

  • has_hh_bias (bool, default: True) – whether the all_weights argument includes a hidden-hidden bias

Returns:

  • output – output array of shape (seq_len, batch, num_directions * hidden_size) or (batch, seq_len, num_directions * hidden_size), depending on batch_first

  • h_outs – final hidden state of shape (num_layers * num_directions, batch, hidden_size)

  • c_outs – final cell state of shape (num_layers * num_directions, batch, hidden_size)

ivy.lstm_update(x, init_h, init_c, kernel, recurrent_kernel, /, *, bias=None, recurrent_bias=None, time_major=False)[source]#

Perform long-short term memory update by unrolling time dimension of input array.

Parameters:

  • x (Union[Array, NativeArray]) – input tensor of LSTM layer [batch_shape, t, in] if time_major=False, else [t, batch_shape, in].

  • init_h (Union[Array, NativeArray]) – initial state tensor for the cell output [batch_shape, out].

  • init_c (Union[Array, NativeArray]) – initial state tensor for the cell hidden state [batch_shape, out].

  • kernel (Union[Array, NativeArray]) – weights for cell kernel [in, 4 x out].

  • recurrent_kernel (Union[Array, NativeArray]) – weights for cell recurrent kernel [out, 4 x out].

  • bias (Optional[Union[Array, NativeArray]], default: None) – bias for cell kernel [4 x out]. (Default value = None)

  • recurrent_bias (Optional[Union[Array, NativeArray]], default: None) – bias for cell recurrent kernel [4 x out]. (Default value = None)

  • time_major (bool, default: False) – whether or not the input tensor x has the time dimension before batch dim.

Return type:

Tuple[Array, Tuple[Array, Array]]

Returns:

ret – hidden state for all timesteps of shape [batch_shape,t,out] if time_major is False, else [t, batch_shape, out], and a tuple containing the final cell states, both of shape [batch_shape,out].

ivy.multi_head_attention(query, /, *, key=None, value=None, batch_first=True, num_heads=8, scale=None, attention_mask=None, in_proj_weights=None, q_proj_weights=None, k_proj_weights=None, v_proj_weights=None, out_proj_weights=None, in_proj_bias=None, out_proj_bias=None, is_causal=False, key_padding_mask=None, bias_k=None, bias_v=None, static_k=None, static_v=None, add_zero_attn=False, return_attention_weights=False, average_attention_weights=True, dropout=0.0, training=False, out=None)[source]#

Apply multi-head attention to inputs x. This is an implementation of multi-headed attention as described in the paper “Attention is all you Need” (Vaswani et al., 2017). If query, key, value are the same, then this is self-attention. Each timestep in query attends to the corresponding sequence in key, and returns a fixed-width vector. This layer first projects query, key and value. These are (effectively) a list of tensors of length num_attention_heads, where the corresponding shapes are (batch_size, <query dimensions>, key_dim), (batch_size, <key/value dimensions>, key_dim), (batch_size, <key/value dimensions>, value_dim). Then, the query and key tensors are dot-producted and scaled. These are softmaxed to obtain attention probabilities. The value tensors are then interpolated by these probabilities, then concatenated back to a single tensor. Finally, the result tensor with the last dimension as value_dim can take a linear projection and return.

Parameters:

  • query (Union[Array, NativeArray]) – The query embeddings. Shape: (L, Q) or (N, L, Q), where L is the number of queries, N is the batch size, Q is the query embedding dimension.

  • key (Optional[Union[Array, NativeArray]], default: None) – The key embeddings. Shape: (S, K) or (N, S, K), where S is the number of keys, N is the batch size, K is the key embedding dimension.

  • value (Optional[Union[Array, NativeArray]], default: None) – The value embeddings. Shape (S, V) or (N, S, V), where S is the number of keys, N is the batch size, V is the value embedding dimension.

  • batch_first (bool, default: True) – If False, query, key and value will have shapes (L, N, Q), (S, N, K) and (S, N, V) respectively (if batched).

  • num_heads (int, default: 8) – The number of attention heads to use.

  • scale (Optional[float], default: None) – The value by which to scale the query-key similarity measure before softmax.

  • attention_mask (Optional[Union[Array, NativeArray]], default: None) – The mask to apply to the query-key values. Shape: (L, S) or (N*num_heads, L, S).

  • in_proj_weights (Optional[Union[Array, NativeArray]], default: None) – The weights used to project query, key and value. Shape: (3*E, E’), where E is the new embedding dimension and E’ is the input embedding dimension, i.e. E’ = Q = K = V.

  • q_proj_weights (Optional[Union[Array, NativeArray]], default: None) – The weights used to project query if in_proj_weights is None. Shape: (E, Q).

  • k_proj_weights (Optional[Union[Array, NativeArray]], default: None) – The weights used to project key if in_proj_weights is None. Shape: (E, K).

  • v_proj_weights (Optional[Union[Array, NativeArray]], default: None) – The weights used to project value if in_proj_weights is None. Shape: (E, V).

  • out_proj_weights (Optional[Union[Array, NativeArray]], default: None) – The weights used to project the attention output. Shape: (O, E), where O is the output embedding dimension.

  • in_proj_bias (Optional[Union[Array, NativeArray]], default: None) – The bias used when projecting query, key and value. Shape: (3*E,).

  • out_proj_bias (Optional[Union[Array, NativeArray]], default: None) – The bias used when projecting the output. Shape: (O,).

  • is_causal (bool, default: False) – If True, use a causal attention mask and ignore the provided attention_mask.

  • key_padding_mask (Optional[Union[Array, NativeArray]], default: None) – A binary mask to apply to the key sequence. Shape: (S,) or (N, S).

  • bias_k (Optional[Union[Array, NativeArray]], default: None) – An additional bias added to the key sequence. Shape: (E,).

  • bias_v (Optional[Union[Array, NativeArray]], default: None) – An additional bias added to the value sequence. Shape: (E,).

  • static_k (Optional[Union[Array, NativeArray]], default: None) – A static key to be used in the attention operators. Shape: (N*num_heads, S, E//num_heads).

  • static_v (Optional[Union[Array, NativeArray]], default: None) – A static value to be used in the attention operators. Shape: (N*num_heads, S, E//num_heads).

  • add_zero_attn (bool, default: False) – A boolean flag indicating whether to add a batch of zeros to key and value.

  • return_attention_weights (bool, default: False) – If True, return the attention weights alongside the attention output.

  • average_attention_weights (bool, default: True) – If True, the returned attention weights will be averaged across heads. Otherwise, the attention weights will be provided separately per head. Note that this flag only has an effect when return_attention_weights=True.

  • dropout (float, default: 0.0) – Specifies the dropout probability. Dropout is applied on the attention weights.

  • training (bool, default: False) – If True, dropout is used, otherwise dropout is not activated.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Union[Array, NativeArray]

Returns:

  • ret – The output following the application of multi-head attention. Either output or (output, attention_weights). output will have shape (L, E) if the inputs were unbatched or (N, L, E) otherwise, and attention_weights will have shape (L, S) or (N, L, S) respectively. If batch_first is False and the inputs were batched, the output will have shape (L, N, E).

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

ivy.nms(boxes, scores=None, iou_threshold=0.5, max_output_size=None, score_threshold=-inf)[source]#
ivy.roi_align(input, boxes, output_size, spatial_scale=1.0, sampling_ratio=-1, aligned=False)[source]#
ivy.scaled_dot_product_attention(query, key, value, /, *, scale=None, mask=None, dropout_p=0.0, is_causal=False, training=False, out=None)[source]#

Apply scaled dot product attention to inputs x using optional mask.

Parameters:

  • query (Union[Array, NativeArray]) – The queries input array. The shape of queries input array should be in [batch_shape,num_queries,feat_dim]. The queries input array should have the same size as keys and values.

  • key (Union[Array, NativeArray]) – The keys input array. The shape of keys input array should be in [batch_shape,num_keys,feat_dim]. The keys input array should have the same size as queries and values.

  • value (Union[Array, NativeArray]) – The values input array. The shape of values input should be in [batch_shape,num_keys,feat_dim]. The values input array should have the same size as queries and keys.

  • scale (Optional[float], default: None) – The scale float value. The scale float value is used to scale the query-key pairs before softmax.

  • mask (Optional[Union[Array, NativeArray]], default: None) – The mask input array. The mask to apply to the query-key values. Default is None. The shape of mask input should be in [batch_shape,num_queries,num_keys].

  • dropout_p (Optional[float], default: 0.0) – Specifies the dropout probability, if greater than 0.0, dropout is applied

  • is_causal (Optional[bool], default: False) – If true, assumes causal attention masking and errors if both mask and is_causal are set.

  • training (Optional[bool], default: False) – If True, dropout is used, otherwise dropout is not activated.

  • out (Optional[Array], default: None) – optional output array, for writing the result to. It must have a shape that the inputs broadcast to.

Return type:

Array

Returns:

  • ret – The output following application of scaled dot-product attention. The output array is the weighted sum produced by the attention score and value. The shape of output array is [batch_shape,num_queries,feat_dim] .

  • Both the description and the type hints above assumes an array input for simplicity,

  • but this function is nestable, and therefore also accepts ivy.Container

  • instances in place of any of the arguments.

Examples

With ivy.Array input:

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.],[4.4, 5.6]]])
>>> k = ivy.array([[[0.6, 1.5], [2.4, 3.3],[4.2, 5.1]]])
>>> v = ivy.array([[[0.4, 1.3], [2.2, 3.1],[4.3, 5.3]]])
>>> result = ivy.scaled_dot_product_attention(q,
...                                           k,
...                                           v,
...                                           scale=1,
...                                           dropout_p=0.1,
...                                           is_causal=True,
...                                           training=True)
>>> print(result)

ivy.array([[[0.40000001, 1.29999995], … [2.19994521, 3.09994531], … [4.30000019, 5.30000019]]])

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.],[4.4, 5.6]]])
>>> k = ivy.array([[[0.6, 1.5], [2.4, 3.3],[4.2, 5.1]]])
>>> v = ivy.array([[[0.4, 1.3], [2.2, 3.1],[4.3, 5.3]]])
>>> mask = ivy.array([[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0],[0.0, 0.0, 0.0]]])
>>> result = ivy.scaled_dot_product_attention(q,k,v,scale=1,mask=mask)
>>> print(result)
ivy.array([[[2.30000019, 3.23333359],

[2.30000019, 3.23333359], [2.30000019, 3.23333359]]])

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.], [4.4, 5.6]]])
>>> k = ivy.array([[[0.6, 1.5], [2.4, 3.3], [4.2, 5.1]]])
>>> v = ivy.array([[[0.4, 1.3], [2.2, 3.1], [4.3, 5.3]]])
>>> out = ivy.zeros(shape=(1, 3, 2))
>>> ivy.scaled_dot_product_attention(q,
...                                  k,
...                                  v,
...                                  scale=1,
...                                  dropout_p=0.1,
...                                  is_causal=True,
...                                  training=True,
...                                  out=out)
>>> print(out)

ivy.array([[[0.40000001, 1.29999995], … [2.19994521, 3.09994531], … [4.30000019, 5.30000019]]])

>>> q = ivy.native_array([[[0.2, 1.], [2.2, 3.],[4.4, 5.6]]])
>>> k = ivy.native_array([[[0.6, 1.5], [2.4, 3.3],[4.2, 5.1]]])
>>> v = ivy.native_array([[[0.4, 1.3], [2.2, 3.1],[4.3, 5.3]]])
>>> mask = ivy.native_array([[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0],[0.0, 0.0, 0.0]]])
>>> result = ivy.scaled_dot_product_attention(q,k,v,scale=1,mask=mask)
>>> print(result)

ivy.array([[[2.30000019, 3.23333359], … [2.30000019, 3.23333359], … [2.30000019, 3.23333359]]])

>>> q = ivy.native_array([[[0.2, 1.], [2.2, 3.], [4.4, 5.6]]])
>>> k = ivy.native_array([[[0.6, 1.5], [2.4, 3.3], [4.2, 5.1]]])
>>> v = ivy.native_array([[[0.4, 1.3], [2.2, 3.1], [4.3, 5.3]]])
>>> out = ivy.zeros(shape=(1, 3, 2))
>>> ivy.scaled_dot_product_attention(q,
...                                  k,
...                                  v,
...                                  scale=1,
...                                  dropout_p=0.1,
...                                  is_causal=True,
...                                  training=True,
...                                  out=out)
>>> print(out)

ivy.array([[[0.40000001, 1.29999995], … [2.19994521, 3.09994531], … [4.30000019, 5.30000019]]])

With ivy.Container input:

>>> q = ivy.Container(a=ivy.array([[[0.2, 1.], [2.7, 3.], [4.4, 5.6]]]),
...                   b=ivy.array([[[1.2, 1.], [2.2, 3.], [4.4, 5.6]]]))
>>> k = ivy.Container(a=ivy.array([[[4.2, 1.], [2.2, 3.3], [4.4, 5.6]]]),
...                   b=ivy.array([[[3.2, 1.], [2.2, 3.6], [4.0, 5.6]]]))
>>> v = ivy.Container(a=ivy.array([[[5.2, 1.], [2.1, 3.], [4.4, 5.6]]]),
...                   b=ivy.array([[[0.2, 1.], [2.2, 3.], [4.4, 5.6]]]))
>>> result = ivy.scaled_dot_product_attention(q,
...                                           k,
...                                           v,
...                                           scale=1,
...                                           dropout_p=0.1,
...                                           is_causal=True,
...                                           training=True)
>>> print(result)
{
    a: ivy.array([[[5.19999981, 1.],
    ...            [2.59249449, 2.68226194],
    ...            [4.4000001, 5.5999999]]]),
    b: ivy.array([[[0.2, 1.],
    ...            [2.19603825, 2.9960382],
    ...            [4.4000001, 5.5999999]]])
}

>>> q = ivy.Container(a=ivy.array([[[0.2, 1.], [2.7, 3.], [4.4, 5.6]]]),
...                   b=ivy.array([[[1.2, 1.], [2.2, 3.], [4.4, 5.6]]]))
>>> k = ivy.Container(a=ivy.array([[[4.2, 1.], [2.2, 3.3], [4.4, 5.6]]]),
...                   b=ivy.array([[[3.2, 1.], [2.2, 3.6], [4.0, 5.6]]]))
>>> v = ivy.Container(a=ivy.array([[[5.2, 1.], [2.1, 3.], [4.4, 5.6]]]),
...                   b=ivy.array([[[0.2, 1.], [2.2, 3.], [4.4, 5.6]]]))
>>> mask = ivy.Container(
...     a=ivy.array([[[1.0, 1.0, 1.0],[1.0, 1.0, 1.0],[1.0, 1.0, 1.0]]]),
...     b=ivy.array([[[1.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0,1.0]]])
... )
>>> result = ivy.scaled_dot_product_attention(q,k,v,scale=1,mask=mask)
>>> print(result)
{
    a: ivy.array([[[4.26894283, 5.40236187],
    ...            [4.39999437, 5.59999037],
    ...            [4.4000001, 5.5999999]]]),
    b: ivy.array([[[4.35046196, 5.54282808],
    ...            [4.39989519, 5.5998764],
    ...            [4.4000001, 5.5999999]]])
}

With a mix of ivy.Array and ivy.NativeArray inputs:

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.],[4.4, 5.6]]])
>>> k = ivy.native_array([[[0.6, 1.5], [2.4, 3.3],[4.2, 5.1]]])
>>> v = ivy.native_array([[[0.4, 1.3], [2.2, 3.1],[4.3, 5.3]]])
>>> result = ivy.scaled_dot_product_attention(q,
...                                            k,
...                                            v,
...                                            scale=1,
...                                            dropout_p=0.1,
...                                            is_causal=True,
...                                            training=True)
>>> print(result)

ivy.array([[[0.40000001, 1.29999995], … [2.19994521, 3.09994531], … [4.30000019, 5.30000019]]])

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.], [4.4, 5.6]]])
>>> k = ivy.native_array([[[0.6, 1.5], [2.4, 3.3], [4.2, 5.1]]])
>>> v = ivy.native_array([[[0.4, 1.3], [2.2, 3.1], [4.3, 5.3]]])
>>> out = ivy.zeros(shape=(1, 3, 2))
>>> ivy.scaled_dot_product_attention(q,k,v,scale=1,out=out)
>>> print(out)
ivy.array([[[4.03946018, 5.0280633 ],
...         [4.29981947, 5.29981089],
...         [4.30000019, 5.30000019]]])

With a mix of ivy.Array and ivy.Container inputs:

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.],[4.4, 5.6]]])
>>> k = ivy.Container(a=ivy.array([[[4.2, 1.], [2.2, 3.3], [4.4, 5.6]]]),
...                   b=ivy.array([[[3.2, 1.], [2.2, 3.6], [4.0, 5.6]]]))
>>> v = ivy.array([[[0.4, 1.3], [2.2, 3.1], [4.3, 5.3]]])
>>> result = ivy.scaled_dot_product_attention(q,k,v,scale=1,is_causal=True)
>>> print(result)
{
    a: ivy.array([[[0.40000001, 1.29999995],
    ...            [2.06345534, 2.9634552],
    ...            [4.30000019, 5.30000019]]]),
    b: ivy.array([[[0.40000001, 1.29999995],
    ...            [2.19336844, 3.09336829],
    ...            [4.30000019, 5.30000019]]])
}
With a mix of :class:`ivy.Array` and :class:`ivy.Container` inputs:

>>> q = ivy.array([[[0.2, 1.], [2.2, 3.],[4.4, 5.6]]])
>>> k = ivy.Container(a=ivy.array([[[4.2, 1.], [2.2, 3.3],[4.4, 5.6]]]),
...                   b=ivy.array([[[3.2, 1.], [2.2, 3.6],[4.0, 5.6]]]))
>>> v = ivy.array([[[0.4, 1.3], [2.2, 3.1],[4.3, 5.3]]])
>>> mask = ivy.native_array([[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]])
>>> result = ivy.scaled_dot_product_attention(q,
...                                           k,
...                                           v,
...                                           scale=1,
...                                           mask=mask,
...                                           dropout_p=0.1,
...                                           training=True)
>>> print(result)
{
    a: ivy.array([[[2.30000019, 3.23333359],
    ...            [2.30000019, 3.23333359],
    ...            [2.30000019, 3.23333359]]]),
    b: ivy.array([[[2.30000019, 3.23333359],
    ...            [2.30000019, 3.23333359],
    ...            [2.30000019, 3.23333359]]])
}

This should have hopefully given you an overview of the layers submodule, if you have any questions, please feel free to reach out on our discord!

On this page