# Tensor operations and primitives (with Tensorflow)

For fluently reading and writing custom TensorFlow code (e.g., custom loss functions, custom layers, …) a good understanding of basic tensor operations is necessary. This article gives an overview of operations, which are commonly found in the wild.

## Terminology

A tensor is a multi-dimensional data structure (basically another name for a multi-dimensional array). Tensors (`tf.Tensor`) have a fixed `shape`, which defines the number of `dimensions` (aka `axis`) and the number of items within each dimension. The total number of dimensions is defined as the `rank` of a tensor. The total number of items within the data structure states the size of the tensor. Furthermore the tensor has a defined datatype (e.g., `tf.int32`, `tf.float32`) and all items must be of this datatype.

The following code snippet creates a tensor with the `size of 9` and the `rank of 2`. The shape is `(3, 3)` and the datatype is implicitly set to `tf.int32`.

``tensor = tf.constant([    [1, 2, 3],    [4, 5, 6],    [7, 8, 9]])print('shape: {}'.format(tensor.shape))print('datatype: {}'.format(tensor.dtype))print('rank: {}'.format(tf.rank(tensor)))print('size: {}'.format(tf.size(tensor)))``

Output:

``shape: (3, 3)datatype: <dtype: 'int32'>rank: 2size: 9``

Based on the rank different mathematical names are defined for the tensor data structure. That is, a tensor with rank 0 is called a `scalar`, a tensor with rank 1 is called a `vector`, a tensor with rank 2 is called a `matrix`. Everything which has a rank higher than 2 is a `tensor`.

## Creating tensors

`tf.constant` creates a immutable tensor from a tensor like data structure (e.g., a python list):

``tensor = tf.constant([    [1, 2, 3],    [4, 5, 6],    [7, 8, 9]])print(tensor)``

Output:

``tf.Tensor([[1 2 3] [4 5 6] [7 8 9]], shape=(3, 3), dtype=int32)``

`tf.range` creates a tensor similarly to a python range definition, with a `start`, `end` (optional) and `delta` (optional, default=1) value.

``tensor = tf.range(1, 8, 2)print(tensor)``

Output:

``tf.Tensor([1 3 5 7], shape=(4,), dtype=int32)``

`tf.fill` creates a tensor based on a specified `shape` and sets a given `value` for each item. `tf.ones` and `tf.zeros` are similar to `tf.fill`, however instead of an arbitrary value `1` or `0` is used. Furthermore `tf.ones_like` and `tf.zeros_like` create a tensor with `1` or `0` based on the shape of an existing tensor.

``tensor = tf.fill([3, 2], 5)print(tensor)``

Output:

``tf.Tensor([[5 5] [5 5] [5 5]], shape=(3, 2), dtype=int32)``

## Indexing and slicing

Similar to `numpy` or python data structures certain indices can be accessed or parts of a tensor can be selected by slicing.

Important syntax for indexing and slicing:

• negative indices count backward from the end of the tensor dimension
• slicing can be used with `:`
• ... includes all dimension, which is a shorthand for using `:` repeatingly
``tensor = tf.constant([    [        [1, 2, 3],        [4, 5, 6]    ],    [        [3, 2, 1],        [6, 5, 4]    ]])print(tensor[:, :, -1], tensor[..., -1])``

Output:

``tf.Tensor([[3 6] [1 4]], shape=(2, 2), dtype=int32)tf.Tensor([[3 6] [1 4]], shape=(2, 2), dtype=int32)``

## Mathematical/Arithmetic operations

There are many mathematical operations implemented for tensors. Some are also overloaded with respective operators.

In the following code snippet several element-wise operations are demonstrated. Generally, for element-wise operations the tensors require the same shape. `Broadcasting`, which is explained in the next section allows for element-wise operations which don’t require the same shape.

``a = tf.constant([1, 2, 3])b = tf.constant([4, 5, 6])print(a/b, tf.truediv(a, b))print(a+b, tf.add(a, b))print(a-b, tf.subtract(a, b))print(a*b, tf.multiply(a, b))``

Output:

``tf.Tensor([0.25 0.4  0.5 ], shape=(3,), dtype=float64) tf.Tensor([0.25 0.4  0.5 ], shape=(3,), dtype=float64)tf.Tensor([5 7 9], shape=(3,), dtype=int32) tf.Tensor([5 7 9], shape=(3,), dtype=int32)tf.Tensor([-3 -3 -3], shape=(3,), dtype=int32) tf.Tensor([-3 -3 -3], shape=(3,), dtype=int32)tf.Tensor([ 4 10 18], shape=(3,), dtype=int32) tf.Tensor([ 4 10 18], shape=(3,), dtype=int32)``

Also the matrix multiplication is implemented:

``a = tf.constant([    [1, 2, 3],    [4, 5, 6]])b = tf.constant([    ,    ,    ])print(a @ b, tf.matmul(a, b))``

Output:

``tf.Tensor([ ], shape=(2, 1), dtype=int32)tf.Tensor([ ], shape=(2, 1), dtype=int32)``

Broadcasting enables element-wise operations of tensors with different shapes. Thereby the tensor with the smaller shape for a certain dimension is “streched”, in order to execute the operation.

``a = tf.constant([1, 2, 3, 4])b = tf.constant()print(a + b)print(a * b)``

Output:

``tf.Tensor([6 7 8 9], shape=(4,), dtype=int32)tf.Tensor([ 5 10 15 20], shape=(4,), dtype=int32)``

In the example above the tensor `b` is broadcast to fit the shape of tensor `a`. The tensor would be broadcast into the following structure: `[5, 5, 5, 5]`.

## Reshaping tensors

The memory layout of a `tf.Tensor` is organized in `row-major` ordering (C-Style). When reshaping a tensor the actual data in the memory is not changed, only the index is reorganized. Therefore the input and output tensors of a reshaping operation must have the same size.

The `tf.reshape` operation is taking a tensor as input and additionally a shape definition:

``tensor = tf.constant([    [        [1., 6., 4.],        [2., 5., 1.]    ],    [        [3., 8., 3.],        [4., 9., 8.]    ],])# reshape: (2, 2, 3) >>> (12, )result1 = tf.reshape(tensor, )print(result1)# reshape: (2, 2, 3) >>> (4, 1, 1, 3)result2 = tf.reshape(tensor, [4, 1, 1, 3])print(result2)``

Output:

``tf.Tensor([1. 6. 4. 2. 5. 1. 3. 8. 3. 4. 9. 8.], shape=(12,), dtype=float32)tf.Tensor([[[[1. 6. 4.]]] [[[2. 5. 1.]]] [[[3. 8. 3.]]] [[[4. 9. 8.]]]], shape=(4, 1, 1, 3), dtype=float32)``

The `tf.expand_dims` and the `tf.squeeze` operations add or remove dimensions of length 1.

## Tiling

Create an output tensor by tiling an input tensor according to a given specification. The `tf.tile` operation requires an `input` `tensor` and a multiples specification. The multiples specification is a one-dimensional tensor and the length of the dimension must be the same as the rank of the input tensor.

``tensor = tf.constant([    [        [1, 2, 3],        [4, 5, 6]    ]])print(tf.tile(tensor, tf.constant([1, 3, 2])))``

Output:

``tf.Tensor([[[1 2 3 1 2 3]  [4 5 6 4 5 6]  [1 2 3 1 2 3]  [4 5 6 4 5 6]  [1 2 3 1 2 3]  [4 5 6 4 5 6]]], shape=(1, 6, 6), dtype=int32)``

The `tf.tile` operation replicates the given tensor of shape `(1, 2, 3)` according to the multiples specification `[1, 3, 2]`. That is, axis 0 is replicated once, axis 1 is replicated three times and axis 2 is replicated twice.

## Repeat

The `tf.repeat` operation repeats tensor items on a specified axis according to a specified pattern. The length of the repeat pattern must be of the same length as the selected axis in the input tensor shape.

``tensor = tf.constant([    [1, 2, 3],    [3, 4, 5]])print(tf.repeat(tensor, [2, 3], 0))print(tf.repeat(tensor, [2, 4, 1], 1))``

Output:

``tf.Tensor([[1 2 3] [1 2 3] [3 4 5] [3 4 5] [3 4 5]], shape=(5, 3), dtype=int32)tf.Tensor([[1 1 2 2 2 2 3] [3 3 4 4 4 4 5]], shape=(2, 7), dtype=int32)``

## Concat

The `tf.concat` operation “merges” tensors along a specified axis. The axis could also be specified with a negative number (similar to python when using a negative index for a list).

``t1 = [[1, 2, 3], [4, 5, 6]]t2 = [[7, 8, 9], [10, 11, 12]]# concat along the 0-axisprint(tf.concat([t1, t2], 0))# concat along the 1-axisprint(tf.concat([t1, t2], 1))``

Output:

``tf.Tensor([[ 1  2  3] [ 4  5  6] [ 7  8  9] [10 11 12]], shape=(4, 3), dtype=int32)tf.Tensor([[ 1  2  3  7  8  9] [ 4  5  6 10 11 12]], shape=(2, 6), dtype=int32)``

## Transpose

In the case of a 2-D tensor the `tf.transpose` operation performs a classic matrix transposition.

``tensor = tf.constant([    [1, 2, 3],    [4, 5, 6]])print(tf.transpose(tensor))``

Output:

``tf.Tensor([[1 4] [2 5] [3 6]], shape=(3, 2), dtype=int32)``

For tensors of a rank greater than 2 a permutation specification can be defined. This specification states the “position” of every index in the resulting tensor.

In the example a tensor with the shape `(2, 2, 3)` is transposed according to a permutation specification `[0, 2, 1]`. The original order of the axes is `[0, 1, 2]`, thus the transposition “swaps” the second with the third axis.

``tensor = tf.constant([[[1,  2,  3],                       [4,  5,  6]],                      [[7,  8,  9],                       [10, 11, 12]]])print(tf.transpose(tensor, [0, 2, 1]))``

Output:

``tf.Tensor([[[ 1  4]  [ 2  5]  [ 3  6]] [[ 7 10]  [ 8 11]  [ 9 12]]], shape=(2, 3, 2), dtype=int32)``