# Tensor operations and primitives (with Tensorflow)

For fluently reading and writing custom TensorFlow code (e.g., custom loss functions, custom layers, …) a good understanding of basic tensor operations is necessary. This article gives an overview of operations, which are commonly found in the wild.

## Terminology

A tensor is a multi-dimensional data structure (basically another name for a multi-dimensional array). Tensors (`tf.Tensor`

) have a fixed `shape`

, which defines the number of `dimensions`

(aka `axis`

) and the number of items within each dimension. The total number of dimensions is defined as the `rank`

of a tensor. The total number of items within the data structure states the size of the tensor. Furthermore the tensor has a defined datatype (e.g., `tf.int32`

, `tf.float32`

) and all items must be of this datatype.

The following code snippet creates a tensor with the `size of 9`

and the `rank of 2`

. The shape is `(3, 3)`

and the datatype is implicitly set to `tf.int32`

.

`tensor = tf.constant([`

[1, 2, 3],

[4, 5, 6],

[7, 8, 9]

])

print('shape: {}'.format(tensor.shape))

print('datatype: {}'.format(tensor.dtype))

print('rank: {}'.format(tf.rank(tensor)))

print('size: {}'.format(tf.size(tensor)))

Output:

`shape: (3, 3)`

datatype: <dtype: 'int32'>

rank: 2

size: 9

Based on the rank different mathematical names are defined for the tensor data structure. That is, a tensor with rank 0 is called a `scalar`

, a tensor with rank 1 is called a `vector`

, a tensor with rank 2 is called a `matrix`

. Everything which has a rank higher than 2 is a `tensor`

.

## Creating tensors

`tf.constant`

creates a immutable tensor from a tensor like data structure (e.g., a python list):

`tensor = tf.constant([`

[1, 2, 3],

[4, 5, 6],

[7, 8, 9]

])

print(tensor)

Output:

`tf.Tensor(`

[[1 2 3]

[4 5 6]

[7 8 9]], shape=(3, 3), dtype=int32)

`tf.range`

creates a tensor similarly to a python range definition, with a `start`

, `end`

(optional) and `delta`

(optional, default=1) value.

`tensor = tf.range(1, 8, 2)`

print(tensor)

Output:

`tf.Tensor([1 3 5 7], shape=(4,), dtype=int32)`

`tf.fill`

creates a tensor based on a specified `shape`

and sets a given `value`

for each item. `tf.ones`

and `tf.zeros`

are similar to `tf.fill`

, however instead of an arbitrary value `1`

or `0`

is used. Furthermore `tf.ones_like`

and `tf.zeros_like`

create a tensor with `1`

or `0`

based on the shape of an existing tensor.

`tensor = tf.fill([3, 2], 5)`

print(tensor)

Output:

`tf.Tensor(`

[[5 5]

[5 5]

[5 5]], shape=(3, 2), dtype=int32)

## Indexing and slicing

Similar to `numpy`

or python data structures certain indices can be accessed or parts of a tensor can be selected by slicing.

Important syntax for indexing and slicing:

- negative indices count backward from the end of the tensor dimension
- slicing can be used with
`:`

- ... includes all dimension, which is a shorthand for using
`:`

repeatingly

`tensor = tf.constant([`

[

[1, 2, 3],

[4, 5, 6]

],

[

[3, 2, 1],

[6, 5, 4]

]

])

print(tensor[:, :, -1], tensor[..., -1])

Output:

`tf.Tensor(`

[[3 6]

[1 4]], shape=(2, 2), dtype=int32)

tf.Tensor(

[[3 6]

[1 4]], shape=(2, 2), dtype=int32)

## Mathematical/Arithmetic operations

There are many mathematical operations implemented for tensors. Some are also overloaded with respective operators.

In the following code snippet several element-wise operations are demonstrated. Generally, for element-wise operations the tensors require the same shape. `Broadcasting`

, which is explained in the next section allows for element-wise operations which don’t require the same shape.

`a = tf.constant([1, 2, 3])`

b = tf.constant([4, 5, 6])

print(a/b, tf.truediv(a, b))

print(a+b, tf.add(a, b))

print(a-b, tf.subtract(a, b))

print(a*b, tf.multiply(a, b))

Output:

`tf.Tensor([0.25 0.4 0.5 ], shape=(3,), dtype=float64) tf.Tensor([0.25 0.4 0.5 ], shape=(3,), dtype=float64)`

tf.Tensor([5 7 9], shape=(3,), dtype=int32) tf.Tensor([5 7 9], shape=(3,), dtype=int32)

tf.Tensor([-3 -3 -3], shape=(3,), dtype=int32) tf.Tensor([-3 -3 -3], shape=(3,), dtype=int32)

tf.Tensor([ 4 10 18], shape=(3,), dtype=int32) tf.Tensor([ 4 10 18], shape=(3,), dtype=int32)

Also the matrix multiplication is implemented:

`a = tf.constant([`

[1, 2, 3],

[4, 5, 6]

])

b = tf.constant([

[4],

[6],

[8]

])

print(a @ b, tf.matmul(a, b))

Output:

`tf.Tensor(`

[[40]

[94]], shape=(2, 1), dtype=int32)

tf.Tensor(

[[40]

[94]], shape=(2, 1), dtype=int32)

## Broadcasting

Broadcasting enables element-wise operations of tensors with different shapes. Thereby the tensor with the smaller shape for a certain dimension is “streched”, in order to execute the operation.

`a = tf.constant([1, 2, 3, 4])`

b = tf.constant([5])

print(a + b)

print(a * b)

Output:

`tf.Tensor([6 7 8 9], shape=(4,), dtype=int32)`

tf.Tensor([ 5 10 15 20], shape=(4,), dtype=int32)

In the example above the tensor `b`

is broadcast to fit the shape of tensor `a`

. The tensor would be broadcast into the following structure: `[5, 5, 5, 5]`

.

## Reshaping tensors

The memory layout of a `tf.Tensor`

is organized in `row-major`

ordering (C-Style). When reshaping a tensor the actual data in the memory is not changed, only the index is reorganized. Therefore the input and output tensors of a reshaping operation must have the same size.

The `tf.reshape`

operation is taking a tensor as input and additionally a shape definition:

`tensor = tf.constant([`

[

[1., 6., 4.],

[2., 5., 1.]

],

[

[3., 8., 3.],

[4., 9., 8.]

],

])

# reshape: (2, 2, 3) >>> (12, )

result1 = tf.reshape(tensor, [12])

print(result1)

# reshape: (2, 2, 3) >>> (4, 1, 1, 3)

result2 = tf.reshape(tensor, [4, 1, 1, 3])

print(result2)

Output:

`tf.Tensor([1. 6. 4. 2. 5. 1. 3. 8. 3. 4. 9. 8.], shape=(12,), dtype=float32)`

tf.Tensor(

[[[[1. 6. 4.]]]

[[[2. 5. 1.]]]

[[[3. 8. 3.]]]

[[[4. 9. 8.]]]], shape=(4, 1, 1, 3), dtype=float32)

The `tf.expand_dims`

and the `tf.squeeze`

operations add or remove dimensions of length 1.

## Tiling

Create an output tensor by tiling an input tensor according to a given specification. The `tf.tile`

operation requires an `input`

`tensor`

and a multiples specification. The multiples specification is a one-dimensional tensor and the length of the dimension must be the same as the rank of the input tensor.

`tensor = tf.constant([`

[

[1, 2, 3],

[4, 5, 6]

]

])

print(tf.tile(tensor, tf.constant([1, 3, 2])))

Output:

`tf.Tensor(`

[[[1 2 3 1 2 3]

[4 5 6 4 5 6]

[1 2 3 1 2 3]

[4 5 6 4 5 6]

[1 2 3 1 2 3]

[4 5 6 4 5 6]]], shape=(1, 6, 6), dtype=int32)

The `tf.tile`

operation replicates the given tensor of shape `(1, 2, 3)`

according to the multiples specification `[1, 3, 2]`

. That is, axis 0 is replicated once, axis 1 is replicated three times and axis 2 is replicated twice.

## Repeat

The `tf.repeat`

operation repeats tensor items on a specified axis according to a specified pattern. The length of the repeat pattern must be of the same length as the selected axis in the input tensor shape.

`tensor = tf.constant([`

[1, 2, 3],

[3, 4, 5]

])

print(tf.repeat(tensor, [2, 3], 0))

print(tf.repeat(tensor, [2, 4, 1], 1))

Output:

`tf.Tensor(`

[[1 2 3]

[1 2 3]

[3 4 5]

[3 4 5]

[3 4 5]], shape=(5, 3), dtype=int32)

tf.Tensor(

[[1 1 2 2 2 2 3]

[3 3 4 4 4 4 5]], shape=(2, 7), dtype=int32)

## Concat

The `tf.concat`

operation “merges” tensors along a specified axis. The axis could also be specified with a negative number (similar to python when using a negative index for a list).

`t1 = [[1, 2, 3], [4, 5, 6]]`

t2 = [[7, 8, 9], [10, 11, 12]]

# concat along the 0-axis

print(tf.concat([t1, t2], 0))

# concat along the 1-axis

print(tf.concat([t1, t2], 1))

Output:

`tf.Tensor(`

[[ 1 2 3]

[ 4 5 6]

[ 7 8 9]

[10 11 12]], shape=(4, 3), dtype=int32)

tf.Tensor(

[[ 1 2 3 7 8 9]

[ 4 5 6 10 11 12]], shape=(2, 6), dtype=int32)

## Transpose

In the case of a 2-D tensor the `tf.transpose`

operation performs a classic matrix transposition.

`tensor = tf.constant([`

[1, 2, 3],

[4, 5, 6]

])

print(tf.transpose(tensor))

Output:

`tf.Tensor(`

[[1 4]

[2 5]

[3 6]], shape=(3, 2), dtype=int32)

For tensors of a rank greater than 2 a permutation specification can be defined. This specification states the “position” of every index in the resulting tensor.

In the example a tensor with the shape `(2, 2, 3)`

is transposed according to a permutation specification `[0, 2, 1]`

. The original order of the axes is `[0, 1, 2]`

, thus the transposition “swaps” the second with the third axis.

`tensor = tf.constant([[[1, 2, 3],`

[4, 5, 6]],

[[7, 8, 9],

[10, 11, 12]]])

print(tf.transpose(tensor, [0, 2, 1]))

Output:

`tf.Tensor(`

[[[ 1 4]

[ 2 5]

[ 3 6]]

[[ 7 10]

[ 8 11]

[ 9 12]]], shape=(2, 3, 2), dtype=int32)