I’ve lately been tinkering with deep studying fashions in Tensorflow, and have accordingly been launched to managing knowledge as tensors.

As a Knowledge Engineer that works all day in tables that I can simply slice, cube, and visualize, I had completely no instinct round working with tensors, and I appeared to consistently run into the identical errors that, particularly at first, went approach over my head.

Nonetheless, deep diving them has taught me loads about tensors and TensorFlow, and I wished to consolidate these learnings right here to make use of as a reference.

In case you have a favourite error, answer, or debugging tip, please go away a remark!

Earlier than we dive into the errors themselves, I wished to doc a couple of of the lightweight, easy bits and items of code that I’ve discovered useful in debugging. (Though it should be acknowledged for authorized causes that we in fact at all times debug with official debugging options and by no means simply dozens of print statements 🙂)

Seeing inside our Tensorflow Datasets

First off, our precise knowledge. Once we print a Dataframe or SELECT * in SQL, we see the information! Once we print a tensor dataset we see…

<_TensorSliceDataset element_spec=(TensorSpec(form=(2, 3), dtype=tf.int32, identify=None), TensorSpec(form=(1, 1), dtype=tf.int32, identify=None))>

That is all fairly helpful info, but it surely doesn’t assist us perceive what’s truly occurring in our knowledge.

To print a single tensor inside the execution graph we are able to leverage tf.print. This text is a superb deep dive into tf.print that I extremely suggest when you plan to make use of it typically: Utilizing tf.Print() in TensorFlow

However when working with Tensorflow datasets throughout improvement, typically we have to see a couple of values at a time. For that we are able to loop by means of and print particular person items of knowledge like this:

# Generate dummy 2D knowledge
np.random.seed(42)
num_samples = 100
num_features = 5
X_data = np.random.rand(num_samples, num_features).astype(np.float32)
y_data = 2 * X_data[:, 0] + 3 * X_data[:, 1] - 1.5 * X_data[:, 2] + 0.5 * X_data[:, 3] + np.random.randn(num_samples)
# Flip it right into a Tensorflow Dataset
dataset = tf.knowledge.Dataset.from_tensor_slices((X_data, y_data))
# Print the primary 10 rows
for i, (options, label) in enumerate(dataset.take(10)):
print(f"Row {i + 1}: Options - {options.numpy()}, Label - {label.numpy()}")

We will additionally use skip to get to a selected index:

mini_dataset = dataset.skip(100).take(20)
for i, (options, label) in enumerate(mini_dataset):
print(f"Row {i + 1}: Options - {options.numpy()}, Label - {label.numpy()}")

Figuring out our tensors’ specs

When working with tensors we additionally have to know their form, rank, dimension, and knowledge kind (if a few of that vocabulary is unfamiliar, because it was to me initially, don’t fear, we’ll get again to it later within the article). Anyway, beneath are a couple of traces of code to assemble this info:

# Create a pattern tensor
sample_tensor = tf.fixed([[1, 2, 3], [4, 5, 6]])
# Get the scale of the tensor (whole variety of parts)
tensor_size = tf.measurement(sample_tensor).numpy()
# Get the rank of the tensor
tensor_rank = tf.rank(sample_tensor).numpy()
# Get the form of the tensor
tensor_shape = sample_tensor.form
# Get the scale of the tensor
tensor_dimensions = sample_tensor.form.as_list()
# Print the outcomes
print("Tensor Dimension:", tensor_size)
print("Tensor Rank:", tensor_rank)
print("Tensor Form:", tensor_shape)
print("Tensor Dimensions:", tensor_dimensions)

The above outputs:

Tensor Dimension: 6
Tensor Rank: 2
Tensor Form: (2, 3)
Tensor Dimensions: [2, 3]

Augmenting mannequin.abstract()

Lastly, its is at all times useful to have the ability to see how knowledge is transferring by means of a mannequin, and the way form adjustments all through inputs and outputs between layers. The supply of many an error shall be a mismatch between these anticipated enter and output shapes and the form of a given tensor.

mannequin.abstract() in fact will get the job executed, however we are able to complement that info with the next snippet, which provides a bit extra context with mannequin and layer inputs and outputs:

print("###################Enter Form and Datatype#####################")
[print(i.shape, i.dtype) for i in model.inputs]
print("###################Output Form and Datatype#####################")
[print(o.shape, o.dtype) for o in model.outputs]
print("###################Layer Enter Form and Datatype#####################")
[print(l.name, l.input, l.dtype) for l in model.layers]

So let’s soar into some errors!

Rank

ValueError: Form should be rank x however is rank y….

Okay, to start with, what’s a rank? Rank is simply the unit of dimensionality we use to explain tensors. A rank 0 tensor is a scalar worth; a rank one tensor is a vector; a rank two is a matrix, and so forth for all n dimensional constructions.

Take for instance a 5 dimensional tensor.

rank_5_tensor = tf.fixed([[[[[1, 2], [3, 4]], [[5, 6], [7, 8]]], [[[9, 10], [11, 12]], [[13, 14], [15, 16]]]],
[[[[17, 18], [19, 20]], [[21, 22], [23, 24]]], [[[25, 26], [27, 28]], [[29, 30], [31, 32]]]]])
print("nRank 5 Tensor:", rank_5_tensor.form)

Rank 5 Tensor: (2, 2, 2, 2, 2)

The code above exhibits that every dimension of the 5 has a measurement of two. If we wished to index it, we might accomplish that alongside any of those axes. To get on the final aspect, 32, we might run one thing like:

rank_5_tensor.numpy()[1][1][1][1][1]

The official tensor documentation has some actually useful visualizations to make this a bit extra understandable.

Again to the error: it’s simply flagging that the tensor supplied is a special dimension than what is predicted to a specific perform. For instance if the error declares that the “Form should be rank 1 however is rank 0…” it implies that we’re offering a scalar worth, and it expects a 1-D tensor.

Take the instance beneath the place we try to multiply tensors along with the matmul methodology.

import tensorflow as tf
import numpy as np
# Create a TensorFlow dataset with random matrices
num_samples = 5
matrix_size = 3
dataset = tf.knowledge.Dataset.from_tensor_slices(np.random.rand(num_samples, matrix_size, matrix_size))
mul = [1,2,3,4,5,6]
# Outline a perform that makes use of tf.matmul
def matmul_function(matrix):
return tf.matmul(matrix, mul)
# Apply the matmul_function to the dataset utilizing map
result_dataset = dataset.map(matmul_function)

If we take a peek on the documentation, matmul expects no less than a rank 2 tensor, so multiplying the matrix by [1,2,3,4,5,6], which is simply an array, will elevate this error.

ValueError: Form should be rank 2 however is rank 1 for '{{node MatMul}} = MatMul[T=DT_DOUBLE, transpose_a=false, transpose_b=false](args_0, MatMul/b)' with enter shapes: [3,3], [2].

An amazing first step for this error is to dive into the documentation and perceive what the perform you might be utilizing is in search of (Right here’s a pleasant record of the capabilities out there on tensors: raw_ops.

Then use the rank methodology to find out what we are literally offering.

print(tf.rank(mul))

tf.Tensor(1, form=(), dtype=int32)

So far as fixes go, tf.reshape is commonly a superb possibility to start out with. Let’s take a quick second to speak a little bit bit about tf.reshape, since it will likely be a devoted companion all through our Tensorflow journey: tf.reshape(tensor, form, identify=None)

Reshape merely takes within the tensor we wish to reshape, and one other tensor containing what we wish the form of the output to be. For instance, let’s reshape our multiplication enter:

mul = [1,2,3,4,5,6]
tf.reshape(mul, [3, 2]).numpy()

array([[1, 2],
[3, 4],
[5, 6]], dtype=int32)

Our variable will flip right into a (3,2) tensor (3 rows, 2 columns). A fast word, tf.reshape(t, [3, -1]).numpy() will produce the identical factor as a result of the -1 tells Tensorflow to compute the scale of the dimension such that the overall measurement stays fixed. The variety of parts within the form tensor is the rank.

As soon as we create a tensor with the correct rank, our multiplication will work simply fantastic!

Form

ValueError: Enter of layer is incompatible with layer….

Having an intuitive understanding of tensor form, and the way it interacts and adjustments throughout mannequin layers has made life with deep studying considerably simpler

First, getting primary vocab out of the way in which: the form of a tensor refers back to the variety of parts alongside every dimension, or axis of the tensor. For instance, a 2D tensor with 3 rows and 4 columns has a form of (3, 4).

So what can go improper with form? Glad you requested, fairly a couple of issues!

Firstly the form and rank of your coaching knowledge should match the enter form anticipated by the enter layer. Let’s check out an instance, a primary CNN:

import tensorflow as tf
from tensorflow.keras import layers, fashions
# Create a perform to generate pattern knowledge
def generate_sample_data(num_samples=100):
for _ in vary(num_samples):
options = tf.random.regular(form=(64, 64, 3))
labels = tf.one_hot(tf.random.uniform(form=(), maxval=10, dtype=tf.int32), depth=10)
yield options, labels
# Create a TensorFlow dataset utilizing the generator perform
sample_dataset = tf.knowledge.Dataset.from_generator(generate_sample_data, output_signature=(tf.TensorSpec(form=(64, 64, 3), dtype=tf.float32), tf.TensorSpec(form=(10,), dtype=tf.float32)))
# Create a CNN mannequin with an enter layer anticipating (128, 128, 3)
mannequin = fashions.Sequential()
mannequin.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(128, 128, 3)))
mannequin.add(layers.MaxPooling2D((2, 2)))
mannequin.add(layers.Conv2D(64, (3, 3), activation='relu'))
mannequin.add(layers.MaxPooling2D((2, 2)))
mannequin.add(layers.Conv2D(64, (3, 3), activation='relu'))
mannequin.add(layers.Flatten())
mannequin.add(layers.Dense(64, activation='relu'))
mannequin.add(layers.Dense(10, activation='softmax'))
# Compile the mannequin
mannequin.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
# Match the mannequin utilizing the dataset
mannequin.match(sample_dataset.batch(32).repeat(), epochs=5, steps_per_epoch=100, validation_steps=20)

Attempting to run the code above will end in:

ValueError: Enter 0 of layer "sequential_5" is incompatible with the layer: anticipated form=(None, 128, 128, 3), discovered form=(None, 64, 64, 3)

It’s because our mannequin is anticipating the enter tensor to be of the form (128, 128, 3) and our generated knowledge is (64, 64, 3).

In a scenario like this, our good pal, reshape, or one other Tensorflow perform, resize, might help. If, as within the case above we’re working with photos, we are able to merely run resize or change the expectations of our mannequin’s enter:

def resize_image(picture, label):
resized_image = tf.picture.resize(picture, measurement=target_shape)
return resized_image, label
# Apply the resize perform to the whole dataset
resized_dataset = sample_dataset.map(resize_image)

On this context, it’s useful to know a little bit about how frequent kinds of fashions and mannequin layers anticipate enter of various shapes, so let’s take a little bit detour.

Deep Neural Networks of Dense layers soak up 1 dimensional tensors (or 2 dimensional, relying on whether or not you embrace batch measurement, however we’ll discuss batch measurement in a bit) of the format (feature_size, ) the place feature_size is the variety of options in every pattern.

Convolutional Neural Networks soak up knowledge representing photos, utilizing 3 dimensional tensors of (width, peak, channels) the place channels are the colour scheme, 1 for grey scale, and three for RBG.

And eventually, Recurrent Neural Networks comparable to LTSMs soak up 2 dimensions (time steps, feature_size)

However again to errors! One other frequent wrongdoer in Tensorflow form errors has to do with how form adjustments as knowledge passes by means of the mannequin layers. As beforehand talked about, totally different layers soak up totally different enter shapes, they usually also can reshape output.

Returning to our CNN instance from above, let’s break it once more, by seeing what occurs once we take away the Flatten layer. If we attempt to run the code we’ll see

ValueError: Shapes (None, 10) and (None, 28, 28, 10) are incompatible

That is the place printing all of our mannequin enter and output shapes together with our knowledge shapes is useful to assist us pinpoint the place there’s a mismatch.

mannequin.abstract() will present us

Layer (kind) Output Form Param #
=================================================================
conv2d_15 (Conv2D) (None, 126, 126, 32) 896
max_pooling2d_10 (MaxPooli (None, 63, 63, 32) 0
ng2D)
conv2d_16 (Conv2D) (None, 61, 61, 64) 18496
max_pooling2d_11 (MaxPooling2D) (None, 30, 30, 64) 0
conv2d_17 (Conv2D) (None, 28, 28, 64) 36928
flatten_5 (Flatten) (None, 50176) 0
dense_13 (Dense) (None, 64) 3211328
dense_14 (Dense) (None, 10) 650
=================================================================
Whole params: 3268298 (12.47 MB)
Trainable params: 3268298 (12.47 MB)
Non-trainable params: 0 (0.00 Byte)

And our additional diagnostic will reveal

###################Enter Form and Datatype#####################
(None, 128, 128, 3) <dtype: 'float32'>
###################Output Form and Datatype#####################
(None, 10) <dtype: 'float32'>
###################Layer Enter Form and Datatype#####################
conv2d_15 KerasTensor(type_spec=TensorSpec(form=(None, 128, 128, 3), dtype=tf.float32, identify='conv2d_15_input'), identify='conv2d_15_input', description="created by layer 'conv2d_15_input'") float32
max_pooling2d_10 KerasTensor(type_spec=TensorSpec(form=(None, 126, 126, 32), dtype=tf.float32, identify=None), identify='conv2d_15/Relu:0', description="created by layer 'conv2d_15'") float32
conv2d_16 KerasTensor(type_spec=TensorSpec(form=(None, 63, 63, 32), dtype=tf.float32, identify=None), identify='max_pooling2d_10/MaxPool:0', description="created by layer 'max_pooling2d_10'") float32
max_pooling2d_11 KerasTensor(type_spec=TensorSpec(form=(None, 61, 61, 64), dtype=tf.float32, identify=None), identify='conv2d_16/Relu:0', description="created by layer 'conv2d_16'") float32
conv2d_17 KerasTensor(type_spec=TensorSpec(form=(None, 30, 30, 64), dtype=tf.float32, identify=None), identify='max_pooling2d_11/MaxPool:0', description="created by layer 'max_pooling2d_11'") float32
flatten_5 KerasTensor(type_spec=TensorSpec(form=(None, 28, 28, 64), dtype=tf.float32, identify=None), identify='conv2d_17/Relu:0', description="created by layer 'conv2d_17'") float32
dense_13 KerasTensor(type_spec=TensorSpec(form=(None, 50176), dtype=tf.float32, identify=None), identify='flatten_5/Reshape:0', description="created by layer 'flatten_5'") float32
dense_14 KerasTensor(type_spec=TensorSpec(form=(None, 64), dtype=tf.float32, identify=None), identify='dense_13/Relu:0', description="created by layer 'dense_13'") float32

It’s lots of output, however we are able to see that dense_13 layer is in search of enter of (None, 50176) form. Nonetheless, conv2d_17 layer outputs (None, 28, 28, 64)

Flatten layers rework the multi-dimensional output from earlier layers right into a one-dimensional (flat) vector that the Dense layer expects.

Conv2d and Max Pooling layers change their enter knowledge in different attention-grabbing methods as nicely, however these are out of scope for this text. For an superior breakdown check out: Final Information to Enter form and Mannequin Complexity in Neural Networks

However what about batch measurement?! I haven’t forgotten!

If we break our code another time by eradicating the .batch(32) from the dataset in mannequin.match we’ll get the error:

ValueError: Enter 0 of layer "sequential_10" is incompatible with the layer: anticipated form=(None, 128, 128, 3), discovered form=(128, 128, 3)

That’s as a result of, the primary dimension of a layer’s enter is reserved for the batch measurement or variety of samples that we wish the mannequin to work by means of at a time. For an excellent deep dive learn by means of Distinction between batch and epoch.

Batch measurement defaults to None previous to becoming, as we are able to see within the mannequin abstract output, and our mannequin expects us to set it elsewhere, relying on how we tune the hyperparameter. We will additionally power it in our enter layer through the use of batch_input_size as an alternative of input_size, however that decreases our flexibility in testing out totally different values.

Sort

TypeError: Did not convert object of kind to Tensor. Unsupported object kind

Lastly, let’s discuss a bit about some knowledge kind specifics in Tensors.

The error above is one other, that, when you’re used to working in database programs with tables constructed from all types of knowledge, could be a bit baffling, but it surely is among the extra easy to diagnose and repair, though there are a few frequent causes to look out for.

The principle situation is that, though tensors assist quite a lot of knowledge sorts, once we convert a NumPy array to tensors (a standard circulation inside deep studying), the datatypes should be floats. The script beneath initializes a contrived instance of a dataframe with None and with string knowledge factors. Let’s stroll by means of some situation and fixes for this instance:

import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow.keras.layers import Dense
from tensorflow.keras.fashions import Sequential
knowledge = [
[None, 0.2, '0.3'],
[0.1, None, '0.3'],
[0.1, 0.2, '0.3'],
]
X_train = pd.DataFrame(knowledge=knowledge, columns=["x1", "x2", "x3"])
y_train = pd.DataFrame(knowledge=[1, 0, 1], columns=["y"])
# Create a TensorFlow dataset
train_dataset = tf.knowledge.Dataset.from_tensor_slices((X_train.to_numpy(), y_train.to_numpy()))
# Outline the mannequin
mannequin = Sequential()
mannequin.add(Dense(1, input_dim=X_train.form[1], activation='sigmoid'))
mannequin.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
# Match the mannequin utilizing the TensorFlow dataset
mannequin.match(train_dataset.batch(3), epochs=3)

Working this code will flag to us that:

ValueError: Did not convert a NumPy array to a Tensor (Unsupported object kind float).

The obvious situation is that you’re sending in a NumPy array that comprises some non float kind, an object. In case you have an precise column of categorical knowledge, there are various methods to transform that to numeric knowledge (One shot encoding, and many others) however that’s out of scope for this dialogue.

We will decide that if we run print(X_train.dtypes), which can inform us what’s in our dataframe that Tensorflow doesn’t like.

x1 float64
x2 float64
x3 object
dtype: object

If we’re operating into non float knowledge factors, the road beneath will magically resolve all of our issues:

X_train = np.asarray(X_train).astype('float32')

One other factor to test for is when you’ve got None or np.nan anyplace.

To search out out we are able to use a couple of traces of code comparable to:

null_mask = X_train.isnull().any(axis=1)
null_rows = X_train[null_mask]
print(null_rows)

Which tells us that now we have nulls on rows 0 and 1:

x1 x2 x3
0 NaN 0.2 0.3
1 0.1 NaN 0.3

If that’s the case, and that’s anticipated/intentional we have to exchange these values with an appropriate various. Fillna might help us right here.

X_train.fillna(worth=0, inplace=True)

With these adjustments to the code beneath, our NumPy array will efficiently convert to a tensor dataset and we are able to practice our mannequin!

I typically discover that I study essentially the most a couple of specific expertise when I’ve to work by means of errors, and I hope this has been considerably useful to you too!

In case you have cool ideas and tips or enjoyable Tensorflow errors please move them alongside!