Tensorflow，tf.reshape 导致“变量不存在梯度”

Question

我正在使用 Tensorflow/Keras（Windows 上的 TF 版本 2.1，Python 3.7）编写一个完全连接的层，但我发现如果我在乘以它之前重塑我的权重张量，那么 Tensorflow 似乎无法计算即使我只是将渐变重塑为自己的形状。考虑以下层代码：

import tensorflow as tf
import numpy as np

class FCLayer(tf.keras.layers.Layer):
    def __init__(self,output_size,cause_error = False):
        super(FCLayer,self).__init__()
        self.output_size = output_size
        self.cause_error = cause_error

    def build(self,input_shape): 
        self.input_size = input_shape[1]        
        weights = self.add_weight(shape=(self.input_size,
                                         self.output_size),
                                 initializer='random_normal',
                                 trainable=True)

        if self.cause_error:
            self.weights2 = tf.reshape( weights,
                                        shape = (self.input_size,
                                                 self.output_size))
        else:
            self.weights2 = weights

    def call(self, inputs):
        return tf.matmul(inputs, self.weights2)    

如果这与cause_error = True一起使用，那么在mnist上训练4个时期时我会得到以下输出（下面包含特定的训练代码）：

Train on 60000 samples, validate on 10000 samples
Epoch 1/4
WARNING:tensorflow:Gradients do not exist for variables ['sequential/dummy_layer/Variable:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['sequential/dummy_layer/Variable:0'] when minimizing the loss.
60000/60000 [==============================] - 1s 20us/sample - loss: 2.4131 - accuracy: 0.0722 - val_loss: 2.3963 - val_accuracy: 0.0834
Epoch 2/4
60000/60000 [==============================] - 1s 12us/sample - loss: 2.4122 - accuracy: 0.0722 - val_loss: 2.3953 - val_accuracy: 0.0836
Epoch 3/4
60000/60000 [==============================] - 1s 12us/sample - loss: 2.4112 - accuracy: 0.0724 - val_loss: 2.3944 - val_accuracy: 0.0838
Epoch 4/4
60000/60000 [==============================] - 1s 13us/sample - loss: 2.4102 - accuracy: 0.0725 - val_loss: 2.3933 - val_accuracy: 0.0839

这只是一个警告，但很明显模型并没有真正改进，显然它需要这些梯度。

如果我设置了 cause_error=False，我会得到预期的输出（没有警告，适度的改进）：

Train on 60000 samples, validate on 10000 samples
Epoch 1/4
60000/60000 [==============================] - 1s 16us/sample - loss: 2.3671 - accuracy: 0.1527 - val_loss: 2.3445 - val_accuracy: 0.1508
Epoch 2/4
60000/60000 [==============================] - 1s 12us/sample - loss: 2.3293 - accuracy: 0.1596 - val_loss: 2.3072 - val_accuracy: 0.1610
Epoch 3/4
60000/60000 [==============================] - 1s 13us/sample - loss: 2.2939 - accuracy: 0.1683 - val_loss: 2.2722 - val_accuracy: 0.1720
Epoch 4/4
60000/60000 [==============================] - 1s 13us/sample - loss: 2.2609 - accuracy: 0.1784 - val_loss: 2.2397 - val_accuracy: 0.1847

我怀疑我需要以某种方式告诉 Tensorflow 跟踪梯度，但我不太确定如何。当我使用 tf.matmul 时，它似乎会自动完成，而且我很确定这种代码曾经在 TF 1 中工作过。

我用来执行的具体代码是（改编自 mnist 教程）：

batch_size = 128
num_classes = 10
epochs = 4

# input image dimensions
img_rows, img_cols = 28, 28

# the data, split between train and test sets
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()


x_train = x_train.reshape(x_train.shape[0], img_rows* img_cols)
x_test = x_test.reshape(x_test.shape[0], img_rows*img_cols)
input_shape = (img_rows * img_cols)

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
y_train = tf.keras.utils.to_categorical(y_train, num_classes)
y_test = tf.keras.utils.to_categorical(y_test, num_classes)

model = tf.keras.models.Sequential()

dummy_layer = FCLayer(10, cause_error = True)
model.add( dummy_layer )
model.add( tf.keras.layers.Dense(10, activation='softmax') )

model.compile(loss=tf.keras.losses.categorical_crossentropy,
              optimizer=tf.keras.optimizers.Adadelta(),
              metrics=['accuracy'])

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))

Answer 1

该问题与 Eager Execution TF 2.0 相关——任何诸如此类的操作tf.reshape在遇到时立即运行。build对于给定的模型只调用一次。现在，发生的事情是您正在创建一个 tensor weights2，它是tensor的重塑版本，tf.Variable weights但本身不是a tf.Variable（操作通常返回张量，而不是变量）。因为这是在急切执行中发生的，所以不会保留此“记录”并且weights2与weights. 因此，当它在模型调用中使用时，weights无法更新。在这种else情况下不会发生这种情况，因为在这里，weights2只是指代实际tf.Variable weights.

解决这个问题的两种方法：

1. 使用assigninbuild进行适当的重塑（注意，我使用self.w因为self.weights是 Keras 层的保留名称）：

   def build(self,input_shape): 
       self.input_size = input_shape[1]        
       self.w = self.add_weight(shape=(self.input_size,
                                             self.output_size),
                                      initializer='random_normal',
                                      trainable=True)
   
       if self.cause_error:
           self.w.assign(tf.reshape(self.w,
                                      shape = (self.input_size,
                                               self.output_size)))
   

这不会导致错误/警告，但它可能不是您想要的，因为您正在修改weights丢失的原始。我想您更愿意weights在每次调用时使用 的修改版本。在这种情况下，在call方法中执行：

class FCLayer(tf.keras.layers.Layer):
    def __init__(self,output_size,cause_error = False):
        super(FCLayer,self).__init__()
        self.output_size = output_size
        self.cause_error = cause_error

    def build(self,input_shape): 
        self.input_size = input_shape[1]        
        self.w = self.add_weight(shape=(self.input_size,
                                          self.output_size),
                                   initializer='random_normal',
                                   trainable=True)
    def call(self, inputs):
        weights2 = tf.reshape(self.w, (self.input_size, self.output_size)
        return tf.matmul(inputs, weights2)

这是有效的，因为现在reshape操作是模型调用图的一部分，即我们可以回溯weights2实际来自weights，并且梯度可以流动。