Tensorflow 2.0 结合 CNN + LSTM

Question

如何在 Tensorflow 2.0 / Keras 中的（扁平化）conv2d 层之后添加 LSTM 层？我的训练输入数据具有以下形状（大小、序列长度、高度、宽度、通道）。对于卷积层，我一次只能处理一张图像，对于 LSTM 层，我需要一系列特征。有没有办法在 LSTM 层之前重塑您的数据，以便您可以将两者结合起来？

Answer 1

从您提供的形状概述来看(size, sequence_length, height, width, channels)，您似乎每个标签都有图像序列。为此，我们通常使用Conv3D. 我附上下面的示例代码：

import tensorflow as tf

SIZE = 64
SEQUENCE_LENGTH = 50
HEIGHT = 128
WIDTH = 128
CHANNELS = 3

data = tf.random.normal((SIZE, SEQUENCE_LENGTH, HEIGHT, WIDTH, CHANNELS))

input = tf.keras.layers.Input((SEQUENCE_LENGTH, HEIGHT, WIDTH, CHANNELS))
hidden = tf.keras.layers.Conv3D(32, (3, 3, 3))(input)
hidden = tf.keras.layers.Reshape((-1, 32))(hidden)
hidden = tf.keras.layers.LSTM(200)(hidden)

model = tf.keras.models.Model(inputs=input, outputs=hidden)
model.summary()

输出：

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 50, 128, 128, 3)] 0         
_________________________________________________________________
conv3d (Conv3D)              (None, 48, 126, 126, 32)  2624      
_________________________________________________________________
reshape (Reshape)            (None, None, 32)          0         
_________________________________________________________________
lstm (LSTM)                  (None, 200)               186400    
=================================================================
Total params: 189,024
Trainable params: 189,024
Non-trainable params: 0

如果您仍然想使用Conv2D在您的情况下不推荐的功能，您将必须执行如下所示的操作。基本上，您在高度维度上附加图像序列，这将使您放松时间维度。

import tensorflow as tf

SIZE = 64
SEQUENCE_LENGTH = 50
HEIGHT = 128
WIDTH = 128
CHANNELS = 3

data = tf.random.normal((SIZE, SEQUENCE_LENGTH, HEIGHT, WIDTH, CHANNELS))

input = tf.keras.layers.Input((SEQUENCE_LENGTH, HEIGHT, WIDTH, CHANNELS))
hidden = tf.keras.layers.Reshape((SEQUENCE_LENGTH * HEIGHT, WIDTH, CHANNELS))(input)
hidden = tf.keras.layers.Conv2D(32, (3, 3))(hidden)
hidden = tf.keras.layers.Reshape((-1, 32))(hidden)
hidden = tf.keras.layers.LSTM(200)(hidden)

model = tf.keras.models.Model(inputs=input, outputs=hidden)
model.summary()

输出：

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 50, 128, 128, 3)] 0         
_________________________________________________________________
reshape (Reshape)            (None, 6400, 128, 3)      0         
_________________________________________________________________
conv2d (Conv2D)              (None, 6398, 126, 32)     896       
_________________________________________________________________
reshape_1 (Reshape)          (None, None, 32)          0         
_________________________________________________________________
lstm (LSTM)                  (None, 200)               186400    
=================================================================
Total params: 187,296
Trainable params: 187,296
Non-trainable params: 0
_________________________________________________________________