Keras fit_generator：getitem 方法的意外使用

Question

Keras fit_generator：getitem 方法的意外使用

我正在使用 fit_generator 函数来训练我的模型，并希望验证我的数据是否按预期构建和使用。我从 keras.utils.Sequence() 派生的类实现了方法__getitem__，__len__并且on_epoch_end看起来像这样：

class PairwiseSequence(Sequence):
"""Generator that returns a combination of simulations (over a
parametrizable amount of timesteps) and the corresponding metric distance.

pair_list: List of pairwise combinations of simulations
results: dictionary with results for the metric distance between
         simulation pairs
sim_files: List of filenames representing single timesteps
batch_size: number of samples to process in a single interference run
"""

def __init__(self, pair_list, results, mean, std, train=False, sim_files=None,
             batch_size=1):
    self.pair_list = pair_list
    self.results = results
    self.batch_size = batch_size
    self.sim_files = sim_files
    self.mean = mean
    self.std = std
    self.train = train

def __len__(self):
    return math.ceil(len(self.pair_list) / self.batch_size)

def __getitem__(self, idx):
    dummy = LOADING_METHOD(self.pair_list[0][0], self.sim_files)
    x_1 = np.zeros(tuple([self.batch_size]) + dummy.shape)
    x_2 = np.zeros(tuple([self.batch_size]) + dummy.shape)
    y = np.zeros((self.batch_size, 1))

    if self.train:
        #print((idx * self.batch_size + i) % len(self.pair_list), ',')
        print("training idx:", idx)
    else:
        print("validation idx:", idx)

        for i in range(0, self.batch_size):
        (sim1, sim2) = self.pair_list[(idx * self.batch_size + i) %
                                      len(self.pair_list)]
        x_1[i] = LOADING_METHOD(sim1, self.sim_files) - self.mean[0]
        x_1[i] /= self.std[0]
        x_2[i] = LOADING_METHOD(sim2, self.sim_files) - self.mean[1]
        x_2[i] /= self.std[1]
        y[i] = self.results[frozenset((sim1.ensemble, sim2.ensemble))]
    return [x_1, x_2], y

def on_epoch_end(self):
    if self.train:
        print("training generator: epoch end")
    else:
        print("validation generator: epoch end")
    #random.shuffle(self.pair_list)

Run Code Online (Sandbox Code Playgroud)

此类用作训练和验证数据的生成器（两个单独的实例）。

正如您所看到的，我正在将纪元结束时idx的参数__getitem__和一些通知打印到控制台。我按如下方式调用 fit_generator （关闭多处理）：

history_callback = model.fit_generator(
    generator=train_gen,
    steps_per_epoch=len(train_gen),
    epochs=epochs,
    verbose=0,
    callbacks=callbacks,
    validation_data=valid_gen,
    validation_steps=len(valid_gen),
    workers=1,
    use_multiprocessing=False,
    shuffle=False
)

Run Code Online (Sandbox Code Playgroud)

我也不再对数据进行洗牌。通过此配置，我预计idx从 0 到 len(generator)，然后on_epoch_end调用。我有 372 个用于训练的样本和 93 个用于验证的样本，batch_size 12idx应该从 0 到 30（训练数据），分别从 0 到 7（验证数据）。但__getitem__被调用的次数比我预期的要多，并且on_epoch_end在中间也被调用！控制台输出如下所示：

batch_size: 12
len(train_gen): 31
len(valid_gen): 8
2018-02-14 08:45:09.041929: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
training idx: 0
training idx: 1
training idx: 2
training idx: 3
training idx: 4
training idx: 5
training idx: 6
training idx: 7
training idx: 8
training idx: 9
training idx: 10
training idx: 11
training idx: 12
training idx: 13
training idx: 14
training idx: 15
training idx: 16
training idx: 17
training idx: 18
training idx: 19
training idx: 20
training generator: epoch end
training idx: 21
training idx: 22
training idx: 23
training idx: 24
training idx: 25
training idx: 26
training idx: 27
training idx: 28
training idx: 29
training idx: 30
training idx: 0
validation generator: epoch end
validation idx: 0
training idx: 1
validation idx: 1
training idx: 2
validation idx: 2
training idx: 3
validation idx: 3
training idx: 4
validation idx: 4
training idx: 5
validation idx: 5
validation generator: epoch end
training idx: 6
validation idx: 6
training idx: 7
validation idx: 7
training idx: 8
validation idx: 0
training idx: 9
validation idx: 1
training idx: 10
validation idx: 2
validation idx: 3
validation idx: 4
validation idx: 5
validation idx: 6
validation idx: 7
validation idx: 0
validation idx: 1
validation idx: 2
Epoch 00000: val_loss improved from inf to 10512.69922, saving model to /home/stefan/vcs/MA/code/results/test/TB_dummy_distance_10513.hdf5
training idx: 11
training idx: 12
training idx: 13
training idx: 14
training idx: 15
training idx: 16
training idx: 17
training idx: 18
training idx: 19
training idx: 20
training generator: epoch end
training idx: 21
training idx: 22
training idx: 23
training idx: 24
training idx: 25
training idx: 26
training idx: 27
training idx: 28
training idx: 29
training idx: 30
training idx: 0
validation generator: epoch end
validation idx: 0
training idx: 1
validation idx: 1
training idx: 2
validation idx: 2
training idx: 3
validation idx: 3
training idx: 4
validation idx: 4
training idx: 5
validation idx: 5
validation generator: epoch end
training idx: 6
validation idx: 6
training idx: 7
validation idx: 7
validation idx: 0
training idx: 8
validation idx: 1
training idx: 9
validation idx: 2
training idx: 10
validation idx: 3
validation idx: 4
validation idx: 5
validation idx: 6
validation idx: 7
validation idx: 0
validation idx: 1
validation idx: 2
Epoch 00001: val_loss improved from 10512.69922 to 5905.95929, saving model to /home/stefan/vcs/MA/code/results/test/TB_dummy_distance_5906.hdf5

Run Code Online (Sandbox Code Playgroud)

fit_generator 如何使用__getitem__和on_epoch_end方法？在第一个纪元开始之前，它是否也调用这些方法来获取一些用于权重初始化的样本数据？这种行为是由某种缓存引起的吗？

非常感谢任何帮助！先感谢您！

更新：

出于测试目的，我将max_queue_size的参数更改fit_generator为 1。这是最终的终端输出：

batch_size: 12
len(train_gen): 31
len(valid_gen): 8
2018-02-14 10:10:40.001065: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
training idx: 0
training idx: 1
training idx: 2
training idx: 3
training idx: 4
training idx: 5
training idx: 6
training idx: 7
training idx: 8
training idx: 9
training idx: 10
training idx: 11
training idx: 12
training idx: 13
training idx: 14
training idx: 15
training idx: 16
training idx: 17
training idx: 18
training idx: 19
training idx: 20
training idx: 21
training idx: 22
training idx: 23
training idx: 24
training idx: 25
training idx: 26
training idx: 27
training idx: 28
training idx: 29
training idx: 30
training generator: epoch end
training idx: 0
training idx: 1
validation idx: 0
validation idx: 1
validation idx: 2
validation idx: 3
validation idx: 4
validation idx: 5
validation idx: 6
validation generator: epoch end
validation idx: 7
validation idx: 0
validation idx: 1
Epoch 00000: val_loss improved from inf to 18090.34473, saving model to /home/stefan/vcs/MA/code/results/test/TB_dummy_distance_18090.hdf5
training idx: 2
training idx: 3
training idx: 4
training idx: 5
training idx: 6
training idx: 7
training idx: 8
training idx: 9
training idx: 10
training idx: 11
training idx: 12
training idx: 13
training idx: 14
training idx: 15
training idx: 16
training idx: 17
training idx: 18
training idx: 19
training idx: 20
training idx: 21
training idx: 22
training idx: 23
training idx: 24
training idx: 25
training idx: 26
training idx: 27
training idx: 28
training idx: 29
training idx: 30
training generator: epoch end
training idx: 0
training idx: 1
validation idx: 0
validation idx: 1
validation idx: 2
validation idx: 3
validation idx: 4
validation idx: 5
validation idx: 6
validation generator: epoch end
validation idx: 7
validation idx: 0
validation idx: 1
Epoch 00001: val_loss did not improve

Run Code Online (Sandbox Code Playgroud)

现在至少在第一个时期所有训练样本都会被查询。但对于第二个 epoch 中的验证数据和训练数据，on_epoch_end仍然会被提前调用。

Answer 1

naa*_*iii 0

下面的代码将适合你

def gen(train_data):
    print('generator initiated')
    #Define a batch size
    batch_size = 64

    #Complete length of data
    data_size = len(train_data)

    #Total number of batches will be created
    num_batches = int(data_size / batch_size)


    if (num_batches*batch_size) < data_size:
         num_batches += 1
    while True:
        cnt=0
        for i in range(num_batches):
            start_index = cnt * batch_size
            end_index = min((cnt + 1) * batch_size, data_size)
            cnt +=1

            //Do some preprocessing 
            x_train_padded = add_pad(x_train,3,pad)
            x_train_padded = np.array(x_train_padded)

            yield (x_train_padded,y_train_padded)


fun_model.fit_generator(gen(train_data),steps_per_epoch =int(len(train_data)/64),nb_epoch=50,callbacks=callbacks_list, verbose=2,shuffle=True)

Run Code Online (Sandbox Code Playgroud)

感谢您的回答，但我必须使用 keras.Sequence() 实例而不是普通生成器，以避免在使用多处理时出现重复数据。我仅出于调试目的停用多处理。另外，这并不能解释为什么经常调用 `__getitem__` （以及中间的 `on_epoch_end`）。 (3认同)

归档时间：	7 年，9 月前
查看次数：	3797 次
最近记录：	6 年前

Keras fit_generator：__getitem__ 方法的意外使用

更新：

Keras fit_generator：getitem 方法的意外使用