Keras - 用于图像和蒙版的大型数据集的生成器

Question

我正在尝试构建一个具有输入和输出(掩码)图像的模型.由于数据集的大小和我有限的内存,我尝试使用Keras文档中引入的Generator方法:

# Provide the same seed and keyword arguments to the fit and flow methods
seed = 1

image_generator = image_datagen.flow_from_directory(
    'data/images',
    class_mode=None,
    seed=seed)

mask_generator = mask_datagen.flow_from_directory(
    'data/masks',
    class_mode=None,
    seed=seed)

# combine generators into one which yields image and masks
train_generator = zip(image_generator, mask_generator)

model.fit_generator(
    train_generator,
    samples_per_epoch=2000,
    nb_epoch=50)

一切似乎都有效,除非代码到达这一行:

train_generator = zip(image_generator, mask_generator)

似乎将这两个列表明确压缩的过程使它们生成了它们的内容,并且系统开始消耗大量的RAM,直到它耗尽内存.

使用Generators的目的是避免在这段代码正好相反的情况下耗尽RAM.

有什么方法可以解决这个问题吗？

Answer 1

您可以用户itertools.izip()返回迭代器而不是列表.

itertools.izip(*iterables)

Make an iterator that aggregates elements from each of the iterables. Like zip() except that it returns an iterator instead of a list. Used for lock-step iteration over several iterables at a time.