使用Keras model.fit_generator生成器

Question

我最初尝试generator在编写用于训练Keras模型的自定义生成器时使用语法.所以我yield编辑__next__.但是,当我尝试训练我的模式时,model.fit_generator我会得到一个错误,我的生成器不是迭代器.的修复是改变yield到return其中还必要rejiggering的逻辑__next__以跟踪状态.与让yield我为我工作相比,这相当麻烦.

有没有办法可以让这个工作yield？如果我必须使用return语句,我将需要编写几个迭代器,这些迭代器必须具有非常笨重的逻辑.

Answer 1

我无法调试您的代码,因为您没有发布它,但我缩写了我为语义分段项目编写的自定义数据生成器,供您用作模板:

def generate_data(directory, batch_size):
    """Replaces Keras' native ImageDataGenerator."""
    i = 0
    file_list = os.listdir(directory)
    while True:
        image_batch = []
        for b in range(batch_size):
            if i == len(file_list):
                i = 0
                random.shuffle(file_list)
            sample = file_list[i]
            i += 1
            image = cv2.resize(cv2.imread(sample[0]), INPUT_SHAPE)
            image_batch.append((image.astype(float) - 128) / 128)

        yield np.array(image_batch)

用法:

model.fit_generator(
    generate_data('~/my_data', batch_size),
    steps_per_epoch=len(os.listdir('~/my_data')) // batch_size)

Answer 2

我最近与Keras的生成器一起玩，最后设法编写了一个示例。它使用随机数据，因此尝试在其上教授NN毫无意义，但这是为Keras使用python生成器的一个很好的例证。

产生一些数据

import numpy as np
import pandas as pd
data = np.random.rand(200,2)
expected = np.random.randint(2, size=200).reshape(-1,1)

dataFrame = pd.DataFrame(data, columns = ['a','b'])
expectedFrame = pd.DataFrame(expected, columns = ['expected'])

dataFrameTrain, dataFrameTest = dataFrame[:100],dataFrame[-100:]
expectedFrameTrain, expectedFrameTest = expectedFrame[:100],expectedFrame[-100:]

发电机

def generator(X_data, y_data, batch_size):

  samples_per_epoch = X_data.shape[0]
  number_of_batches = samples_per_epoch/batch_size
  counter=0

  while 1:

    X_batch = np.array(X_data[batch_size*counter:batch_size*(counter+1)]).astype('float32')
    y_batch = np.array(y_data[batch_size*counter:batch_size*(counter+1)]).astype('float32')
    counter += 1
    yield X_batch,y_batch

    #restart counter to yeild data in the next epoch as well
    if counter >= number_of_batches:
        counter = 0

凯拉斯模型

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten, Reshape
from keras.layers.convolutional import Convolution1D, Convolution2D, MaxPooling2D
from keras.utils import np_utils


 model = Sequential()
 model.add(Dense(12, activation='relu', input_dim=dataFrame.shape[1]))
 model.add(Dense(1, activation='sigmoid'))


 model.compile(loss='binary_crossentropy', optimizer='adadelta', metrics=['accuracy'])

 #Train the model using generator vs using the full batch
 batch_size = 8

 model.fit_generator(generator(dataFrameTrain,expectedFrameTrain,batch_size), epochs=3,steps_per_epoch = dataFrame.shape[0]/batch_size, validation_data=generator(dataFrameTest,expectedFrameTest,batch_size*2),validation_steps=dataFrame.shape[0]/batch_size*2)

 #without generator
 #model.fit(x = np.array(dataFrame), y = np.array(expected), batch_size = batch_size, epochs = 3)

输出量

Epoch 1/3
25/25 [==============================] - 3s - loss: 0.7297 - acc: 0.4750 - 
val_loss: 0.7183 - val_acc: 0.5000
Epoch 2/3
25/25 [==============================] - 0s - loss: 0.7213 - acc: 0.3750 - 
val_loss: 0.7117 - val_acc: 0.5000
Epoch 3/3
25/25 [==============================] - 0s - loss: 0.7132 - acc: 0.3750 - 
val_loss: 0.7065 - val_acc: 0.5000