使用Keras的python生成器线程安全性

Question

我正在使用Keras进行一些ML并使用此生成器来处理数据和标签:

def createBatchGenerator(driving_log,batch_size=32):
    batch_images = np.zeros((batch_size, 66, 200, 3))
    batch_steering = np.zeros(batch_size)
    while 1:
        for i in range(batch_size):
            x,y = get_preprocessed_row(driving_log)
            batch_images[i]=x
            batch_steering[i]=y
        yield batch_images, batch_steering

当我在本地使用它时运行正常,但是当我在带有GPU的AWS g2.2xlarge上运行它时,我收到此错误"ValueError:generator already execution".有人可以帮我解决这个问题吗？

Answer 1

您需要创建一个可以支持多线程的生成器,以确保一次由两个线程调用生成器:

import threading

class createBatchGenerator:

    def __init__(self, driving_log,batch_size=32):
        self.driving_log = driving_log
        self.batch_size = batch_size
        self.lock = threading.Lock()

    def __iter__(self):
        return self

    def __next__(self):
        with self.lock:
           batch_images = np.zeros((batch_size, 66, 200, 3))
           batch_steering = np.zeros(batch_size)

           for i in range(self.batch_size):
               x,y = get_preprocessed_row(self.driving_log)
               batch_images[i]=x
               batch_steering[i]=y
           return batch_images, batch_steering