Parallel fitting of multiple Keras Models on single GPU

Question

I'm trying to fit multiple small Keras models in parallel on a single GPU. Because of reasons i need to get them out of a list and train them one step at a time. Since I was not lucky with the standard multiprocessing module i use pathos.

What I tried to do is something like this:

from pathos.multiprocessing import ProcessPool as Pool
import tensorflow as tf
import keras.backend as K

def multiprocess_step(self, model):
    K.set_session(sess)
    with sess.graph.as_default():
        model = step(model, sess)
        return model

def step(model, sess):
    K.set_session(sess)
    with sess.graph.as_default():
        model.fit(x=data['X_train'], y=data['y_train'],
               batch_size=batch_size
               validation_data=(data['X_test'], data['y_test']), 
               verbose=verbose,
               shuffle=True,
               initial_epoch=self.step_num - 1)
        return model

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
config.gpu_options.visible_device_list = "0"
sess = tf.Session(config=config)

K.set_session(sess)
with sess.graph.as_default():
    pool = Pool(8).map
    model_list = pool(multiprocess_step, model_list)

but whatever I try I keep getting an error claiming that the models dont seem to be on the same graph...

ValueError: Tensor("training/RMSprop/Variable:0", shape=(25, 352), dtype=float32_ref) must be from the same graph as Tensor("RMSprop/rho/read:0", shape=(), dtype=float32).

The exception originates in the model.fit() row so I must have done something wrong with the assignment of the session graph even though I tried to set that in every possible location?

Does anyone have experience with something similar?

Answer 1

Keras 问题跟踪器上提出了以下建议。与使用多处理相比，我不确定该方法的相对优点。

in_1 = Input()
lstm_1 = LSTM(...)(in_1)
out_1 = Dense(...)(lstm_1)

in_2 = Input()
lstm_2 = LSTM(...)(in_2)
out_2 = Dense(...)(lstm_2)

model_1 = Model(input=in_1, output=out_1)
model_2 = Model(input=in_2, output=out_2)

model = Model(input = [in_1, in_2], output = [out_1, out_2])
model.compile(...)
model.fit(...)

model_1.predict(...)
model_2.predict(...)