我希望并行运行几个模拟实例,但每个模拟都有自己独立的数据集.
目前我实现如下:
P = mp.Pool(ncpus) # Generate pool of workers
for j in range(nrun): # Generate processes
sim = MDF.Simulation(tstep, temp, time, writeout, boundaryxy, boundaryz, relax, insert, lat,savetemp)
lattice = MDF.Lattice(tstep, temp, time, writeout, boundaryxy, boundaryz, relax, insert, lat, kb, ks, kbs, a, p, q, massL, randinit, initvel, parangle,scaletemp,savetemp)
adatom1 = MDF.Adatom(tstep, temp, time, writeout, boundaryxy, boundaryz, relax, insert, lat, ra, massa, amorse, bmorse, r0, z0, name, lattice, samplerate,savetemp)
P.apply_async(run,(j,sim,lattice,adatom1),callback=After) # run simulation and ISF analysis in each process
P.close() …我在用 multiprocessing.Pool()
这是我想要的池:
def insert_and_process(file_to_process,db):
db = DAL("path_to_mysql" + db)
#Table Definations
db.table.insert(**parse_file(file_to_process))
return True
if __name__=="__main__":
file_list=os.listdir(".")
P = Pool(processes=4)
P.map(insert_and_process,file_list,db) # here having problem.
我想传递2个参数我想要做的只是初始化4个DB连接(这里将尝试在每个函数调用上创建连接,因此可能有数百万个连接并导致IO Freezed死亡).如果我可以为每个进程创建4个数据库连接和1个,那就可以.
Pool有什么解决方案吗？还是我应该抛弃它？
编辑:
从你们两个人的帮助下我得到了这个:
args=zip(f,cycle(dbs))
Out[-]:
[('f1', 'db1'),
('f2', 'db2'),
('f3', 'db3'),
('f4', 'db4'),
('f5', 'db1'),
('f6', 'db2'),
('f7', 'db3'),
('f8', 'db4'),
('f9', 'db1'),
('f10', 'db2'),
('f11', 'db3'),
('f12', 'db4')]
所以这里它将如何工作,我将数据库连接代码移动到主要级别并执行此操作:
def process_and_insert(args):
#Table Definations
args[1].table.insert(**parse_file(args[0]))
return True
if __name__=="__main__":
file_list=os.listdir(".")
P = Pool(processes=4)
dbs = [DAL("path_to_mysql/database") for i in range(0,3)]
args=zip(file_list,cycle(dbs))
P.map(insert_and_process,args) …当我尝试使用简单的模型进行预测时，我得到了以下错误：
在图表中找不到在feed_devices或fetch_devices中指定的张量输入_1：0
在行：
seatbelt_model.predict(image_arr, verbose=1)
在代码中：
from tensorflow import keras
import tensorflow as tf
import numpy as np
graph = tf.get_default_graph()
seatbelt_model = keras.models.load_model(filepath='./graphs/seatbelt_A_3_81.h5')
class SeatbeltPredictor:
INPUT_SHAPE = (-1, 120, 160, 1)
@staticmethod
def predict_seatbelt(image_arr):
with graph.as_default():
image_arr = np.array(image_arr).reshape(SeatbeltPredictor.INPUT_SHAPE)
predicted_labels = seatbelt_model.predict(image_arr, verbose=1)
return predicted_labels
该模型具有以下形状：
input_layer = keras.layers.Input(shape=(IMAGE_HEIGHT, IMAGE_WIDTH, 1))
conv_0 = keras.layers.Conv2D(filters=32, kernel_size=[5, 5], activation=tf.nn.relu, padding="SAME")(input_layer)
pool_0 = keras.layers.MaxPool2D(pool_size=[2, 2], strides=2, padding="VALID")(conv_0)
conv_1 = keras.layers.Conv2D(filters=32, kernel_size=[5, 5], activation=tf.nn.relu, padding="SAME")(pool_0)
pool_1 = keras.layers.MaxPool2D(pool_size=[2, 2], strides=2, padding="VALID")(conv_1)
flat_0 = …