我正在尝试使用 concurrent.futures 模块将一个需要很长时间的进程拆分为多个进程。附上下面的代码
主功能:
with concurrent.futures.ProcessPoolExecutor() as executor:
for idx, score in zip([idx for idx in range(dataframe.shape[0])],executor.map(get_max_fuzzy_score,[dataframe[idx:idx+1] for idx in range(dataframe.shape[0])])):
print('processing '+str(idx+1)+' of '+str(dataframe.shape[0]+1))
dataframe['max_row_score'].iloc[idx] = score
Run Code Online (Sandbox Code Playgroud)
get_max_fuzzy_score 函数:
def get_max_fuzzy_score(picklepath_or_list, df):
import numpy as np
extracted_text_columns = list(df.filter(regex='extracted_text').columns)
data_list = [df[data].iloc[0] for data in extracted_text_columns if not df[data].isnull().values.any()]
try:
size = len(picklepath_or_list)
section_snippet_list = picklepath_or_list
except:
section_snippet_list = pickle.load(open(picklepath_or_list,'rb'))
scores = []
for section_snippet in section_snippet_list:
for data in data_list:
scores.append(fuzz.partial_ratio(data,section_snippet))
score = max(scores)
return score
Run Code Online (Sandbox Code Playgroud)
该函数采用几列的值,并从先前构建的列表中返回最大模糊分数。
这是我得到的错误: …
parallel-processing distributed-computing python-3.x concurrent.futures python-multiprocessing
python-3.x ×1