这是我用来将pandas.DataFrame对象的行上的apply函数进行派发的代码:
from multiprocessing import cpu_count, Pool
from functools import partial
def parallel_applymap_df(df: DataFrame, func, num_cores=cpu_count(),**kargs):
partitions = np.linspace(0, len(df), num_cores + 1, dtype=np.int64)
df_split = [df.iloc[partitions[i]:partitions[i + 1]] for i in range(num_cores)]
pool = Pool(num_cores)
series = pd.concat(pool.map(partial(apply_wrapper, func=func, **kargs), df_split))
pool.close()
pool.join()
return series
Run Code Online (Sandbox Code Playgroud)
它适用于200 000行的子样本,但是当我尝试完整的200 000 000个示例时,出现以下错误消息:
~/anaconda3/lib/python3.6/site-packages/multiprocess/connection.py in _send_bytes(self, buf)
394 n = len(buf)
395 # For wire compatibility with 3.2 and lower
—> 396 header = struct.pack("!i", n)
397 if n > 16384:
398 # The …Run Code Online (Sandbox Code Playgroud)