小编Ber*_*rdG的帖子

Python multiprocess.Pool.map无法处理大型数组。

这是我用来将pandas.DataFrame对象的行上的apply函数进行派发的代码:

from multiprocessing import cpu_count, Pool
from functools import partial

def parallel_applymap_df(df: DataFrame, func, num_cores=cpu_count(),**kargs):

partitions = np.linspace(0, len(df), num_cores + 1, dtype=np.int64)
df_split = [df.iloc[partitions[i]:partitions[i + 1]] for i in range(num_cores)]
pool = Pool(num_cores)
series = pd.concat(pool.map(partial(apply_wrapper, func=func, **kargs), df_split))
pool.close()
pool.join()

return series
Run Code Online (Sandbox Code Playgroud)

它适用于200 000行的子样本,但是当我尝试完整的200 000 000个示例时,出现以下错误消息:

~/anaconda3/lib/python3.6/site-packages/multiprocess/connection.py in _send_bytes(self, buf)
394         n = len(buf)
395         # For wire compatibility with 3.2 and lower
—> 396         header = struct.pack("!i", n)
397         if n > 16384:
398             # The …
Run Code Online (Sandbox Code Playgroud)

python multiprocess pandas

6
推荐指数
1
解决办法
248
查看次数

标签 统计

multiprocess ×1

pandas ×1

python ×1