小编Luc*_*sel的帖子

OSError:[Errno 12]使用python多处理池时无法分配内存

我正在尝试使用Python将一个函数并行应用于5个交叉验证集,并multiprocessing针对不同的参数值重复执行此操作,如下所示:

import pandas as pd
import numpy as np
import multiprocessing as mp
from sklearn.model_selection import StratifiedKFold

#simulated datasets
X = pd.DataFrame(np.random.randint(2, size=(3348,868), dtype='int8'))
y = pd.Series(np.random.randint(2, size=3348, dtype='int64'))

#dummy function to apply
def _work(args):
    del(args)

for C in np.arange(0.0,2.0e-3,1.0e-6):
    splitter = StratifiedKFold(n_splits=5)
    with mp.Pool(processes=5) as pool:
        pool_results = \
            pool.map(
                func=_work,
                iterable=((C,X.iloc[train_index],X.iloc[test_index]) for train_index, test_index in splitter.split(X, y))
            )
Run Code Online (Sandbox Code Playgroud)

但是在执行过程中,出现以下错误:

Traceback (most recent call last):
  File "mre.py", line 19, in <module>
    with mp.Pool(processes=5) as pool:
  File "/usr/lib/python3.5/multiprocessing/context.py", …
Run Code Online (Sandbox Code Playgroud)

python multiprocessing scikit-learn

2
推荐指数
1
解决办法
1219
查看次数

标签 统计

multiprocessing ×1

python ×1

scikit-learn ×1