我的Parallel用途:
sdf = Parallel(n_jobs=3, verbose=1, pre_dispatch='1.5*n_jobs')(delayed(char_etl)(x,k1,k2,k3) for x in X)
Run Code Online (Sandbox Code Playgroud)
X是一个列表.char_etl,是一个字符串匹配函数。
我使用的版本是:python 2.7.11、centos 6.7、joblib 0.11
每次都卡住了,像这样:
[Parallel(n_jobs=3)]: Done 89 tasks | elapsed: 10.9s
[Parallel(n_jobs=3)]: Done 258 tasks | elapsed: 46.8s
[Parallel(n_jobs=3)]: Done 552 tasks | elapsed: 1.7min
[Parallel(n_jobs=3)]: Done 902 tasks | elapsed: 2.7min
[Parallel(n_jobs=3)]: Done 997 out of 1000 | elapsed: 2.9min remaining: 0.5s
Thanks.
Run Code Online (Sandbox Code Playgroud)