标签: python-pool

我试图将列表作为参数传递给pool.map(co_refresh, input_list).但是,pool.map没有触发功能co_refresh.并且没有错误返回.看起来这个过程就在那里.

原始代码:

from multiprocessing import Pool
import pandas as pd
import os

account='xxx'
password='xxx'
threads=5
co_links='file.csv'

input_list=[]

pool = Pool(processes=threads)
def co_refresh(url, account, password, outputfile):

    print(url + ' : ' + account + ' : ' + password + ' : ' + outputfile)

    return;

link_pool = pd.read_csv(co_links, skipinitialspace = True)

for i, row in link_pool.iterrows():

    ln = (row.URL, account, password, os.path.join('e:/', row.File_Name.split('.')[0] + '.csv'))

    input_list.append(ln)

pool.map(co_refresh, input_list)

pool.close()

但是,它从未触发过该功能co_refresh.如何将列表作为参数传递给我的函数？

旧问题(简体): …

我对Python一点都不熟悉,我经常做Ruby或JS.但我需要在运行Python的系统上编写基准测试脚本.我要做的是创建一个小脚本,获取文件大小和线程数,并写一个随机缓冲区.这是我在摆弄2个小时后得到的:

from multiprocessing import Pool
import os, sys

def writeBuf(buf):
    def write(n):
        f = open(os.path.join(directory, 'n' + str(n)), 'w')
        try:
            f.write(buf)
            f.flush()
            os.fsync(f.fileno)
        finally:
            f.close()
    return write

if __name__ == '__main__':
    targetDir = sys.argv[1]
    numThreads = int(sys.argv[2])
    numKiloBytes = int(sys.argv[3])
    numFiles = int(102400 / numKiloBytes)

    buf = os.urandom(numKiloBytes * 1024)

    directory = os.path.join(targetDir, str(numKiloBytes) + 'k')
    if not os.path.exists(directory):
        os.makedirs(directory)

    with Pool(processes=numThreads) as pool:
        pool.map(writeBuf(buf), range(numFiles))

但它抛出了错误: AttributeError: Can't pickle local object 'writeBuf.<locals>.write'

我之前尝试过write没有闭包,但是当我尝试在__name__ == '__main__'部件内部定义函数时出现错误.省略这 …

我正在使用

Zope - 2.13.19
Python - 2.6.8

下面的代码在手动运行时有效,但在外部方法时则无效.它会引发以下错误.我在做概念错误的事情吗？

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/opt/python2.6/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/opt/python2.6/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/python2.6/lib/python2.6/multiprocessing/pool.py", line 225, in _handle_tasks
    put(task)
TypeError: expected string or Unicode object, NoneType found


import time
from multiprocessing import Pool
import logging


def func(name):
    print 'hello %s,' % name
    time.sleep(5)
    print 'nice to meet you.'


def get_data():
    pool = Pool(processes=2)
    pool.map(func, ('frank', 'justin', 'osi', 'thomas'))

我正在尝试在Windows 10 Intel Core i7-8550U处理器上使用Python 3.7.4进行python多处理。
我正在用两个函数测试多进程，一个函数使用基本sleep（），另一个函数使用sklearn的matthews_corrcoef。多重处理可使用睡眠功能，但不能使用sklearn功能。

import numpy as np
from sklearn.metrics import matthews_corrcoef
import time
import concurrent.futures
from multiprocessing import Process, Pool
from functools import partial
import warnings
import sys

class Runner():
  def sleeper(self, pred, man, thr = None):
    return time.sleep(2)

  def mcc_score(self, pred, man, thr = None):
    warnings.filterwarnings("ignore")
    return matthews_corrcoef(pred, man)

  def pool(self, func):
    t1 = time.perf_counter()
    p = Pool()
    meth = partial(func, pred, man)
    res = p.map(meth, thres)
    p.close()

    t2 = time.perf_counter()
    print(f'Pool {func.__name__} {round((t2-t1), 3)} …