Python:无法启动新线程.<100活动线程

Question

我收到以下错误:

----- Match 93028: ------ Patch 5.11 ------78 Threads Active
----- Match 93029: ------ Patch 5.11 ------77 Threads Active
----- Match 93030: ------ Patch 5.11 ------76 Threads Active
----- Match 93031: ------ Patch 5.11 ------71 Threads Active
----- Match 93032: ------ Patch 5.11 ------55 Threads Active
----- Match 93033: ------ Patch 5.11 ------56 Threads Active
----- Match 93034: ------ Patch 5.11 ------57 Threads Active
----- Match 93035: ------ Patch 5.11 ------58 Threads Active
----- Match 93036: ------ Patch 5.11 ------59 Threads Active
Traceback (most recent call last):
  File "pulldata.py", line 91, in <module>
    getPatchData('5.11', '511')
  File "pulldata.py", line 64, in getPatchData
    matchThread.start()
  File "/usr/lib/python3.4/threading.py", line 850, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

通常这是由于打开太多线程引起的,但正如您所看到的那样,我也打印了活动的线程数.有<100个活动线程,所以我不确定问题是什么.这是相关代码:

slot = threading.BoundedSemaphore(value=1000)
def getMatchData(index,match,patch):
    global requestsSent
    global logfile
    print("----- Match {0}: ------ Patch {1} ------{2} Threads Active".format(index,patch,threading.active_count()))
    logfile.write("Parsing Match {0} for patch {1}:\n".format(index,patch))

    #match is a class. get is a function that sends a request to the server and returns a request object from where I get the json response.
    data = match.get().json()

    #processdata

    slot.release()

def getPatchData(patch, name):
    global logfile
    threads = []
    matches = getAllMatches(patch)
    for index, match in enumerate(matches):
        slot.acquire()
        matchThread = threading.Thread(target=getMatchData, args=(index,match,patch))
        threads.append(matchThread)
        matchThread.start()
        for t in threads:
            if not t.isAlive():
                threads.remove(t)

    for t in threads:
        t.join()

插槽信号量应该限制活动线程的数量,但我认为无论如何我都没有达到1000个线程.之前我假设这个错误是由于我的线程数组指向线程引起的,所以我添加了代码,当它们不再活动时将它们从数组中删除.

我无法理解为什么当只有59个活动线程时我无法启动新线程.

另外,有没有更好的方法来实现我想要做的事情？每个线程都向API发送请求.我尝试没有并发,但我甚至没有接近我的速率限制.

Answer 1

我遇到了类似的问题,这就是我如何解决它.

不确定OP使用的操作系统,但在Linux上,每个用户的进程数通常有限制.你可以用ulimit -u(或者ulimit -a)看到它.该定义有点用词不当,因为限制实际上是OS线程(或LWP)的数量.(参见接受的答复:https://superuser.com/questions/376532/does-gnu-linux-counts-processes-and-threads-together-when-i-limit-their-number)

在我的系统上,限制似乎设置为400(但可以由管理员更改).

您可以使用以下命令查看所有线程的列表:

ps -fLu <your_username>

在我的例子中,我的python应用程序将引发与OP报告的相同的异常,但threading.active_count()将返回7.

事实证明,我之前的会话中有很多遗留的过程(我曾经有点过于敏锐nohup......),每个都有几个线程,在系统中闲逛.删除它们摆脱了线程创建错误.