Python Multiprocessing Queue when set to infinite is capped at 32768 (2^15)

Question

I have a list with about 800000 elements (small strings) that are loaded into a Queue which is then consumed by different worker processes from a multiprocessing pool. I've found out that both in PyPy and in Python (2.7 and 3.6 respectively), even though I've set the Queue's maxsize explicitly to 0, the Queue in both cases is capped at 32768 elements at any given time, and therefore blocks on the 32768th element.

Why is this happening? I thought they were supposed to be infinite if maxsize is <= 0? I've gone over this StackOverflow question Python Queue raising Full even when infinite but it's the only one of such nature. Is there something else I may be missing?

I've tried the implementations of a multiprocessing Queue where I load a million integers and the queue.put(val) method always blocks on the 32768th value.

from multiprocessing import Queue
q = Queue(maxsize=0)
for i in range(int(1e7)):
    q.put(i)
    print(i)

I was expecting to be able to insert all 1 million integers into the Queue, but as it turns out, it's unable to hold them all as it blocks on the 32768th integer. I'd love to have some light shed on the specifics of why this may be happening, which may already have been answered in the other StackOverflow question linked above, but it seems that the user that made the answer asked whether we were using a 32bit Python distribution, which is not my case, as I am using a 64bit Python distribution in both cases, as can be seen here (for PyPy with 2.7.13, which is what I'm using in my project):

Python 2.7.13 (990cef41fe11, Mar 21 2019, 12:15:10)
[PyPy 7.1.0 with GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys;print("%x" % sys.maxsize, sys.maxsize > 2**32)
('7fffffffffffffff', True)

UPDATE:

I've noticed something very interesting. This happened while running the Queue on MacOS, but I ran the code in a docker container with Linux and the queue was effectively loaded in full with all 800000 elements at once! It seems that this has something to do with MacOS.

Answer 1

多处理队列挂起意味着您很可能超过了最大信号量计数；即 OSX 不能算那么高，并隐含地将您限制为 2^15 - 1。

我不确定源代码的可靠性/最新性如何，但这似乎与 osx 的最大值一致（https://github.com/st3fan/osx-10.9/blob/master/xnu- 2422.1.72/bsd/sys/semaphore.h )

编辑/证明：

明确尝试使用比 2^15 - 1 更大的限制失败：

>>> Queue(maxsize=2**15)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/context.py", line 102, in Queue
    return Queue(maxsize, ctx=self.get_context())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/queues.py", line 48, in __init__
    self._sem = ctx.BoundedSemaphore(maxsize)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/context.py", line 87, in BoundedSemaphore
    return BoundedSemaphore(value, ctx=self.get_context())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/synchronize.py", line 145, in __init__
    SemLock.__init__(self, SEMAPHORE, value, value, ctx=ctx)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/synchronize.py", line 59, in __init__
    unlink_now)
OSError: [Errno 22] Invalid argument

在您的情况下，您已尝试隐式创建一个无限大小 (<=0) 的队列，但这并不是实际发生的情况，并且应用了 OSX 限制：https : //github.com/python/cpython/blob/master/ Lib/multiprocessing/queues.py#L37