使用 dask 分布式时，Pycharm 调试器会抛出 Bad 文件描述符错误

Question

我正在使用最轻量/简单的dask多处理，即非集群本地Client：

from distributed import Client
client = Client()

即便如此：第一个调用实例会产生dask.bag.compute()以下结果：

Connected to pydev debugger (build 191.7141.48)
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 383, in _on_run
    r = self.sock.recv(1024)
OSError: [Errno 9] Bad file descriptor
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 383, in _on_run
    r = self.sock.recv(1024)
OSError: [Errno 9] Bad file descriptor
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 383, in _on_run
    r = self.sock.recv(1024)
OSError: [Errno 9] Bad file descriptor

结果是，您或多或少可以抛硬币决定程序是否会继续执行或因通信异常而出错。当翻转出现“反面”时，会发生以下情况：

Connected to pydev debugger (build 191.7141.48)
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 383, in _on_run
    r = self.sock.recv(1024)
OSError: [Errno 9] Bad file descriptor
Process ForkServerProcess-3:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/distributed/core.py", line 178, in __init__
    from .counter import Digest
ImportError: cannot import name 'Digest' from 'distributed.counter' (/usr/local/lib/python3.7/site-packages/distributed/counter.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/site-packages/distributed/process.py", line 181, in _run
    target(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/distributed/nanny.py", line 587, in _run
    worker = Worker(*worker_args, **worker_kwargs)
  File "/usr/local/lib/python3.7/site-packages/distributed/worker.py", line 552, in __init__
    **kwargs
  File "/usr/local/lib/python3.7/site-packages/distributed/node.py", line 76, in __init__
    io_loop=self.io_loop,
  File "/usr/local/lib/python3.7/site-packages/distributed/core.py", line 180, in __init__
    self.digests = defaultdict(partial(Digest, loop=self.io_loop))
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/distributed/utils.py", line 179, in ignoring
    yield
SystemError: error return without exception set
distributed.nanny - WARNING - Worker process 20417 exited with status 1
Traceback (most recent call last):
  File "_pydevd_frame_eval/pydevd_frame_evaluator_darwin_37_64.pyx", line 95, in _pydevd_frame_eval.pydevd_frame_evaluator_darwin_37_64.get_bytecode_while_frame_eval
KeyError: '/usr/local/lib/python3.7/site-packages/distributed/bokeh/__init__.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1758, in <module>
    main()
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1752, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1147, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/git/huddl/python/hamspam/enron.py", line 205, in <module>
    client = Client()
  File "/usr/local/lib/python3.7/site-packages/distributed/client.py", line 712, in __init__
    self.start(timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/distributed/client.py", line 858, in start
    sync(self.loop, self._start, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/distributed/utils.py", line 331, in sync
    six.reraise(*error[0])
  File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/distributed/utils.py", line 316, in f
    result[0] = yield future
  File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
    value = future.result()
  File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 736, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/usr/local/lib/python3.7/site-packages/distributed/client.py", line 928, in _start
    yield self.cluster
  File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
    value = future.result()
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py", line 603, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 736, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/usr/local/lib/python3.7/site-packages/distributed/deploy/local.py", line 284, in _start
    yield [self._start_worker(**self.worker_kwargs) for i in range(n_workers)]
  File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
    value = future.result()
  File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
    result_list.append(f.result())
  File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.send(value)
  File "/usr/local/lib/python3.7/site-packages/distributed/deploy/local.py", line 316, in _start_worker
    raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start

对此有什么建议吗？

尝试使用某种LocalCluster模式时会出现更多问题/并发症 - 但这将被保存到另一个问题中。