我知道以前也有人问过类似的问题,但他们的解决方案并不是很有帮助。我想最好的解决方案可能更具体于每个集群配置,因此我在此处提供有关我的集群和错误的更多详细信息。
import dask.dataframe as dd
import dask.bag as db
import json
from dask.distributed import Client, LocalCluster
cluster = LocalCluster()
client = Client(cluster)
Run Code Online (Sandbox Code Playgroud)
这是我的集群设置
cluster.scheduler
Run Code Online (Sandbox Code Playgroud)
#输出:
Scheduler: tcp://127.0.0.1:35367 workers: 8 cores: 48 tasks: 0
cluster.workers
Run Code Online (Sandbox Code Playgroud)
#输出:
{0: <Nanny: tcp://127.0.0.1:43789, threads: 6>,
1: <Nanny: tcp://127.0.0.1:41375, threads: 6>,
2: <Nanny: tcp://127.0.0.1:42577, threads: 6>,
3: <Nanny: tcp://127.0.0.1:40171, threads: 6>,
4: <Nanny: tcp://127.0.0.1:32867, threads: 6>,
5: <Nanny: tcp://127.0.0.1:46529, threads: 6>,
6: <Nanny: tcp://127.0.0.1:41535, threads: 6>,
7: <Nanny: tcp://127.0.0.1:39645, threads: 6>}
client
Run Code Online (Sandbox Code Playgroud)
#输出
Client
Scheduler: tcp://127.0.0.1:35367 …Run Code Online (Sandbox Code Playgroud) memory-management cluster-computing worker bigdata dask-dataframe