错误:找不到对象'.doSnowGlobals'？

Question

我正在尝试并行化4个节点上的代码(type ="SOCK").这是我的代码.

library(itertools)
library(foreach)
library(doParallel)
library(parallel)

workers <- ip address of 4 nodes
cl = makePSOCKcluster(workers, master="ip address of master")
registerDoParallel(cl)

z <- read.csv("ProcessedData.csv", header=TRUE, as.is=TRUE)
z <- as.matrix(z)


system.time({
  chunks <- getDoParWorkers()
  b <- foreach (these = isplitIndices(nrow(z),
                                      chunks=chunks),
                .combine = c) %dopar% {
                  a <- rep(0, length(these))
                  for (i in 1:length(these)) {
                    a[i] <- mean(z[these[i],])
                  }
                  a
                }
})

我收到此错误:

4个节点产生错误; 第一个错误:找不到对象'.doSnowGlobals'.

如果我使用doMC即使用相同的机器核心,此代码运行正常.但是当我尝试使用其他计算机进行并行计算时,我得到了上述错误.当我将其更改为registerDoSNOW时,错误仍然存​​在.

snow和DoSNOW是否在群集中工作？我可以使用snow在localhost上创建节点,但不能在群集上创建节点.谁在那里用雪？

Answer 1

如果任何worker无法加载doParallel包,则会出现此错误.您可以通过将doParallel安装到某个目录并通过".libPaths"将主服务器指向它来实现这一点:

> .libPaths('~/R/lib.test')
> library(doParallel)
> cl <- makePSOCKcluster(3, outfile='')
starting worker pid=26240 on localhost:11566 at 13:47:59.470
starting worker pid=26248 on localhost:11566 at 13:47:59.667
starting worker pid=26256 on localhost:11566 at 13:47:59.864
> registerDoParallel(cl)
> foreach(i=1:10) %dopar% i
Warning: namespace ‘doParallel’ is not available and has been replaced
by .GlobalEnv when processing object ‘’
Warning: namespace ‘doParallel’ is not available and has been replaced
by .GlobalEnv when processing object ‘’
Warning: namespace ‘doParallel’ is not available and has been replaced
by .GlobalEnv when processing object ‘’
Error in checkForRemoteErrors(lapply(cl, recvResult)) : 
  3 nodes produced errors; first error: object '.doSnowGlobals' not found

当doParallel函数在worker上反序列化时会发出警告.执行该函数时会发生错误并尝试访问在doParallel命名空间中定义的.doSnowGlobal,而不是在.GlobalEnv中.

您还可以通过执行以下命令来验证worker上的doParallel是否可用:

> clusterEvalQ(cl, library(doParallel))
Error in checkForRemoteErrors(lapply(cl, recvResult)) : 
  3 nodes produced errors; first error: there is no package called ‘doParallel’

Answer 2

@Steve Weston 的答案的一个具体情况是，当您的工作人员无法加载给定的包（例如 doParallel）时，因为该包位于 Packrat 项目内。将软件包安装到系统库或工作人员能够找到它们的其他位置。

Answer 3

要在每个工作程序上设置库路径，您可以运行：

clusterEvalQ(cl, .libPaths("Your library path"))