使用parallel的parLapply:无法访问并行代码中的变量

Question

我最近有一台带有多个内核的计算机,我正在学习使用并行计算.我非常精通lapply并被告知parLapply工作非常相似.我虽然没有正确操作.似乎我必须明确地将所有内容放在其中parLapply以使其工作(即要使用的函数,变量等).使用lapply它从父环境读取,parLapply似乎不这样做.所以在我下面的例子中,我可以通过将所有信息放在里面来使一切工作,parLapply但如果我在用户定义的函数中使用它,我就无法明确地放入text.var其中parLapply.

library(parallel)
text.var <- rep("I like cake and ice cream so much!", 20)
ntv <- length(text.var)
gc.rate <- 10

pos <-  function(i) {
    paste(sapply(strsplit(tolower(i), " "), nchar), collapse=" | ")
}

lapply(seq_len(ntv), function(i) {
        x <- pos(text.var[i])
        if (i%%gc.rate==0) gc()
        return(x)
    }

)

#doesn't work
cl <- makeCluster(mc <- getOption("cl.cores", 4))
parLapply(cl, seq_len(ntv), function(i) {
        x <- pos(text.var[i])
        if (i%%gc.rate==0) gc()
        return(x)
    }

)

#does work but have to specify all the stuff inside parLapply
cl <- makeCluster(mc <- getOption("cl.cores", 4))
parLapply(cl, seq_len(ntv), function(i) {
        ######stuff I have to put inside parLapply##########
        text.var <- rep("I like cake and ice cream so much!", 20)
        ntv <- length(text.var)
        gc.rate <- 10
        pos <-  function(i) {
            paste(sapply(strsplit(tolower(i), " "), nchar), collapse=" | ")
        }
        ######stuff I have to put inside parLapply##########
        x <- pos(text.var[i])
        if (i%%gc.rate==0) gc()
        return(x)
    }
)

如何传递text.var,ntv,gc.rate,并pos以parLapply没有明确地把它们里面？(我猜你以某种方式将它们作为列表传递)

PS windows 7机器所以我需要用parLapply我认为

Answer 1

您需要将这些变量导出到集群中的其他R进程:

cl <- makeCluster(mc <- getOption("cl.cores", 4))
clusterExport(cl=cl, varlist=c("text.var", "ntv", "gc.rate", "pos"))

Answer 2

Martin Morgan提供的另一种方法也适用于此.

此方法直接在parLapply调用中将对象提供给集群中的每个节点,而无需使用集群导出:

library(parallel)
text.var <- rep("I like cake and ice cream so much!", 20)
ntv <- length(text.var)
gc.rate <- 10

pos <-  function(i) {
    paste(sapply(strsplit(tolower(i), " "), nchar), collapse=" | ")
}

cl <- makeCluster(mc <- getOption("cl.cores", 4))
parLapply(cl, seq_len(ntv), function(i, pos, text.var, ntv, gc.rate) {
        x <- pos(text.var[i])
        if (i%%gc.rate==0) gc()
        return(x)
    }, pos, text.var, ntv, gc.rate
)