如何在R中写入map reduce？

Question

我是R.的新手.我知道如何在Java中编写map reduce.我想在R中尝试相同.所以任何人都可以帮助提供任何samle代码,并且R中的MapReduce是否有任何固定格式

请发送除此之外的任何链接:https://github.com/RevolutionAnalytics/RHadoop/wiki/Tutorial

任何示例代码都会更有帮助.

Answer 1

当您想用 Java 以外的语言实现映射缩减（使用 Hadoop）时，您可以使用称为流式处理的功能。然后数据通过 STDIN (readLines()) 馈送到映射器，通过 STDOUT(cat()) 返回到 Hadoop，然后通过 STDIN (readLines()) 再次馈送到减速器，最后通过 STDOUT (cat()) 模糊化。

以下代码摘自我撰写的一篇关于使用 R for Hadoop 编写映射缩减作业的文章。该代码应该计算 2 克，但我想说它足够简单，足以了解 MapReduce 方面发生了什么。

# map.R

library(stringdist, quietly=TRUE)

input <- file("stdin", "r")

while(length(line <- readLines(input, n=1, warn=FALSE)) > 0) {
   # in case of empty lines
   # more sophisticated defensive code makes sense here
   if(nchar(line) == 0) break

   fields <- unlist(strsplit(line, "\t"))

   # extract 2-grams
   d <- qgrams(tolower(fields[4]), q=2)

   for(i in 1:ncol(d)) {
     # language / 2-gram / count
     cat(fields[2], "\t", colnames(d)[i], "\t", d[1,i], "\n")
   }
}

close(input)

-

# reduce.R

input <- file("stdin", "r")

# initialize variables that keep
# track of the state

is_first_line <- TRUE

while(length(line <- readLines(input, n=1, warn=FALSE)) > 0) {
   line <- unlist(strsplit(line, "\t"))
   # current line belongs to previous
   # line's key pair
   if(!is_first_line &&
      prev_lang == line[1] &&
      prev_2gram == line[2]) {
        sum <- sum + as.integer(line[3])
   }
   # current line belongs either to a
   # new key pair or is first line
   else {
     # new key pair - so output the last
     # key pair's result
     if(!is_first_line) {
       # language / 2-gram / count
       cat(prev_lang,"\t",prev_2gram,"\t",sum,"\n")
     }
     # initialize state trackers
     prev_lang <- line[1]
     prev_2gram <- line[2]
     sum <- as.integer(line[3])
     is_first_line <- FALSE
   }
}

# the final record
cat(prev_lang,"\t",prev_2gram, "\t", sum, "\n")

close(input)

http://www.joyofdata.de/blog/mapreduce-r-hadoop-amazon-emr/