R:内存耗尽,如何循环遍历行？

Question

我有一个700.000+行的数据帧(myDF),每行有两列,id和text.该文本有140个字符文本(推文),我想运行一个情绪分析,我从网上得到它们.但是,无论我尝试什么,我都会在4gb内存的macbook上出现内存问题.

我在想,也许我可以遍历行,例如前10行,然后是第10行......等等.(即使批量为100,我也会遇到问题)这会解决问题吗？以这种方式循环的最佳方法是什么？

我在这里发布我的代码:

library(plyr)
library(stringr)

# function score.sentiment
score.sentiment = function(sentences, pos.words, neg.words, .progress='none')
{
   # Parameters
   # sentences: vector of text to score
   # pos.words: vector of words of postive sentiment
   # neg.words: vector of words of negative sentiment
   # .progress: passed to laply() to control of progress bar

   # create simple array of scores with laply
   scores = laply(sentences,
   function(sentence, pos.words, neg.words)
   {

      # split sentence into words with str_split (stringr package)
      word.list = str_split(sentence, "\\s+")
      words = unlist(word.list)

      # compare words to the dictionaries of positive & negative terms
      pos.matches = match(words, pos.words)
      neg.matches = match(words, neg.words)

      # get the position of the matched term or NA
      # we just want a TRUE/FALSE
      pos.matches = !is.na(pos.matches)
      neg.matches = !is.na(neg.matches)

      # final score
    score = sum(pos.matches)- sum(neg.matches)
      return(score)
      }, pos.words, neg.words, .progress=.progress )

   # data frame with scores for each sentence
   scores.df = data.frame(text=sentences, score=scores)
   return(scores.df)
}

# import positive and negative words
pos = readLines("positive_words.txt")
neg = readLines("negative_words.txt")

# apply function score.sentiment


myDF$scores = score.sentiment(myDF$text, pos, neg, .progress='text') 

Answer 1

对于700,000个140个字符的句子来说,4 GB听起来就足够了.计算情绪分数的另一种方法可能是更多的记忆和时间效率和/或更容易分解成块.而不是处理每个句子,将整组句子分成单词

words <- str_split(sentences, "\\s+")

然后确定每个句子中有多少个单词,并创建单个单词向量

len <- sapply(words, length)
words <- unlist(words, use.names=FALSE)

通过重用words变量,我释放了之前用于重新循环的内存(不需要显式调用垃圾收集器,这与@ cryo111中的建议相反!).你可以找到一个单词是否在pos.words,而不用担心NAs words %in% pos.words.但是我们可以有点聪明并计算这个逻辑向量的累积和,然后将每个句子中最后一个单词的累积和进行子集化

cumsum(words %in% pos.words)[len]

并计算单词的数量作为其衍生物

pos.match <- diff(c(0, cumsum(words %in% pos.words)[len]))

这是pos.match你得分的一部分.所以

scores <- diff(c(0, cumsum(words %in% pos.words)[len])) - 
          diff(c(0, cumsum(words %in% neg.words)[len]))

就是这样.

score_sentiment <-
    function(sentences, pos.words, neg.words)
{
    words <- str_split(sentences, "\\s+")
    len <- sapply(words, length)
    words <- unlist(words, use.names=FALSE)
    diff(c(0, cumsum(words %in% pos.words)[len])) - 
      diff(c(0, cumsum(words %in% neg.words)[len]))
}

这里的意图是一次性处理你的所有句子

myDF$scores <- score_sentiment(myDF$text, pos, neg)

这避免了与向量化解决方案相比,虽然与@joran所指示的正确实现的lapply朋友相比并不具有本质上低效的循环,但是效率非常低.可能不会被复制到此处,并且返回(仅)该分数不会浪费我们已经知道的记忆返回信息(句子).最大的记忆将是和.sentencessentenceswords

如果内存仍然存在问题,那么我将创建一个索引,可用于将文本拆分为更小的组,并计算每个组的得分

nGroups <- 10 ## i.e., about 70k sentences / group
idx <- seq_along(myDF$text)
grp <- split(idx, cut(idx, nGroups, labels=FALSE))
scorel <- lapply(grp, function(i) score_sentiment(myDF$text[i], pos, neg))
myDF$scores <- unlist(scorel, use.names=FALSE)

确保首先确实myDF$text是一个角色,例如,myDF$test <- as.character(myDF$test)