在R中查找ngrams并在语料库中比较ngrams

Question

我正在开始使用R中的tm软件包,所以请耐心等待,并为大文本墙做道歉.我创造了一个相当大的社会主义/共产主义宣传语料库,并希望提取新创造的政治术语(多个词,例如"斗争 - 批评 - 转型运动").

这是一个两步的问题,一个关于我的代码到目前为止,一个关于我应该如何继续.

第1步:为此,我想首先确定一些常见的ngram.但是我很早就陷入了困境.这是我一直在做的事情:

library(tm)
library(RWeka)

a  <-Corpus(DirSource("/mycorpora/1965"), readerControl = list(language="lat")) # that dir is full of txt files
summary(a)  
a <- tm_map(a, removeNumbers)
a <- tm_map(a, removePunctuation)
a <- tm_map(a , stripWhitespace)
a <- tm_map(a, tolower)
a <- tm_map(a, removeWords, stopwords("english")) 
a <- tm_map(a, stemDocument, language = "english") 
# everything works fine so far, so I start playing around with what I have
adtm <-DocumentTermMatrix(a) 
adtm <- removeSparseTerms(adtm, 0.75)

inspect(adtm) 

findFreqTerms(adtm, lowfreq=10) # find terms with a frequency higher than 10

findAssocs(adtm, "usa",.5) # just looking for some associations  
findAssocs(adtm, "china",.5)

# ... and so on, and so forth, all of this works fine

我加载到R中的语料库工作正常,我投入的大多数函数.我没有遇到任何问题从我的语料库创建TDM,找到频繁的单词,关联,创建单词云等等.但是当我尝试使用tm FAQ中概述的方法使用识别ngrams时,我显然在使用tdm构造函数时犯了一些错误:

# Trigram

TrigramTokenizer <- function(x) NGramTokenizer(x, 
                                Weka_control(min = 3, max = 3))

tdm <- TermDocumentMatrix(a, control = list(tokenize = TrigramTokenizer))

inspect(tdm)

我收到此错误消息:

Error in rep(seq_along(x), sapply(tflist, length)) : 
invalid 'times' argument
In addition: Warning message:
In is.na(x) : is.na() applied to non-(list or vector) of type 'NULL'

有任何想法吗？"a"不是正确的类/对象吗？我糊涂了.我认为这里有一个根本性的错误,但我没有看到它.:(

第2步:然后,当我将语料库与其他语料库进行比较时,我想确定明显过多的ngrams.例如,我可以将我的语料库与大型标准英语语料库进行比较.或者我创建可以相互比较的子集(例如苏联与中国共产党的术语).你对我这样做有什么建议吗？我应该研究哪些脚本/函数？只是一些想法或指针会很棒.

谢谢你的耐心!

Answer 1

我无法重现您的问题,您使用的是最新版本的R,tm,RWeka等吗？

require(tm)
a <- Corpus(DirSource("C:\\Downloads\\Only1965\\Only1965"))
summary(a)  
a <- tm_map(a, removeNumbers)
a <- tm_map(a, removePunctuation)
a <- tm_map(a , stripWhitespace)
a <- tm_map(a, tolower)
a <- tm_map(a, removeWords, stopwords("english")) 
# a <- tm_map(a, stemDocument, language = "english") 
# I also got it to work with stemming, but it takes so long...
adtm <-DocumentTermMatrix(a) 
adtm <- removeSparseTerms(adtm, 0.75)

inspect(adtm) 

findFreqTerms(adtm, lowfreq=10) # find terms with a frequency higher than 10
findAssocs(adtm, "usa",.5) # just looking for some associations  
findAssocs(adtm, "china",.5)

# Trigrams
require(RWeka)
TrigramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))
tdm <- TermDocumentMatrix(a, control = list(tokenize = TrigramTokenizer))
tdm <- removeSparseTerms(tdm, 0.75)
inspect(tdm[1:5,1:5])

而这就是我得到的

A term-document matrix (5 terms, 5 documents)

Non-/sparse entries: 11/14
Sparsity           : 56%
Maximal term length: 28 
Weighting          : term frequency (tf)

                                   Docs
Terms                               PR1965-01.txt PR1965-02.txt PR1965-03.txt
  †chinese press                              0             0             0
  †renmin ribao                               0             1             1
  — renmin ribao                              2             5             2
  “ chinese people                            0             0             0
  “renmin ribaoâ€\u009d editorial             0             1             0
  etc. 

关于你的第二步,这里有一些指向有用的开头:

http://quantifyingmemory.blogspot.com/2013/02/mapping-significant-textual-differences.html http://tedunderwood.com/2012/08/14/where-to-start-with-text-mining/和这是他的代码https://dl.dropboxusercontent.com/u/4713959/Neuchatel/NassrProgram.R