将单词列表转换为这些单词出现的频率列表

Question

我正在使用各种单词列表进行大量工作.

请考虑我的以下问题:

docText={"settlement", "new", "beginnings", "wildwood", "settlement", "book",
"excerpt", "agnes", "leffler", "perry", "my", "mother", "junetta", 
"hally", "leffler", "brought", "my", "brother", "frank", "and", "me", 
"to", "edmonton", "from", "monmouth", "illinois", "mrs", "matilda", 
"groff", "accompanied", "us", "her", "husband", "joseph", "groff", 
"my", "father", "george", "leffler", "and", "my", "uncle", "andrew", 
"henderson", "were", "already", "in", "edmonton", "they", "came", 
"in", "1910", "we", "arrived", "july", "1", "1911", "the", "sun", 
"was", "shining", "when", "we", "arrived", "however", "it", "had", 
"been", "raining", "for", "days", "and", "it", "was", "very", 
"muddy", "especially", "around", "the", "cn", "train"}

searchWords={"the","for","my","and","me","and","we"}

这些列表的要长得多(说,在250个字searchWords列表,并docText为约12000字).

现在,我有能力通过执行以下操作来计算给定单词的频率:

docFrequency=Sort[Tally[docText],#1[[2]]>#2[[2]]&];    
Flatten[Cases[docFrequency,{"settlement",_}]][[2]]

但是我被挂断的地方是我要创建特定列表.具体而言,将单词列表转换为出现这些单词的频率列表的问题.我已经尝试用Do循环来做这个但是碰到了墙.

我想通过docText与searchWords和其外观的绝对频率更换docText的每个元素.即,由于"结算"出现两次,它将被列表中的2替换,而由于"我的"出现3次,它将变为3.列表将是2,1,1,1,2和等等.

我怀疑,答案就在某个地方If[]和Map[]？

这听起来很奇怪,但我试图预处理一组有关术语频率信息的信息......

---

增加清晰度(我希望):

这是一个更好的例子.

searchWords={"0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "a", "A", "about", 
"above", "across", "after", "again", "against", "all", "almost", 
"alone", "along", "already", "also", "although", "always", "among", 
"an", "and", "another", "any", "anyone", "anything", "anywhere", 
"are", "around", "as", "at", "b", "B", "back", "be", "became", 
"because", "become", "becomes", "been", "before", "behind", "being", 
"between", "both", "but", "by", "c", "C", "can", "cannot", "could", 
"d", "D", "do", "done", "down", "during", "e", "E", "each", "either", 
"enough", "even", "ever", "every", "everyone", "everything", 
"everywhere", "f", "F", "few", "find", "first", "for", "four", 
"from", "full", "further", "g", "G", "get", "give", "go", "h", "H", 
"had", "has", "have", "he", "her", "here", "herself", "him", 
"himself", "his", "how", "however", "i", "I", "if", "in", "interest", 
"into", "is", "it", "its", "itself", "j", "J", "k", "K", "keep", "l", 
"L", "last", "least", "less", "m", "M", "made", "many", "may", "me", 
"might", "more", "most", "mostly", "much", "must", "my", "myself", 
"n", "N", "never", "next", "no", "nobody", "noone", "not", "nothing", 
"now", "nowhere", "o", "O", "of", "off", "often", "on", "once", 
"one", "only", "or", "other", "others", "our", "out", "over", "p", 
"P", "part", "per", "perhaps", "put", "q", "Q", "r", "R", "rather", 
"s", "S", "same", "see", "seem", "seemed", "seeming", "seems", 
"several", "she", "should", "show", "side", "since", "so", "some", 
"someone", "something", "somewhere", "still", "such", "t", "T", 
"take", "than", "that", "the", "their", "them", "then", "there", 
"therefore", "these", "they", "this", "those", "though", "three", 
"through", "thus", "to", "together", "too", "toward", "two", "u", 
"U", "under", "until", "up", "upon", "us", "v", "V", "very", "w", 
"W", "was", "we", "well", "were", "what", "when", "where", "whether", 
"which", "while", "who", "whole", "whose", "why", "will", "with", 
"within", "without", "would", "x", "X", "y", "Y", "yet", "you", 
"your", "yours", "z", "Z"}

这些是自动生成的停用词WordData[].所以我想将这些单词与docText进行比较.由于"结算"不是其中的一部分searchWords,因此它会显示为0.但由于"我的"是其中的一部分searchWords,它会弹出计数(所以我可以告诉给定单词出现的次数).

我真的非常感谢你的帮助 - 我很期待参加一些正式的课程,因为我碰到了我能够真正解释我想做什么的能力!

Answer 1

我们可以用以下方法替换searchWords0中没有出现的所有docText内容:

preprocessedDocText = 
   Replace[docText, 
     Dispatch@Append[Thread[searchWords -> searchWords], _ -> 0], {1}]

我们可以按频率替换剩余的单词:

replaceTable = Dispatch[Rule @@@ Tally[docText]];

preprocessedDocText /. replaceTable

Dispatch预处理规则列表(->)并在后续使用中显着加快替换.

我没有对大数据进行基准测试,但Dispatch应提供良好的加速.