我在SO上发现了上一个问题:N-gram:解释+ 2个应用程序.OP给出了这个例子并询问它是否正确:
Sentence: "I live in NY."
word level bigrams (2 for n): "# I', "I live", "live in", "in NY", 'NY #'
character level bigrams (2 for n): "#I", "I#", "#l", "li", "iv", "ve", "e#", "#i", "in", "n#", "#N", "NY", "Y#"
When you have this array of n-gram-parts, you drop the duplicate ones and add a counter for each part giving the frequency:
word level bigrams: [1, 1, 1, 1, 1]
character level bigrams: [2, 1, 1, …至少可以考虑使用3种类型的n-gram来表示文本文档:
我不清楚哪一个应该用于给定的任务(聚类,分类等).我在某处读到,当文字包含拼写错误时,字符级别的n-gram优于字级n-gram,因此"Mary loves dogs"仍然类似于"Mary lpves dogs".
选择"正确"表示还有其他标准需要考虑吗？