Java使用中的Stanford POS标记

Question

Mar 9, 2011 1:22:06 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+FFFD, decimal: 65533)
Mar 9, 2011 1:22:06 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+FFFD, decimal: 65533)
Mar 9, 2011 1:22:06 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+FFFD, decimal: 65533)
Mar 9, 2011 1:22:06 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+FFFD, decimal: 65533)
Mar 9, 2011 1:22:06 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+FFFD, decimal: 65533)
Mar 9, 2011 1:22:06 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+FFFD, decimal: 65533)
Mar 9, 2011 1:22:06 PM edu.stanford.nlp.process.PTBLexer next
WARNING: Untokenizable: ? (U+FFFD, decimal: 65533)

当我想将POS标签分配给句子时,这些是我得到的错误.我从文件中读取句子.最初(少数句子)我没有得到这个错误(即不可言说的),但在读完一些句子之后会出现这个错误.我使用POS标签的v2.0(即2009),模型是left3words.

Answer 1

我同意Yuval - 一个字符编码问题,但最常见的情况是文件是单字节编码,例如ISO-8859-1而标记器试图用UTF-8读取它.请参阅Wikipedia上对U + FFFD的讨论.