如何从 spaCy 标记重建原始文本，即使在具有复杂空格和标点符号的情况下

Question

' '.join(token_list) 在连续出现多个空格和标点符号的情况下不会重建原始文本。

例如：

from spacy.tokenizer import Tokenizer
from spacy.lang.en import English

nlp = English()
# Create a blank Tokenizer with just the English vocab
tokenizerSpaCy = Tokenizer(nlp.vocab)

context_text = 'this    is a     test \n \n \t\t test for    \n testing  -  ./l \t'

contextSpaCyToksSpaCyObj = tokenizerSpaCy(context_text)
spaCy_toks = [i.text for i in contextSpaCyToksSpaCyObj]

reconstruct = ' '.join(spaCy_toks)
reconstruct == context_text

>False 

是否有一种既定的方法可以从 spaCy 令牌重建原始文本？

既定答案应适用于此边缘情况文本（您可以通过单击“改进此问题”按钮直接获取源代码）

" UNCLASSIFIED U.S. Department of State Case No. F-2014-20439 Doc No. C05795279 Date: 01/07/2016\n\n\n RELEASE IN PART\n B5, B6\n\n\n\n\nFrom: H <hrod17@clintonemail.com>\nSent: Monday, July 23, 2012 7:26 AM\nTo: 'millscd @state.gov'\nCc: 'DanielJJ@state.gov.; 'hanleymr@state.gov'\nSubject Re: S speech this morning\n\n\n\n Waiting to hear if Monica can come by and pick up at 8 to take to Josh. If I don't hear from her, can you send B5\nsomeone else?\n\n Original Message ----\nFrom: Mills, Cheryl D [MillsCD@state.gov]\nSent: Monday, July 23, 2012 07:23 AM\nTo: H\nCc: Daniel, Joshua J <Daniel1.1@state.gov>\nSubject: FW: S speech this morning\n\nSee below\n\n B5\n\ncdm\n\n Original Message\nFrom: Shah, Rajiv (AID/A) B6\nSent: Monday, July 23, 2012 7:19 AM\nTo: Mills, Cheryl D\nCc: Daniel, Joshua.'\nSubject: S speech this morning\n\nHi cheryl,\n\nI look fwd to attending the speech this morning.\n\nI had one last minute request - I understand that in the final version there is no reference to the child survival call to\naction, but their is a reference to family planning efforts. Could you and josh try to make sure there is some specific\nreference to the call to action?\n\nAlso, in terms of acknowledgements it would be good to note torn friedan's leadership as everyone is sensitive to our ghi\ntransition and we want to continue to send the usaid-pepfar-cdc working together public message. I don't know if he is\nthere, but wanted to flag.\n\nLook forward to it.\n\nRaj\n\n\n\n\n UNCLASSIFIED U.S. Department of State Case No. F-2014-20439 Doc No. C05795279 Date: 01/07/2016\n\x0c"

Answer 1

您可以通过更改代码中的两行来轻松完成此操作：

spaCy_toks = [i.text + i.whitespace_ for i in contextSpaCyToksSpaCyObj]
reconstruct = ''.join(spaCy_toks)

基本上，spaCy 中的每个标记都知道它后面是否有空格。因此，token.whitespace_默认情况下，您会调用而不是在太空中加入它们。