Python，sklearn，it-idf 如何按“####”分割，默认空格

Question

使用sklean tf-idf，默认使用空间分割

corpus = [  
'This is the first document.',  
'This is the second second document.',  
'And the third one.',  
'Is this the first document?'
]    

vectorizer = CountVectorizer()   
X = vectorizer.fit_transform(corpus)

但是，我想使用这种形式：

enter code herecorpus = [  
'This####is####the####first####document.',  
'This####is####the####second####second####document.'
]
vectorizer = CountVectorizer()   
X = vectorizer.fit_transform(corpus)
tfidf=transformer.fit_transform(vectorizer.fit_transform(documents))
word=vectorizer.get_feature_names()
weight=tfidf.toarray()

怎么做？

Answer 1

使用自定义标记器：

def four_pounds_tokenizer(s):
   return s.split('####')

vectorizer = CountVectorizer(tokenizer=four_pounds_tokenizer)
X = vectorizer.fit_transform(corpus)