语料库的TfidfVectorizer无法适应内存

Question

我想基于一个无法适应内存的语料库来构建一个tf-idf模型.我阅读了教程但是语料库似乎立即被加载:

from sklearn.feature_extraction.text import TfidfVectorizer
corpus = ["doc1", "doc2", "doc3"]
vectorizer = TfidfVectorizer(min_df=1)
vectorizer.fit(corpus)

我想知道我是否可以逐个将文档加载到内存中而不是加载所有文档.

Answer 1

是的,你可以,只需使你的语料库成为迭代器.例如,如果您的文档驻留在光盘上,则可以定义一个迭代器,该迭代器将文件名列表作为参数,并逐个返回文档,而不会一次将所有内容都加载到内存中.

from sklearn.feature_extraction.text import TfidfVectorizer

def make_corpus(doc_files):
    for doc in doc_files:
        yield load_doc_from_file(doc) #load_doc_from_file is a custom function for loading a doc from file

file_list = ... # list of files you want to load
corpus = make_corpus(file_list)
vectorizer = TfidfVectorizer(min_df=1)
vectorizer.fit(corpus)