Doc2vec:如何获取文档向量

Question

如何使用Doc2vec获取两个文本文档的文档向量？我是新手,所以如果有人能指出我正确的方向/帮助我一些教程会很有帮助

我正在使用gensim.

doc1=["This is a sentence","This is another sentence"]
documents1=[doc.strip().split(" ") for doc in doc1 ]
model = doc2vec.Doc2Vec(documents1, size = 100, window = 300, min_count = 10, workers=4)

我明白了

AttributeError:'list'对象没有属性'words'

每当我跑这个.

Answer 1

如果要训练Doc2Vec模型,您的数据集需要包含单词列表(类似于Word2Vec格式)和标签(文档ID).它还可以包含一些其他信息(有关更多信息,请参阅https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/doc2vec-IMDB.ipynb).

# Import libraries

from gensim.models import doc2vec
from collections import namedtuple

# Load data

doc1 = ["This is a sentence", "This is another sentence"]

# Transform data (you can add more data preprocessing steps) 

docs = []
analyzedDocument = namedtuple('AnalyzedDocument', 'words tags')
for i, text in enumerate(doc1):
    words = text.lower().split()
    tags = [i]
    docs.append(analyzedDocument(words, tags))

# Train model (set min_count = 1, if you want the model to work with the provided example data set)

model = doc2vec.Doc2Vec(docs, size = 100, window = 300, min_count = 1, workers = 4)

# Get the vectors

model.docvecs[0]
model.docvecs[1]

更新(如何训练时代):Doc2Vec函数包含__CODE__和__CODE__参数,但这意味着学习速率在一个时期内衰减__CODE__到__CODE__.要训​​练多个时期,请手动设置学习率,如下所示:

# Import libraries

from gensim.models import doc2vec
from collections import namedtuple

# Load data

doc1 = ["This is a sentence", "This is another sentence"]

# Transform data (you can add more data preprocessing steps) 

docs = []
analyzedDocument = namedtuple('AnalyzedDocument', 'words tags')
for i, text in enumerate(doc1):
    words = text.lower().split()
    tags = [i]
    docs.append(analyzedDocument(words, tags))

# Train model (set min_count = 1, if you want the model to work with the provided example data set)

model = doc2vec.Doc2Vec(docs, size = 100, window = 300, min_count = 1, workers = 4)

# Get the vectors

model.docvecs[0]
model.docvecs[1]

Answer 2

Gensim已更新.LabeledSentence的语法不包含标签.现在有标签 - 请参阅LabeledSentence的文档https://radimrehurek.com/gensim/models/doc2vec.html

但是,@ bee2502是正确的

docvec = model.docvecs[99] 

它应该是训练模型的第100个向量值,它适用于整数和字符串.

Answer 3

doc=["This is a sentence","This is another sentence"]
documents=[doc.strip().split(" ") for doc in doc1 ]
model = doc2vec.Doc2Vec(documents, size = 100, window = 300, min_count = 10, workers=4)

我得到了AttributeError:'list'对象没有属性'words',因为Doc2vec()的输入文档没有正确的LabeledSentence格式.我希望下面的例子可以帮助您理解格式.

documents = LabeledSentence(words=[u'some', u'words', u'here'], labels=[u'SENT_1']) 

更多细节在这里:http: //rare-technologies.com/doc2vec-tutorial/但是,我通过使用TaggedLineDocument()从文件中获取输入数据来解决问题.
文件格式:一个文档=一行=一个TaggedDocument对象.预期单词已经过预处理并由空格分隔,标签是从文档行号自动构造的.

sentences=doc2vec.TaggedLineDocument(file_path)
model = doc2vec.Doc2Vec(sentences,size = 100, window = 300, min_count = 10, workers=4)

获取文档向量:您可以使用docvecs.更多细节:https://radimrehurek.com/gensim/models/doc2vec.html#gensim.models.doc2vec.TaggedDocument

docvec = model.docvecs[99] 

其中99是我们想要的矢量的文档ID.如果标签是整数格式(默认情况下,如果使用TaggedLineDocument()加载),请像我一样直接使用整数id.如果标签是字符串格式,请使用"SENT_99".这类似于Word2vec