如何使用gensim LDA获取文档的完整主题分发？

Question

当我训练我的lda模型时

dictionary = corpora.Dictionary(data)
corpus = [dictionary.doc2bow(doc) for doc in data]
num_cores = multiprocessing.cpu_count()
num_topics = 50
lda = LdaMulticore(corpus, num_topics=num_topics, id2word=dictionary, 
workers=num_cores, alpha=1e-5, eta=5e-1)

我希望num_topics为每个文档获得所有内容的完整主题分发.也就是说,在这种特殊情况下,我希望每个文档都有50个主题有助于分发,我希望能够访问所有50个主题的贡献.如果严格遵守LDA的数学,这个输出就是LDA应该做的.然而,仅gensim输出超过一定的阈值,如图主题在这里.例如,如果我尝试

lda[corpus[89]]
>>> [(2, 0.38951721864890398), (9, 0.15438596408262636), (37, 0.45607443684895665)]

它只显示了对文档89贡献最大的3个主题.我在上面的链接中尝试了解决方案,但这对我不起作用.我仍然得到相同的输出:

theta, _ = lda.inference(corpus)
theta /= theta.sum(axis=1)[:, None]

产生相同的输出,即每个文档只有2,3个主题.

我的问题是如何更改此阈值,以便我可以访问每个文档的完整主题分布？无论主题对文档的贡献多么微不足道,我如何访问完整的主题分发？我想要完整分发的原因是我可以在文档的分发之间执行KL相似性搜索.

提前致谢

Answer 1

似乎没有人回复,所以我会尽力回答这个问题,因为我可以给出gensim 文档.

minimum_probability在训练模型以获得所需结果时,您似乎需要将参数设置为0.0:

lda = LdaMulticore(corpus=corpus, num_topics=num_topics, id2word=dictionary, workers=num_cores, alpha=1e-5, eta=5e-1,
              minimum_probability=0.0)

lda[corpus[233]]
>>> [(0, 5.8821799358842424e-07),
 (1, 5.8821799358842424e-07),
 (2, 5.8821799358842424e-07),
 (3, 5.8821799358842424e-07),
 (4, 5.8821799358842424e-07),
 (5, 5.8821799358842424e-07),
 (6, 5.8821799358842424e-07),
 (7, 5.8821799358842424e-07),
 (8, 5.8821799358842424e-07),
 (9, 5.8821799358842424e-07),
 (10, 5.8821799358842424e-07),
 (11, 5.8821799358842424e-07),
 (12, 5.8821799358842424e-07),
 (13, 5.8821799358842424e-07),
 (14, 5.8821799358842424e-07),
 (15, 5.8821799358842424e-07),
 (16, 5.8821799358842424e-07),
 (17, 5.8821799358842424e-07),
 (18, 5.8821799358842424e-07),
 (19, 5.8821799358842424e-07),
 (20, 5.8821799358842424e-07),
 (21, 5.8821799358842424e-07),
 (22, 5.8821799358842424e-07),
 (23, 5.8821799358842424e-07),
 (24, 5.8821799358842424e-07),
 (25, 5.8821799358842424e-07),
 (26, 5.8821799358842424e-07),
 (27, 0.99997117731831464),
 (28, 5.8821799358842424e-07),
 (29, 5.8821799358842424e-07),
 (30, 5.8821799358842424e-07),
 (31, 5.8821799358842424e-07),
 (32, 5.8821799358842424e-07),
 (33, 5.8821799358842424e-07),
 (34, 5.8821799358842424e-07),
 (35, 5.8821799358842424e-07),
 (36, 5.8821799358842424e-07),
 (37, 5.8821799358842424e-07),
 (38, 5.8821799358842424e-07),
 (39, 5.8821799358842424e-07),
 (40, 5.8821799358842424e-07),
 (41, 5.8821799358842424e-07),
 (42, 5.8821799358842424e-07),
 (43, 5.8821799358842424e-07),
 (44, 5.8821799358842424e-07),
 (45, 5.8821799358842424e-07),
 (46, 5.8821799358842424e-07),
 (47, 5.8821799358842424e-07),
 (48, 5.8821799358842424e-07),
 (49, 5.8821799358842424e-07)]

Answer 2

如果它可以帮助别人：

训练好LDA模型后，如果想要获取文档的所有主题，且不限制较低的阈值，则应在调用get_document_topics方法时将minimum_probability设置为0。

ldaModel.get_document_topics(bagOfWordOfADocument, minimum_probability=0.0)