如何在Python中快速计算大量向量的余弦相似度？

Question

我有一组10万个向量,我需要根据余弦相似性检索前25个最接近的向量.

Scipy和Sklearn有计算余弦距离/相似度2向量的实现,但我需要计算100k X 100k大小的余弦Sim然后取出前25.python计算中有任何快速实现吗？

根据@Silmathoron建议,这就是我正在做的事情 -

#vectors is a list of vectors of size : 100K x 400 i.e. 100K vectors each of dimenions 400
vectors = numpy.array(vectors)  
similarity = numpy.dot(vectors, vectors.T)


# squared magnitude of preference vectors (number of occurrences)
square_mag = numpy.diag(similarity)

# inverse squared magnitude
inv_square_mag = 1 / square_mag

# if it doesn't occur, set it's inverse magnitude to zero (instead of inf)
inv_square_mag[numpy.isinf(inv_square_mag)] = 0

# inverse of the magnitude
inv_mag = numpy.sqrt(inv_square_mag)

# cosine similarity (elementwise multiply by inverse magnitudes)
cosine = similarity * inv_mag
cosine = cosine.T * inv_mag

k = 26

box_plot_file = file("box_data.csv","w+")

for sim,query in itertools.izip(cosine,queries):
    k_largest = heapq.nlargest(k, sim)
    k_largest = map(str,k_largest)
    result = query + "," + ",".join(k_largest) + "\n"
    box_plot_file.write(result)
box_plot_file.close()

Answer 1

我会首先尝试更智能的算法，而不是加速暴力（计算所有向量对）。如果您的向量维度较低，KDTree 可能会起作用，scipy.spatial.KDTree()。如果它们是高维度的，那么您可能首先需要一个随机投影： http ://scikit-learn.org/stable/modules/random_projection.html