Par*_*val 6 python scipy dbscan scikit-learn
我正在尝试使用scikit-learn的DBSCAN实现来集群化一堆文档.首先,我使用scikit-learn的TfidfVectorizer创建TF-IDF矩阵(它是numpy.float64类型的163405x13029稀疏矩阵).然后我尝试聚类该矩阵的特定子集.当子集较小时(例如,最多几千行),事情就可以正常工作.但是我得到了大的子集(有几万行)ValueError: could not convert integer scalar.
这是完整的追溯(idxs是一个索引列表):
ValueError Traceback (most recent call last)
<ipython-input-1-73ee366d8de5> in <module>()
193 # use descriptions to clusterize items
194 ncm_clusterizer = DBSCAN()
--> 195 ncm_clusterizer.fit_predict(tfidf[idxs])
196 idxs_clusters = list(zip(idxs, ncm_clusterizer.labels_))
197 for e in idxs_clusters:
/usr/local/lib/python3.4/site-packages/sklearn/cluster/dbscan_.py in fit_predict(self, X, y, sample_weight)
294 cluster labels
295 """
--> 296 self.fit(X, sample_weight=sample_weight)
297 return self.labels_
/usr/local/lib/python3.4/site-packages/sklearn/cluster/dbscan_.py in fit(self, X, y, sample_weight)
264 X = check_array(X, accept_sparse='csr')
265 clust = dbscan(X, sample_weight=sample_weight,
--> 266 **self.get_params())
267 self.core_sample_indices_, self.labels_ = clust
268 if len(self.core_sample_indices_):
/usr/local/lib/python3.4/site-packages/sklearn/cluster/dbscan_.py in dbscan(X, eps, min_samples, metric, algorithm, leaf_size, p, sample_weight, n_jobs)
136 # This has worst case O(n^2) memory complexity
137 neighborhoods = neighbors_model.radius_neighbors(X, eps,
--> 138 return_distance=False)
139
140 if sample_weight is None:
/usr/local/lib/python3.4/site-packages/sklearn/neighbors/base.py in radius_neighbors(self, X, radius, return_distance)
584 if self.effective_metric_ == 'euclidean':
585 dist = pairwise_distances(X, self._fit_X, 'euclidean',
--> 586 n_jobs=self.n_jobs, squared=True)
587 radius *= radius
588 else:
/usr/local/lib/python3.4/site-packages/sklearn/metrics/pairwise.py in pairwise_distances(X, Y, metric, n_jobs, **kwds)
1238 func = partial(distance.cdist, metric=metric, **kwds)
1239
-> 1240 return _parallel_pairwise(X, Y, func, n_jobs, **kwds)
1241
1242
/usr/local/lib/python3.4/site-packages/sklearn/metrics/pairwise.py in _parallel_pairwise(X, Y, func, n_jobs, **kwds)
1081 if n_jobs == 1:
1082 # Special case to avoid picklability checks in delayed
-> 1083 return func(X, Y, **kwds)
1084
1085 # TODO: in some cases, backend='threading' may be appropriate
/usr/local/lib/python3.4/site-packages/sklearn/metrics/pairwise.py in euclidean_distances(X, Y, Y_norm_squared, squared, X_norm_squared)
243 YY = row_norms(Y, squared=True)[np.newaxis, :]
244
--> 245 distances = safe_sparse_dot(X, Y.T, dense_output=True)
246 distances *= -2
247 distances += XX
/usr/local/lib/python3.4/site-packages/sklearn/utils/extmath.py in safe_sparse_dot(a, b, dense_output)
184 ret = a * b
185 if dense_output and hasattr(ret, "toarray"):
--> 186 ret = ret.toarray()
187 return ret
188 else:
/usr/local/lib/python3.4/site-packages/scipy/sparse/compressed.py in toarray(self, order, out)
918 def toarray(self, order=None, out=None):
919 """See the docstring for `spmatrix.toarray`."""
--> 920 return self.tocoo(copy=False).toarray(order=order, out=out)
921
922 ##############################################################
/usr/local/lib/python3.4/site-packages/scipy/sparse/coo.py in toarray(self, order, out)
256 M,N = self.shape
257 coo_todense(M, N, self.nnz, self.row, self.col, self.data,
--> 258 B.ravel('A'), fortran)
259 return B
260
ValueError: could not convert integer scalar
我正在使用Python 3.4.3(在Red Hat上),scipy 0.18.1和scikit-learn 0.18.1.
谷歌搜索我发现一个错误修复,显然解决了其他类型的稀疏矩阵(如csr)相同的问题,但不是为了咕咕声.
我已尝试向DBSCAN提供稀疏半径邻域图(而不是特征矩阵),如此处所示,但同样的错误发生.
我尝试过HDBSCAN,但同样的错误发生了.
我该如何修复此问题或绕过此问题？
即使实现允许，DBSCAN也可能会对如此高维的数据产生不好的结果（从统计的角度来看，由于维数灾难）。
相反，我建议您使用该类TruncatedSVD将 TF-IDF 特征向量的维数减少到 50 或 100 个分量，然后应用于DBSCAN结果。