我刚刚发现了熊猫的力量而我喜欢它,但我无法弄清楚这个问题:
我有一个DataFrame df.head():
lon lat h filename time
0 19.961216 80.617627 -0.077165 60048 2002-05-15 12:59:31.717467
1 19.923916 80.614847 -0.018689 60048 2002-05-15 12:59:31.831467
2 19.849396 80.609257 -0.089205 60048 2002-05-15 12:59:32.059467
3 19.830776 80.607857 0.076485 60048 2002-05-15 12:59:32.116467
4 19.570708 80.588183 0.162943 60048 2002-05-15 12:59:32.888467
我想将我的数据分组为九天
gb = df.groupby(pd.TimeGrouper(key='time', freq='9D'))
第一组:
2002-05-15 12:59:31.717467 lon lat h filename time
0 19.961216 80.617627 -0.077165 60048 2002-05-15 12:59:31.717467
1 19.923916 80.614847 -0.018689 60048 2002-05-15 12:59:31.831467
2 19.849396 80.609257 -0.089205 60048 2002-05-15 12:59:32.059467
3 19.830776 …我有大量数据，我想对其进行 kmean 分类。数据集太大了，我无法将文件加载到内存中。
我的想法是像训练数据集一样对数据集的某些部分运行分类，然后将分类应用到数据集的其余部分。
import pandas as pd
import pickle
from sklearn.cluster import KMeans
frames = [pd.read_hdf(fin) for fin in ifiles]
data = pd.concat(frames, ignore_index=True, axis=0)
data.dropna(inplace=True)
k = 12
x = pd.concat(data['A'], data['B'], data['C'], axis=1, keys=['A','B','C'])
model = KMeans(n_clusters=k, random_state=0, n_jobs = -2)
model.fit(x)
pickle.dump(model, open(filename, 'wb'))
x 看起来像这样：
array([[-2.26732099, 0.24895614, 2.34840191],
[-2.26732099, 0.22270912, 1.88942378],
[-1.99246557, 0.04154312, 2.63458941],
...,
[-4.29596287, 1.97036309, -0.22767511],
[-4.26055474, 1.72347591, -0.18185197],
[-4.15980382, 1.73176239, -0.30781225]])
该模型如下所示：
KMeans(algorithm='auto', copy_x=True, init='k-means++', max_iter=300,
n_clusters=12, n_init=10, n_jobs=-2, precompute_distances='auto',
random_state=0, tol=0.0001, …