平衡numpy数组与过采样

Question

请帮我找一个干净的方法来创建一个现有的新阵列.如果任何类的示例数小于类中的最大示例数,则应对其进行过采样.样本应取自原始数组(无论是随机还是顺序)

比方说,初始数组是这样的:

[  2,  29,  30,   1]
[  5,  50,  46,   0]
[  1,   7,  89,   1]
[  0,  10,  92,   9]
[  4,  11,   8,   1]
[  3,  92,   1,   0]

最后一列包含类:

classes = [ 0,  1,  9]

类的分布如下:

distrib = [2, 3, 1]

我需要的是创建一个具有相同数量的所有类的样本的新数组,从原始数组中随机获取,例如

[  5,  50,  46,   0]
[  3,  92,   1,   0]
[  5,  50,  46,   0] # one example added
[  2,  29,  30,   1]
[  1,   7,  89,   1]
[  4,  11,   8,   1]
[  0,  10,  92,   9]
[  0,  10,  92,   9] # two examples
[  0,  10,  92,   9] # added

Answer 1

以下代码可以满足您的需求:

a = np.array([[  2,  29,  30,   1],
              [  5,  50,  46,   0],
              [  1,   7,  89,   1],
              [  0,  10,  92,   9],
              [  4,  11,   8,   1],
              [  3,  92,   1,   0]])

unq, unq_idx = np.unique(a[:, -1], return_inverse=True)
unq_cnt = np.bincount(unq_idx)
cnt = np.max(unq_cnt)
out = np.empty((cnt*len(unq),) + a.shape[1:], a.dtype)
for j in xrange(len(unq)):
    indices = np.random.choice(np.where(unq_idx==j)[0], cnt)
    out[j*cnt:(j+1)*cnt] = a[indices]

>>> out
array([[ 5, 50, 46,  0],
       [ 5, 50, 46,  0],
       [ 5, 50, 46,  0],
       [ 1,  7, 89,  1],
       [ 4, 11,  8,  1],
       [ 2, 29, 30,  1],
       [ 0, 10, 92,  9],
       [ 0, 10, 92,  9],
       [ 0, 10, 92,  9]])

当numpy 1.9发布时,或者你从开发分支编译时,前两行可以压缩成:

unq, unq_idx, unq_cnt = np.unique(a[:, -1], return_inverse=True,
                                  return_counts=True)

请注意,方式np.random.choice有效,不能保证原始数组的所有行都将出现在输出中,如上例所示.如果需要,您可以执行以下操作:

unq, unq_idx = np.unique(a[:, -1], return_inverse=True)
unq_cnt = np.bincount(unq_idx)
cnt = np.max(unq_cnt)
out = np.empty((cnt*len(unq) - len(a),) + a.shape[1:], a.dtype)
slices = np.concatenate(([0], np.cumsum(cnt - unq_cnt)))
for j in xrange(len(unq)):
    indices = np.random.choice(np.where(unq_idx==j)[0], cnt - unq_cnt[j])
    out[slices[j]:slices[j+1]] = a[indices]
out = np.vstack((a, out))

>>> out
array([[ 2, 29, 30,  1],
       [ 5, 50, 46,  0],
       [ 1,  7, 89,  1],
       [ 0, 10, 92,  9],
       [ 4, 11,  8,  1],
       [ 3, 92,  1,  0],
       [ 5, 50, 46,  0],
       [ 0, 10, 92,  9],
       [ 0, 10, 92,  9]])

Answer 2

这给出了每个类的概率相等的随机分布:

distrib = np.bincount(a[:,-1])
prob = 1/distrib[a[:, -1]].astype(float)
prob /= prob.sum()

In [38]: a[np.random.choice(np.arange(len(a)), size=np.count_nonzero(distrib)*distrib.max(), p=prob)]
Out[38]: 
array([[ 5, 50, 46,  0],
       [ 4, 11,  8,  1],
       [ 0, 10, 92,  9],
       [ 0, 10, 92,  9],
       [ 2, 29, 30,  1],
       [ 0, 10, 92,  9],
       [ 3, 92,  1,  0],
       [ 1,  7, 89,  1],
       [ 1,  7, 89,  1]])

每个类具有相同的概率,不保证相等的发生率.