如何在numpy savetxt中格式化,使零仅保存为"0"

Question

我正在将numpy稀疏数组(已删除)保存到csv中.结果是我有一个3GB的csv.问题是95%的细胞是0.0000.我用过fmt='%5.4f'.如何格式化和保存,使零保存为0,非零浮点数以'%5.4f'格式保存？如果我能做到这一点,我相信我可以将3GB降至300MB.

我在用

np.savetxt('foo.csv', arrayDense, fmt='%5.4f', delimiter = ',')

感谢和问候

Answer 1

如果你看一下源代码np.savetxt,你会看到,虽然有很多代码可以处理Python 2和Python 3之间的参数和差异,但它最终是一个简单的python循环,其中行每行都被格式化并写入文件.因此,如果你自己编写,你就不会失去任何表现.例如,这是一个写下紧凑零的简化函数:

def savetxt_compact(fname, x, fmt="%.6g", delimiter=','):
    with open(fname, 'w') as fh:
        for row in x:
            line = delimiter.join("0" if value == 0 else fmt % value for value in row)
            fh.write(line + '\n')

例如:

In [70]: x
Out[70]: 
array([[ 0.        ,  0.        ,  0.        ,  0.        ,  1.2345    ],
       [ 0.        ,  9.87654321,  0.        ,  0.        ,  0.        ],
       [ 0.        ,  3.14159265,  0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ]])

In [71]: savetxt_compact('foo.csv', x, fmt='%.4f')

In [72]: !cat foo.csv
0,0,0,0,1.2345
0,9.8765,0,0,0
0,3.1416,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0

然后,只要您编写自己的savetxt函数,您也可以使它处理稀疏矩阵,因此您不必在保存之前将其转换为(密集)numpy数组.(I假设稀疏数组实现使用从稀疏表示之一scipy.sparse.)在下面的函数,唯一的变化是从... for value in row到... for value in row.A[0].

def savetxt_sparse_compact(fname, x, fmt="%.6g", delimiter=','):
    with open(fname, 'w') as fh:
        for row in x:
            line = delimiter.join("0" if value == 0 else fmt % value for value in row.A[0])
            fh.write(line + '\n')

例:

In [112]: a
Out[112]: 
<6x5 sparse matrix of type '<type 'numpy.float64'>'
    with 3 stored elements in Compressed Sparse Row format>

In [113]: a.A
Out[113]: 
array([[ 0.        ,  0.        ,  0.        ,  0.        ,  1.2345    ],
       [ 0.        ,  9.87654321,  0.        ,  0.        ,  0.        ],
       [ 0.        ,  3.14159265,  0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ]])

In [114]: savetxt_sparse_compact('foo.csv', a, fmt='%.4f')

In [115]: !cat foo.csv
0,0,0,0,1.2345
0,9.8765,0,0,0
0,3.1416,0,0,0
0,0,0,0,0
0,0,0,0,0
0,0,0,0,0

Answer 2

另一个可以满足您要求的简单选项是'g'说明符.如果你更关心有效数字而不是更多关于看到x个数字的数字,并且不介意它在科学和浮点数之间切换,这很好地解决了问题.例如:

np.savetxt("foo.csv", arrayDense, fmt='%5.4g', delimiter=',') 

如果arrayDense是这样的:

matrix([[ -5.54900000e-01,   0.00000000e+00,   0.00000000e+00],
    [  0.00000000e+00,   3.43560000e-08,   0.00000000e+00],
    [  0.00000000e+00,   0.00000000e+00,   3.43422000e+01]])

你的方式会产生:

-0.5549,0.0000,0.0000
0.0000,0.0000,0.0000
0.0000,0.0000,34.3422

以上将反过来:

-0.5549,    0,    0
0,3.436e-08,    0
0,    0,34.34

这种方式也更灵活.请注意,使用'g'代替'f',您不会丢失数据(即3.4356e-08而不是0.0000).这显然取决于您设置精度的方式.