ali*_*i_m 45 python arrays performance numpy
我正在寻找一种最有效的方法来确定一个大型数组是否包含至少一个非零值.乍一看np.any似乎是这项工作的明显工具,但对大型阵列来说似乎意外地缓慢.
考虑这种极端情况:
first = np.zeros(1E3,dtype=np.bool)
last = np.zeros(1E3,dtype=np.bool)
first[0] = True
last[-1] = True
# test 1
%timeit np.any(first)
>>> 100000 loops, best of 3: 6.36 us per loop
# test 2
%timeit np.any(last)
>>> 100000 loops, best of 3: 6.95 us per loop
至少np.any在这里似乎做了一些模糊的事情 - 如果非零值是数组中的第一个True,那么在返回之前不应该考虑任何其他值,所以我希望测试1比测试2稍快一些.
但是,当我们使阵列更大时会发生什么？
first = np.zeros(1E9,dtype=np.bool)
last = np.zeros(1E9,dtype=np.bool)
first[0] = True
last[-1] = True
# test 3
%timeit np.any(first)
>>> 10 loops, best of 3: 21.6 ms per loop
# test 4
%timeit np.any(last)
>>> 1 loops, best of 3: 739 ms per loop
正如预期的那样,测试4比测试3慢很多.但是,在测试3中
np.any,仍然只需要检查单个元素的值,
first以便知道它包含至少一个非零值.那么,为什么测试3比测试1慢得多？
我正在使用Numpy的开发版本(1.8.0.dev-e11cd9b),但我使用Numpy 1.7.1获得完全相同的计时结果.我正在运行64位Linux,Python 2.7.4.我的系统基本上是空闲的(我正在运行一个IPython会话,一个浏览器和一个文本编辑器),而且我绝对不会打交换.我还在另一台运行Numpy 1.7.1的机器上复制了结果.
使用Numpy 1.6.2我得到的测试1和3的时间大约为1.85us,因此jorgeca说在这方面Numpy 1.6.2和1.7.1 1.7.0 之间似乎有一些性能回归.
在JF Sebastian和jorgeca的带领下,我使用np.all了一个零数组进行了更多的基准测试,这应该相当于调用np.any第一个元素为一的数组.
测试脚本:
import timeit
import numpy as np
print 'Numpy v%s' %np.version.full_version
stmt = "np.all(x)"
for ii in xrange(10):
setup = "import numpy as np; x = np.zeros(%d,dtype=np.bool)" %(10**ii)
timer = timeit.Timer(stmt,setup)
n,r = 1,3
t = np.min(timer.repeat(r,n))
while t < 0.2:
n *= 10
t = np.min(timer.repeat(r,n))
t /= n
if t < 1E-3:
timestr = "%1.3f us" %(t*1E6)
elif t < 1:
timestr = "%1.3f ms" %(t*1E3)
else:
timestr = "%1.3f s" %t
print "Array size: 1E%i, %i loops, best of %i: %s/loop" %(ii,n,r,timestr)
结果:
Numpy v1.6.2
Array size: 1E0, 1000000 loops, best of 3: 1.738 us/loop
Array size: 1E1, 1000000 loops, best of 3: 1.845 us/loop
Array size: 1E2, 1000000 loops, best of 3: 1.862 us/loop
Array size: 1E3, 1000000 loops, best of 3: 1.858 us/loop
Array size: 1E4, 1000000 loops, best of 3: 1.864 us/loop
Array size: 1E5, 1000000 loops, best of 3: 1.882 us/loop
Array size: 1E6, 1000000 loops, best of 3: 1.866 us/loop
Array size: 1E7, 1000000 loops, best of 3: 1.853 us/loop
Array size: 1E8, 1000000 loops, best of 3: 1.860 us/loop
Array size: 1E9, 1000000 loops, best of 3: 1.854 us/loop
Numpy v1.7.0
Array size: 1E0, 100000 loops, best of 3: 5.881 us/loop
Array size: 1E1, 100000 loops, best of 3: 5.831 us/loop
Array size: 1E2, 100000 loops, best of 3: 5.924 us/loop
Array size: 1E3, 100000 loops, best of 3: 5.864 us/loop
Array size: 1E4, 100000 loops, best of 3: 5.997 us/loop
Array size: 1E5, 100000 loops, best of 3: 6.979 us/loop
Array size: 1E6, 100000 loops, best of 3: 17.196 us/loop
Array size: 1E7, 10000 loops, best of 3: 116.162 us/loop
Array size: 1E8, 1000 loops, best of 3: 1.112 ms/loop
Array size: 1E9, 100 loops, best of 3: 11.061 ms/loop
Numpy v1.7.1
Array size: 1E0, 100000 loops, best of 3: 6.216 us/loop
Array size: 1E1, 100000 loops, best of 3: 6.257 us/loop
Array size: 1E2, 100000 loops, best of 3: 6.318 us/loop
Array size: 1E3, 100000 loops, best of 3: 6.247 us/loop
Array size: 1E4, 100000 loops, best of 3: 6.492 us/loop
Array size: 1E5, 100000 loops, best of 3: 7.406 us/loop
Array size: 1E6, 100000 loops, best of 3: 17.426 us/loop
Array size: 1E7, 10000 loops, best of 3: 115.946 us/loop
Array size: 1E8, 1000 loops, best of 3: 1.102 ms/loop
Array size: 1E9, 100 loops, best of 3: 10.987 ms/loop
Numpy v1.8.0.dev-e11cd9b
Array size: 1E0, 100000 loops, best of 3: 6.357 us/loop
Array size: 1E1, 100000 loops, best of 3: 6.399 us/loop
Array size: 1E2, 100000 loops, best of 3: 6.425 us/loop
Array size: 1E3, 100000 loops, best of 3: 6.397 us/loop
Array size: 1E4, 100000 loops, best of 3: 6.596 us/loop
Array size: 1E5, 100000 loops, best of 3: 7.569 us/loop
Array size: 1E6, 100000 loops, best of 3: 17.445 us/loop
Array size: 1E7, 10000 loops, best of 3: 115.109 us/loop
Array size: 1E8, 1000 loops, best of 3: 1.094 ms/loop
Array size: 1E9, 100 loops, best of 3: 10.840 ms/loop
根据seberg的评论,我尝试使用np.float32数组进行相同的测试而不是np.bool.在这种情况下,Numpy 1.6.2也会随着数组大小的增加而减速:
Numpy v1.6.2
Array size: 1E0, 100000 loops, best of 3: 3.503 us/loop
Array size: 1E1, 100000 loops, best of 3: 3.597 us/loop
Array size: 1E2, 100000 loops, best of 3: 3.742 us/loop
Array size: 1E3, 100000 loops, best of 3: 4.745 us/loop
Array size: 1E4, 100000 loops, best of 3: 14.533 us/loop
Array size: 1E5, 10000 loops, best of 3: 112.463 us/loop
Array size: 1E6, 1000 loops, best of 3: 1.101 ms/loop
Array size: 1E7, 100 loops, best of 3: 11.724 ms/loop
Array size: 1E8, 10 loops, best of 3: 116.924 ms/loop
Array size: 1E9, 1 loops, best of 3: 1.168 s/loop
Numpy v1.7.1
Array size: 1E0, 100000 loops, best of 3: 6.548 us/loop
Array size: 1E1, 100000 loops, best of 3: 6.546 us/loop
Array size: 1E2, 100000 loops, best of 3: 6.804 us/loop
Array size: 1E3, 100000 loops, best of 3: 7.784 us/loop
Array size: 1E4, 100000 loops, best of 3: 17.946 us/loop
Array size: 1E5, 10000 loops, best of 3: 117.235 us/loop
Array size: 1E6, 1000 loops, best of 3: 1.096 ms/loop
Array size: 1E7, 100 loops, best of 3: 12.328 ms/loop
Array size: 1E8, 10 loops, best of 3: 118.431 ms/loop
Array size: 1E9, 1 loops, best of 3: 1.172 s/loop
为什么会这样？与布尔情况一样,np.all仍然只需要在返回之前检查第一个元素,因此时间应该仍然是数组大小的常量.
Jus*_*eel 29
正如评论中所猜测的那样,我可以确认数组的处理是以块的形式完成的.首先,我将向您展示代码中的内容,然后我将向您展示如何更改块大小以及这样做对基准测试的影响.
np.all(x)与x.all()相同.all()实际上调用了np.core.umath.logical_and.reduce(x).
如果你想深入了解numpy源代码,我将尝试引导你找到使用缓冲区/块大小.包含我们将要查看的所有代码的文件夹是numpy/core/src/umath /.
ufunc_object.c中的PyUFunc_Reduce()是处理reduce的C函数.在PyUFunc_Reduce()中,通过PyUFunc_GetPyValues()函数(ufunc_object.c)在某些全局字典中查找reduce的值,可以找到块或缓冲区的大小.在我的机器上并从开发分支编译,块大小为8192.在reduction.c中调用PyUFunc_ReduceWrapper()来设置迭代器(步长等于块大小)并调用传入的循环函数是ufunc_object.c中的reduce_loop().
reduce_loop()基本上只使用迭代器并为每个块调用另一个innerloop()函数.innerloop函数位于loops.c.src中.对于布尔数组和我们的all/logical_and的情况,适当的内部循环函数是BOOL_logical_and.您可以通过搜索BOOLEAN LOOPS找到正确的功能,然后它是下面的第二个功能(由于这里使用的模板式编程很难找到).在那里你会发现每个块实际上都在进行短路.
您可以使用np.getbuffersize()获取块/缓冲区大小.对我来说,它返回8192而不手动设置它与我通过在代码中打印出缓冲区大小找到的匹配.您可以使用np.setbuffersize()来更改块大小.
我将您的基准代码更改为以下内容:
import timeit
import numpy as np
print 'Numpy v%s' %np.version.full_version
stmt = "np.all(x)"
for ii in xrange(9):
setup = "import numpy as np; x = np.zeros(%d,dtype=np.bool); np.setbufsize(%d)" %(10**ii, max(8192, min(10**ii, 10**7)))
timer = timeit.Timer(stmt,setup)
n,r = 1,3
t = np.min(timer.repeat(r,n))
while t < 0.2:
n *= 10
t = np.min(timer.repeat(r,n))
t /= n
if t < 1E-3:
timestr = "%1.3f us" %(t*1E6)
elif t < 1:
timestr = "%1.3f ms" %(t*1E3)
else:
timestr = "%1.3f s" %t
print "Array size: 1E%i, %i loops, best of %i: %s/loop" %(ii,n,r,timestr)
Numpy不喜欢缓冲区大小太小或太大,所以我确保它不会小于8192或大于1E7,因为Numpy不喜欢1E8的缓冲区大小.否则,我将缓冲区大小设置为正在处理的数组的大小.我只升级到1E8,因为我的机器目前只有4GB的内存.结果如下:
Numpy v1.8.0.dev-2a5c2c8
Array size: 1E0, 100000 loops, best of 3: 5.351 us/loop
Array size: 1E1, 100000 loops, best of 3: 5.390 us/loop
Array size: 1E2, 100000 loops, best of 3: 5.366 us/loop
Array size: 1E3, 100000 loops, best of 3: 5.360 us/loop
Array size: 1E4, 100000 loops, best of 3: 5.433 us/loop
Array size: 1E5, 100000 loops, best of 3: 5.400 us/loop
Array size: 1E6, 100000 loops, best of 3: 5.397 us/loop
Array size: 1E7, 100000 loops, best of 3: 5.381 us/loop
Array size: 1E8, 100000 loops, best of 3: 6.126 us/loop
由于缓冲区大小有多大限制,因此在最后一个时间点有一个小的上升,因为有多个块被处理.