在cython中迭代数组,列表比np.array更快？

Question

TLDR:在cython中,为什么(或何时？)迭代一个numpy数组比迭代python列表更快？

一般来说:我之前使用过Cython并且能够在天真的python impl上获得巨大的速度,但是,弄清楚究竟需要做什么似乎并非无足轻重.

考虑以下3个sum()函数的实现.它们位于一个名为'cy'的cython文件中(很明显,这里有np.sum(),但除了我的观点之外......)

天真的蟒蛇:

def sum_naive(A):
   s = 0
   for a in A:
       s += a
   return s

Cython的函数需要python列表:

def sum_list(A):
    cdef unsigned long s = 0
    for a in A:
        s += a
    return s

Cython的函数需要一个numpy数组.

def sum_np(np.ndarray[np.int64_t, ndim=1] A):
    cdef unsigned long s = 0
    for a in A:
        s += a
    return s

我希望,在运行时间方面,sum_np <sum_list <sum_naive,但是,下面的脚本演示相反(的完整性,我加np.sum())

N = 1000000
v_np = np.array(range(N))
v_list = range(N)

%timeit cy.sum_naive(v_list)
%timeit cy.sum_naive(v_np)
%timeit cy.sum_list(v_list)
%timeit cy.sum_np(v_np)
%timeit v_np.sum()

结果:

In [18]: %timeit cyMatching.sum_naive(v_list)
100 loops, best of 3: 18.7 ms per loop

In [19]: %timeit cyMatching.sum_naive(v_np)
1 loops, best of 3: 389 ms per loop

In [20]: %timeit cyMatching.sum_list(v_list)
10 loops, best of 3: 82.9 ms per loop

In [21]: %timeit cyMatching.sum_np(v_np)
1 loops, best of 3: 1.14 s per loop

In [22]: %timeit v_np.sum()
1000 loops, best of 3: 659 us per loop

这是怎么回事？为什么cython + numpy变慢？

PS
我确实使用
#cython:boundscheck = False
#cython:wraparound = False

Answer 1

有一种更好的方法在cython中实现这一点,至少在我的机器上打败np.sum它,因为它避免了类型检查和其他在处理任意数组时numpy通常必须做的事情:

#cython.wraparound=False
#cython.boundscheck=False
cimport numpy as np

def sum_np(np.ndarray[np.int64_t, ndim=1] A):
    cdef unsigned long s = 0
    for a in A:
        s += a
    return s

def sum_np2(np.int64_t[::1] A):
    cdef:
        unsigned long s = 0
        size_t k

    for k in range(A.shape[0]):
        s += A[k]

    return s

然后是时间:

N = 1000000
v_np = np.array(range(N))
v_list = range(N)

%timeit sum(v_list)
%timeit sum_naive(v_list)
%timeit np.sum(v_np)
%timeit sum_np(v_np)
%timeit sum_np2(v_np)
10 loops, best of 3: 19.5 ms per loop
10 loops, best of 3: 64.9 ms per loop
1000 loops, best of 3: 1.62 ms per loop
1 loops, best of 3: 1.7 s per loop
1000 loops, best of 3: 1.42 ms per loop

您不希望通过Python样式迭代numpy数组,而是使用索引访问元素,因为它可以转换为纯C,而不是依赖于Python API.