自建扩展模块比内置c模块慢

Question

自建扩展模块比内置c模块慢

MSe*_*ert 9 python performance python-c-api

要学习如何创建C扩展,我决定只复制一个内置.c文件(在本例中itertoolsmodule.c)并将其放在我的包中.我只能从更改的名称在模块内部itertools来mypkg.

然后我编译它(Windows 10,MSVC Community 14)setuptools.Extension:

from setuptools import setup, Extension

itertools_module = Extension('mypkg.itertoolscopy',
                              sources=['src/itertoolsmodulecopy.c'])

setup(...
      ext_modules=[itertools_module])

Run Code Online (Sandbox Code Playgroud)

默认使用编译器标志/c /nologo /Ox /W3 /GL /DNDEBUG /MD,我在某处读取这些默认值等于python编译方式的设置.但是我使用conda(64位设置),所以这可能不一定是真的.

这一切都进展顺利 - 但是一个基准测试filterfalse显示它几乎比内置的慢2倍:

import mypkg
import itertools

import random

a = [random.random() for _ in range(500000)]
func = None

%timeit list(filter(func, a))
100 loops, best of 3: 3.42 ms per loop
%timeit list(itertools.filterfalse(func, a))
100 loops, best of 3: 3.41 ms per loop
%timeit list(mypkg.filterfalse(func, a))
100 loops, best of 3: 6.77 ms per loop

Run Code Online (Sandbox Code Playgroud)

但是,对于较小的迭代,差异也会变小:

a = [random.random() for _ in range(500)]  # 1 / 1000 of the elements

%timeit list(filter(func, a))
100000 loops, best of 3: 9.66 µs per loop
%timeit list(itertools.filterfalse(func, a))
100000 loops, best of 3: 10.8 µs per loop
%timeit list(mypkg.filterfalse(func, a))
100000 loops, best of 3: 14.4 µs per loop

Run Code Online (Sandbox Code Playgroud)

我无法解释这种速度上的差异,但我不得不承认我对C编译码不太熟悉.我不知道实际上是什么让它慢了.

在python 2.7上使用ifilterand ifilterfalse和2.7版本的itertoolsmodule.c文件结果相同.

有谁知道是什么让代码比内置函数更糟糕,以及如何加速它？

Answer 1

Ant*_*ile 6

我自己对这个问题感到好奇,我打算尝试重现这个问题.虽然OP是在Windows上,但我在linux上尝试这个稍微容易一些.我最终在Windows上尝试过,但我会引导你完成我的工作!

建立

我做了一个小测试工具,它是一个shell脚本,但它让其他人更容易尝试我正在尝试的东西:D

test.sh

#!/usr/bin/env bash
set -euxo pipefail
rm -rf itertoolsmodule.c setup.py venv

PYTHON=3.5
FUNCTION=filterfalse
INIT=PyInit_
#PYTHON=2.7
#FUNCTION=ifilterfalse
#INIT=init

wget "https://raw.githubusercontent.com/python/cpython/$PYTHON/Modules/itertoolsmodule.c"
sed -i "s/${INIT}itertools/${INIT}_myitertools/" itertoolsmodule.c
sed -i 's/"itertools"/"_myitertools"/' itertoolsmodule.c

cat > setup.py << EOF
from setuptools import setup, Extension
mod = Extension('_myitertools', ['itertoolsmodule.c'])
setup(name='foo', ext_modules=[mod])
EOF

virtualenv venv -ppython"$PYTHON"
venv/bin/pip install . -v

cat > test.py << EOF
import _myitertools
import itertools
import random
import time


a = [random.random() for _ in range(500000)]
iterations = range(10)
seconds = 5


def builtins_filter():
    for _ in iterations:
        list(filter(None, a))

_itertools_filterfalse = itertools.$FUNCTION
def itertools_filterfalse():
    for _ in iterations:
        list(_itertools_filterfalse(None, a))

_myitertools_filterfalse = _myitertools.$FUNCTION
def myitertools_filterfalse():
    for _ in iterations:
        list(_myitertools_filterfalse(None, a))


def runbench(func):
    start = time.time()
    end = start + seconds
    iterations = 0
    while time.time() < end:
        func()
        iterations += 1
    return iterations


for func in (builtins_filter, itertools_filterfalse, myitertools_filterfalse):
    print('*' * 79)
    print(func.__name__)
    print('{} iterations in {} seconds'.format(runbench(func), seconds))
EOF

Run Code Online (Sandbox Code Playgroud)

ubuntu16.04 x86_64 python3.5.2(stock,apt)

(我剪掉了(imo)不重要的部分):

$ ./test.sh
+ rm -rf itertoolsmodule.c setup.py venv
+ PYTHON=3.5
+ FUNCTION=filterfalse
+ INIT=PyInit_

...

+ venv/bin/pip install . -v

...

    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/tmp/foo/venv/include/python3.5m -c itertoolsmodule.c -o build/temp.linux-x86_64-3.5/itertoolsmodule.o
    creating build/lib.linux-x86_64-3.5
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/itertoolsmodule.o -o build/lib.linux-x86_64-3.5/_myitertools.cpython-35m-x86_64-linux-gnu.so

...

+ venv/bin/python test.py
*******************************************************************************
builtins_filter
1401 iterations in 50 seconds
*******************************************************************************
itertools_filterfalse
1977 iterations in 50 seconds
*******************************************************************************
myitertools_filterfalse
1981 iterations in 50 seconds

Run Code Online (Sandbox Code Playgroud)

ubuntu16.04 x86_64 python2.7.12(stock,apt)

+ rm -rf itertoolsmodule.c setup.py venv
+ PYTHON=2.7
+ FUNCTION=ifilterfalse
+ INIT=init

...

+ venv/bin/pip install . -v

...

    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c itertoolsmodule.c -o build/temp.linux-x86_64-2.7/itertoolsmodule.o
    creating build/lib.linux-x86_64-2.7
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/itertoolsmodule.o -o build/lib.linux-x86_64-2.7/_myitertools.so

...

+ venv/bin/python test.py
*******************************************************************************
builtins_filter
871 iterations in 50 seconds
*******************************************************************************
itertools_filterfalse
1918 iterations in 50 seconds
*******************************************************************************
myitertools_filterfalse
1863 iterations in 50 seconds

Run Code Online (Sandbox Code Playgroud)

视窗!

对于Windows,我稍微更改了脚本,因此使用C:\Python##\python.exe(使用mysysgit 构建virtualenvs,因此我有一些unix工具集(bash等)).将事物从bin更改为Scripts(用于virtualenv)等.我没有/使用conda所以这些只是在Windows 10上的股票python

windows 10 python 2.7.9(股票,msi安装程序)

+ rm -rf itertoolsmodule.c setup.py venv
+ PYTHON=2.7
+ FUNCTION=ifilterfalse
+ INIT=init

...

+ venv/Scripts/pip install . -v

...

    C:\Users\Anthony\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\include -Ic:\users\anthony\appdata\local\temp\foo\venv\PC /Tcitertoolsmodule.c /Fobuild\temp.win32-2.7\Release\itertoolsmodule.obj
itertoolsmodule.c
    creating build\lib.win32-2.7
    C:\Users\Anthony\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python27\Libs /LIBPATH:c:\users\anthony\appdata\local\temp\foo\venv\libs /LIBPATH:c:\users\anthony\appdata\local\temp\foo\venv\PCbuild /EXPORT:init_myitertools build\temp.win32-2.7\Release\itertoolsmodule.obj /OUT:build\lib.win32-2.7\_myitertools.pyd /IMPLIB:build\temp.win32-2.7\Release\_myitertools.lib /MANIFESTFILE:build\temp.win32-2.7\Release\_myitertools.pyd.manifest

...

+ venv/Scripts/python test.py
*******************************************************************************
builtins_filter
914 iterations in 50 seconds
*******************************************************************************
itertools_filterfalse
2352 iterations in 50 seconds
*******************************************************************************
myitertools_filterfalse
2266 iterations in 50 seconds

Run Code Online (Sandbox Code Playgroud)

windows 10 python3.5.1(股票,msi安装程序)

+ rm -rf itertoolsmodule.c setup.py venv
+ PYTHON=3.5
+ FUNCTION=filterfalse
+ INIT=PyInit_

...

+ venv/Scripts/pip install . -v

...

    D:\Programs\VS2015\VC\BIN\amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Python35\include -IC:\Python35\include -ID:\Programs\VS2015\VC\INCLUDE -ID:\Programs\VS2015\VC\ATLMFC\INCLUDE "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\\shared" "-IC:\Program Files (x86)\Windows Kits\8.1\include\\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\\winrt" /Tcitertoolsmodule.c /Fobuild\temp.win-amd64-3.5\Release\itertoolsmodule.obj
itertoolsmodule.c
    creating C:\Temp\pip-1fnf27jo-build\build\lib.win-amd64-3.5
    D:\Programs\VS2015\VC\BIN\amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Python35\Libs /LIBPATH:c:\users\anthony\appdata\local\temp\foo\venv\libs /LIBPATH:c:\users\anthony\appdata\local\temp\foo\venv\PCbuild\amd64 /LIBPATH:D:\Programs\VS2015\VC\LIB\amd64 /LIBPATH:D:\Programs\VS2015\VC\ATLMFC\LIB\amd64 "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.10240.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\8.1\lib\winv6.3\um\x64" /EXPORT:PyInit__myitertools build\temp.win-amd64-3.5\Release\itertoolsmodule.obj /OUT:build\lib.win-amd64-3.5\_myitertools.cp35-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.5\Release\_myitertools.cp35-win_amd64.lib

...

+ venv/Scripts/python test.py
*******************************************************************************
builtins_filter
658 iterations in 50 seconds
*******************************************************************************
itertools_filterfalse
2601 iterations in 50 seconds
*******************************************************************************
myitertools_filterfalse
2715 iterations in 50 seconds

Run Code Online (Sandbox Code Playgroud)

结论

至少,我用stock python测试表明扩展模块没有表现出不同的性能特征.

我花了半个小时在这上面,并没有产生繁殖.希望这有助于下一个尝试这个的穷人.我只能猜测conda正在进行一些额外的优化,然后发送一个pyconfig.h文件,该文件位于用于编译的标志之上.虽然说实话,我还没有冒险进入康茄园空间,所以我不知道他们的生态系统是如何运作的

归档时间：	9 年，3 月前
查看次数：	248 次
最近记录：	9 年前