如何在Cython中的新式缓冲区对象中包装C指针和长度？

Question

我在Cython中编写了一个Python 2.7扩展模块. 如何创建一个实现新式缓冲区接口的Python对象,该接口包含C库提供给我的大块内存？ 内存块只是一串字节,而不是结构或多维数组.我给了一个const void *指针和一个长度,以及一些指针保持有效的细节.

我无法复制内存 - 这会破坏我的应用程序的性能.

使用我可以简单使用的旧式缓冲区对象PyBuffer_FromMemory(),但我似乎无法找到一种类似的简单方法来生成新式缓冲区对象.

我是否必须创建自己的实现缓冲区接口的类？或者Cython提供了一种简单的方法吗？

我已经阅读了Cython文档中的Unicode和Passing Strings和Typed Memoryviews页面,但文档不精确且不够完整,并且没有类似于我想要做的示例.

这是我尝试过的(test.pyx):

from libc.stdlib cimport malloc
from libc.string cimport memcpy

## pretend that this function is in some C library and that it does
## something interesting.  (this function is unrelated to the problem
## I'm experiencing -- this is just an example function that returns a
## chunk of memory that I want to wrap in an object that follows the
## new buffer protocol.)
cdef void dummy_function(const void **p, size_t *l):
    cdef void *tmp = malloc(17)
    memcpy(tmp, "some test\0 bytes", 17)
    p[0] = tmp
    l[0] = 17

cpdef getbuf():
    cdef const void *cstr
    cdef size_t l
    dummy_function(&cstr, &l)

    ## error: test.pyx:21:20: Invalid base type for memoryview slice: void
    #cdef const void[:] ret = cstr[:l]

    ## error: test.pyx:24:9: Assignment to const 'ret'
    #cdef const char[:] ret = cstr[:l]

    ## error: test.pyx:27:27: Cannot convert 'void const *' to memoryviewslice
    #cdef char[:] ret = cstr[:l]

    ## this next attempt cythonizes, but raises an exception:
    ## $ python -c 'import test; test.getbuf()'
    ## Traceback (most recent call last):
    ##   File "<string>", line 1, in <module>
    ##   File "test.pyx", line 15, in test.getbuf (test.c:1411)
    ##   File "test.pyx", line 38, in test.getbuf (test.c:1350)
    ##   File "stringsource", line 614, in View.MemoryView.memoryview_cwrapper (test.c:6763)
    ##   File "stringsource", line 321, in View.MemoryView.memoryview.__cinit__ (test.c:3309)
    ## BufferError: Object is not writable.
    cdef char[:] ret = (<const char *>cstr)[:l]

    ## this raises the same exception as above
    #cdef char[:] ret = (<char *>cstr)[:l]

    return ret

Answer 1

您可以通过定义和特殊方法来定义实现缓冲区协议的扩展类型.例如:__getbuffer____releasebuffer__

from cpython.buffer cimport PyBuffer_FillInfo
from libc.stdlib cimport free, malloc
from libc.string cimport memcpy

cdef void dummy_function(const void **p, size_t *l):
    cdef void *tmp = malloc(17)
    memcpy(tmp, "some test\0 bytes", 17)
    p[0] = tmp
    l[0] = 17

cdef void free_dummy_data(const void *p, size_t l, void *arg):
    free(<void *>p)

cpdef getbuf():
    cdef const void *p
    cdef size_t l
    dummy_function(&p, &l)
    return MemBuf_init(p, l, &free_dummy_data, NULL)

ctypedef void dealloc_callback(const void *p, size_t l, void *arg)

cdef class MemBuf:
    cdef const void *p
    cdef size_t l
    cdef dealloc_callback *dealloc_cb_p
    cdef void *dealloc_cb_arg

    def __getbuffer__(self, Py_buffer *view, int flags):
        PyBuffer_FillInfo(view, self, <void *>self.p, self.l, 1, flags)
    def __releasebuffer__(self, Py_buffer *view):
        pass

    def __dealloc__(self):
        if self.dealloc_cb_p != NULL:
            self.dealloc_cb_p(self.p, self.l, self.dealloc_cb_arg)

# Call this instead of constructing a MemBuf directly.  The __cinit__
# and __init__ methods can only take Python objects, so the real
# constructor is here.  See:
# https://mail.python.org/pipermail/cython-devel/2012-June/002734.html
cdef MemBuf MemBuf_init(const void *p, size_t l,
                        dealloc_callback *dealloc_cb_p,
                        void *dealloc_cb_arg):
    cdef MemBuf ret = MemBuf()
    ret.p = p
    ret.l = l
    ret.dealloc_cb_p = dealloc_cb_p
    ret.dealloc_cb_arg = dealloc_cb_arg
    return ret

使用上面的(命名test.pyx),您将获得以下行为:

$ python -c 'import test; print repr(memoryview(test.getbuf()).tobytes())'
'some test\x00 bytes\x00'

我不知道是否有更简单的方法.

Answer 2

正如@RichardHansen 在他的自我回答中正确观察到的那样，您想要的是一个实现缓冲区协议的类，并且具有一个合适的析构函数来管理内存。

Cython 实际上以 的形式提供了一个相当轻量级的内置类cython.view.array，因此无需创建自己的类。它实际上记录在您链接的页面中，但为了提供适合您的情况的快速示例：

# at the top of your file
from cython.view cimport array

# ...

# after the call to dummy_function
my_array = array(shape=(l,), itemsize=sizeof(char), format='b',  # or capital B depending on if it's signed
                 allocate_buffer=False)
my_array.data = cstr
my_array.callback_free_data = free

cdef char[:] ret = my_array

只是为了引起注意几个位：allocate_buffer设置为，False因为您要在 中分配自己的位cstr。设置callback_free_data确保free使用标准库函数。