如何从ctypes C函数获取打印输出到Jupyter/IPython笔记本？

Question

介绍

假设我有这个C代码:

#include <stdio.h>

// Of course, these functions are simplified for the purposes of this question.
// The actual functions are more complex and may receive additional arguments.

void printout() {
    puts("Hello");
}
void printhere(FILE* f) {
    fputs("Hello\n", f);
}

我正在编译为共享对象(DLL): gcc -Wall -std=c99 -fPIC -shared example.c -o example.so

然后我将它导入到在Jupyter或IPython笔记本中运行的Python 3.x中:

import ctypes
example = ctypes.cdll.LoadLibrary('./example.so')

printout = example.printout
printout.argtypes = ()
printout.restype = None

printhere = example.printhere
printhere.argtypes = (ctypes.c_void_p)  # Should have been FILE* instead
printhere.restype = None

题

如何执行两个printout()和printhere()C函数(通过ctypes)并获取在Jupyter/IPython笔记本中打印的输出？

如果可能的话,我想避免编写更多的C代码.我更喜欢纯Python解决方案.

我也希望避免写入临时文件.但是,写入管道/插座可能是合理的.

预期的状态,当前的状态

如果我在一个Notebook单元格中键入以下代码:

print("Hi")           # Python-style print
printout()            # C-style print
printhere(something)  # C-style print
print("Bye")          # Python-style print

我想得到这个输出:

Hi
Hello
Hello
Bye

但是,相反,我只在笔记本中获得了Python风格的输出结果.C风格的输出被打印到启动笔记本进程的终端.

研究

据我所知,在Jupyter/IPython笔记本中,sys.stdout它不是任何文件的包装器:

import sys

sys.stdout

# Output in command-line Python/IPython shell:
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
# Output in IPython Notebook:
<IPython.kernel.zmq.iostream.OutStream at 0x7f39c6930438>
# Output in Jupyter:
<ipykernel.iostream.OutStream at 0x7f6dc8f2de80>

sys.stdout.fileno()

# Output in command-line Python/IPython shell:
1
# Output in command-line Jupyter and IPython notebook:
UnsupportedOperation: IOStream has no fileno.

相关问题和链接:

• Python ctypes:Python文件对象< - > C FILE*
• Python 3替换PyFile_AsFile
• 使用fopen,fwrite和fclose通过ctypes
• Python ctypes DLL stdout
• Python:Popen的StringIO - 缺少fileno()in的解决方法StringIO,但仅适用于subprocess.Popen.

以下两个链接使用涉及创建临时文件的类似解决方案.但是,在实现此类解决方案时必须小心,以确保以正确的顺序打印Python样式的输出和C样式的输出.

• 如何防止C共享库在python中的stdout上打印？
• 在Python中重定向各种stdout

是否可以避免临时文件？

我尝试使用C找到一个解决方案,open_memstream()然后分配返回FILE*的stdout,但由于stdout无法分配,它无法正常工作.

然后我尝试获取fileno()返回的流open_memstream(),但我不能,因为它没有文件描述符.

然后我看了一下freopen(),但它的API需要传递一个文件名.

然后我查看了Python的标准库并找到了tempfile.SpooledTemporaryFile(),它是内存中的临时文件类对象.但是,一旦fileno()调用它就会被写入磁盘.

到目前为止,我找不到任何仅限内存的解决方案.最有可能的是,无论如何我们都需要使用临时文件.(这不是什么大问题,但只是一些额外的开销和额外的清理,我宁愿避免.)

有可能使用os.pipe(),但这似乎很难没有分叉.

Answer 1

我终于找到了解决方案。它需要将整个单元包装在上下文管理器中（或仅包装C代码）。它还使用一个临时文件，因为如果不使用它，我将找不到任何解决方案。

完整的笔记本可作为GitHub Gist获得：https : //gist.github.com/denilsonsa/9c8f5c44bf2038fd000f

第1部分：使用Python准备C库

import ctypes

# use_errno parameter is optional, because I'm not checking errno anyway.
libc = ctypes.CDLL(ctypes.util.find_library('c'), use_errno=True)

class FILE(ctypes.Structure):
    pass

FILE_p = ctypes.POINTER(FILE)

# Alternatively, we can just use:
# FILE_p = ctypes.c_void_p

# These variables, defined inside the C library, are readonly.
cstdin = FILE_p.in_dll(libc, 'stdin')
cstdout = FILE_p.in_dll(libc, 'stdout')
cstderr = FILE_p.in_dll(libc, 'stderr')

# C function to disable buffering.
csetbuf = libc.setbuf
csetbuf.argtypes = (FILE_p, ctypes.c_char_p)
csetbuf.restype = None

# C function to flush the C library buffer.
cfflush = libc.fflush
cfflush.argtypes = (FILE_p,)
cfflush.restype = ctypes.c_int

第2部分：构建我们自己的上下文管理器以捕获stdout

import io
import os
import sys
import tempfile
from contextlib import contextmanager

@contextmanager
def capture_c_stdout(encoding='utf8'):
    # Flushing, it's a good practice.
    sys.stdout.flush()
    cfflush(cstdout)

    # We need to use a actual file because we need the file descriptor number.
    with tempfile.TemporaryFile(buffering=0) as temp:
        # Saving a copy of the original stdout.
        prev_sys_stdout = sys.stdout
        prev_stdout_fd = os.dup(1)
        os.close(1)

        # Duplicating the temporary file fd into the stdout fd.
        # In other words, replacing the stdout.
        os.dup2(temp.fileno(), 1)

        # Replacing sys.stdout for Python code.
        #
        # IPython Notebook version of sys.stdout is actually an
        # in-memory OutStream, so it does not have a file descriptor.
        # We need to replace sys.stdout so that interleaved Python
        # and C output gets captured in the correct order.
        #
        # We enable line_buffering to force a flush after each line.
        # And write_through to force all data to be passed through the
        # wrapper directly into the binary temporary file.
        temp_wrapper = io.TextIOWrapper(
            temp, encoding=encoding, line_buffering=True, write_through=True)
        sys.stdout = temp_wrapper

        # Disabling buffering of C stdout.
        csetbuf(cstdout, None)

        yield

        # Must flush to clear the C library buffer.
        cfflush(cstdout)

        # Restoring stdout.
        os.dup2(prev_stdout_fd, 1)
        os.close(prev_stdout_fd)
        sys.stdout = prev_sys_stdout

        # Printing the captured output.
        temp_wrapper.seek(0)
        print(temp_wrapper.read(), end='')

有趣的部分：使用它！

libfoo = ctypes.CDLL('./foo.so')

printout = libfoo.printout
printout.argtypes = ()
printout.restype = None

printhere = libfoo.printhere
printhere.argtypes = (FILE_p,)
printhere.restype = None


print('Python Before capturing')
printout()  # Not captured, goes to the terminal

with capture_c_stdout():
    print('Python First')
    printout()
    print('Python Second')
    printhere(cstdout)
    print('Python Third')

print('Python After capturing')
printout()  # Not captured, goes to the terminal

输出：

Python Before capturing
Python First
C printout puts
Python Second
C printhere fputs
Python Third
Python After capturing

学分和进一步的工作

该解决方案是阅读我在问题上链接的所有链接的结果，以及大量的反复试验。

此解决方案仅重定向stdout，同时重定向stdout和可能会很有趣stderr。现在，我将其作为练习留给读者。;）

另外，此解决方案中没有异常处理（至少现在还没有）。