exec() 调用中非常大的 python 函数定义使 Django 崩溃，但不会使直接执行的 Python 代码崩溃

Question

我有一个非常大（约 40 万行）的 Python 函数，我试图通过exec()调用来定义它。如果我运行以下 Python 脚本：

exec("""def blah()
# 400k lines of IF/THEN/ELSE
""", globals())
blah()

通过从命令行调用 Python，它工作正常。

但是，如果我在 Django 实例中执行相同操作，它会在没有任何错误消息或堆栈跟踪的情况下使服务器崩溃，我只能假设这是由于分段错误造成的。

Django runserver 和上面的脚本都在同一个 Conda 环境中运行，并且都有无限的可用堆栈（通过resource.getrlimit在 Django 中打印出来确认）。

这是我的完整ulimit -a输出：

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515017
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

启动 Django 服务器的命令序列如下：

source activate <conda env name>
python manage.py runserver

这是导致崩溃的 shell 输入/输出：

(faf) [pymaster@t9dpyths3 faf]$ python manage.py runserver 9000
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
August 04, 2020 - 08:25:19
Django version 3.0.3, using settings 'faf.settings'
Starting development server at http://127.0.0.1:9000/
Quit the server with CONTROL-C.
[04/Aug/2020 08:25:25] "GET /projects/ HTTP/1.1" 200 13847
[04/Aug/2020 08:26:49] "PUT /projects/projectname/ HTTP/1.1" 200 76  # This event triggers the exec
(faf) [pymaster@t9dpyths3 faf]$

Answer 1

这听起来像是一项分而治之的工作！

将您的 exec 块分成几个部分以查找失败的位置，尝试捕获BaseException而不是Exception转储进度

如果您认为遇到了段错误，可以使用示例来处理它signal.signal(signalnum, handler)

由于它们保证是一个包含的逻辑块，因此您可以通过拆分 atdef和if语句来开始执行新块。如果大多数if语句都处于最高范围，您应该能够直接对它们进行拆分，否则将需要一些额外的范围检测。

import signal
import sys

CONTENT_AND_POS = {
    "text_lines": [],    # first attempt is exec("") without if
    "block_line_no": 1,  # first block should be at line 1+
}

def report(text_lines, line_no, msg=""):
    """ display progress to the console """
    print("running code block at {}:{}\n{}".format(
        line_no, msg, text_lines))  # NOTE reordered from args

def signal_handler_segfault(signum, frame):
    """ try to show where the segfault occurred """
    report(
        "\n".join(CONTENT_AND_POS["text_lines"]),
        CONTENT_AND_POS["block_line_no"],
        "SIGNAL {}".format(signum)
    )
    sys.exit("caught segfault")

# initial setup
signal.signal(signal.SIGSEGV, signal_handler_segfault)
path_code_to_exec = sys.argv[1]  # consider argparse
print("reading from {}".format(path_code_to_exec))

# main entrypoint
with open(path_code_to_exec) as fh:
    for line_no, line in enumerate(fh, 1):  # files are iterable by-line
        if line.startswith(("def", "if")):  # new block to try
            text_exec_block = "\n".join(CONTENT_AND_POS["text_lines"])
            try:
                exec(text_exec_block, globals())
            except BaseException as ex:
                report(
                    text_exec_block,
                    CONTENT_AND_POS["block_line_no"],
                    str(repr(ex)))
                # catching BaseException will squash exit, ctrl+C, et al.
                sys.exit("caught BaseException")
            # reset for the next block
            CONTENT_AND_POS["block_line_no"] = line_no  # new block begins
            CONTENT_AND_POS["text_lines"].clear()
        # continue with new or existing block
        CONTENT_AND_POS["text_lines"].append(line)

    # execute the last block (which is otherwise missed)
    exec_text_lines(
        CONTENT_AND_POS["text_lines"],
        CONTENT_AND_POS["block_line_no"]
    )

print("successfully executed {} lines".format(line_no))

如果这仍然无声地结束，请在执行之前输出每个块的行号。您可能需要写入文件或sys.stdout/stderr确保输出不会丢失