那些遇到过的问题

从子进程调用时,Python程序永远挂起

Don*_*fft 35 python subprocess

pip测试套件使用子进程调用来运行集成测试.最近放置了PR,删除了一些旧的兼容性代码.特别是它b()用显式使用b""文字替换了一个函数.然而,这似乎打破了特定子进程调用永远挂起的地方.更糟糕的是,它只会永远挂在Python 3.3(可能只有Python 3.3.5)上,并且不能轻易地在Travis之外复制.

相关的拉动请求:

其他Pull请求也会出现类似的问题,但是在不同版本的Python和不同的测试用例中它们会失败.这些Pull请求是:

另一位用户今天在IRC上向我报告了一个类似的问题,他们说他们可以在Ubuntu 14.04上使用Python 3.3从死神(但不是在OSX上)本地再现它,而不仅仅是在特拉维斯,就像我迄今为止大部分都能做到的那样.他们给我发了一些重现的步骤:

$ git clone git@github.com:xavfernandez/pip.git
$ cd pip
$ git checkout debug_stuck
$ pip install pytest==2.5.2 scripttest==1.3 virtualenv==1.11.6 mock==1.0.1 pretend==1.0.8 setuptools==4.0
$ # The below should pass just fine
$ py.test -k test_env_vars_override_config_file -v -s
$ # Now edit pip/req/req_set.py and remove method remove_me_to_block or change its content to print('KO') or pass
$ # The below should hang forever
$ py.test -k test_env_vars_override_config_file -v -s
Run Code Online (Sandbox Code Playgroud)

在上面的例子中,该remove_me_to_block方法不是在任何地方调用的,只是它的存在足以使测试不被阻塞,并且它的不存在(或改变它的内容)足以使测试块永远存在.

大多数调试都是在这个PR(https://github.com/pypa/pip/pull/1901)中进行了更改.在测试通过之前推送了一次提交,直到应用了这个特定的提交 - https://github.com/dstufft/pip/commit/d296df620916b4cd2379d9fab988cbc088e28fe0.具体来说,要么改变要使用,b'\r\n'要么(entry + endline).encode("utf-8")触发它,但这些事情都不在执行路径中pip install -vvv INITools,而是它无法执行的命令.

在试图追查问题的时候,我注意到如果我"something".encode("utf8")用(lambda: "something")().encode("utf8")它取代至少一个电话就行了.

尝试调试时的另一个问题是,我尝试了各种各样的事情(添加打印语句,无操作atexit函数,使用异常的异步子进程)将简单地将问题从特定Python版本的特定测试用例转移到不同的不同版本的Python上的测试用例.

我知道subprocess如果您subprocess.Popen().stdout/stderr/stdin直接读/写,模块可能会死锁.但是,此代码使用的communicate()方法应该解决这些问题.在wait()调用内部,communicate()进程会永远挂起,等待pip进程退出.

其他信息:

  • 非常heisenbug-ey,我已经设法让它消失或根据不应该对它产生任何影响的各种事物进行转变.
  • 我已经跟踪了pip内部的执行,一直到代码路径的末尾,直到sys.exit()被调用.
  • 更换sys.exit()与os._exit()修复所有挂的问题,但是我宁愿这样做,因为我们则跳过清理Python解释器一样.
  • 没有其他线程在运行(已通过验证threading.enumerate).
  • 我有一些变化的组合,即使没有subprocess.PIPE用于stdout/stderr/stdin,它也会挂起,但是如果不使用它们,其他组合将不会挂起(或者它将转移到不同的测试用例/ python版).
  • 它似乎与时间无关,任何特定提交将在影响测试用例/ Pythons上100%失败或在0%时间失败.
  • 通常情况下,更改的代码甚至不会被pip子流程中的特定代码路径执行,但是仅仅存在变化似乎会破坏它.
  • 我已经尝试禁用字节码生成使用PYTHONDONTWRITEBYTECODE=1并且在一个组合中产生效果,但在其他组合中它没有效果.
  • 子进程调用的命令不会在每次调用时挂起(类似的命令是通过测试套件发出的)但是它总是挂在特定提交的完全相同的位置.
  • 到目前为止,我已经完全无法subproccess在测试套件中重现这一点,但是我不知道它是否与之相关.

我完全不知道是什么造成了这种情况.

更新#1

使用faulthandler.dump_traceback_later()我得到了这个结果:

Timeout (0:00:05)!
Current thread 0x00007f417bd92740:
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
  [ Duplicate Lines Snipped ]
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/requests/packages/urllib3/response.py", line 287 in closed
Timeout (0:00:05)!
Current thread 0x00007f417bd92740:
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
  [ Duplicate Lines Snipped ]
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/requests/packages/urllib3/response.py", line 287 in closed
Timeout (0:00:05)!
Current thread 0x00007f417bd92740:
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
  [ Duplicate Lines Snipped ]
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
Timeout (0:00:05)!
Current thread 0x00007f417bd92740:
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
  [ Duplicate Lines Snipped ]
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
Timeout (0:00:05)!
Current thread 0x00007f417bd92740:
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
  [ Duplicate Lines Snipped ]
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
Timeout (0:00:05)!
Current thread 0x00007f417bd92740:
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
  [ Duplicate Lines Snipped ]
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
Timeout (0:00:05)!
Current thread 0x00007f417bd92740:
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
  [ Duplicate Lines Snipped ]
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
Timeout (0:00:05)!
Current thread 0x00007f417bd92740:
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
  [ Duplicate Lines Snipped ]
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/requests/packages/urllib3/response.py", line 285 in closed
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
  [ Duplicate Lines Snipped ]
  File "/tmp/pytest-10/test_env_vars_override_config_file0/pip_src/pip/_vendor/cachecontrol/filewrapper.py", line 24 in __getattr__
Run Code Online (Sandbox Code Playgroud)

这告诉我,问题可能与垃圾收集和urllib3有关吗？的Filewrapper在pip._vendor.cachecontrol.filewrapper被用作围绕urllib3响应对象(其子类的包装io.IOBase),使我们可以三通的read()方法来存储在缓冲器中的每个读取调用的结果以及其返回,然后一旦该文件已经被完全消耗运行与该缓冲区内容的回调,以便我们可以将项目存储在缓存中.这可能会以某种方式与GC进行交互吗？

更新#2

如果我def __del__(self): pass向Filewrapper类添加一个方法,那么在我尝试的情况下一切正常.我进行了测试以确保这不是因为我碰巧定义了一个方法(有时会"修改"它),通过将其更改为def __del2__(self): pass并再次开始失败.我不确定为什么这种方法可以正常工作,而无操作__del__方法似乎不太理想.

更新#3

import gc; gc.set_debug(gc.DEBUG_UNCOLLECTABLE)在执行挂起的pip命令期间将打印的东西添加到stderr两次,它们是:

gc: uncollectable <CallbackFileWrapper 0x7f66385c1cd0>
gc: uncollectable <dict 0x7f663821d5a8>
gc: uncollectable <functools.partial 0x7f663831de10>
gc: uncollectable <_io.BytesIO 0x7f663804dd50>
gc: uncollectable <method 0x7f6638219170>
gc: uncollectable <tuple 0x7f663852bd40>
gc: uncollectable <HTTPResponse 0x7f663831c7d0>
gc: uncollectable <PreparedRequest 0x7f66385c1a90>
gc: uncollectable <dict 0x7f663852cb48>
gc: uncollectable <dict 0x7f6637fdcab8>
gc: uncollectable <HTTPHeaderDict 0x7f663831cb90>
gc: uncollectable <CaseInsensitiveDict 0x7f66385c1ad0>
gc: uncollectable <dict 0x7f6638218ab8>
gc: uncollectable <RequestsCookieJar 0x7f663805d7d0>
gc: uncollectable <dict 0x7f66382140e0>
gc: uncollectable <dict 0x7f6638218680>
gc: uncollectable <list 0x7f6638218e18>
gc: uncollectable <dict 0x7f6637f14878>
gc: uncollectable <dict 0x7f663852c5a8>
gc: uncollectable <dict 0x7f663852cb00>
gc: uncollectable <method 0x7f6638219d88>
gc: uncollectable <DefaultCookiePolicy 0x7f663805d590>
gc: uncollectable <list 0x7f6637f14518>
gc: uncollectable <list 0x7f6637f285a8>
gc: uncollectable <list 0x7f6637f144d0>
gc: uncollectable <list 0x7f6637f14ab8>
gc: uncollectable <list 0x7f6637f28098>
gc: uncollectable <list 0x7f6637f14c20>
gc: uncollectable <list 0x7f6637f145a8>
gc: uncollectable <list 0x7f6637f14440>
gc: uncollectable <list 0x7f663852c560>
gc: uncollectable <list 0x7f6637f26170>
gc: uncollectable <list 0x7f663821e4d0>
gc: uncollectable <list 0x7f6637f2d050>
gc: uncollectable <list 0x7f6637f14fc8>
gc: uncollectable <list 0x7f6637f142d8>
gc: uncollectable <list 0x7f663821d050>
gc: uncollectable <list 0x7f6637f14128>
gc: uncollectable <tuple 0x7f6637fa8d40>
gc: uncollectable <tuple 0x7f66382189e0>
gc: uncollectable <tuple 0x7f66382183f8>
gc: uncollectable <tuple 0x7f663866cc68>
gc: uncollectable <tuple 0x7f6637f1e710>
gc: uncollectable <tuple 0x7f6637fc77a0>
gc: uncollectable <tuple 0x7f6637f289e0>
gc: uncollectable <tuple 0x7f6637f19f80>
gc: uncollectable <tuple 0x7f6638534d40>
gc: uncollectable <tuple 0x7f6637f259e0>
gc: uncollectable <tuple 0x7f6637f1c7a0>
gc: uncollectable <tuple 0x7f6637fc8c20>
gc: uncollectable <tuple 0x7f6638603878>
gc: uncollectable <tuple 0x7f6637f23440>
gc: uncollectable <tuple 0x7f663852c248>
gc: uncollectable <tuple 0x7f6637f2a0e0>
gc: uncollectable <tuple 0x7f66386a6ea8>
gc: uncollectable <tuple 0x7f663852f9e0>
gc: uncollectable <tuple 0x7f6637f28560>
Run Code Online (Sandbox Code Playgroud)

然后

gc: uncollectable <CallbackFileWrapper 0x7f66385c1350>
gc: uncollectable <dict 0x7f6638c33320>
gc: uncollectable <HTTPResponse 0x7f66385c1590>
gc: uncollectable <functools.partial 0x7f6637f03ec0>
gc: uncollectable <_io.BytesIO 0x7f663804d600>
gc: uncollectable <dict 0x7f6637f1f680>
gc: uncollectable <method 0x7f663902d3b0>
gc: uncollectable <tuple 0x7f663852be18>
gc: uncollectable <HTTPMessage 0x7f66385c1c10>
gc: uncollectable <HTTPResponse 0x7f66385c1450>
gc: uncollectable <PreparedRequest 0x7f66385cac50>
gc: uncollectable <dict 0x7f6637f2f248>
gc: uncollectable <dict 0x7f6637f28b90>
gc: uncollectable <dict 0x7f6637f1e638>
gc: uncollectable <list 0x7f6637f26cb0>
gc: uncollectable <list 0x7f6637f2f638>
gc: uncollectable <HTTPHeaderDict 0x7f66385c1f90>
gc: uncollectable <CaseInsensitiveDict 0x7f66385b2890>
gc: uncollectable <dict 0x7f6638bd9200>
gc: uncollectable <RequestsCookieJar 0x7f663805da50>
gc: uncollectable <dict 0x7f6637f28a28>
gc: uncollectable <dict 0x7f663853aa28>
gc: uncollectable <list 0x7f663853a6c8>
gc: uncollectable <dict 0x7f6638ede5f0>
gc: uncollectable <dict 0x7f6637f285f0>
gc: uncollectable <dict 0x7f663853a4d0>
gc: uncollectable <method 0x7f663911f710>
gc: uncollectable <DefaultCookiePolicy 0x7f663805d210>
gc: uncollectable <list 0x7f6637f28ab8>
gc: uncollectable <list 0x7f6638215050>
gc: uncollectable <list 0x7f663853a200>
gc: uncollectable <list 0x7f6638215a28>
gc: uncollectable <list 0x7f663853a950>
gc: uncollectable <list 0x7f663853a998>
gc: uncollectable <list 0x7f6637f21638>
gc: uncollectable <list 0x7f6637f0cd40>
gc: uncollectable <list 0x7f663853ac68>
gc: uncollectable <list 0x7f6637f22c68>
gc: uncollectable <list 0x7f663853a170>
gc: uncollectable <list 0x7f6637fa6a28>
gc: uncollectable <list 0x7f66382153b0>
gc: uncollectable <list 0x7f66386a5e60>
gc: uncollectable <list 0x7f663852f2d8>
gc: uncollectable <list 0x7f66386a3320>
    [<pip._vendor.cachecontrol.filewrapper.CallbackFileWrapper object at 0x7f66385c1cd0>, <pip._vendor.cachecontrol.filewrapper.CallbackFileWrapper object at 0x7f66385c1350>]
Run Code Online (Sandbox Code Playgroud)

这是有用的信息吗？我之前从未使用过这面旗帜所以我不知道这是不寻常的.

hay*_*ypo 19

在Python 2中,如果一组对象在链(引用循环)中链接在一起,并且至少一个对象有一个__del__方法,则垃圾收集器不会删除这些对象.如果你有一个参考周期,添加一个__del__()方法可能只是隐藏错误(解决方法错误).

根据您的更新#3,看起来您有这样的问题.

归档时间：

查看次数：

1909 次

最近记录：

12 年，1 月 前
相关归档
如何使用请求下载图像 336
如何在类子类化时运行代码？ 95
Matplotlib Legends不工作 78
Django CSRF Coo​​kie未设置 73
3D矢量的旋转？ 64
virtualenvwrapper函数在shell脚本中不可用 56
如何从(非Web)python客户端访问经过身份验证的Google App Engine服务？ 52
Python等效于内联函数或宏 49
在Python和sqlite中转义字符 44
从check_output中抑制stdout,但将其写入日志 3
难疑归档
如何在JavaScript中替换所有出现的字符串 4081
从Git存储库中删除文件而不从本地文件系统中删除它 2892
什么是serialVersionUID,我为什么要使用它？ 2880
.prop()vs .attr() 2249
JavaScript对象的长度 2224
在JavaScript中定义枚举的首选语法是什么？ 1982
编译用于高放射性环境的应用程序 1414
你怎么能加速Eclipse？ 1258
使用pip安装特定的软件包版本 1199
命名类 - 如何避免将所有内容称为"<WhatEver> Manager"？ 1147