"当使用python的子进程和rsync时,"filedescriptor超出select()中的范围"

Question

以下代码用于将上传的图片同步到另一个地方.它工作正常,但经过一段时间(大约10天)后,服务无法使用,显示错误:'filedescriptor超出了select()'范围,但重启服务解决了问题.

# sync.py

def sync_file(source_pic, hashval, retry_num=3):

    pic_path = os.path.join(gen_dir(hashval), os.path.split(source_pic)[1])
    filename = tempfile.mkstemp()[1]
    with open(filename, 'w') as f:
        f.write(pic_path)

    for sync_path in options.sync_paths:
        try_num = 0
        rsync_cmd = ['rsync','-au', '--files-from='+filename, options.pic_path, sync_path]

        while try_num < retry_num:
            proc = subprocess.Popen(rsync_cmd,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            stdout_value, stderr_value = proc.communicate()

            if len(stderr_value) != 0:
                logger.error('sync failed: %s' % stderr_value)
                try_num = try_num + 1
                #raise UploadException('sync failed')
            else:
                break

    os.remove(filename)

日志信息:

File "/path/to/sync.py", line 25, in sync_file
    stdout_value, stderr_value = proc.communicate()
File "/usr/lib/python2.6/subprocess.py", line 691, in communicate
    return self._communicate(input)
File "/usr/lib/python2.6/subprocess.py", line 1211, in _communicate
    rlist, wlist, xlist = select.select(read_set, write_set, [])
    ValueError: filedescriptor out of range in select()

是否存在导致错误的未封闭文件描述符？似乎子进程没有关闭文件描述符,所以当它运行1024次时,文件描述符超出范围.(我们使用python 2.6,子进程被强制使用select.select(),它有1024个文件描述符的限制,甚至epoll可用)

Answer 1

您可以手动关闭文件描述符。拨打电话后communicate，拨打proc.stderr.close()和proc.stdout.close()。