相关疑难解决方法(0)

请求response.iter_content()得到不完整的文件(1024MB而不是1.5GB)?

您好我一直在使用此代码段从网站下载文件,到目前为止,小于1GB的文件都很好.但我注意到1.5GB的文件不完整

# s is requests session object
r = s.get(fileUrl, headers=headers, stream=True)

start_time = time.time()
with open(local_filename, 'wb') as f:
    count = 1
    block_size = 512
    try:
        total_size = int(r.headers.get('content-length'))
        print 'file total size :',total_size
    except TypeError:
        print 'using dummy length !!!'
        total_size = 10000000

    for chunk in r.iter_content(chunk_size=block_size):

        if chunk:  # filter out keep-alive new chunks

            duration = time.time() - start_time
            progress_size = int(count * block_size)
            if duration == 0:
                duration = 0.1
            speed = int(progress_size / (1024 * …
Run Code Online (Sandbox Code Playgroud)

python urllib web-scraping python-requests

10
推荐指数
1
解决办法
6360
查看次数

标签 统计

python ×1

python-requests ×1

urllib ×1

web-scraping ×1