GCloud 上传 httplib2.RedirectMissingLocation：重定向但响应缺少位置：标头

Question

我正在尝试使用一个简单的 python 程序将一个小文件上传到 gcloud

client = storage.Client(project=GCLOUD_PROJECT)
bucket = client.get_bucket(GCLOUD_BUCKET)
blob = bucket.blob(GCLOUD_FILE_ON_CLOUD)
blob.upload_from_filename(GCLOUD_FILE_LOCAL)

它一直工作到最近，并且发生了一些变化。现在，每当我上传大于 5MB 的文件时，都会出现以下错误。小于或等于 5MB 的文件通过。大小不足以实现可续传，是吗？

Traceback (most recent call last):
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/gcloud_upload.py", line 40, in <module>
    blob.upload_from_filename(GCLOUD_FILE_LOCAL)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/storage/blob.py", line 597, in upload_from_filename
    encryption_key=encryption_key, client=client)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/storage/blob.py", line 543, in upload_from_file
    http_response = upload.stream_file(use_chunks=True)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1086, in stream_file
    response = send_func(self.stream.tell())
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1215, in _send_chunk
    return self._send_media_request(request, end)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1125, in _send_media_request
    self.bytes_http, request, retries=self.num_retries)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/http_wrapper.py", line 423, in make_api_request
    check_response_func=check_response_func)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/http_wrapper.py", line 371, in _make_api_request_no_retry
    redirections=redirections, connection_type=connection_type)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/httplib2/__init__.py", line 1986, in request
    cachekey,
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/httplib2/__init__.py", line 1685, in _request
    content,
httplib2.RedirectMissingLocation: Redirected but the response is missing a Location: header.

当我调试时，我看到以下内容。

{
 'content-type': 'text/plain; charset=utf-8', 
 'range': 'bytes=0-1048575', 
 'content-length': '0', 
 'date': 'Sun, 19 Jan 2020 23:52:13 GMT', 
 'server': 'UploadServer', 
 'alt-svc': 'quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000', 
 'status': '308'}

我的 pip 列表供参考。操作系统是 MAC OSx。我也在 Linux 上测试过。同样的问题。

adal                     1.2.2    
bson                     0.5.8    
cachetools               3.1.1    
certifi                  2019.9.11
cffi                     1.13.1   
chardet                  3.0.4    
cryptography             2.8      
dnspython                1.16.0   
gcloud                   0.18.3   
gitdb2                   2.0.6    
GitPython                3.0.5    
google-auth              1.6.3    
googleapis-common-protos 1.51.0   
httplib2                 0.16.0   
idna                     2.7      
itsdangerous             1.1.0    
MarkupSafe               1.1.1    
oauth2client             4.1.3    
oauthlib                 3.1.0    
pip                      19.0.3   
protobuf                 3.11.2   
pyasn1                   0.4.7    
pyasn1-modules           0.2.7    
pycparser                2.19     
PyJWT                    1.7.1    
python-dateutil          2.8.0    
requests                 2.19.1   
requests-oauthlib        1.2.0    
rsa                      4.0      
setuptools               40.8.0   
six                      1.12.0   
smmap2                   2.0.5    
urllib3                  1.23     
websocket-client         0.56.0   
Werkzeug                 0.16.0   

这是上周的工作。最近有什么变化吗？

Answer 1

解决方案

gcloud包已被弃用两次并且与httplib2>=0.16. 正确的解决方案是使用google-cloud-*包族。

google-api-python-client>=1.7.12 正在使用redirect_codes API，请升级，它可以正常工作。

httplib2 v0.17.0 刚刚发布，能够修改被视为重定向的响应代码集。如果您可以修改创建Http对象的代码，这是最好的选择：

http = httplib2.Http()
http.redirect_codes = http.redirect_codes - {308}

如果这是不可能的，请编辑您的 requirements.txt 以锁定 httplib2<0.16.0

很长的故事

谷歌云存储服务器使用 HTTP 308进行特殊的可恢复上传功能，这有点类似于“重试相同的方法到同一位置”，但不完全相同。

以上（可能）是 PyPI 包google-resumable-media 的基本原理，它被 gcloud 相关包的更新版本使用，并以类似的方式处理 200 和 308，与通用 HTTP 客户端不同。

历史背景：

• 2016包 gcloud被弃用，支持包google-cloud
• 2017 google-cloud-python将 HTTP 传输从 httplib2切换到请求
• 2018包 google-cloud再次被弃用，取而代之的是 google-cloud-* 包
• 2020 httplib2的v0.16每308个重定向获得支持RFC7538

对不起，坏消息。作为 HTTP 爱好者，我偏向于 308 支持。如果您有更好的想法如何更优雅地处理这种情况，请联系。

Answer 2

我用以下方法解决了这个问题：

pip install httplib2==0.15.0

pip install google-api-python-client==1.6

编辑：加载速度更快是：pip install httplib2==0.15.0 pip install google-api-python-client==1.7.11

Answer 3

将您的httplib2版本降级到0.15.0. 在 python google-cloud-sdk 上为我工作。