为什么upload_from_file Google Cloud Storage Function 会引发超时错误？

Question

我创建了一个对我有用的函数，它将文件上传到 Google Cloud Storage。

问题是当我的朋友尝试使用他本地机器上的相同代码将同一个文件上传到同一个存储桶时，他收到timeout error。他的互联网非常好，他应该能够在他的连接中毫无问题地上传文件。

知道为什么会这样吗？

def upload_to_cloud(file_path):
    """
        saves a file in the google storage. As Google requires audio files greater than 60 seconds to be saved on cloud before processing
        It always saves in 'audio-files-bucket' (folder)
        Input:
            Path of file to be saved
        Output:
            URI of the saved file
    """
    print("Uploading to cloud...")
    client = storage.Client().from_service_account_json(KEY_PATH)
    bucket = client.get_bucket('audio-files-bucket')
    file_name = str(file_path).split('\\')[-1]
    print(file_name)
    blob = bucket.blob(file_name)
    f = open(file_path, 'rb')
    blob.upload_from_file(f)
    f.close()
    print("uploaded at: ", "gs://audio-files-bucket/{}".format(file_name))
    return "gs://audio-files-bucket/{}".format(file_name)

它通过upload_from_file(f)行中的超时异常。

我们尝试使用upload_from_filename 函数，但仍然出现同样的错误。

Answer 1

现在，您可以在合并此PR时使用 blob 添加自定义超时，默认值为 60 秒。因此，如果您的互联网连接速度很慢，您可以为上传操作添加自定义超时。

def upload_blob(bucket_name, source_file_name, destination_blob_name):
  
    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(destination_blob_name)

    blob.upload_from_filename(source_file_name, timeout=300)

    print(
        "File {} uploaded to {}.".format(
            source_file_name, destination_blob_name
        )
    )

Answer 2

该问题通过减小 blob 的块大小来解决。代码改为：

def upload_to_cloud(file_path):
    """
        saves a file in the google storage. As Google requires audio files greater than 60 seconds to be saved on cloud before processing
        It always saves in 'audio-files-bucket' (folder)
        Input:
            Path of file to be saved
        Output:
            URI of the saved file
    """
    print("Uploading to cloud...")
    client = storage.Client().from_service_account_json(KEY_PATH)
    bucket = client.get_bucket('audio-files-bucket')
    file_name = str(file_path).split('\\')[-1]
    print(file_name)
    blob = bucket.blob(file_name)

    ## For slow upload speed
    storage.blob._DEFAULT_CHUNKSIZE = 2097152 # 1024 * 1024 B * 2 = 2 MB
    storage.blob._MAX_MULTIPART_SIZE = 2097152 # 2 MB

    with open(file_path, 'rb') as f:
        blob.upload_from_file(f)
    print("uploaded at: ", "gs://audio-files-bucket/{}".format(file_name))
    return "gs://audio-files-bucket/{}".format(file_name)