我们正在尝试将文件上传到Google Cloud Storage中,然后再将其移到BigQuery中,但是在某些上传过程中,我们经常会遇到“ 500 Internal Server Error”或“ 410 Gone”(以下是原始消息)。
我们使用的是官方SDK,并添加了带指数补偿的重试功能,但错误始终存在。请问您有什么建议吗?
这是我们上传(scala)的方式:
val credential = new GoogleCredential().setAccessToken(accessToken)
val requestInitializer = new HttpRequestInitializer() {
def initialize(request: HttpRequest): Unit = {
credential.initialize(request)
// to avoid read timed out exception
request.setConnectTimeout(200000)
request.setReadTimeout(200000)
request.setIOExceptionHandler(new
HttpBackOffIOExceptionHandler(new ExponentialBackOff()))
request.setUnsuccessfulResponseHandler(new
HttpBackOffUnsuccessfulResponseHandler(new ExponentialBackOff()))
}
}
val storage = new Storage.Builder(
new NetHttpTransport,
JacksonFactory.getDefaultInstance,
requestInitializer
).setApplicationName("MyAppHere").build
val objectMetadata = new StorageObject()
.setBucket(bucketName)
.setName(distantFileName)
val isc = new InputStreamContent("binary/octet-stream", fis)
val length = isc.getLength
val insertObject = storage.objects().insert(bucketName, objectMetadata, isc)
// …Run Code Online (Sandbox Code Playgroud)