等待 S3 上传池连接超时

Question

我正在尝试从 7 台机器上传大量文件。在每台机器中，我运行 6 个线程来上传到 S3 。当我从一台机器上运行上传时，它工作正常，但当我在 7 台机器上运行时，它开始失败。

我在其余机器上遇到以下错误。

错误 - AmazonClientException com.amazonaws.SdkClientException：无法执行 HTTP 请求：等待来自池的连接超时

我上传到 S3 的小文件总数 = 1659328

每个线程中的记录数 = 276554

那么我必须关闭 TransferManager 吗？如果是的话我应该如何关闭它？我的应用程序是多线程的。当我调用tm.shutdownNow();时其他线程将无法使用它。

这是我要上传到 S3 的代码。

AWSCredentials credential = new ProfileCredentialsProvider("skjfffkjg-Prod-ServiceUser").getCredentials();
        AmazonS3Client s3Client = (AmazonS3Client) AmazonS3ClientBuilder.standard().withRegion("us-east-1")
                .withCredentials(new AWSStaticCredentialsProvider(credential)).withForceGlobalBucketAccessEnabled(true)
                .build();

        s3Client.getClientConfiguration().setMaxConnections(100);

上传到S3方法

public void uploadToToS3() {
        _logger.info("Number of record to be processed in current thread: : " + records.size());


        TransferManager tm = new TransferManager(s3Client);

        MultipleFileUpload upload = tm.uploadFileList(bucketName, "", new File(fileLocation), records);

        if (upload.isDone() == false) {
            System.out.println("Transfer: " + upload.getDescription());
            System.out.println("  - State: " + upload.getState());
            System.out.println("  - Progress: " + upload.getProgress().getBytesTransferred());
        }
        try {
            upload.waitForCompletion();
        } catch (AmazonServiceException e1) {
            _logger.error("AmazonServiceException " + e1.toString());
        } catch (AmazonClientException e1) {
            _logger.error("AmazonClientException " + e1.toString());
        } catch (InterruptedException e1) {
            _logger.error("InterruptedException " + e1.toString());
        }
        System.out.println("Is Upload completed Successfully ="+upload.isDone());

        for (File file : records) {
            try {
                Files.delete(FileSystems.getDefault().getPath(file.getAbsolutePath()));
            } catch (IOException e) {
                _logger.error("IOException in file delete: " + e.toString());
                System.exit(1);
                _logger.error("IOException: " + e.toString());
            }
        }

        _logger.info("Calling Transfer manager shutdown");
        // tm.shutdownNow();
    }

我是否必须关闭任何内容才能顺利上传？

Answer 1

当您拥有 S3 对象时，您需要中止并关闭它 - 就像您中止打开的连接或关闭文件读取器等一样。

因此，您需要确保正确关闭对象请求。顺便说一句，增加最大连接数可能不是最佳解决方案。

有一个 AWS API 允许中止 S3 操作。我相信您需要添加“finally”块才能控制异常期间的上传。

这是我将使用的片段：

public void uploadToToS3() {
        _logger.info("Number of record to be processed in current thread: : " + records.size());
    TransferManager tm = new TransferManager(s3Client);

    MultipleFileUpload upload = tm.uploadFileList(bucketName, "", new File(fileLocation), records);

    if (upload.isDone() == false) {
        System.out.println("Transfer: " + upload.getDescription());
        System.out.println("  - State: " + upload.getState());
        System.out.println("  - Progress: " + upload.getProgress().getBytesTransferred());
    }
    try {
        upload.waitForCompletion();
    } catch (AmazonServiceException e1) {
        _logger.error("AmazonServiceException " + e1.toString());
    } catch (AmazonClientException e1) {
        _logger.error("AmazonClientException " + e1.toString());
    } catch (InterruptedException e1) {
        _logger.error("InterruptedException " + e1.toString());
    } 

    // ************ THIS IS THE MODIFICATION ************ 
    finally {
        upload.getSubTransfers().forEach(s -> s.abort());
    }
    // *************************************************** 

    System.out.println("Is Upload completed Successfully ="+upload.isDone());
    for (File file : records) {
        try {
            Files.delete(FileSystems.getDefault().getPath(file.getAbsolutePath()));
        } catch (IOException e) {
            _logger.error("IOException in file delete: " + e.toString());
            System.exit(1);
            _logger.error("IOException: " + e.toString());
        }
    }

    _logger.info("Calling Transfer manager shutdown");
    // tm.shutdownNow();
}