AWS SDK S3套接字已关闭异常

Question

我的应用程序使用了近10个线程,每个线程每分钟可能会产生7,000个Put请求到S3.(我在一个功能强大的EC2盒子上运行它可以很好地处理负载.)它可以运行近一个小时,但是,一小时后,得到Unable to execute HTTP request: Socket Closed例外:

        http.AmazonHttpClient: Unable to execute HTTP request: Socket Closed
    java.net.SocketException: Socket Closed
    at java.net.AbstractPlainSocketImpl.setOption(AbstractPlainSocketImpl.java:206)
    at java.net.Socket.setSoTimeout(Socket.java:1105)
    at sun.security.ssl.SSLSocketImpl.setSoTimeout(SSLSocketImpl.java:2414)
    at org.apache.http.impl.io.SocketInputBuffer.isDataAvailable(SocketInputBuffer.java:106)
    at org.apache.http.impl.AbstractHttpClientConnection.isResponseAvailable(AbstractHttpClientConnection.java:246)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.isResponseAvailable(ManagedClientConnectionImpl.java:180)
    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
    at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:47)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:446)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3641)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1438)
    at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:128)
    at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:120)
    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.upload(UploadMonitor.java:176)
    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:134)
    at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:50)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

Put Requests使用AWS SDK TransferManager异步完成.我想,在一个put请求完全完成所需的时间内,大约有10个是异步进行的.

谷歌搜索该异常,我发现了两个可能的原因:

1. MaxConnections的限制.我把它从默认的50到3000提升到无济于事.
2. 过早的垃圾收集.我已经尝试保留对UploadTransferManager返回的对象的引用(在并发队列中),并且再次没有帮助.

我怎样才能解决这个问题？再次,该应用程序运行良好近一个小时,但一直在大约一个小时后撞到这堵墙.(我在EC2上的Amazon AMI Linux上运行.)

更新

• 除AWS SDK之外的任何代码都不会触及套接字,甚至不知道它们.所有HTTP工作都是通过AWS SDK完成的.
• 因此,如果某些东西正在关闭它们,那么它必须是AWS SDK中的内容.
• 代码在EC2服务器上运行; 没有理由期待EC2和S3之间存在任何类型的网络连接问题,并且当然没有理由它们应该可预测地发生(每次运行一小时后)

Answer 1

导致此异常的原因只有一个。您或您的框架关闭了套接字，然后继续使用它。