Ant*_*ton 5 amazon-s3 amazon-ec2 amazon-web-services
I'm trying to move a large file (~1.75GB) from S3 to EC2 instance in the same region.
Via AWS command line I can do it in ~13 seconds.
aws s3 cp s3://mybucket/myfile.txt ~
Run Code Online (Sandbox Code Playgroud)
wget from the same EC2 instance (if I make the file public) does it in 34 seconds.
wget https://mybucket.s3.amazonaws.com/myfile.txt
Run Code Online (Sandbox Code Playgroud)
My application via REST API can download it with the same speed as wget.
The instance is m5a.2xlarge which should have "Up to 10 Gbps" network. If my math is right 10Gbps should be around 1 gigabyte per second, so under 2 seconds to get the file?
UPD: speedtest-cli shows 850Mbit/s.
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by BroadAspect (Ashburn, VA) [0.98 km]: 1.387 ms
Testing download speed................................................................................
Download: 850.94 Mbit/s
Testing upload speed......................................................................................................
Upload: 929.41 Mbit/s
Run Code Online (Sandbox Code Playgroud)
UPD2: Enhanced Networking is enabled
[ec2-user@ip-172-30-0-196 ~]$ ethtool -i eth0
driver: ena
version: 2.0.2g
Run Code Online (Sandbox Code Playgroud)
UPD3: Tried couple of more instance types. Smallest/cheapest ARM a1.medium and i3en.6xlarge (I/O optimized, "guaranteed" 25Gbps)
Time to copy 1.75GB file from S3 to EC2 instance in the same region:
|instance type|Network Gbps|aws s3 cp|wget|
|-------------|------------|---------|----|
|m5a.2xlarge | Up to 10 | 13 | 34 |
|a1.medium | Up to 10 | 26 | 34 |
|i3en.6xlarge | 25 | 7.5 | 18 |
Run Code Online (Sandbox Code Playgroud)
What's interesting - switching to IO - optimized instance improves the transfer speed, but it's still not anywhere near [file size]/[instance Gbps].
UPD4: AWS S3 configuration
AWS S3 CLI tool offers few configurations options such as:
aws configure set default.s3.max_concurrent_requests 10
aws configure set default.s3.multipart_chunksize 8MB
Run Code Online (Sandbox Code Playgroud)
Numbers above are defaults. After playing with these for a bit, the best combination I found for m5a.2xlarge is 3 concurrent requests, 180MB chunk size. That brings the download time down to 9 seconds or ~204MiB/sec. Better, but still quite far from (theoretical?) maximum.
拥有 10Gbps 并不能保证另一端以该速度向您发送数据。
在配置了 ENA 的当前一代 EC2 实例类型(支持高达 25 Gbps 的 S3)之前,最大值为 5 Gbps。
您可以进行一些额外的测试,例如 EC2 到 EC2 或 EC2 到 speedtest.net(或同等测试)。
顺便说一句,如果您确实要下载 1.75GB 的文本文件,那么压缩会有所帮助。
| 归档时间: |
|
| 查看次数: |
292 次 |
| 最近记录: |