将AWS s3文件读取为Java代码

Question

我试图从AWS s3读取一个文件到我的java代码:

  File file = new File("s3n://mybucket/myfile.txt");
  FileInputStream fileInput = new FileInputStream(file);

然后我收到一个错误:

java.io.FileNotFoundException:s3n:/mybucket/myfile.txt(没有这样的文件或目录),位于java.io.FileInputStream的java.io.FileInputStream.open(Native Method).(FileInputStream.java:146)

---

有没有办法从AWS s3打开/读取文件？非常感谢!

Answer 1

Java中的"File"类不理解S3存在.以下是从AWS文档中读取文件的示例:

AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());        
S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, key));
InputStream objectData = object.getObjectContent();
// Process the objectData stream.
objectData.close();

Answer 2

在 2019 年，有一种更优的方法可以从 S3 读取文件：

private final AmazonS3 amazonS3Client = AmazonS3ClientBuilder.standard().build();

private Collection<String> loadFileFromS3() {
    try (final S3Object s3Object = amazonS3Client.getObject(BUCKET_NAME,
                                                            FILE_NAME);
        final InputStreamReader streamReader = new InputStreamReader(s3Object.getObjectContent(), StandardCharsets.UTF_8);
        final BufferedReader reader = new BufferedReader(streamReader)) {
        return reader.lines().collect(Collectors.toSet());
    } catch (final IOException e) {
        log.error(e.getMessage(), e)
        return Collections.emptySet();
    }
}

Answer 3

我们还可以使用software.amazon.awssdk:s3

 //Assuming the credentials are read from Environment Variables, so no hardcoding here

    S3Client client = S3Client.builder()
                        .region(regionSelected)
                        .build();
    
    GetObjectRequest getObjectRequest = GetObjectRequest.builder()
                    .bucket(bucketName)
                    .key(fileName)
                    .build();
    
    ResponseInputStream<GetObjectResponse> responseInputStream = client.getObject(getObjectRequest);

    InputStream stream = new ByteArrayInputStream(responseInputStream.readAllBytes());
    
    
    System.out.println("Content :"+ new String(responseInputStream.readAllBytes(), StandardCharsets.UTF_8));

Answer 4

在java中读取S3文件的步骤可以是：

1. 创建 AmazonS3Client。
2. 使用存储桶名称和密钥创建 S3Object。
3. 使用S3Object创建缓冲区读取器并逐行读取文件。

1>>>

    BasicAWSCredentials awsCreds = new BasicAWSCredentials("accessKey", "secretKey");
    AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
            .withCredentials(new AWSStaticCredentialsProvider(awsCreds))
            .withRegion("region_name_here").build();  

2>>>

   S3Object object = s3Client.getObject(new GetObjectRequest("bucketName", "key"));

3>>>

   BufferedReader reader = new BufferedReader(new InputStreamReader(object.getObjectContent()));

    String s = null;
    while ((s = reader.readLine()) != null)
    {
        System.out.println(s);
        //your business logic here
    }

谢谢。

Answer 5

如果文件的内容是字符串，则可以使用getObjectAsString。否则，您可以使用IOUtils.toByteArray将getObjectContent()文件的内容读入字节数组。

显然，这些最适合用于可以轻松装入内存的小型 S3 对象。

private final AmazonS3 amazonS3Client = AmazonS3ClientBuilder.standard().build();

private String loadStringFromS3() {
    try {
        return amazonS3Client.getObjectAsString(BUCKET_NAME, FILE_NAME);
    } catch (final IOException e) {
        log.error(e.getMessage(), e)
        return null;
    }
}

private byte[] loadDataFromS3() {
    try (final S3Object s3Object = amazonS3Client.getObject(BUCKET_NAME, FILE_NAME)) {
        return IOUtils.toByteArray(s3Object.getObjectContent());
    } catch (final IOException e) {
        log.error(e.getMessage(), e)
        return null;
    } finally {
        IOUtils.closeQuietly(object, log);
    }
}