小编Dhi*_*jeb的帖子

使用 Spark 和 Java 从 s3 读取文件时无法连接到服务端点

我需要将 S3 存储桶中的文件读入 Spark 数据集。我使用了正确的 secretKey 和 accessKey,我也尝试了端点配置,但出现此错误:

com.amazonaws.SdkClientException: Failed to connect to service endpoint: 
 at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)
 at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)

 ... 74 more



java.nio.file.AccessDeniedException: datalakedbr: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Failed to connect to service endpoint: 

 at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:187)
 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
 at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
 at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
 at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Failed to connect to service endpoint: 
 at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:159)
Run Code Online (Sandbox Code Playgroud)

这是使用的方法:

    parkSession sparkSession = SparkSession.builder()
            .master("local").appName("readFile")
            .config("fs.s3a.awsAccessKeyId", "key") …
Run Code Online (Sandbox Code Playgroud)

java amazon-s3 amazon-web-services apache-spark

0
推荐指数
1
解决办法
2703
查看次数