R连接到AWS Athena

Question

我正在尝试根据我在线阅读的内容连接到AWS Athena,但我遇到了问题.

采取的步骤

• 更新Java
• 用accesskey/secretKey替换user/pass
• 用user/pass传递accesskey/secretKey

有任何想法吗？

错误信息:

.jcall中的错误(drv @ jdrv,"Ljava/sql/Connection;","connect",as.character(url)[1],:java.sql.SQLException:必须提供AWS accessId/secretKey或AWS凭证提供程序

系统信息

  sysname                           release                                       version 
  "Linux"                           "4.4.0-62-generic"      "#83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017" 
  nodename                          machine                                        login
 "ip-***-**-**-***"                 "x86_64"                                      "unknown" 
  user                              effective_user 
 "rstudio"                          "rstudio"

代码https://www.r-bloggers.com/interacting-with-amazon-athena-from-r/

library(RJDBC)

URL <- 'https://s3.amazonaws.com/athena-downloads/drivers/AthenaJDBC41-1.0.0.jar'
fil <- basename(URL)
if (!file.exists(fil)) download.file(URL, fil)

drv <- JDBC(driverClass="com.amazonaws.athena.jdbc.AthenaDriver", fil, identifier.quote="'")

con <- jdbcConnection <- dbConnect(drv, 'jdbc:awsathena://athena.us-east-1.amazonaws.com:443/',
                                   s3_staging_dir="s3://mybucket",
                                   user=Sys.getenv("myuser"),
                                   password=Sys.getenv("mypassword"))

Answer 1

Athena JDBC驱动程序期望您的AWS Access Key Id为user,而密钥为password:

accessKeyId <- "your access key id..."
secretKey <- "your secret key..."

jdbcConnection <- dbConnect(
  drv, 
  'jdbc:awsathena://athena.us-east-1.amazonaws.com:443',
  s3_staging_dir="s3://mybucket",
  user=accessKeyId,
  password=secretKey
)

R-bloggers文章使用Sys.getenv("ATHENA_USER")和从环境变量中获取那些Sys.getenv("ATHENA_PASSWORD"),但这是可选的.

更新:使用带有来自R的Athena驱动程序的凭据提供程序

@Sam是正确的,凭据提供程序是处理AWS凭据的最佳实践.我推荐使用DefaultCredentialsProviderChain,它涵盖了从CLI配置文件,环境变量等加载凭据的几个选项.

1. 下载AWS SDK for Java,特别是来自(lib)的SDK jar 和第三方依赖项jars(third-party/lib)的目录.

2. 添加一些R代码将所有jar文件添加到rJava的类路径中

   # Load JAR Files
   library("rJava")
   
   .jinit()
   
   # Load AWS SDK jar
   .jaddClassPath("/path/to/aws-java-sdk-1.11.98/lib/aws-java-sdk-1.11.98.jar")
   
   # Add Third-Party JARs
   jarFilePaths <- dir("/path/to/aws-java-sdk-1.11.98/third-party/lib/", full.names=TRUE, pattern=".jar")
   for(i in 1:length(jarFilePaths)) {
       .jaddClassPath(jarFilePaths[i])
   }
   

3. 配置Athena驱动程序以按名称加载凭据提供程序类

   athenaConn <- dbConnect(
     athenaDriver, 
     'jdbc:awsathena://athena.us-east-1.amazonaws.com:443',
     s3_staging_dir="s3://mybucket",
     aws_credentials_provider_class="com.amazonaws.auth.DefaultAWSCredentialsProviderChain"
     )
   

设置类路径是关键.当dbConnect被执行时,雅典娜驱动程序将尝试从JAR文件加载指定的类,而这将加载所有的依赖.如果类路径不包含SDK JAR,您将看到如下错误:

.jcall中的错误(drv @ jdrv,"Ljava/sql/Connection;","connect",as.character(url)[1],:java.lang.NoClassDefFoundError:无法初始化类com.amazonaws.auth.DefaultAWSCredentialsProviderChain

如果没有第三方JAR引用,您可能会看到如下错误:

.jcall中的错误(drv @ jdrv,"Ljava/sql/Connection;","connect",as.character(url)[1],:java.lang.NoClassDefFoundError:org/apache/commons/logging/LogFactory