小编awe*_*law的帖子

我在 EMR 上使用 Spark 2.4.0，并尝试使用 AWS Glue 数据目录在 s3 中存储简单的数据帧。代码如下：

val peopleTable = spark.sql("select * from emrdb.testtableemr")
val filtered = peopleTable.filter("name = 'Andrzej'")
filtered.repartition(1).write.format("hive").mode("append").saveAsTable("emrdb.testtableemr")

上面的代码按预期工作 - 数据被过滤并存储在与 AWS Glue 表 emrdb.testtableemr 链接的s3 目录中。我遇到的问题是：虽然它工作正常，但仍会引发以下异常

scala> filtered.repartition(1).write.format("hive").mode("append").saveAsTable("emrdb.testtableemr")
org.apache.spark.sql.AnalysisException: java.lang.IllegalArgumentException: Can not create a Path from an empty string;
  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
  at org.apache.spark.sql.hive.HiveExternalCatalog.loadTable(HiveExternalCatalog.scala:843)
  at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.loadTable(ExternalCatalogWithListener.scala:159)
  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:259)
  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)
  at org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand.run(CreateHiveTableAsSelectCommand.scala:66)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
  at …

"The event is not supported for notifications (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: CF3108325F3C9B60; S3 Extended Request ID: wcWzRXUu7YJn/BVnPDtOx7yBHllhIPELEwsTweqVcfwLw1hkR2iDiSmQbxeL3Hrtp7Kv58ujS2s=; Proxy: null)"

请参阅以下来自 AWS Mgm 控制台的 CloudFormation 事件：

下面是我的 AWS SAMtemplate.yaml文件：

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: An AWS Serverless Specification template describing your function.
Resources:
  olatexOrdersInputDirectory:
    Type: 'AWS::S3::Bucket'
  olatexXlsxOrderLoader:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: olatexXlsxOrderLoader/index.handler
      Runtime: nodejs12.x
      Description: ''
      MemorySize: 128
      Timeout: 15
      Policies:
      - AWSLambdaBasicExecutionRole
      - …