如何在Spark 2.0中启用笛卡尔联接？

Question

我必须在Spark 2.0中交叉加入2个数据帧我遇到以下错误:

用户类抛出异常:

org.apache.spark.sql.AnalysisException: Cartesian joins could be prohibitively expensive and are disabled by default. To explicitly enable them, please set spark.sql.crossJoin.enabled = true; 

请帮我在哪里设置这个配置,我在eclipse中编码.

Answer 1

由于错误消息明确指出您需要设置spark.sql.crossJoin.enabled = true为spark配置

您可以设置如下相同的内容:

val sparkConf = new SparkConf().setAppName("Test")
sparkConf.set("spark.sql.crossJoin.enabled", "true")

然后SparkSession通过传递获得或创建SparkConf

val sparkSession = SparkSession.builder().config(sparkConf).getOrCreate()

那你加入......