亚马逊emr上的jupyterhub docker中的pyspark3,pyspark和spark kearnels似乎不允许自动完成函数名称或doc字符串,shift-tab.有没有人注意到这种行为？
我用jupyterhub和spark启动了一个集群.我为pyspark或pyspark3创建了一个新的笔记本.
它似乎是在docker中使用conda.我试图升级所有,但只是打破了一切.
我正在运行 Pyspark 脚本以将数据帧写入 jupyter Notebook 中的 csv，如下所示：
df.coalesce(1).write.csv('Data1.csv',header = 'true')
运行一个小时后，我收到以下错误。
错误：来自http://.....session 的无效状态代码未激活。
我的配置是这样的：
spark.conf.set("spark.dynamicAllocation.enabled","true")
spark.conf.set("shuffle.service.enabled","true")
spark.conf.set("spark.dynamicAllocation.minExecutors",6)
spark.conf.set("spark.executor.heartbeatInterval","3600s")
spark.conf.set("spark.cores.max", "4")
spark.conf.set("spark.sql.tungsten.enabled", "true")
spark.conf.set("spark.eventLog.enabled", "true")
spark.conf.set("spark.app.id", "Logs")
spark.conf.set("spark.io.compression.codec", "snappy")
spark.conf.set("spark.rdd.compress", "true")
spark.conf.set("spark.executor.instances", "6")
spark.conf.set("spark.executor.memory", '20g')
spark.conf.set("hive.exec.dynamic.partition", "true")
spark.conf.set("hive.exec.dynamic.partition.mode", "nonstrict")
spark.conf.set("spark.driver.allowMultipleContexts", "true")
spark.conf.set("spark.master", "yarn")
spark.conf.set("spark.driver.memory", "20G")
spark.conf.set("spark.executor.instances", "32")
spark.conf.set("spark.executor.memory", "32G")
spark.conf.set("spark.driver.maxResultSize", "40G")
spark.conf.set("spark.executor.cores", "5")
我检查了容器节点，错误是：
ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Container marked as failed:container_e836_1556653519610_3661867_01_000005 on host: ylpd1205.kmdc.att.com. Exit status: 143. Diagnostics: Container killed …我正在创建一个EMR集群，并使用jupyter Notebook运行一些Spark任务。我的任务在执行大约1小时后死亡，错误是：
An error was encountered:
Invalid status code '400' from https://xxx.xx.x.xxx:18888/sessions/0/statements/20 with error payload: "requirement failed: Session isn't active."
我的理解是，它与Livy配置有关livy.server.session.timeout，但是我不知道如何在集群的引导程序中进行设置（我需要在引导程序中进行设置，因为创建的集群没有ssh访问权限）
提前谢谢