Eri*_*let 1 scala apache-spark
我正在使用 Azure Databricks 和 Scala。我想 show() 一个数据框，但我收到了一个我无法理解的错误，我想解决它。我的代码行是：
println("----------------------------------------------------------------Printing schema")
df.printSchema()
println("----------------------------------------------------------------Printing dataframe")
df.show()
println("----------------------------------------------------------------Error before")
标准输出如下，消息“---------------------------------------------------- -------------------------错误之前”它不会出现。
> ----------------------------------------------------------------Printing schema
> root
> |-- processed: integer (nullable = false)
> |-- processDatetime: string (nullable = false)
> |-- executionDatetime: string (nullable = false)
> |-- executionSource: string (nullable = false)
> |-- executionAppName: string (nullable = false)
>
> ----------------------------------------------------------------Printing dataframe
> 2020-02-18T14:19:00.069+0000: [GC (Allocation Failure) [PSYoungGen: 1497248K->191833K(1789440K)] 2023293K->717886K(6063104K),
> 0.0823288 secs] [Times: user=0.18 sys=0.02, real=0.09 secs]
> 2020-02-18T14:19:40.823+0000: [GC (Allocation Failure) [PSYoungGen: 1637209K->195574K(1640960K)] 2163262K->721635K(5914624K),
> 0.0483384 secs] [Times: user=0.17 sys=0.00, real=0.05 secs]
> 2020-02-18T14:19:44.843+0000: [GC (Allocation Failure) [PSYoungGen: 1640950K->139092K(1809920K)] 2167011K->665161K(6083584K),
> 0.0301711 secs] [Times: user=0.11 sys=0.00, real=0.03 secs]
> 2020-02-18T14:19:50.910+0000: Track exception: Job aborted due to stage failure: Task 59 in stage 62.0 failed 4 times, most recent
> failure: Lost task 59.3 in stage 62.0 (TID 2672, 10.139.64.6, executor
> 1): java.lang.IllegalArgumentException: Illegal sequence boundaries:
> 1581897600000000 to 1581811200000000 by 86400000000
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage23.processNext(Unknown
> Source)
> at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$15$$anon$2.hasNext(WholeStageCodegenExec.scala:659)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
> at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
> at org.apache.spark.scheduler.Task.doRunTask(Task.scala:139)
> at org.apache.spark.scheduler.Task.run(Task.scala:112)
> at org.apache.spark.executor.Executor$TaskRunner$$anonfun$13.apply(Executor.scala:497)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1526)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:503)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> Driver stacktrace:.
> 2020-02-18T14:19:50.925+0000: Track message: Process finished with exit code 1. Metric: Writer. Value: 1.0.
如果没有看到你的代码，很难准确地知道，但我有一个类似的错误，另一个答案（关于int超出范围）让我误入歧途。
您得到的内容java.lang.IllegalArgumentException令人困惑，但实际上非常具体：
Illegal sequence boundaries: 1581897600000000 to 1581811200000000 by 86400000000\n此错误抱怨您正在使用sequence()Spark SQL 函数，并且告诉它从1581897600000000到1581811200000000 by 86400000000。由于数字很大，很容易错过，但这是一条从较大数字开始的指令通过增加一个正整数变为一个较小的数。例如，从12到6 x 3。
根据DataBricks 文档，这是不允许的：
\n\n\n\n\n\n
\n- 开始 - 一个表达式。范围的开始。
\n- stop - 一个表达式。范围结束（含）。
\n- 步骤 - 可选表达式。\n 范围的步长。默认情况下，如果 start 小于\n 或等于 stop，则步长为 1，否则为 -1。对于时间 \n 序列来说，\xe2\x80\x99 分别为 1 天和 -1 天。如果 start 大于 stop 则步长必须为负，反之亦然。
\n
此外，我认为其他答案对该int专栏的关注具有误导性。非法序列错误中提到的大数字看起来像是来自日期列。您没有任何 DateType 列，但您的字符串列的命名类似于日期列；假设您在序列函数中使用它们，并且它们被强制转换为日期。