Spark SQL没有正确转换时区

Question

使用Scala 2.10.4和spark 1.5.1和spark 1.6

sqlContext.sql(
  """
    |select id,
    |to_date(from_utc_timestamp(from_unixtime(at), 'US/Pacific')),
    |from_utc_timestamp(from_unixtime(at), 'US/Pacific'),
    |from_unixtime(at),
    |to_date(from_unixtime(at)),
    | at
    |from events
    | limit 100
  """.stripMargin).collect().foreach(println)

Spark-Submit选项: --driver-java-options '-Duser.timezone=US/Pacific'

结果:

[56d2a9573bc4b5c38453eae7,2016-02-28,2016-02-27 16:01:27.0,2016-02-28 08:01:27,2016-02-28,1456646487]
[56d2aa1bfd2460183a571762,2016-02-28,2016-02-27 16:04:43.0,2016-02-28 08:04:43,2016-02-28,1456646683]
[56d2aaa9eb63bbb63456d5b5,2016-02-28,2016-02-27 16:07:05.0,2016-02-28 08:07:05,2016-02-28,1456646825]
[56d2aab15a21fa5f4c4f42a7,2016-02-28,2016-02-27 16:07:13.0,2016-02-28 08:07:13,2016-02-28,1456646833]
[56d2aac8aeeee48b74531af0,2016-02-28,2016-02-27 16:07:36.0,2016-02-28 08:07:36,2016-02-28,1456646856]
[56d2ab1d87fd3f4f72567788,2016-02-28,2016-02-27 16:09:01.0,2016-02-28 08:09:01,2016-02-28,1456646941]

美国/太平洋地区的时间应该是2016-02-28 00:01:27等等,但有些时间是两次减去"8"小时

Answer 1

经过一段时间的阅读后得出以下结论:

• Spark-Sql不支持日期时间,也不支持时区
• 使用时间戳是唯一的解决方案
• from_unixtime(at)正确解析纪元时间,只是由于时区而将其打印为字符串会改变它.可以安全地假设 from_unixtime它将正确转换它(虽然打印它可能会显示不同的结果)
• from_utc_timestamp 将时间戳转移(不仅仅转换)到该时区,在这种情况下,它将减去8小时到(-08:00)以来的时间
• 打印sql结果会使时区参数变得混乱