如何从pyspark中的时间戳中提取时间？

Question

我需要使用 pyspark 从时间戳（这是数据帧中的一列）中提取时间。假设这是时间戳 2019-01-03T18:21:39 ，我只想提取时间“18:21:39”，以便它始终以这种方式显示“01:01:01”

df = spark.createDataFrame(["2020-06-17T00:44:30","2020-06-17T06:06:56","2020-06-17T15:04:34"],StringType()).toDF('datetime')

df=df.select(df['datetime'].cast(TimestampType()))

我尝试了如下但没有得到预期的结果

df1=df.withColumn('time',concat(hour(df['datetime']),lit(":"),minute(df['datetime']),lit(":"),second(df['datetime'])))

display(df1)

+-------------------+-------+
|           datetime|   time|
+-------------------+-------+
|2020-06-17 00:44:30|0:44:30|
|2020-06-17 06:06:56| 6:6:56|
|2020-06-17 15:04:34|15:4:34|
+-------------------+-------+

我的结果是这样的6:6:56，但我希望它们是06:06:56

Answer 1

使用date_format函数。

from pyspark.sql.types import StringType

df = spark \
  .createDataFrame(["2020-06-17T00:44:30","2020-06-17T06:06:56","2020-06-17T15:04:34"], StringType()) \
  .toDF('datetime')

from pyspark.sql.functions import date_format
q = df.withColumn('time', date_format('datetime', 'HH:mm:ss'))

>>> q.show()
+-------------------+--------+
|           datetime|    time|
+-------------------+--------+
|2020-06-17T00:44:30|00:44:30|
|2020-06-17T06:06:56|06:06:56|
|2020-06-17T15:04:34|15:04:34|
+-------------------+--------+