如何在字符串中创建spark中的TimestampType列

Question

我有一些数据包含在如下的字符串数组中(仅用于示例):

val myArray = Array("1499955986039", "1499955986051", "1499955986122")

我想将我的列表映射到Timestamp数组,以便创建一个RDD(myRdd),然后创建一个这样的数据帧

val df = createdataframe(myRdd, StructType(StructField("myTymeStamp", TimestampType,true)

我的问题不是如何创建Rdd,而是如何用毫秒时间戳替换字符串.你有什么主意吗？谢谢

Answer 1

用途java.sql.Timestamp:

val myArray = Array("1499955986039", "1499955986051", "1499955986122")
import java.sql.Timestamp    
import org.apache.spark.sql.Row
import org.apache.spark.sql.types.{StructType, StructField, TimestampType}

val rdd = sc.parallelize(myArray).map(s => Row(new Timestamp(s.toLong)))

val schema = StructType(Array(StructField("myTymeStamp", TimestampType, true)))

spark.createDataFrame(rdd, schema)
// res25: org.apache.spark.sql.DataFrame = [myTymeStamp: timestamp]