小编And*_*nin的帖子

如何在pyspark中使用带有两列的date_add?

我有一个包含一些列的数据框:

+------------+--------+----------+----------+
|country_name| ID_user|birth_date|      psdt|
+------------+--------+----------+----------+
|      ??????|16460783|       486|1970-01-01|
|      ??????|16467391|      4669|1970-01-01|
|      ??????|16467889|      6861|1970-01-01|
|   ?????????|16468013|      5360|1970-01-01|
|      ??????|16471027|      6311|1970-01-01|
|      ??????|16474162|      5567|1970-01-01|
|      ??????|16476386|      4351|1970-01-01|
|      ??????|16481067|      3831|1970-01-01|
|   ?????????|16485965|     -2369|1970-01-01|
|    ????????|16486027|      5864|1970-01-01|
+------------+--------+----------+----------+
only showing top 10 rows
Run Code Online (Sandbox Code Playgroud)

我需要用“birth_date”添加“psdt”。我写了这段代码,但 (sf.date_add) 不起作用:

resultbirthDF =(
        resultDF
        .select(sf.col("country_name"),
                sf.col("ID_user"),
                sf.col("birth_date"),
                sf.lit(past_datetr).alias("psdt")
               )
        .withColumn("birth_datetrue",sf.date_add(sf.to_date(sf.col("psdt")),sf.col("birth_date")))
    ).show(10)

'Column' object is not callable
Traceback (most recent call last):
  File "/volumes/disk1/yarn/local/usercache/livy/appcache/application_1573843665329_0786/container_e05_1573843665329_0786_01_000001/pyspark.zip/pyspark/sql/functions.py", line 1006, in date_add
    return Column(sc._jvm.functions.date_add(_to_java_column(start), days))
Run Code Online (Sandbox Code Playgroud)

如何解决这个问题呢?

apache-spark apache-spark-sql pyspark

3
推荐指数
1
解决办法
2703
查看次数

标签 统计

apache-spark ×1

apache-spark-sql ×1

pyspark ×1