我在pyspark中使用自定义函数来检查spark数据帧中每一行的条件,如果condition为true则添加列.
代码如下:
from pyspark.sql.types import *
from pyspark.sql.functions import *
from pyspark.sql import Row
def customFunction(row):
if (row.prod.isNull()):
prod_1 = "new prod"
return (row + Row(prod_1))
else:
prod_1 = row.prod
return (row + Row(prod_1))
sdf = sdf_temp.map(customFunction)
sdf.show()
Run Code Online (Sandbox Code Playgroud)
我在下面提到了错误:
AttributeError:'unicode'对象没有属性'isNull'
如何检查自定义函数中当前行中特定列的空值?
isnull user-defined-functions apache-spark pyspark spark-dataframe