我正在尝试比较两个 pandas 数据帧,但出现错误,因为“DataFrame”对象没有属性“withColumn”。可能是什么问题?
import pandas as pd
import pyspark.sql.functions as F
pd_df=pd.DataFrame(df.dtypes,columns=['column','data_type'])
pd_df1=pd.DataFrame(df1.dtypes,columns=['column','data_type'])
pd.merge(pd_df,pd_df1, on='column', how='outer'
).withColumn(
"result",
F.when(F.col("data_type_x") == 'NaN','new attribute'.otherwise('old attribute')))
.select(
"column",
"data_type_x",
"data_type_y",
"result"
)
Run Code Online (Sandbox Code Playgroud)
df 和 df1 是一些数据框