pyspark:向数据框Row元素添加一个新字段

Question

我有以下元素:

a = Row(ts=1465326926253, myid=u'1234567', mytype=u'good') 

Row是火花数据框Row类.我可以在a中附加一个新字段,所以看起来像:

a = Row(ts=1465326926253, myid=u'1234567', mytype=u'good', name = u'john') 

谢谢!

Answer 1

这是一个有效的更新答案.首先,您必须创建一个字典,然后更新dict,然后将其写入pyspark Row.

代码如下:

from pyspark.sql import Row

#Creating the pysql row
row = Row(field1=12345, field2=0.0123, field3=u'Last Field')

#Convert to python dict
temp = row.asDict()

#Do whatever you want to the dict. Like adding a new field or etc.
temp["field4"] = "it worked!"

# Save or output the row to a pyspark rdd
output = Row(**temp)

#How it looks
output

In [1]:
Row(field1=12345, field2=0.0123, field3=u'Last Field', field4='it worked!')

Answer 2

您无法将新字段添加到中Row。Row是的子类tuple

from pyspark.sql import Row

issubclass(Row, tuple)
## True

isinstance(Row(), tuple)
## True

和Python tuples是不可变的。您所能做的就是创建一个新的：

row = Row(ts=1465326926253, myid=u'1234567', mytype=u'good') 

# In legacy Python: Row(name=u"john", **row.asDict())
Row(**row.asDict(), name=u"john") 
## Row(myid='1234567', mytype='good', name='john', ts=1465326926253)

请注意，Row保持其字段按名称排序。