替换 pyspark 数据框中列名称中的字符

Question

我在 Pyspark 中有一个如下所示的数据框

df = spark.createDataFrame([(2,'john',1,1),
                            (2,'john',1,2),
                            (3,'pete',8,3),
                            (3,'pete',8,4),
                            (5,'steve',9,5)],
                           ['id','/na/me','val/ue', 'rank/'])

df.show()

+---+------+------+-----+
| id|/na/me|val/ue|rank/|
+---+------+------+-----+
|  2|  john|     1|    1|
|  2|  john|     1|    2|
|  3|  pete|     8|    3|
|  3|  pete|     8|    4|
|  5| steve|     9|    5|
+---+------+------+-----+

/现在，在这个数据框中，我想替换scrore 下的列名称_。但是，如果/出现在列名称的开头或结尾，则删除 ，/但不要替换为_。

我已经做了如下

for name in df.schema.names:
  df = df.withColumnRenamed(name, name.replace('/', '_'))


>>> df
DataFrame[id: bigint, _na_me: string, val_ue: bigint, rank_: bigint]


>>>df.show()
+---+------+------+-----+
| id|_na_me|val_ue|rank_|
+---+------+------+-----+
|  2|  john|     1|    1|
|  2|  john|     1|    2|
|  3|  pete|     8|    3|
|  3|  pete|     8|    4|
|  5| steve|     9|    5|
+---+------+------+-----+

我怎样才能达到下面我想要的结果

+---+------+------+-----+
| id| na_me|val_ue| rank|
+---+------+------+-----+
|  2|  john|     1|    1|
|  2|  john|     1|    2|
|  3|  pete|     8|    3|
|  3|  pete|     8|    4|
|  5| steve|     9|    5|
+---+------+------+-----+

Answer 1

尝试regular expression以 python 方式替换（re.sub）。

import re
cols=[re.sub(r'(^_|_$)','',f.replace("/","_")) for f in df.columns]

df = spark.createDataFrame([(2,'john',1,1),
                            (2,'john',1,2),
                            (3,'pete',8,3),
                            (3,'pete',8,4),
                            (5,'steve',9,5)],
                           ['id','/na/me','val/ue', 'rank/'])

df.toDF(*cols).show()
#+---+-----+------+----+
#| id|na_me|val_ue|rank|
#+---+-----+------+----+
#|  2| john|     1|   1|
#|  2| john|     1|   2|
#|  3| pete|     8|   3|
#|  3| pete|     8|   4|
#|  5|steve|     9|   5|
#+---+-----+------+----+

#or using for loop on schema.names
for name in df.schema.names:
  df = df.withColumnRenamed(name, re.sub(r'(^_|_$)','',name.replace('/', '_')))

df.show()
#+---+-----+------+----+
#| id|na_me|val_ue|rank|
#+---+-----+------+----+
#|  2| john|     1|   1|
#|  2| john|     1|   2|
#|  3| pete|     8|   3|
#|  3| pete|     8|   4|
#|  5|steve|     9|   5|
#+---+-----+------+----+