从字符串 PySpark Dataframe 列中删除正则表达式

Question

我需要从 pyspark 数据框中的字符串列中删除正则表达式

df = spark.createDataFrame(
    [
        ("Dog 10H03", "10H03"),
        ("Cat 09H24 eats rat", "09H24"),
        ("Mouse 09H45 runs away", "09H45"),
        ("Mouse 09H45 enters room", "09H45"),
    ],
    ["Animal", "Time"],
)

时间戳（例如10H03）是必须删除的正则表达式。

+--------------------+------------------+-----+
|              Animal| Animal_strip_time| Time|
+--------------------+------------------+-----+
|           Dog 10H03|              Dog |10H03|
|  Cat 09H24 eats rat|     Cat  eats rat|09H24|
|Mouse 09H45 runs ...|  Mouse  runs away|09H45|
|Mouse 09H45 enter...|Mouse  enters room|09H45|
+--------------------+------------------+-----+

该列中的时间戳Time可能与该列中的时间戳不同Animal。因此，它不能用于匹配字符串。

正则表达式应遵循 XXHXX 模式，其中 X 是 0-9 之间的数字

Answer 1

这应该可以完成工作：

from pyspark.sql import functions as F

df = df.withColumn("Animal_strip_time", F.regexp_replace("Animal", r"\d\dH\d\d", ""))

df.show()
+--------------------+-----+------------------+                                 
|              Animal| Time| Animal_strip_time|
+--------------------+-----+------------------+
|           Dog 10H03|10H03|              Dog |
|  Cat 09H24 eats rat|09H24|     Cat  eats rat|
|Mouse 09H45 runs ...|09H45|  Mouse  runs away|
|Mouse 09H45 enter...|09H45|Mouse  enters room|
+--------------------+-----+------------------+