小编toh*_*ong的帖子

在另一个数据帧上执行UDF时，如何引用pyspark数据帧？

这是一个虚拟的例子。我创建了两个dataframes scores和lastnames，并在各躺着一个列，它是在两个dataframes相同。在应用的UDF中scores，我要过滤lastnames并返回在中找到的字符串lastname。

from pyspark import SparkContext
from pyspark import SparkConf
from pyspark.sql import SQLContext
from pyspark.sql.types import *

sc = SparkContext("local")
sqlCtx = SQLContext(sc)


# Generate Random Data
import itertools
import random
student_ids = ['student1', 'student2', 'student3']
subjects = ['Math', 'Biology', 'Chemistry', 'Physics']
random.seed(1)
data = []

for (student_id, subject) in itertools.product(student_ids, subjects):
    data.append((student_id, subject, random.randint(0, 100)))

from pyspark.sql.types import StructType, StructField, IntegerType, StringType
schema = StructType([
            StructField("student_id", StringType(), nullable=False), …