如何从PySpark DataFrame获取随机行？

Question

如何从PySpark DataFrame中获取随机行？我只看到sample()以分数作为参数的方法.将此分数设置1/numberOfRows为随机结果,有时我不会得到任何行.

On RRD有一个方法takeSample(),它将您希望样本包含的元素数作为参数.我知道这可能很慢,因为你必须计算每个分区,但有没有办法在DataFrame上得到这样的东西？

Answer 1

你可以简单地打电话takeSample给RDD:

df = sqlContext.createDataFrame(
    [(1, "a"), (2, "b"), (3, "c"), (4, "d")], ("k", "v"))
df.rdd.takeSample(False, 1, seed=0)
## [Row(k=3, v='c')]

如果你不想收集,你可以简单地采取更高的分数和限制:

df.sample(False, 0.1, seed=0).limit(1)

Answer 2

不同类型的样品

随机抽样 % 有替换和无替换的数据

import pyspark.sql.functions as F
#Randomly sample 50% of the data without replacement
sample1 = df.sample(False, 0.5, seed=0)

#Randomly sample 50% of the data with replacement
sample1 = df.sample(True, 0.5, seed=0)

#Take another sample exlcuding records from previous sample using Anti Join
sample2 = df.join(sample1, on='ID', how='left_anti').sample(False, 0.5, seed=0)

#Take another sample exlcuding records from previous sample using Where
sample1_ids = [row['ID'] for row in sample1.ID]
sample2 = df.where(~F.col('ID').isin(sample1_ids)).sample(False, 0.5, seed=0)

#Generate a startfied sample of the data across column(s)
#Sampling is probabilistic and thus cannot guarantee an exact number of rows
fractions = {
        'NJ': 0.5, #Take about 50% of records where state = NJ
    'NY': 0.25, #Take about 25% of records where state = NY
    'VA': 0.1, #Take about 10% of records where state = VA
}
stratified_sample = df.sampleBy(F.col('state'), fractions, seed=0)