在PySpark中,重新分区模块具有一个可选的column参数,该参数当然将通过该键对您的数据框进行重新分区。
我的问题是-没有钥匙时,Spark如何重新分区?我无法进一步深入研究源代码,以找到通过Spark本身进行处理的地方。
def repartition(self, numPartitions, *cols):
"""
Returns a new :class:`DataFrame` partitioned by the given partitioning expressions. The
resulting DataFrame is hash partitioned.
:param numPartitions:
can be an int to specify the target number of partitions or a Column.
If it is a Column, it will be used as the first partitioning column. If not specified,
the default number of partitions is used.
.. versionchanged:: 1.6
Added optional arguments to specify the partitioning columns. Also made numPartitions
optional if partitioning …Run Code Online (Sandbox Code Playgroud)