pyspark.pandas.DataFrame.spark.repartition

spark.repartition(num_partitions: int) → ps.DataFrame

Returns a new DataFrame partitioned by the given partitioning expressions. The resulting DataFrame is hash partitioned.

Parameters

num_partitionsint

The target number of partitions.

Returns

DataFrame

Examples

>>> psdf = ps.DataFrame({"age": [5, 5, 2, 2],
...         "name": ["Bob", "Bob", "Alice", "Alice"]}).set_index("age")
>>> psdf.sort_index()  
      name
age
2    Alice
2    Alice
5      Bob
5      Bob
>>> new_psdf = psdf.spark.repartition(7)
>>> new_psdf.to_spark().rdd.getNumPartitions()
7
>>> new_psdf.sort_index()   
      name
age
2    Alice
2    Alice
5      Bob
5      Bob

pyspark.pandas.DataFrame.spark.apply pyspark.pandas.DataFrame.spark.coalesce