pyspark.sql.datasource.DataSourceStreamReader.partitions#
- DataSourceStreamReader.partitions(start, end)[source]#
Returns a list of InputPartition given the start and end offsets. Each InputPartition represents a data split that can be processed by one Spark task. This may be called with an empty offset range when start == end, in that case the method should return an empty sequence of InputPartition.
- Parameters:
- startdict
The start offset of the microbatch to plan partitioning.
- enddict
The end offset of the microbatch to plan partitioning.
- Returns:
- sequence of
InputPartitions A sequence of partitions for this data source. Each partition value must be an instance of InputPartition or a subclass of it.
- sequence of