pyspark.streaming.DStream.transform

DStream.transform(func: Union[Callable[[pyspark.rdd.RDD[T]], pyspark.rdd.RDD[U]], Callable[[datetime.datetime, pyspark.rdd.RDD[T]], pyspark.rdd.RDD[U]]]) → pyspark.streaming.dstream.TransformedDStream[U][source]

Return a new DStream in which each RDD is generated by applying a function on each RDD of this DStream.

func can have one argument of rdd, or have two arguments of (time, rdd)