pyspark.streaming.DStream.updateStateByKey

DStream.updateStateByKey(updateFunc, numPartitions=None, initialRDD=None)

Return a new “state” DStream where the state for each key is updated by applying the given function on the previous state of the key and the new values of the key.

Parameters

updateFuncfunction

State update function. If this function returns None, then corresponding state key-value pair will be eliminated.

pyspark.streaming.DStream.union pyspark.streaming.DStream.window