pyspark.sql.streaming.StreamingQuery.recentProgress

property StreamingQuery.recentProgress

Returns an array of the most recent [[StreamingQueryProgress]] updates for this query. The number of progress updates retained for each stream is configured by Spark session configuration spark.sql.streaming.numRecentProgressUpdates.

New in version 2.1.0.

Returns

list

List of dict which is the most recent StreamingQueryProgress updates for this query.

Examples

>>> sdf = spark.readStream.format("rate").load()
>>> sq = sdf.writeStream.format('memory').queryName('this_query').start()

Get an array of the most recent query progress updates for this query

>>> sq.recentProgress 
[...]

>>> sq.stop()

pyspark.sql.streaming.StreamingQuery.processAllAvailable pyspark.sql.streaming.StreamingQuery.runId