pyspark.sql.streaming.StreamingQueryManager.addListener#
- StreamingQueryManager.addListener(listener)[source]#
Register a
StreamingQueryListener
to receive up-calls for life cycle events ofStreamingQuery
.New in version 3.4.0.
Changed in version 3.5.0: Supports Spark Connect.
- Parameters
- listener
StreamingQueryListener
A
StreamingQueryListener
to receive up-calls for life cycle events ofStreamingQuery
.
- listener
Notes
This function behaves differently in Spark Connect mode. In Connect, the provided functions doesn’t have access to variables defined outside of it. Also in Connect, you need to use self.spark to access spark session. Using spark would throw an exception. In short, if you want to use spark session inside the listener, please use self.spark in Connect mode, and use spark otherwise.
Examples
>>> from pyspark.sql.streaming import StreamingQueryListener >>> class TestListener(StreamingQueryListener): ... def onQueryStarted(self, event): ... pass ... ... def onQueryProgress(self, event): ... pass ... ... def onQueryIdle(self, event): ... pass ... ... def onQueryTerminated(self, event): ... pass ... >>> test_listener = TestListener()
Register streaming query listener
>>> spark.streams.addListener(test_listener)
Deregister streaming query listener
>>> spark.streams.removeListener(test_listener)