pyspark.sql.SparkSession.builder.getOrCreate

builder.getOrCreate() → pyspark.sql.session.SparkSession

Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder.

New in version 2.0.0.

Changed in version 3.4.0: Supports Spark Connect.

Returns

SparkSession

Examples

This method first checks whether there is a valid global default SparkSession, and if yes, return that one. If no valid global default SparkSession exists, the method creates a new SparkSession and assigns the newly created SparkSession as the global default.

>>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
>>> s1.conf.get("k1") == "v1"
True

The configuration of the SparkSession can be changed afterwards

>>> s1.conf.set("k1", "v1_new")
>>> s1.conf.get("k1") == "v1_new"
True

In case an existing SparkSession is returned, the config options specified in this builder will be applied to the existing SparkSession.

>>> s2 = SparkSession.builder.config("k2", "v2").getOrCreate()
>>> s1.conf.get("k1") == s2.conf.get("k1") == "v1_new"
True
>>> s1.conf.get("k2") == s2.conf.get("k2") == "v2"
True

pyspark.sql.SparkSession.builder.enableHiveSupport pyspark.sql.SparkSession.builder.master