pyspark.sql.SparkSession.builder.getOrCreate

builder.getOrCreate() → pyspark.sql.session.SparkSession

Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder.

New in version 2.0.0.

Examples

This method first checks whether there is a valid global default SparkSession, and if yes, return that one. If no valid global default SparkSession exists, the method creates a new SparkSession and assigns the newly created SparkSession as the global default.

>>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
>>> s1.conf.get("k1") == "v1"
True

In case an existing SparkSession is returned, the config options specified in this builder will be applied to the existing SparkSession.

>>> s2 = SparkSession.builder.config("k2", "v2").getOrCreate()
>>> s1.conf.get("k1") == s2.conf.get("k1")
True
>>> s1.conf.get("k2") == s2.conf.get("k2")
True

pyspark.sql.SparkSession.builder.enableHiveSupport pyspark.sql.SparkSession.builder.master