pyspark.SparkContext.setCheckpointDir

SparkContext.setCheckpointDir(dirName)

Set the directory under which RDDs are going to be checkpointed. The directory must be an HDFS path if running on a cluster.

New in version 0.7.0.

Parameters

dirNamestr

path to the directory where checkpoint files will be stored (must be HDFS path if running in cluster)

See also

SparkContext.getCheckpointDir()

RDD.checkpoint()

RDD.getCheckpointFile()