pyspark.SparkContext.setCheckpointDir

SparkContext.setCheckpointDir(dirName: str) → None

Set the directory under which RDDs are going to be checkpointed. The directory must be an HDFS path if running on a cluster.

New in version 0.7.0.

Parameters

dirNamestr

path to the directory where checkpoint files will be stored (must be HDFS path if running in cluster)

See also

SparkContext.getCheckpointDir()

RDD.checkpoint()

RDD.getCheckpointFile()

pyspark.SparkContext.sequenceFile pyspark.SparkContext.setJobDescription