pyspark.SparkFiles.get

classmethod SparkFiles.get(filename: str) → str

Get the absolute path of a file added through SparkContext.addFile() or SparkContext.addPyFile().

New in version 0.7.0.

Parameters

filenamestr

file that are added to resources

Returns

str

the absolute path of the file

See also

SparkFiles.getRootDirectory()

SparkContext.addFile()

SparkContext.addPyFile()

SparkContext.listFiles()

Examples

>>> import os
>>> import tempfile
>>> from pyspark import SparkFiles

>>> with tempfile.TemporaryDirectory() as d:
...     path1 = os.path.join(d, "test.txt")
...     with open(path1, "w") as f:
...         _ = f.write("100")
...
...     sc.addFile(path1)
...     file_list1 = sorted(sc.listFiles)
...
...     def func1(iterator):
...         path = SparkFiles.get("test.txt")
...         assert path.startswith(SparkFiles.getRootDirectory())
...         return [path]
...
...     path_list1 = sc.parallelize([1, 2, 3, 4]).mapPartitions(func1).collect()
...
...     path2 = os.path.join(d, "test.py")
...     with open(path2, "w") as f:
...         _ = f.write("import pyspark")
...
...     # py files
...     sc.addPyFile(path2)
...     file_list2 = sorted(sc.listFiles)
...
...     def func2(iterator):
...         path = SparkFiles.get("test.py")
...         assert path.startswith(SparkFiles.getRootDirectory())
...         return [path]
...
...     path_list2 = sc.parallelize([1, 2, 3, 4]).mapPartitions(func2).collect()
>>> file_list1
['file:/.../test.txt']
>>> set(path_list1)
{'.../test.txt'}
>>> file_list2
['file:/.../test.py', 'file:/.../test.txt']
>>> set(path_list2)
{'.../test.py'}

pyspark.SparkConf.toDebugString pyspark.SparkFiles.getRootDirectory