pyspark.pandas.read_delta

pyspark.pandas.read_delta(path: str, version: Optional[str] = None, timestamp: Optional[str] = None, index_col: Union[str, List[str], None] = None, **options: Any) → pyspark.pandas.frame.DataFrame

Read a Delta Lake table on some file system and return a DataFrame.

If the Delta Lake table is already stored in the catalog (aka the metastore), use ‘read_table’.

Parameters

pathstring

Path to the Delta Lake table.

versionstring, optional

Specifies the table version (based on Delta’s internal transaction version) to read from, using Delta’s time travel feature. This sets Delta’s ‘versionAsOf’ option. Note that this paramter and timestamp paramter cannot be used together, otherwise it will raise a ValueError.

timestampstring, optional

Specifies the table version (based on timestamp) to read from, using Delta’s time travel feature. This must be a valid date or timestamp string in Spark, and sets Delta’s ‘timestampAsOf’ option. Note that this paramter and version paramter cannot be used together, otherwise it will raise a ValueError.

index_colstr or list of str, optional, default: None

Index column of table in Spark.

options

Additional options that can be passed onto Delta.

Returns

DataFrame

See also

DataFrame.to_delta

read_table

read_spark_io

read_parquet

Examples

>>> ps.range(1).to_delta('%s/read_delta/foo' % path)  
>>> ps.read_delta('%s/read_delta/foo' % path)  
   id
0   0

>>> ps.range(10, 15, num_partitions=1).to_delta('%s/read_delta/foo' % path,
...                                             mode='overwrite')  
>>> ps.read_delta('%s/read_delta/foo' % path)  
   id
0  10
1  11
2  12
3  13
4  14

>>> ps.read_delta('%s/read_delta/foo' % path, version=0)  
   id
0   0

You can preserve the index in the roundtrip as below.

>>> ps.range(10, 15, num_partitions=1).to_delta(
...     '%s/read_delta/bar' % path, index_col="index")  
>>> ps.read_delta('%s/read_delta/bar' % path, index_col="index")  
       id
index
0      10
1      11
2      12
3      13
4      14

pyspark.pandas.DataFrame.to_table pyspark.pandas.DataFrame.to_delta