pyspark.pandas.read_csv#

pyspark.pandas.read_csv(path, sep=',', header='infer', names=None, index_col=None, usecols=None, dtype=None, nrows=None, parse_dates=False, quotechar=None, escapechar=None, comment=None, encoding=None, **options)[source]#

Read CSV (comma-separated) file into DataFrame or Series.

Parameters
pathstr or list

Path(s) of the CSV file(s) to be read.

sepstr, default ‘,’

Delimiter to use. Non empty string.

headerint, default ‘infer’

Whether to use the column names, and the start of the data. Default behavior is to infer the column names: if no names are passed the behavior is identical to header=0 and column names are inferred from the first line of the file, if column names are passed explicitly then the behavior is identical to header=None. Explicitly pass header=0 to be able to replace existing names

namesstr or array-like, optional

List of column names to use. If file contains no header row, then you should explicitly pass header=None. Duplicates in this list will cause an error to be issued. If a string is given, it should be a DDL-formatted string in Spark SQL, which is preferred to avoid schema inference for better performance.

index_col: str or list of str, optional, default: None

Index column of table in Spark.

usecolslist-like or callable, optional

Return a subset of the columns. If list-like, all elements must either be positional (i.e. integer indices into the document columns) or strings that correspond to column names provided either by the user in names or inferred from the document header row(s). If callable, the callable function will be evaluated against the column names, returning names where the callable function evaluates to True.

dtypeType name or dict of column -> type, default None

Data type for data or columns. E.g. {‘a’: np.float64, ‘b’: np.int32} Use str or object together with suitable na_values settings to preserve and not interpret dtype.

nrowsint, default None

Number of rows to read from the CSV file.

parse_datesboolean or list of ints or names or list of lists or dict, default False.

Currently only False is allowed.

quotecharstr (length 1), optional

The character used to denote the start and end of a quoted item. Quoted items can include the delimiter and it will be ignored.

escapecharstr (length 1), default None

One-character string used to escape other characters.

comment: str, optional

Indicates the line should not be parsed.

encoding: str, optional

Indicates the encoding to read file

optionsdict

All other options passed directly into Spark’s data source.

Returns
DataFrame or Series

See also

DataFrame.to_csv

Write DataFrame to a comma-separated values (csv) file.

Examples

>>> ps.read_csv('data.csv')  

Load multiple CSV files as a single DataFrame:

>>> ps.read_csv(['data-01.csv', 'data-02.csv'])