API Reference

This page lists an overview of all public PySpark modules, classes, functions and methods.

Note

Spark SQL, Pandas API on Spark, Structured Streaming, and MLlib (DataFrame-based) support Spark Connect.

• Spark SQL
  • Core Classes
  • Spark Session
  • Configuration
  • Input/Output
  • DataFrame
  • Column
  • Data Types
  • Row
  • Functions
  • Window
  • Grouping
  • Catalog
  • Avro
  • Observation
  • UDF
  • UDTF
  • VariantVal
  • Protobuf
  • Python Data Source
• Pandas API on Spark
  • Input/Output
  • General functions
  • Series
  • DataFrame
  • Index objects
  • Window
  • GroupBy
  • Resampling
  • Options and settings
  • Machine Learning utilities
  • Extensions
  • Testing
• Structured Streaming
  • Core Classes
  • Input/Output
  • Query Management
• MLlib (DataFrame-based)
  • Pipeline APIs
  • Parameters
  • Feature
  • Classification
  • Clustering
  • Functions
  • Vector and Matrix
  • Recommendation
  • Regression
  • Statistics
  • Tuning
  • Evaluation
  • Frequency Pattern Mining
  • Image
  • Distributor
  • Utilities
• MLlib (DataFrame-based) for Spark Connect
  • Pipeline APIs
  • Feature
  • Classification
  • Functions
  • Tuning
  • Evaluation
  • Utilities
• Spark Streaming (Legacy)
  • Core Classes
  • Streaming Management
  • Input and Output
  • Transformations and Actions
  • Kinesis
• MLlib (RDD-based)
  • Classification
  • Clustering
  • Evaluation
  • Feature
  • Frequency Pattern Mining
  • Vector and Matrix
  • Random
  • Recommendation
  • Regression
  • Statistics
  • Tree
  • Utilities
• Spark Core
  • Public Classes
  • Spark Context APIs
  • RDD APIs
  • Broadcast and Accumulator
  • Management
• Resource Management
  • Core Classes
• Errors
  • Classes
  • Methods
• Testing
  • pyspark.testing.assertDataFrameEqual
  • pyspark.testing.assertSchemaEqual