pyspark.resource.ResourceProfile¶

class pyspark.resource.ResourceProfile(_java_resource_profile: Optional[py4j.java_gateway.JavaObject] = None, _exec_req: Optional[Dict[str, pyspark.resource.requests.ExecutorResourceRequest]] = None, _task_req: Optional[Dict[str, pyspark.resource.requests.TaskResourceRequest]] = None)[source]¶

Resource profile to associate with an RDD. A pyspark.resource.ResourceProfile allows the user to specify executor and task requirements for an RDD that will get applied during a stage. This allows the user to change the resource requirements between stages. This is meant to be immutable so user cannot change it after building.

New in version 3.1.0.

Notes

This API is evolving.

Attributes

`executorResources`
`id`
`taskResources`

pyspark.resource.ResourceInformation pyspark.resource.ResourceProfileBuilder