pyspark.resource.ExecutorResourceRequests

class pyspark.resource.ExecutorResourceRequests(_jvm: Optional[py4j.java_gateway.JVMView] = None, _requests: Optional[Dict[str, pyspark.resource.requests.ExecutorResourceRequest]] = None)

A set of Executor resource requests. This is used in conjunction with the pyspark.resource.ResourceProfileBuilder to programmatically specify the resources needed for an RDD that will be applied at the stage level.

New in version 3.1.0.

See also

pyspark.resource.ResourceProfile

Notes

This API is evolving.

Methods

cores(amount)	Specify number of cores per Executor.
memory(amount)	Specify heap memory.
memoryOverhead(amount)	Specify overhead memory.
offheapMemory(amount)	Specify off heap memory.
pysparkMemory(amount)	Specify pyspark memory.
resource(resourceName, amount[, …])	Amount of a particular custom resource(GPU, FPGA, etc) to use.

Attributes

requests

Returns

pyspark.resource.ExecutorResourceRequest pyspark.resource.TaskResourceRequest