pyspark.resource.ResourceProfile#

classpyspark.resource.ResourceProfile(_java_resource_profile=None,_exec_req=None,_task_req=None)[source]#

Resource profile to associate with an RDD. Apyspark.resource.ResourceProfileallows the user to specify executor and task requirements for an RDD that will getapplied during a stage. This allows the user to change the resource requirements betweenstages. This is meant to be immutable so user cannot change it after building.

New in version 3.1.0.

Changed in version 4.0.0:Supports Spark Connect.

Notes

This API is evolving.

Examples

Create Executor resource requests.

>>>executor_requests=(...ExecutorResourceRequests()....cores(2)....memory("6g")....memoryOverhead("1g")....pysparkMemory("2g")....offheapMemory("3g")....resource("gpu",2,"testGpus","nvidia.com")...)

Create task resource requasts.

>>>task_requests=TaskResourceRequests().cpus(2).resource("gpu",2)

Create a resource profile.

>>>builder=ResourceProfileBuilder()>>>resource_profile=builder.require(executor_requests).require(task_requests).build

Create an RDD with the resource profile.

>>>rdd=sc.parallelize(range(10)).withResources(resource_profile)>>>rdd.getResourceProfile()<pyspark.resource.profile.ResourceProfile object ...>>>>rdd.getResourceProfile().taskResources{'cpus': <...TaskResourceRequest...>, 'gpu': <...TaskResourceRequest...>}>>>rdd.getResourceProfile().executorResources{'gpu': <...ExecutorResourceRequest...>, 'cores': <...ExecutorResourceRequest...>, 'offHeap': <...ExecutorResourceRequest...>, 'memoryOverhead': <...ExecutorResourceRequest...>, 'pyspark.memory': <...ExecutorResourceRequest...>, 'memory': <...ExecutorResourceRequest...>}

Attributes

executorResources	Returns
id	Returns
taskResources	Returns