System and user compute profiles

A compute profile indicates which provisioner to use when creating a clusterand specifies the cluster configuration. They also specify the provisionerconfiguration that should be used when creating a cluster.

• To create asystem compute profile, go to theSystem admin page inCloud Data Fusion Studio. This page lists all system compute profilesand lets you create new system compute profiles.
• To create auser compute profile, go to theNamespaceadministration page in Cloud Data Fusion Studio, and then select thenamespace to create the profile in. Then, you can create a profile thatexists only within that namespace.

Compute profile assignment

You can assign compute profiles to batch pipelines in the following ways:

• Assign a default profile for the Cloud Data Fusion instance.
• Assign a default profile for a specific namespace.
• Assign a profile to a batch pipeline to use for runs that are startedmanually.
• Assign a profile to a pipeline schedule.

If a profile is set in the schedule that triggers a run, or if you manually runa pipeline and there's a profile assigned to that pipeline,Cloud Data Fusion uses that compute profile.

If no profile is set, Cloud Data Fusion uses the default profile for thenamespace. If no default profile is set for the namespace,

Cloud Data Fusion uses the system default profile. If no system default isset, the built-in profile is used.

Assign a default compute profile

To assign default profiles to a Cloud Data Fusion namespace or instance, goto the Cloud Data Fusion Studio and clickSystem admin>Configuration>System compute profiles. To select thedefault, click thestar star by the profilename.

Optional: use the Preferences Microservices to set default profiles

• To set the default profile, set a preference on the Cloud Data Fusioninstance with key system.profile.name and valuesystem:<profile-name>.
• To set the default profile for a namespace, set a preference on thechosen namespace with keysystem.profile.name and value<scope>:<profile-name>.

Assign a compute profile for manual runs

To assign a profile to use for manual pipeline runs, follow these steps:

1. Navigate to the pipeline detail page.
2. ClickConfigure > Compute config.
3. Select a profile and clickSave. The selected profile is usedwhenever the pipeline runs manually.

Alternatively, you can use the Preferences Microservices to set the profile formanual runs by setting preference on theDataPipelineWorkflow entity with keysystem.profile.name and value<scope>:<profile-name>.

Assign a compute profile to a schedule

Any time you create a schedule for a pipeline, you can assign a profile to it.Whenever the schedule triggers a pipeline run, it will use that profile for therun. This is true for time schedules and schedules that other pipelinestrigger.

Override a compute profile configuration

When a profile is created, each configuration setting can be made immutable bylocking it. However, if configuration settings are not locked, they can beoverridden at runtime. To override profile configuration, follow these steps:

1. From the Pipeline List page, select the deployed pipeline you want to run.
2. From the Pipeline Details page, clickConfigure.
3. Choose a compute profile and clickCustomize.
4. Change any settings and clickSave.

You can use runtime arguments and schedule properties to modify the clustersize and other settings.

• To override the profile used, set a runtime argument with the keysystem.profile.nameand value<scope>:<profile-name>.
• To override a profile property, set a runtime argument with keysystem.profile.properties.<property-name> and value equal to the value forthat property.

For example, to override thenumWorkerssetting to a value of10, set apreference or runtime argument with the keysystem.profile.properties.numWorkers and the value10.

What's next