This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
Note
Access to this page requires authorization. You can trysigning in orchanging directories.
Access to this page requires authorization. You can trychanging directories.
| automl | Contains automated machine learning classes for Azure Machine Learning SDKv2. Main areas include managing AutoML tasks. |
| constants | This package defines constants used in Azure Machine Learning SDKv2. |
| data_transfer | |
| dsl | |
| entities | Contains entities and SDK objects for Azure Machine Learning SDKv2. Main areas include managing compute targets, creating/managing workspaces and jobs, and submitting/accessing model, runsand run output/logging etc. |
| finetuning | Contains custom model finetuning classes for AzureML SDK V2. |
| identity | Contains Identity Configuration for Azure Machine Learning SDKv2. |
| model_customization | |
| operations | Contains supported operations for Azure Machine Learning SDKv2. Operations are classes contain logic to interact with backend services, usually auto generated operations call. |
| parallel | |
| sweep |
| exceptions | Contains exception module in Azure Machine Learning SDKv2. This includes enums and classes for exceptions. |
| Input | Initialize an Input object. |
| MLClient | A client class to interact with Azure ML services. Use this client to manage Azure ML resources such as workspaces, jobs, models, and so on. |
| MpiDistribution | MPI distribution configuration. |
| Output | Define an output. |
| PyTorchDistribution | PyTorch distribution configuration. |
| RayDistribution | Note This is an experimental class, and may change at any time. Please seehttps://aka.ms/azuremlexperimental for more information. Ray distribution configuration. |
| TensorFlowDistribution | TensorFlow distribution configuration. |
Creates a Command object which can be used inside a dsl.pipeline function or used as a standalone Command job.
command(*, name: str | None = None, description: str | None = None, tags: Dict | None = None, properties: Dict | None = None, display_name: str | None = None, command: str | None = None, experiment_name: str | None = None, environment: str | Environment | None = None, environment_variables: Dict | None = None, distribution: Dict | MpiDistribution | TensorFlowDistribution | PyTorchDistribution | RayDistribution | DistributionConfiguration | None = None, compute: str | None = None, inputs: Dict | None = None, outputs: Dict | None = None, instance_count: int | None = None, instance_type: str | None = None, locations: List[str] | None = None, docker_args: str | List[str] | None = None, shm_size: str | None = None, timeout: int | None = None, code: PathLike | str | None = None, identity: ManagedIdentityConfiguration | AmlTokenConfiguration | UserIdentityConfiguration | None = None, is_deterministic: bool = True, services: Dict[str, JobService | JupyterLabJobService | SshJobService | TensorBoardJobService | VsCodeJobService] | None = None, job_tier: str | None = None, priority: str | None = None, parent_job_name: str | None = None, **kwargs: Any) -> Command| Name | Description |
|---|---|
name | The name of the Command job or component. Default value: None |
description | The description of the Command. Defaults to None. Default value: None |
tags | Tag dictionary. Tags can be added, removed, and updated. Defaults to None. Default value: None |
properties | The job property dictionary. Defaults to None. Default value: None |
display_name | The display name of the job. Defaults to a randomly generated name. Default value: None |
command | The command to be executed. Defaults to None. Default value: None |
experiment_name | The name of the experiment that the job will be created under. Defaults to currentdirectory name. Default value: None |
environment | The environment that the job will run in. Default value: None |
environment_variables | A dictionary of environment variable names and values.These environment variables are set on the process where user script is being executed.Defaults to None. Default value: None |
distribution | The configuration for distributed jobs. Defaults to None. Default value: None |
compute | The compute target the job will run on. Defaults to default compute. Default value: None |
inputs | A mapping of input names to input data sources used in the job. Defaults to None. Default value: None |
outputs | A mapping of output names to output data sources used in the job. Defaults to None. Default value: None |
instance_count | The number of instances or nodes to be used by the compute target. Defaults to 1. Default value: None |
instance_type | The type of VM to be used by the compute target. Default value: None |
locations | The list of locations where the job will run. Default value: None |
docker_args | Extra arguments to pass to the Docker run command. This would override anyparameters that have already been set by the system, or in this section. This parameter is onlysupported for Azure ML compute types. Defaults to None. Default value: None |
shm_size | The size of the Docker container's shared memory block. This should be in theformat of (number)(unit) where the number has to be greater than 0 and the unit can be one ofb(bytes), k(kilobytes), m(megabytes), or g(gigabytes). Default value: None |
timeout | The number, in seconds, after which the job will be cancelled. Default value: None |
code | The source code to run the job. Can be a local path or "http:", "https:", or "azureml:" urlpointing to a remote location. Default value: None |
identity | The identity that the command job will use while running on compute. Default value: None |
is_deterministic | Specifies whether the Command will return the same output given the same input.Defaults to True. When True, if a Command Component is deterministic and has been run before in thecurrent workspace with the same input and settings, it will reuse results from a previously submittedjob when used as a node or step in a pipeline. In that scenario, no compute resources will be used. Default value: True |
services | Optional[dict[str,Union[JobService,JupyterLabJobService,SshJobService,TensorBoardJobService,VsCodeJobService]]] The interactive services for the node. Defaults to None. This is an experimental parameter,and may change at any time. Please seehttps://aka.ms/azuremlexperimental for more information. Default value: None |
job_tier | The job tier. Accepted values are "Spot", "Basic", "Standard", or "Premium". Default value: None |
priority | The priority of the job on the compute. Accepted values are "low", "medium", and "high".Defaults to "medium". Default value: None |
parent_job_name | parent job id for command job Default value: None |
| Type | Description |
|---|---|
A Command object. |
Creating a Command Job using the command() builder method.
from azure.ai.ml import Input, Output, command train_func = command( environment="AzureML-sklearn-1.0-ubuntu20.04-py38-cpu:33", command='echo "hello world"', distribution={"type": "Pytorch", "process_count_per_instance": 2}, inputs={ "training_data": Input(type="uri_folder"), "max_epochs": 20, "learning_rate": 1.8, "learning_rate_schedule": "time-based", }, outputs={"model_output": Output(type="uri_folder")}, )Construct a batch deployment object from yaml file.
load_batch_deployment(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> BatchDeployment| Name | Description |
|---|---|
source Required | The local yaml source of a batch deployment object. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Constructed batch deployment object. |
Construct a batch endpoint object from yaml file.
load_batch_endpoint(source: str | PathLike | IO, relative_origin: str | None = None, *, params_override: List[Dict] | None = None, **kwargs: Any) -> BatchEndpoint| Name | Description |
|---|---|
source Required | The local yaml source of a batch endpoint object. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
| Name | Description |
|---|---|
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Constructed batch endpoint object. |
Constructs a CapabilityHost object from a YAML file.
load_capability_host(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> CapabilityHost| Name | Description |
|---|---|
source Required | A path to a local YAML file or an already-open file object containing a capabilityhost configuration.If the source is a path, it will be opened and read. If the source is an open file, the file will be readdirectly. |
| Name | Description |
|---|---|
relative_origin | The root directory for the YAML. This directory will be used as the origin for deducingthe relative locations of files referenced in the parsed YAML. Defaults to the same directory as source ifsource is a file or file path input. Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Loaded CapabilityHost object. |
| Type | Description |
|---|---|
Raised if CapabilityHost cannot be successfully validated.Details will be provided in the error message. |
Loading a capabilityhost from a YAML config file.
from azure.ai.ml import load_capability_host capability_host = load_capability_host( source="./sdk/ml/azure-ai-ml/tests/test_configs/workspace/ai_workspaces/test_capability_host_hub.yml" )Load component from local or remote to a component function.
load_component(source: PathLike | str | IO | None = None, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> CommandComponent | ParallelComponent | PipelineComponent| Name | Description |
|---|---|
source | The local yaml source of a component. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. Default value: None |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
A Component object |
Loading a Component object from a YAML file, overriding its version to "1.0.2", andregistering it remotely.
from azure.ai.ml import load_component component = load_component( source="./sdk/ml/azure-ai-ml/tests/test_configs/components/helloworld_component.yml", params_override=[{"version": "1.0.2"}], ) registered_component = ml_client.components.create_or_update(component)Construct a compute object from a yaml file.
load_compute(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict[str, str]] | None = None, **kwargs: Any) -> Compute| Name | Description |
|---|---|
source Required | The local yaml source of a compute. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Optional parameters to override in the loaded yaml. Default value: None |
| Type | Description |
|---|---|
Loaded compute object. |
Loading a Compute object from a YAML file and overriding its description.
from azure.ai.ml import load_compute compute = load_compute( "../tests/test_configs/compute/compute-vm.yaml", params_override=[{"description": "loaded from compute-vm.yaml"}], )Construct a connection object from yaml file.
load_connection(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> WorkspaceConnection| Name | Description |
|---|---|
source Required | The local yaml source of a connection object. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
<xref:Connection> | Constructed connection object. |
Construct a data object from yaml file.
load_data(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> Data| Name | Description |
|---|---|
source Required | The local yaml source of a data object. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Constructed Data or DataImport object. |
| Type | Description |
|---|---|
Raised if Data cannot be successfully validated.Details will be provided in the error message. |
Construct a datastore object from a yaml file.
load_datastore(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> Datastore| Name | Description |
|---|---|
source Required | The local yaml source of a datastore. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Loaded datastore object. |
| Type | Description |
|---|---|
Raised if Datastore cannot be successfully validated.Details will be provided in the error message. |
Construct a environment object from yaml file.
load_environment(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> Environment| Name | Description |
|---|---|
source Required | The local yaml source of an environment. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Constructed environment object. |
| Type | Description |
|---|---|
Raised if Environment cannot be successfully validated.Details will be provided in the error message. |
Construct a FeatureSet object from yaml file.
load_feature_set(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> FeatureSet| Name | Description |
|---|---|
source Required | The local yaml source of a FeatureSet object. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Constructed FeatureSet object. |
| Type | Description |
|---|---|
Raised if FeatureSet cannot be successfully validated.Details will be provided in the error message. |
Load a feature store object from a yaml file.
load_feature_store(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> FeatureStore| Name | Description |
|---|---|
source Required | The local yaml source of a feature store. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Loaded feature store object. |
Construct a FeatureStoreEntity object from yaml file.
load_feature_store_entity(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> FeatureStoreEntity| Name | Description |
|---|---|
source Required | The local yaml source of a FeatureStoreEntity object. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Constructed FeatureStoreEntity object. |
| Type | Description |
|---|---|
Raised if FeatureStoreEntity cannot be successfully validated.Details will be provided in the error message. |
Note
This is an experimental method, and may change at any time. Please seehttps://aka.ms/azuremlexperimental for more information.
Constructs a Index object from a YAML file.
load_index(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> Index| Name | Description |
|---|---|
source Required | A path to a local YAML file or an already-open file object containing an index configuration.If the source is a path, it will be opened and read. If the source is an open file, the file will be readdirectly. |
| Name | Description |
|---|---|
relative_origin | The root directory for the YAML. This directory will be used as the origin for deducingthe relative locations of files referenced in the parsed YAML. Defaults to the same directory as source ifsource is a file or file path input. Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
A loaded Index object. |
| Type | Description |
|---|---|
Raised if Index cannot be successfully validated.Details will be provided in the error message. |
Constructs a Job object from a YAML file.
load_job(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> Job| Name | Description |
|---|---|
source Required | A path to a local YAML file or an already-open file object containing a job configuration.If the source is a path, it will be opened and read. If the source is an open file, the file will be readdirectly. |
| Name | Description |
|---|---|
relative_origin | The root directory for the YAML. This directory will be used as the origin for deducingthe relative locations of files referenced in the parsed YAML. Defaults to the same directory as source ifsource is a file or file path input. Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
A loaded Job object. |
| Type | Description |
|---|---|
Raised if Job cannot be successfully validated.Details will be provided in the error message. |
Loading a Job from a YAML config file.
from azure.ai.ml import load_job job = load_job(source="./sdk/ml/azure-ai-ml/tests/test_configs/command_job/command_job_test_local_env.yml")Note
This is an experimental method, and may change at any time. Please seehttps://aka.ms/azuremlexperimental for more information.
load_marketplace_subscription(source: str | PathLike | IO, *, relative_origin: str | None = None, **kwargs: Any) -> MarketplaceSubscription| Name | Description |
|---|---|
source Required |
| Name | Description |
|---|---|
relative_origin | Default value: None |
Constructs a Model object from a YAML file.
load_model(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> Model| Name | Description |
|---|---|
source Required | A path to a local YAML file or an already-open file object containing a job configuration.If the source is a path, it will be opened and read. If the source is an open file, the file will be readdirectly. |
| Name | Description |
|---|---|
relative_origin | The root directory for the YAML. This directory will be used as the origin for deducingthe relative locations of files referenced in the parsed YAML. Defaults to the same directory as source ifsource is a file or file path input. Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
A loaded Model object. |
| Type | Description |
|---|---|
Raised if Job cannot be successfully validated.Details will be provided in the error message. |
Loading a Model from a YAML config file, overriding the name and version parameters.
from azure.ai.ml import load_model model = load_model( source="./sdk/ml/azure-ai-ml/tests/test_configs/model/model_with_stage.yml", params_override=[{"name": "new_model_name"}, {"version": "1"}], )Note
This is an experimental method, and may change at any time. Please seehttps://aka.ms/azuremlexperimental for more information.
Constructs a ModelPackage object from a YAML file.
load_model_package(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> ModelPackage| Name | Description |
|---|---|
source Required | A path to a local YAML file or an already-open file object containing a job configuration.If the source is a path, it will be opened and read. If the source is an open file, the file will be readdirectly. |
| Name | Description |
|---|---|
relative_origin | The root directory for the YAML. This directory will be used as the origin for deducingthe relative locations of files referenced in the parsed YAML. Defaults to the same directory as source ifsource is a file or file path input. Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
A loaded ModelPackage object. |
| Type | Description |
|---|---|
Raised if Job cannot be successfully validated.Details will be provided in the error message. |
Loading a ModelPackage from a YAML config file.
from azure.ai.ml import load_model_package model_package = load_model_package( "./sdk/ml/azure-ai-ml/tests/test_configs/model_package/model_package_simple.yml" )Construct a online deployment object from yaml file.
load_online_deployment(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> OnlineDeployment| Name | Description |
|---|---|
source Required | The local yaml source of an online deployment object. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Constructed online deployment object. |
| Type | Description |
|---|---|
Raised if Online Deployment cannot be successfully validated.Details will be provided in the error message. |
Construct a online endpoint object from yaml file.
load_online_endpoint(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> OnlineEndpoint| Name | Description |
|---|---|
source Required | The local yaml source of an online endpoint object. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Constructed online endpoint object. |
| Type | Description |
|---|---|
Raised if Online Endpoint cannot be successfully validated.Details will be provided in the error message. |
Load a registry object from a yaml file.
load_registry(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> Registry| Name | Description |
|---|---|
source Required | The local yaml source of a registry. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Loaded registry object. |
Note
This is an experimental method, and may change at any time. Please seehttps://aka.ms/azuremlexperimental for more information.
load_serverless_endpoint(source: str | PathLike | IO, *, relative_origin: str | None = None, **kwargs: Any) -> ServerlessEndpoint| Name | Description |
|---|---|
source Required |
| Name | Description |
|---|---|
relative_origin | Default value: None |
Load a workspace object from a yaml file. This includes workspace sub-classeslike hubs and projects.
load_workspace(source: str | PathLike | IO, *, relative_origin: str | None = None, params_override: List[Dict] | None = None, **kwargs: Any) -> Workspace| Name | Description |
|---|---|
source Required | The local yaml source of a workspace. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
params_override | Fields to overwrite on top of the yaml file.Format is [{"field1": "value1"}, {"field2": "value2"}] Default value: None |
| Type | Description |
|---|---|
Loaded workspace object. |
Loading a Workspace from a YAML config file.
from azure.ai.ml import load_workspace ws = load_workspace( "../tests/test_configs/workspace/workspace_min.yaml", params_override=[{"description": "loaded from workspace_min.yaml"}], )Deprecated - use 'load_connection' instead. Construct a connection object from yaml file.
load_workspace_connection(source: str | PathLike | IO, *, relative_origin: str | None = None, **kwargs: Any) -> WorkspaceConnection| Name | Description |
|---|---|
source Required | The local yaml source of a connection object. Must be either apath to a local file, or an already-open file.If the source is a path, it will be open and read.An exception is raised if the file does not exist.If the source is an open file, the file will be read directly,and an exception is raised if the file is not readable. |
| Name | Description |
|---|---|
relative_origin | The origin to be used when deducingthe relative locations of files referenced in the parsed yaml.Defaults to the inputted source's directory if it is a file or file path input.Defaults to "./" if the source is a stream input with no name value. Default value: None |
| Type | Description |
|---|---|
<xref:Connection> | Constructed connection object. |
Creates a Spark object which can be used inside a dsl.pipeline function or used as a standalone Spark job.
spark(*, experiment_name: str | None = None, name: str | None = None, display_name: str | None = None, description: str | None = None, tags: Dict | None = None, code: PathLike | str | None = None, entry: Dict[str, str] | SparkJobEntry | None = None, py_files: List[str] | None = None, jars: List[str] | None = None, files: List[str] | None = None, archives: List[str] | None = None, identity: Dict[str, str] | ManagedIdentity | AmlToken | UserIdentity | None = None, driver_cores: int | None = None, driver_memory: str | None = None, executor_cores: int | None = None, executor_memory: str | None = None, executor_instances: int | None = None, dynamic_allocation_enabled: bool | None = None, dynamic_allocation_min_executors: int | None = None, dynamic_allocation_max_executors: int | None = None, conf: Dict[str, str] | None = None, environment: str | Environment | None = None, inputs: Dict | None = None, outputs: Dict | None = None, args: str | None = None, compute: str | None = None, resources: Dict | SparkResourceConfiguration | None = None, **kwargs: Any) -> Spark| Name | Description |
|---|---|
experiment_name | The name of the experiment the job will be created under. Default value: None |
name | The name of the job. Default value: None |
display_name | The job display name. Default value: None |
description | The description of the job. Defaults to None. Default value: None |
tags | The dictionary of tags for the job. Tags can be added, removed, and updated. Defaults to None. Default value: None |
code | The source code to run the job. Can be a local path or "http:", "https:", or "azureml:" urlpointing to a remote location. Default value: None |
entry | The file or class entry point. Default value: None |
py_files | The list of .zip, .egg or .py files to place on the PYTHONPATH for Python apps.Defaults to None. Default value: None |
jars | The list of .JAR files to include on the driver and executor classpaths. Defaults to None. Default value: None |
files | The list of files to be placed in the working directory of each executor. Defaults to None. Default value: None |
archives | The list of archives to be extracted into the working directory of each executor.Defaults to None. Default value: None |
identity | Optional[Union[dict[str,str],ManagedIdentityConfiguration,AmlTokenConfiguration,UserIdentityConfiguration]] The identity that the Spark job will use while running on compute. Default value: None |
driver_cores | The number of cores to use for the driver process, only in cluster mode. Default value: None |
driver_memory | The amount of memory to use for the driver process, formatted as strings with a size unitsuffix ("k", "m", "g" or "t") (e.g. "512m", "2g"). Default value: None |
executor_cores | The number of cores to use on each executor. Default value: None |
executor_memory | The amount of memory to use per executor process, formatted as strings with a size unitsuffix ("k", "m", "g" or "t") (e.g. "512m", "2g"). Default value: None |
executor_instances | The initial number of executors. Default value: None |
dynamic_allocation_enabled | Whether to use dynamic resource allocation, which scales the number ofexecutors registered with this application up and down based on the workload. Default value: None |
dynamic_allocation_min_executors | The lower bound for the number of executors if dynamic allocation isenabled. Default value: None |
dynamic_allocation_max_executors | The upper bound for the number of executors if dynamic allocation isenabled. Default value: None |
conf | A dictionary with pre-defined Spark configurations key and values. Defaults to None. Default value: None |
environment | The Azure ML environment to run the job in. Default value: None |
inputs | A mapping of input names to input data used in the job. Defaults to None. Default value: None |
outputs | A mapping of output names to output data used in the job. Defaults to None. Default value: None |
args | The arguments for the job. Default value: None |
compute | The compute resource the job runs on. Default value: None |
resources | The compute resource configuration for the job. Default value: None |
| Type | Description |
|---|---|
A Spark object. |
Building a Spark pipeline using the DSL pipeline decorator
from azure.ai.ml import Input, Output, dsl, spark from azure.ai.ml.constants import AssetTypes, InputOutputModes # define the spark task first_step = spark( code="/src", entry={"file": "add_greeting_column.py"}, py_files=["utils.zip"], files=["my_files.txt"], driver_cores=2, driver_memory="1g", executor_cores=1, executor_memory="1g", executor_instances=1, inputs=dict( file_input=Input(path="/dataset/iris.csv", type=AssetTypes.URI_FILE, mode=InputOutputModes.DIRECT) ), args="--file_input ${{inputs.file_input}}", resources={"instance_type": "standard_e4s_v3", "runtime_version": "3.4.0"}, ) second_step = spark( code="/src", entry={"file": "count_by_row.py"}, jars=["scala_project.jar"], files=["my_files.txt"], driver_cores=2, driver_memory="1g", executor_cores=1, executor_memory="1g", executor_instances=1, inputs=dict( file_input=Input(path="/dataset/iris.csv", type=AssetTypes.URI_FILE, mode=InputOutputModes.DIRECT) ), outputs=dict(output=Output(type="uri_folder", mode=InputOutputModes.DIRECT)), args="--file_input ${{inputs.file_input}} --output ${{outputs.output}}", resources={"instance_type": "standard_e4s_v3", "runtime_version": "3.4.0"}, ) # Define pipeline @dsl.pipeline(description="submit a pipeline with spark job") # type:ignore def spark_pipeline_from_builder(data): add_greeting_column = first_step(file_input=data) count_by_row = second_step(file_input=data) return {"output": count_by_row.outputs.output} # type:ignore pipeline = spark_pipeline_from_builder( data=Input(path="/dataset/iris.csv", type=AssetTypes.URI_FILE, mode=InputOutputModes.DIRECT), )Was this page helpful?
Need help with this topic?
Want to try using Ask Learn to clarify or guide you through this topic?
Was this page helpful?
Want to try using Ask Learn to clarify or guide you through this topic?