argo-python-dsl  

Python DSL forArgo Workflows

If you're new to Argo, we recommend checking out the examples in pure YAML. The language is descriptive and the Argoexamples provide an exhaustive explanation.

For a more experienced audience, this DSL grants you the ability to programatically define Argo Workflows in Python which is then translated to the Argo YAML specification.

The DSL makes use of the Argo models defined in theArgo Python client repository. Combining the two approaches we are given the whole low-level control over Argo Workflows.

Installation

pip install argo-workflows-dsl

Getting Started

Hello World

This example demonstrates the simplest functionality. Defining aWorkflow by subclassing theWorkflow class and a single template with the@template decorator.

The entrypoint to the workflow is defined as anentrypoint class property.

DAG: Tasks

This example demonstrates tasks defined via dependencies forming adiamond structure. Tasks are defined using the@task decorator and theymust return a valid template.

The entrypoint is automatically created asmain for the top-level tasks of theWorkflow.

Artifacts

Artifacts can be passed similarly toparameters in three forms:arguments,inputs andoutputs, wherearguments is the default one (simply@artifact or@parameter).

I.e.:inputs.artifact(...)

Both artifacts and parameters are passedone by one, which means that for multiple artifacts (parameters), one should call:

@inputs.artifact(name="artifact", ...)@inputs.parameter(name="parameter_a", ...)@inputs.parameter(...)deffoo(self,artifact:V1alpha1Artifact,prameter_b:V1alpha1Parameter, ...):pass

A complete example:

Going further:closure andscope

This is where it gets quite interesting. So far, we've only scratched the benefits that the Python implementation provides.

What if we want to use native Python code and execute it as a step in the Workflow. What are our options?

Option A) is to reuse the existing mindset, dump the code in a string, pass it as the source to theV1ScriptTemplate model and wrap it with thetemplate decorator.This is illustrated in the following code block:

importtextwrapclassScriptsPython(Workflow):    ...@templatedefgen_random_int(self)->V1alpha1ScriptTemplate:source=textwrap.dedent("""\          import random          i = random.randint(1, 100)          print(i)        """)template=V1alpha1ScriptTemplate(image="python:alpine3.6",name="gen-random-int",command=["python"],source=source        )returntemplate

api_version:argoproj.io/v1alpha1kind:Workflowmetadata:generate_name:scripts-python-name:scripts-pythonspec:entrypoint:main...templates:  -name:gen-random-intscript:command:      -pythonimage:python:alpine3.6name:gen-random-intsource:'import random\ni = random.randint(1, 100)\nprint(i)\n'

Not bad, but also not living up to the full potential. Since we're already writing Python, why would we wrap the code in a string? This is where we introduceclosures.

The logic ofclosures is quite simple. Just wrap the function you want to execute in a container in the@closure decorator. Theclosure then takes care of the rest and returns atemplate (just as the@template decorator).

The only thing we need to take care of is to provide it an image which has the necessary Python dependencies installed and is present in the cluster.

There is a plan to eliminate even this step in the future, but currently it is inavoidable.

Following the previous example:

classScriptsPython(Workflow):    ...@closure(image="python:alpine3.6"    )defgen_random_int()->V1alpha1ScriptTemplate:importrandomi=random.randint(1,100)print(i)

The closure implements theV1alpha1ScriptTemplate, which means that you can pass in things likeresources,env, etc...

Also, make sure that youimport whatever library you are using, the context is not preserved ---closure behaves as a staticmethod and issandboxed from the module scope.

Now, what if we had a function (or a whole script) which is quite big. Wrapping it in a single Python function is not very Pythonic and it gets tedious. This is where we can make use ofscopes.

Say that we, for example, wanted to initialize logging before running ourgen_random_int function.

    ...@closure(scope="main",image="python:alpine3.6"    )defgen_random_int(main)->V1alpha1ScriptTemplate:importrandommain.init_logging()i=random.randint(1,100)print(i)@scope(name="main")definit_logging(level="DEBUG"):importlogginglogging_level=getattr(logging,level,"INFO")logging.getLogger("__main__").setLevel(logging_level)

@closure(scope="main",# <--- provide the closure a scopeimage="python:alpine3.6"    )defgen_random_int(main):# <--- use the scope name

@scope(name="main")# <--- add function to a scopedefinit_logging(level="DEBUG"):

Each function in the given scope is then namespaced by the scope name and injected to the closure.

I.e. the resulting YAML looks like this:

...spec:...templates:    -name:gen-random-intscript:command:        -pythonimage:python:alpine3.6name:gen-random-intsource:|-          import logging          import random          class main:            """Scoped objects injected from scope 'main'."""            @staticmethod            def init_logging(level="DEBUG"):              logging_level = getattr(logging, level, "INFO")              logging.getLogger("__main__").setLevel(logging_level)          main.init_logging()          i = random.randint(1, 100)          print(i)

Submitting with dsl

Workflow

fromargo.workflows.clientimport (ApiClient,Configuration,WorkflowServiceApi,V1alpha1WorkflowCreateRequest)fromargo.workflows.dslimportWorkflowfromargo.workflows.dsl.tasksimport*fromargo.workflows.dsl.templatesimport*classDagDiamond(Workflow):@task@parameter(name="message",value="A")defA(self,message:V1alpha1Parameter)->V1alpha1Template:returnself.echo(message=message)@task@parameter(name="message",value="B")@dependencies(["A"])defB(self,message:V1alpha1Parameter)->V1alpha1Template:returnself.echo(message=message)@task@parameter(name="message",value="C")@dependencies(["A"])defC(self,message:V1alpha1Parameter)->V1alpha1Template:returnself.echo(message=message)@task@parameter(name="message",value="D")@dependencies(["B","C"])defD(self,message:V1alpha1Parameter)->V1alpha1Template:returnself.echo(message=message)@template@inputs.parameter(name="message")defecho(self,message:V1alpha1Parameter)->V1Container:container=V1Container(image="alpine:3.7",name="echo",command=["echo","{{inputs.parameters.message}}"],        )returncontainerif__name__=="__main__":wf=DagDiamond()config=Configuration(host="http://localhost:2746")client=ApiClient(configuration=config)wf.submit(client,'argo')

CronWorkflow

fromargo.workflows.clientimportConfiguration,ApiClientfromargo.workflows.dslimporttemplatefromargo.workflows.dslimportCronWorkflowfromargo.workflows.dsl.templatesimportV1ContainerclassHelloCron(CronWorkflow):entrypoint="whalesay"schedule="0 0 1 1 *"@templatedefwhalesay(self)->V1Container:container=V1Container(image="docker/whalesay:latest",name="whalesay",command=["cowsay"],args=["hello world"],        )returncontainerif__name__=="__main__":wf=HelloCron()print(wf)config=Configuration(host="http://localhost:2746")client=ApiClient(configuration=config)wf.submit(client,"argo")