Posted onApr 21, 2023 • Edited onApr 25, 2023

Part 3. Data reference and resource in plugins

In the third part of the tutorial, we will learn how to use data references, how to use resources and how to connect to external services. We will extend our plugin with the following functionality.

The plugin needs to send the user defined data to be sent to user defined API. We will also have to extend our form with the above data (data, and API endpoint).

Let's get started

Often when coding a plugin, we want to use the data defined by the user. For example, the user references some data from the internal state of the workflow that he wants to an external API. To do this he uses so-called dot notation. Dot notation is a way of specifying the location of the data. It looks like this.

<source>@<path.to.data>

This is a working example:

event@properties.email

It means get the data from the event from itsproperties.email. A full description of all sources and how the data reference works can be foundhere.

In this tutorial we are interested in how to use this entry and retrieve data, and how to put a field in the form that will require a dot notation entry.

Data reference

Let's start with the form. To add a new field to the form we will use thedotPath component and in theform / FromGroup / FormFields section we will add the following code:

FormField(id="data",name="Data to send",description="Please provide data to send",component=FormComponent(type="dotPath",props={"label":"Data to send"}))

This way we get:

form=Form(groups=[FormGroup(name="Event type plugin configuration",description="Define required event type",fields=[FormField(id="event_type",name="Event type",description="Event type to check",component=FormComponent(type="text",props={"label":"Event type"})),FormField(id="data",name="Data to send",description="Please provide data to send",component=FormComponent(type="dotPath",props={"label":"Data to send"}))]),]),

With this, we will get a field of this type in the form.

Then we need to extendinit in register function with adata field and extend an object that will store the data and verify its correctness at the same time.

So I would add to init:

init={"event_type":"","data":""}

and add to theConfiguration object:

classConfiguration(PluginConfig):event_type:strdata:str@validator("event_type")defmust_not_be_empty(cls,value):iflen(value)==0:raiseValueError("Event type can not be empty.")returnvalue@validator("data")# (1)defdata_must_not_be_empty(cls,value):iflen(value)==0:raiseValueError("Data can not be empty.")returnvalue

Validates the data property. It is defined as string, but itmay not be empty string

We have configuration data, now it's time to read data entered in the form and read data fromdot notation.

We do it as follows:

In the run method:

asyncdefrun(self,payload:dict,in_edge=None):dot=self._get_dot_accessor(payload)# (1)data_to_send=dot[self.config.data]# (2)ifself.event.type==self.config.event_type:returnResult(port="MyEvent",value=data_to_send)else:returnResult(port="NotMyEvent",value={})

Get the DotAccessor object that will convert the dot notation to the data.
Convert anything that is defined inconfig.data to the real data from the workflow and assign it to data_to_send

Resource

In many cases, it is necessary to link the plug-in to some resource. It can be e.g. a database, API, etc. Usually, such resources need login credentials. We don't want to keep them in a plugin as they can be used in many plugins. For this purpose, it is necessary to create a resource. The resource consists of credential data and sometimes a URL.

For example, let's add a functionality to our plugin that will send the data pointed by the user to the defined API (a resource).

For this purpose, we will create a resource that will have the following data

{"url":"api-url","method":"api-method","api_key":"api-key"}

The resources are created in thetracardi/service/setup/setup_resources.py directory

We add the following code to theget_resource_types() function

ResourceSettings(id="my-api-url",# (1)name="Custom API URL",# (2)icon="web",# (3)tags=["api"],# (4)config={# (5)"url":"<api-url>","method":"<api-method>","api_key":"<api-key>"},manual="custom_api_url_resource"# (6)),

Resource Id. A unique string that identifies the resource.
Resource name
Resource icon.
The tag that describes the resource. It is used to filter the resources in the plugin form.
Resource configuration*.md file namecontaining documentation on how to get authorization data, e.g. when the resource is an external system.
The documentation file is located in thetracardi-api/docs/resources directory

This is the set of data the system needs to add a resource. After restarting the server, we can click onResources in the Tracardi GUI menu, thenAdd new resource and the list of resources will includeCustom API URL. The resource we just created. The form will also include an object:

{"url":"<api-url>","method":"<api-method>","api_key":"<api-key>"}

which we defined in the config property and which should be filled by the user.

Loading resource

Once we have a resource, we can load it in our plugin. We do this in theset_up method. We add the following two lines
to theset_up method

resource=awaitstorage.driver.resource.load(config.resource.id)# (1)self.credentials=resource.credentials.get_credentials(self,output=MyResourceConfig)# (2)

Read resource id fromconfig.resource.id. Note the objectconfig.resource must be defined in the config
Get data for authorization (credentials), which is an object defined inResourceSettings in theconfig property, and filled in by the user when creating the resource.

The first line reads the resource id fromconfig.resource.id. Note that we do not have such field in the config yet and we need to add it (we'll do it in a moment)

The second line readscredentials from the resource, i.e. the object that we defined in theconfig property of theResourceSettings object above and a user filled when creating the resource.

The wholeset_up method should look like this:

fromtracardi.service.storage.driverimportstorageclassMyPlugin(ActionRunner):config:dictcredentials:MyResourceConfig# (1)asyncdefset_up(self,config):self.config=configresource=awaitstorage.driver.resource.load(config.resource.id)self.credentials=resource.credentials.get_credentials(self,output=MyResourceConfig)# (2)

We have added the credentials' property of typeMyResourceConfig. We'll have to define it yet.
Rest of plugin code.

Validating the resource

We still have some parts missing. We need to validate the data from the resource and extend the configuration so that the user can select the appropriate resource in the form.

To verify the data from the resource and make it readable, we need to create theMyResourceConfig object. I named it in this line:

self.credentials=resource.credentials.get_credentials(self,output=MyResourceConfig)# (1)

Seeoutput = MyResourceConfig

Inoutput=MyResourceConfig, I requested that the credentials loaded from the resource be in the form of aMyResourceConfig object. I don't have it yet, so let's create it. We usually store resource objects in the directory:tracardi/domain/resources

File: tracardi/domain/resources/my_resource_config.py

```pythonfrom pydantic import BaseModel, AnyHttpUrlfrom typing import Optional class MyResourceConfig(BaseModel):    url: AnyHttpUrl    method: str    api_key: Optional[str] = None```

Tip: It must have the same schema as defined in config inResourceSettings. Note that I definedurl asAnyHttpUrl this means it can only accept a string that looks like a URL. On the other hand,api_key is of typeOptional[str], which means that such a property may not be available or be None.

We already have an object, so we have to extend the plug-in configuration with the resource. The resource will be selected from the list of available resources, and it will be identified as the resourcename and id.

Example

{"id":"9bb2a926-b6ae-4cad-9b3c-9380ea7bfede","name":"My API"}

For this purpose, we will add a resource to the plug-in configuration, which, let me remind you, is in theregister function under theinit property.

It should look like this:

defregister()->Plugin:returnPlugin(start=False,spec=Spec(module=__name__,className=MyPlugin.__name__,init={"resource":{"id":"","name":""},"event_type":"","data":""},inputs=["payload"],outputs=["MyEvent","NotMyEvent"],version='0.1',license="MIT",author="Your Name"),metadata=MetaData(name="My first plugin",desc='Checks if the event type is equal to my-event.',group=["Test plugin"]))

We also need to extend the configuration validation object.

fromtracardi.domain.named_entityimportNamedEntityclassConfiguration(PluginConfig):resource:NamedEntity# (1)event_type:strdata:str@validator("event_type")defmust_not_be_empty(cls,value):iflen(value)==0:raiseValueError("Event type can not be empty.")returnvalue@validator("data")defdata_must_not_be_empty(cls,value):iflen(value)==0:raiseValueError("Data can not be empty.")returnvalue

NamedEntity is an object containingid and name. It is already defined in Tracardi.

Resource select field

It remains to add a field to the form that will allow user to select a resource from the list of defined resources. Remember that there may be different types of resources created in the system, so we need to filter them so that only those related to our plugin are on the list of resources.

This is done by pointing to the tag we defined inResourceSettings. In our case it isapi(tags=["api"]).

Now we have all the information. We must extend the form with theresource select field that displays available resources, so the user will be able to select one.

Please add this code toform in theregister function.

FormField(id="resource",name="Resource",description="Select your API resource.",component=FormComponent(type="resource",props={"label":"API Resource","tag":"api"})# (1)),

Notice tag property equal toapi

Theform property should look like this.

form=Form(groups=[FormGroup(name="Event type plugin configuration",description="Define required event type",fields=[FormField(id="resource",name="Resource",description="Select your API resource.",component=FormComponent(type="resource",props={"label":"API Resource","tag":"api"})),FormField(id="event_type",name="Event type",description="Event type to check",component=FormComponent(type="text",props={"label":"Event type"})#)])])