Repository files navigation

This package provides a development kit for creating custom data transformations inData Cloud. It allows you to write your own data processing logic in Python while leveraging Data Cloud's infrastructure for data access and running data transformations, mapping execution into Data Cloud data structures likeData Model Objects andData Lake Objects.

More specifically, this codebase gives you ability to test code locally before pushing to Data Cloud's remote execution engine, greatly reducing how long it takes to develop.

Use of this project with Salesforce is subject to theTERMS OF USE

• Python 3.11 only (currently supported version - if your system version is different, we recommend usingpyenv to configure 3.11)
• Azul Zulu OpenJDK 17.x
• Docker support likeDocker Desktop
• A salesforce org with some DLOs or DMOs with data and this feature enabled (it is not GA)
• Aconnected app

The SDK can be downloaded directly from PyPI withpip:

pip install salesforce-data-customcode

You can verify it was properly installed via CLI:

datacustomcode version

Ensure you have all theprerequisites prepared on your machine.

To get started, create a directory and initialize a new project with the CLI:

mkdir datacloud&&cd datacloudpython3.11 -m venv .venvsource .venv/bin/activatepip install salesforce-data-customcodedatacustomcode init my_package

This will yield all necessary files to get started:

.├── Dockerfile├── README.md├── requirements.txt├── requirements-dev.txt├── payload│   ├── config.json│   ├── entrypoint.py├── jupyterlab.sh└── requirements.txt

• Dockerfile(Do not update) – Development container emulating the remote execution environment.
• requirements-dev.txt(Do not update) – These are the dependencies for the development environment.
• jupyterlab.sh(Do not update) – Helper script for setting up Jupyter.
• requirements.txt – Here you define the requirements that you will need for your script.
• payload – This folder will be compressed and deployed to the remote execution environment.
  • config.json – This config defines permissions on the back and can be generated programmatically withscan CLI method.
  • entrypoint.py – The script that defines the data transformation logic.

A functional entrypoint.py is provided so you can run once you've configured your connected app:

cd my_packagedatacustomcode configuredatacustomcode run ./payload/entrypoint.py

After modifying theentrypoint.py as needed, using any dependencies you add in the.venv virtual environment, you can run this script in Data Cloud:

To Add New Dependencies:

1. Make sure your virtual environment is activated
2. Add dependencies torequirements.txt
3. Runpip install -r requirements.txt
4. The SDK automatically packages all dependencies when you rundatacustomcode zip

datacustomcode scan ./payload/entrypoint.pydatacustomcode deploy --path ./payload --name my_custom_script --cpu-size CPU_L

You can now use the Salesforce Data Cloud UI to find the created Data Transform and use theRun Now button to run it.Once the Data Transform run is successful, check the DLO your script is writing to and verify the correct records were added.

The SDK automatically handles all dependency packaging for Data Cloud deployment. Here's how it works:

1. Add dependencies torequirements.txt - List any Python packages your script needs
2. Install locally - Usepip install -r requirements.txt in your virtual environment
3. Automatic packaging - When you rundatacustomcode zip, the SDK automatically:
   • Packages all dependencies fromrequirements.txt
   • Uses the correct platform and architecture for Data Cloud

No need to worry about platform compatibility - the SDK handles this automatically through the Docker-based packaging process.

.├── payload│   ├── config.json│   ├── entrypoint.py├── files│   ├── data.csv

Your Python dependencies can be packaged as .py files, .zip archives (containing multiple .py files or a Python package structure), or .egg files.

.├── payload│   ├── config.json│   ├── entrypoint.py├── py-files│   ├── moduleA│   │   ├── __init__.py│   │   ├── moduleA.py

Your entry point script will define logic using theClient object which wraps data access layers.

You should only need the following methods:

• find_file_path(file_name) - Returns a file path
• read_dlo(name) – Read from a Data Lake Object by name
• read_dmo(name) – Read from a Data Model Object by name
• write_to_dlo(name, spark_dataframe, write_mode) – Write to a Data Model Object by name with a Spark dataframe
• write_to_dmo(name, spark_dataframe, write_mode) – Write to a Data Lake Object by name with a Spark dataframe

For example:

from datacustomcode import Clientclient = Client()sdf = client.read_dlo('my_DLO')# some transformations# ...client.write_to_dlo('output_DLO')

The Data Cloud Custom Code SDK provides a command-line interface (CLI) with the following commands:

• --debug: Enable debug-level logging

Display the current version of the package.

Configure credentials for connecting to Data Cloud.

Options:

• --profile TEXT: Credential profile name (default: "default")
• --username TEXT: Salesforce username
• --password TEXT: Salesforce password
• --client-id TEXT: Connected App Client ID
• --client-secret TEXT: Connected App Client Secret
• --login-url TEXT: Salesforce login URL
• --dataspace TEXT: Dataspace name (optional, for non-default dataspaces)

Initialize a new development environment with a template.

Argument:

• DIRECTORY: Directory to create project in (default: ".")

Scan a Python file to generate a Data Cloud configuration.

Argument:

• FILENAME: Python file to scan

Options:

• --config TEXT: Path to save the configuration file (default: same directory as FILENAME)
• --dry-run: Preview the configuration without saving to a file

Run an entrypoint file locally for testing.

Argument:

• ENTRYPOINT: Path to entrypoint Python file

Options:

• --config-file TEXT: Path to configuration file
• --dependencies TEXT: Additional dependencies (can be specified multiple times)
• --profile TEXT: Credential profile name (default: "default")

Zip a transformation job in preparation to upload to Data Cloud.

Options:

• --path TEXT: Path to the code directory (default: ".")
• --network TEXT: docker network (default: "default")

Deploy a transformation job to Data Cloud.

Options:

• --profile TEXT: Credential profile name (default: "default")
• --path TEXT: Path to the code directory (default: ".")
• --name TEXT: Name of the transformation job [required]
• --version TEXT: Version of the transformation job (default: "0.0.1")
• --description TEXT: Description of the transformation job (default: "")
• --network TEXT: docker network (default: "default")
• --cpu-size TEXT: CPU size for the deployment (default: "CPU_XL"). Available options: CPU_L(Large), CPU_XL(Extra Large), CPU_2XL(2X Large), CPU_4XL(4X Large)

The SDK provides Docker-based development options that allow you to test your code in an environment that closely resembles Data Cloud's execution environment.

When you initialize a project withdatacustomcode init my_package, aDockerfile is created automatically. This Dockerfile:

• Isn't used during local development with virtual environments
• Becomes active during packaging when you rundatacustomcode zip ordeploy
• Ensures compatibility by using the same base image as Data Cloud
• Handles dependencies automatically regardless of platform differences

Within yourinited package, you will find a.devcontainer folder which allows you to run a docker container while developing inside of it.

Read more about Dev Containers here:https://code.visualstudio.com/docs/devcontainers/containers.

1. Install the VS Code extension "Dev Containers" by microsoft.com.
2. Open your package folder in VS Code, ensuring that the.devcontainer folder isat the root of the File Explorer
3. Bring up the Command Palette (on mac: Cmd + Shift + P), and select "DevContainers: Rebuild and Reopen in Container"
4. Allow the docker image to be built, then you're ready to develop

Once inside the Dev Container:

• Terminal access: Open a terminal within the container
• Run your code: Executedatacustomcode run ./payload/entrypoint.py
• Environment consistency: Your code will run inside a docker container that more closely resembles Data Cloud compute than your machine

Within yourinited package, you will find ajupyterlab.sh file that can open a jupyter notebook for you. Jupyter notebooks, incombination with Data Cloud'sQuery EditorandData Explorer, can be extremely helpful for dataexploration. Instead of running an entire script, one can run one code cell at a time as they discover and experiment with the DLO or DMO data.

You can read more about Jupyter Notebooks here:https://jupyter.org/

1. Within the root project of your package folder, run./jupyterlab.sh start
2. Double-click on "account.ipynb" file, which provides a starting point for a notebook
3. Use shift+enter to execute each cell within the notebook. Add/edit/delete cells of code as needed for your data exploration.
4. Don't forget to run./jupyterlab.sh stop to stop the docker container

1. Log in to salesforce as an admin. In the top right corner, click on the gear icon and go toSetup
2. In the left hand side, search for "App Manager" and select theApp Manager underneathApps
3. Click onNew Connected App in the upper right
4. Fill in the required fields within theBasic Information section
5. Under theAPI (Enable OAuth Settings) section:
   1. Click on the checkbox to Enable OAuth Settings.
   2. Provide a callback URL likehttp://localhost:55555/callback
   3. In the Selected OAuth Scopes, make sure thatrefresh_token,api,cdp_query_api,cdp_profile_api is selected.
   4. Click on Save to save the connected app
6. From the detail page that opens up afterwards, click the "Manage Consumer Details" button to find your client id and client secret
7. Go back toSetup, thenOAuth and OpenID Connect Settings, and enable the "Allow OAuth Username-Password Flows" option

You now have all fields necessary for thedatacustomcode configure command.

If you're working with a non-default dataspace in Salesforce Data Cloud, you can specify the dataspace during configuration:

datacustomcode configure --dataspace my-dataspace

For default dataspaces, you can omit the--dataspace parameter entirely - the SDK will connect to the default dataspace automatically.

• Troubleshooting
• For Contributors