Cloud Storage
The Google Cloud Storage connector lets you connect to a Google Cloud Storage and perform file transfer operations.
Before you begin
Before using the Cloud Storage connector, do the following tasks:
- In your Google Cloud project:
- Ensure that network connectivity is set up. For information about network patterns, seeNetwork connectivity.
- Grant theroles/connectors.admin IAM role to the user configuring the connector.
- Grant the following IAM roles to the service account that you want to use for the connector:
roles/secretmanager.viewerroles/secretmanager.secretAccessorroles/storage.admin
A service account is a special type of Google account intended to represent a non-human user that needs to authenticate and be authorized to access data in Google APIs. If you don't have a service account, you must create a service account. The connector and the service account must belong to the same project. For more information, seeCreating a service account.
- Enable the following services:
secretmanager.googleapis.com(Secret Manager API)connectors.googleapis.com(Connectors API)
To understand how to enable services, seeEnabling services.
If these services or permissions have not been enabled for your project previously, you are prompted to enable them when configuring the connector.
Configure the connector
A connection is specific to a data source. Itmeans that if you have many data sources, you must create a separate connectionfor each data source. To create a connection, do the following:
- In theCloud console, go to theIntegration Connectors > Connections page and then select or create a Google Cloud project.
- Click+ CREATE NEW to open theCreate Connection page.
- In theLocation section, choose the location for the connection.
- Region: Select a location from the drop-down list.
For the list of all the supported regions, seeLocations.
- ClickNEXT.
- Region: Select a location from the drop-down list.
- In theConnection Details section, complete the following:
- Connector: SelectCloud Storage from the drop down list of available Connectors.
- Connector version: Select the Connector version from the drop down list of available versions.
- In theConnection Name field, enter a name for the Connection instance.
Connection names must meet the following criteria:
- Connection names can use letters, numbers, or hyphens.
- Letters must be lower-case.
- Connection names must begin with a letter and end with a letter or number.
- Connection names cannot exceed 49 characters.
- Optionally, enter aDescription for the connection instance.
- Optionally, enableCloud logging, and then select a log level. By default, the log level is set to
Error. - Service Account: Select a service account that has therequired roles.
- Optionally, configure theConnection node settings:
- Minimum number of nodes: Enter the minimum number of connection nodes.
- Maximum number of nodes: Enter the maximum number of connection nodes.
A node is a unit (or replica) of a connection that processes transactions. More nodes are required to process more transactions for a connection and conversely, fewer nodes are required to process fewer transactions. To understand how the nodes affect your connector pricing, see Pricing for connection nodes. If you don't enter any values, by default the minimum nodes are set to 2 (for better availability) and the maximum nodes are set to 50.
Note: You can customize the connection node values only if you are a Pay-as-you-go customer. - Project ID: The ID of the Google Cloud project where the data resides.
- Optionally, click+ ADD LABEL to add a label to the Connection in the form of a key/value pair.
- ClickNEXT.
- Review: Review your connection.
- ClickCreate.
Entities, operations, and actions
All the Integration Connectors provide a layer of abstraction for the objects ofthe connected application. You can access an application's objects only through this abstraction. The abstraction is exposed to you as entities, operations, and actions.
Note: You can view the entities, operations, and actions of a connector in theConnectors task.- Entity:An entity can be thought of as an object, or a collection of properties, in theconnected application or service. The definition of an entity differs from a connector to a connector. For example, in a database connector, tables are the entities, in a file server connector, folders are the entities, and in a messaging system connector, queues are the entities.
However, it is possible that a connector doesn't support or have any entities, in which case the
Entitieslist will be empty. - Operation:An operation is the activity that you can perform on an entity. You can performany of the following operations on an entity:
Selecting an entity from the available list, generates a list ofoperations available for the entity. For a detailed description of the operations, see the Connectors task'sentity operations. However, if a connector doesn't support any of the entity operations, such unsupported operations aren't listed in the
Operationslist. - Action:An action is a first class function that is made available to the integrationthrough the connector interface. An action lets you make changes to an entity or entities, and vary from connector to connector. Normally, an action will have some input parameters, and an output parameter. However, it is possible that a connector doesn't support any action, in which case the
Actionslist will be empty.
System limitations
The Google Cloud Storage connector can process a maximum of 10 transactions per second, pernode, andthrottles any transactions beyond this limit. By default, Integration Connectors allocates 2 nodes (for better availability) for a connection.
For information on the limits applicable to Integration Connectors, seeLimits.
Note: The number ofIntegration Connectors nodes will autoscale dynamically based on your usage. However, if you want to reserve capacity for large volumes without waiting for autoscaling, you can adjust the minimum node value for a connection. More nodes are required to process more transactions for a connection. Conversely, fewer nodes are required if a connection processes fewer transactions. To configure the node values, do the following:- If you are a pay-as-you-go customer, configure the minimum and maximum node value in the edit connection page.
- If you are a subscription based customer,contact support.
The maximum transactions that a node can handle depends on various factors. So, before adjusting the minimum nodes for better throughput, it is recommended you check if your backend systems are set up optimally to handle the required traffic.
Actions
The Google Cloud Storage connection supports the following actions:
Note: The results of all the entity operations and actions will be available as a JSON response in theConnectors task'sconnectorOutputPayload response parameter after you run your integration.DownloadObject action
The following table describes the input parameters of theDownloadObject action.
| Parameter name | Required | Data type | Description |
|---|---|---|---|
| Bucket | Yes | String | Bucket name where the object to be downloaded is present. |
| ObjectFilePath | No | String | Name of the object that should be downloaded. If not specified, all the objects from the specified bucket will be downloaded. If the object to download is present in a child folder of a bucket, you must provide the full path of such object. For example, to download |
| HasBytes | No | Boolean | Whether to download content as bytes. The valid values aretrue orfalse. If set totrue, the content is downloaded as aBase64 encoded string.By default, the |
| UpdatedEndDate | No | Date | The end date range to download objects. If not specified, objects will be downloaded from the specifiedUpdatedStartDate until the present day. |
| UpdatedStartDate | No | Date | The start of the date range to download objects. If not specified, objects will be downloaded from the beginning of the time until theUpdatedEndDate. |
connectorOutputPayload response parameter after you run your integration.For examples on how to configure theDownloadObject action, seeExamples.
UploadObject action
The following table describes the input parameters of theUploadObject action.
| Parameter name | Required | Data type | Description |
|---|---|---|---|
| Bucket | Yes | String | Bucket name where the object will be uploaded. |
| FolderPath | No | String | The path to the folder where the object should be uploaded. |
| ContentBytes | No | String | Content to upload in the form of bytes (Base64 encoded string). |
| HasBytes | No | Boolean | Whether to upload content as bytes. Valid values;true orfalse. If set totrue, the content you want to upload should be a Base64 encoded string.By default, the |
| Content | Yes | String | The content to be uploaded. |
| ObjectName | No | String | Name of the object that will be uploaded. |
For examples on how to configure theUploadObject action, seeExamples.
CopyObject action
The following table describes the input parameters of theCopyObject action.
| Parameter name | Required | Data type | Description |
|---|---|---|---|
| BucketSource | Yes | String | Bucket name from which you want to copy the object. |
| ObjectSource | Yes | String | Full path of the folder where you want to copy the object. |
| BucketDestination | Yes | String | Bucket name to which you want to copy the object. |
| ObjectDestination | No | String | Full path of the destination including the object name. If you don't specify any object name, the name of the source object is retained. |
For examples on how to configure theCopyObject action, seeExamples.
MoveObject action
The following table describes the input parameters of the MoveObject action.
| Parameter name | Required | Data type | Description |
|---|---|---|---|
| BucketSource | Yes | String | Bucket name from which you want to move the object. |
| ObjectSource | Yes | String | Full path of the folder where you want to move the object. |
| BucketDestination | Yes | String | Bucket name to which you want to move the object. |
| ObjectDestination | No | String | Full path of the destination including the object name. If you don't specify any object name, the name of the source object is retained. |
DeleteObject action
The following table describes the input parameters of theDeleteObject action.
| Parameter name | Required | Data type | Description |
|---|---|---|---|
| BucketSource | Yes | String | Bucket name where the object to be deleted is present. |
| ObjectSource | Yes | String | Name of the object that you want to delete. |
| Generation | No | Double | Version of the object to delete. If present, deletes the specified revision of the object as opposed to the latest version, which is the default behaviour. |
| IfGenerationMatch | No | Double | Makes the delete operation conditional on whether the object's current generation matches the given value. Setting this value to |
| IfGenerationNotMatch | No | Double | Makes the delete operation conditional on whether the object's current generation does not match the given value. If no live object exists, the precondition fails. Setting this value to0 makes the operation succeed only if there is a live version of the object. |
| IfMetagenerationMatch | No | Double | Makes the delete operation conditional on whether the object's current metageneration matches the specified value. |
| IfMetagenerationNotMatch | No | Double | Makes the delete operation conditional on whether the object's current metageneration does not match the specified value. |
SignURL action
The following table describes the input parameters of theSignURL action which creates a signed URL for the specified object.
| Parameter name | Required | Data type | Description |
|---|---|---|---|
| Bucket | Yes | String | The bucket name where the object resides. |
| Object | Yes | String | The name of the object to generate the SignedURL for. |
| RequestMethod | No | String | The method that the signed request will use. The default value isGET. |
| Location | No | String | Location of the specified bucket. The default value isauto. |
| ActiveDateTime | No | String | The dateTime when the SignedURL will become active. If not specified the current dateTime will be used. |
| Query | No | String | The query string that must be included when using the SignedURL, if not specified no query string will be used. |
| CustomHeaders | No | String | A comma separated list of name=value of the headers to use with the SignedURL, if not specified not custom headers will be used. |
| ExpiresIn | Yes | String | The expiration time for the SignedURL it should be in the format:1d2h3m4s, the maximum value is7d0h0m0s. |
| HMAC Access ID | Yes | String | The HMAC access ID. For information, seeHMAC keys. |
| HMAC Secret | Yes | String | The HMAC secret. |
Examples
This examples in this section describe the following operations:
- List all objects
- List all objects in a bucket
- List objects using the LIKE filter for name
- List all buckets
- Download an object
- Download a binary object
- Upload a binary object to a bucket
- Upload an object to a bucket
- Upload an object to a folder
- Copy an object
- Move an object
- Delete an object
- Create a signed URL for an object
The following table lists the sample scenarios and the corresponding configuration in the Connectors task:
| Task | Configuration |
|---|---|
| List all objects |
This lists all the objects in all the buckets. The objects are listed in the |
| List all objects in a bucket |
|
| List objects using the LIKE filter for name |
|
| List all buckets |
|
| Download an object |
This example downloads the |
| Download a binary object | The steps to download a binary object are the same as downloading a regular object as previously described. Additionally, you must specify the {"Bucket":"bucket-test-01","ObjectFilePath":"image01.png","HasBytes":true} If the download is successful, the output in the {"Success":"true","ContentBytes":"SGVsbG8gdGVzdCE\u003d"} By default, the If the file, contains special characters, such as ä, Ø, Thành, do the following:
|
| Upload a binary object to a bucket |
This example creates the |
| Upload an object to a bucket |
This example creates the |
| Upload an object to a folder |
This example creates the |
| Copy an object |
This example copies the If the copy is successful, the output in the {"Success":"true"} |
| Move an object |
This example moves the If the copy is successful, the output in the {"Success":"true"} |
| Delete an object |
This example deletes the If the copy is successful, the output in the {"Success":"true"} |
| Create a signed URL for an object |
This example creates a signed URL for the {"Success": "true","SignURL": "https://storage.googleapis.com/example-bucket/cat.jpeg?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=example%40example-project.iam.gserviceaccount.com%2F20181026%2Fus-central1%2Fstorage%2Fgoog4_request&X-Goog-Date=20181026T181309Z&X-Goog-Expires=900&X-Goog-SignedHeaders=host&X-Goog-Signature=247a2aa45f169edf4d187d54e7cc46e4731b1e6273242c4f4c39a1d2507a0e58706e25e3a85a7dbb891d62afa8496def8e260c1db863d9ace85ff0a184b894b117fe46d1225c82f2aa19efd52cf21d3e2022b3b868dcc1aca2741951ed5bf3bb25a34f5e9316a2841e8ff4c530b22ceaa1c5ce09c7cbb5732631510c20580e61723f5594de3aea497f195456a2ff2bdd0d13bad47289d8611b6f9cfeef0c46c91a455b94e90a66924f722292d21e24d31dcfb38ce0c0f353ffa5a9756fc2a9f2b40bc2113206a81e324fc4fd6823a29163fa845c8ae7eca1fcf6e5bb48b3200983c56c5ca81fffb151cca7402beddfc4a76b133447032ea7abedc098d2eb14a7"} |
Considerations
- A downloadable object can have a maximum size of 10 MB.
- You can't upload multiple files using the
UploadObjectaction. You can upload only a single file.
Create connections using Terraform
You can use theTerraformresource to create a new connection.
To learn how to apply or remove a Terraform configuration, seeBasic Terraform commands.
To view a sample terraform template for connection creation, seesample template.
When creating this connection by using Terraform, you must set the following variables in your Terraform configuration file:
| Parameter name | Data type | Required | Description |
|---|---|---|---|
| project_id | STRING | True | The ID of the Google Cloud project where the data resides. |
Use the Cloud Storage connection in an integration
After you create the connection, it becomes available in bothApigee Integration and Application Integration. You can use the connectionin an integration through the Connectors task.
- To understand how to create and use the Connectors task in Apigee Integration, seeConnectors task.
- To understand how to create and use the Connectors task in Application Integration, seeConnectors task.
Get help from the Google Cloud community
You can post your questions and discuss this connector in the Google Cloudcommunity atCloud Forums.What's next
- Understand how tosuspend and resume a connection.
- Understand how tomonitor connector usage.
- Understand how toview connector logs.
Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2026-02-18 UTC.