gcloud beta datastream streams create - creates a Datastream stream

gcloud beta datastream streams create(STREAM :--location=LOCATION)--display-name=DISPLAY_NAME(--backfill-none    |--backfill-all--mongodb-excluded-objects=MONGODB_EXCLUDED_OBJECTS    |--mysql-excluded-objects=MYSQL_EXCLUDED_OBJECTS    |--oracle-excluded-objects=ORACLE_EXCLUDED_OBJECTS    |--postgresql-excluded-objects=POSTGRESQL_EXCLUDED_OBJECTS    |--salesforce-excluded-objects=SALESFORCE_EXCLUDED_OBJECTS    |--spanner-excluded-objects=SPANNER_EXCLUDED_OBJECTS    |--sqlserver-excluded-objects=SQLSERVER_EXCLUDED_OBJECTS)(--destination-name=DESTINATION_NAME (--bigquery-destination-config=BIGQUERY_DESTINATION_CONFIG |--gcs-destination-config=GCS_DESTINATION_CONFIG))(--source-name=SOURCE_NAME (--mongodb-source-config=MONGODB_SOURCE_CONFIG |--mysql-source-config=MYSQL_SOURCE_CONFIG |--oracle-source-config=ORACLE_SOURCE_CONFIG |--postgresql-source-config=POSTGRESQL_SOURCE_CONFIG |--salesforce-source-config=SALESFORCE_SOURCE_CONFIG |--spanner-source-config=SPANNER_SOURCE_CONFIG |--sqlserver-source-config=SQLSERVER_SOURCE_CONFIG))[--labels=[KEY=VALUE,…]][--rule-sets=RULE_SETS][--force    |--validate-only][GCLOUD_WIDE_FLAG …]

Create a Datastream stream. If successful, the response body contains a newlycreated instance of Operation. To get the operation result, call: describeOPERATION

gcloudbetadatastreamstreamscreateSTREAM--location=us-central1--display-name=my-stream--source-name=source--oracle-source-config=source_config.json--destination-name=destination--gcs-destination-config=destination_config.json--backfill-none

To create a stream with a MySQL source and a Cloud Storage destination and thatexcludes some objects from being backfilled:

gcloudbetadatastreamstreamscreateSTREAM--location=us-central1--display-name=my-stream--source-name=source--mysql-source-config=source_config.json--destination-name=destination--gcs-destination-config=destination_config.json--backfill-all--mysql-excluded-objects=excluded_objects.json

Stream resource - The stream to create. The arguments in this group can be usedto specify the attributes of this resource. (NOTE) Some attributes are not givenarguments in this group but can be set in other ways.

To set theproject attribute:

• provide the argumentstream on the command line with a fullyspecified name;
• provide the argument--project on the command line;
• set the propertycore/project.

This must be specified.

STREAM

ID of the stream or fully qualified identifier for the stream.

To set thestream attribute:

• provide the argumentstream on the command line.

This positional argument must be specified if any of the other arguments in thisgroup are specified.

--location=LOCATION

The Cloud location for the stream.

To set thelocation attribute:

• provide the argumentstream on the command line with a fullyspecified name;
• provide the argument--location on the command line.

--display-name=DISPLAY_NAME

Friendly name for the stream.

Exactly one of these must be specified:

--backfill-none

Do not automatically backfill any objects. This flag is equivalent to selectingthe Manual backfill type in the Google Cloud console.

Or at least one of these can be specified:

--backfill-all

Automatically backfill objects included in the stream source configuration.Specific objects can be excluded. This flag is equivalent to selecting theAutomatic backfill type in the Google Cloud console.

At most one of these can be specified:

--mongodb-excluded-objects=MONGODB_EXCLUDED_OBJECTS

Path to a YAML (or JSON) file containing the MongoDB data sources to avoidbackfilling.

The JSON file is formatted as follows, with camelCase field naming:

{"databases":[{"database":"sample_database","collections":[{"collection":"sample_collection","fields":[{"field":"sample_field",}]}]}]}

--mysql-excluded-objects=MYSQL_EXCLUDED_OBJECTS

Path to a YAML (or JSON) file containing the MySQL data sources to avoidbackfilling.

The JSON file is formatted as follows, with camelCase field naming:

{"mysqlDatabases":[{"database":"sample_database","mysqlTables":[{"table":"sample_table","mysqlColumns":[{"column":"sample_column",}]}]}]}

--oracle-excluded-objects=ORACLE_EXCLUDED_OBJECTS

Path to a YAML (or JSON) file containing the Oracle data sources to avoidbackfilling.

The JSON file is formatted as follows, with camelCase field naming:

{"oracleSchemas":[{"schema":"SAMPLE","oracleTables":[{"table":"SAMPLE_TABLE","oracleColumns":[{"column":"COL",}]}]}]}

--postgresql-excluded-objects=POSTGRESQL_EXCLUDED_OBJECTS

Path to a YAML (or JSON) file containing the PostgreSQL data sources to avoidbackfilling.

The JSON file is formatted as follows, with camelCase field naming:

{"postgresqlSchemas":[{"schema":"SAMPLE","postgresqlTables":[{"table":"SAMPLE_TABLE","postgresqlColumns":[{"column":"COL",}]}]}]}

--salesforce-excluded-objects=SALESFORCE_EXCLUDED_OBJECTS

Path to a YAML (or JSON) file containing the Salesforce data sources to avoidbackfilling.

The JSON file is formatted as follows, with camelCase field naming:

{"objects":[{"objectName":"SAMPLE",},{"objectName":"SAMPLE2",}]}

--spanner-excluded-objects=SPANNER_EXCLUDED_OBJECTS

Path to a YAML (or JSON) file containing the Spanner data sources to avoidbackfilling.

The JSON file is formatted as follows, with camelCase field naming:

{"schemas":[{"schema":"SAMPLE_SCHEMA","tables":[{"table":"SAMPLE_TABLE","columns":[{"column":"SAMPLE_COLUMN",}]}]}]}

--sqlserver-excluded-objects=SQLSERVER_EXCLUDED_OBJECTS

Path to a YAML (or JSON) file containing the SQL Server data sources to avoidbackfilling.

The JSON file is formatted as follows, with camelCase field naming:

{"schemas":[{"schema":"SAMPLE","tables":[{"table":"SAMPLE_TABLE","columns":[{"column":"COL",}]}]}]}

This must be specified.

Connection profile resource - Resource ID of the destination connection profile.This represents a Cloud resource. (NOTE) Some attributes are not given argumentsin this group but can be set in other ways.

To set theproject attribute:

• provide the argument--destination-name on the command line with afully specified name;
• provide the argument--project on the command line;
• set the propertycore/project.

To set thelocation attribute:

• provide the argument--destination-name on the command line with afully specified name;
• provide the argument--location on the command line.

This must be specified.

--destination-name=DESTINATION_NAME

ID of the connection_profile or fully qualified identifier for theconnection_profile.

To set theconnection_profile attribute:

• provide the argument--destination-name on the command line.

Exactly one of these must be specified:

--bigquery-destination-config=BIGQUERY_DESTINATION_CONFIG

Path to a YAML (or JSON) file containing the configuration for Google BigQueryDestination Config.

The YAML (or JSON) file should be formatted as follows:

BigQuery configuration with source hierarchy datasets and merge mode (merge modeis by default):

{"sourceHierarchyDatasets":{"datasetTemplate":{"location":"us-central1","datasetIdPrefix":"my_prefix","kmsKeyName":"projects/{project}/locations/{location}/keyRings/{key_ring}/cryptoKeys/{cryptoKey}"}},"merge":{}"dataFreshness":"3600s"}

BigQuery configuration with source hierarchy datasets and append only mode:

{"sourceHierarchyDatasets":{"datasetTemplate":{"location":"us-central1","datasetIdPrefix":"my_prefix","kmsKeyName":"projects/{project}/locations/{location}/keyRings/{key_ring}/cryptoKeys/{cryptoKey}"}},"appendOnly":{}}

BigQuery configuration with single target dataset and merge mode:

{"singleTargetDataset":{"datasetId":"projectId:my_dataset"},"merge":{}"dataFreshness":"3600s"}

BigQuery configuration with Big Lake table configuration:

{"singleTargetDataset":{"datasetId":"projectId:datasetId"},"appendOnly":{},"blmtConfig":{"bucket":"bucketName","tableFormat":"ICEBERG","fileFormat":"PARQUET","connectionName":"projectId.region.connectionName","rootPath":"/root"}}

--gcs-destination-config=GCS_DESTINATION_CONFIG

Path to a YAML (or JSON) file containing the configuration for Google CloudStorage Destination Config.

The JSON file is formatted as follows:

{"path":"some/path","fileRotationMb":5,"fileRotationInterval":"15s","avroFileFormat":{}}

This must be specified.

Connection profile resource - Resource ID of the source connection profile. Thisrepresents a Cloud resource. (NOTE) Some attributes are not given arguments inthis group but can be set in other ways.

To set theproject attribute:

• provide the argument--source-name on the command line with a fullyspecified name;
• provide the argument--project on the command line;
• set the propertycore/project.

To set thelocation attribute:

• provide the argument--source-name on the command line with a fullyspecified name;
• provide the argument--location on the command line.

This must be specified.

--source-name=SOURCE_NAME

ID of the connection_profile or fully qualified identifier for theconnection_profile.

To set theconnection_profile attribute:

• provide the argument--source-name on the command line.

Exactly one of these must be specified:

--mongodb-source-config=MONGODB_SOURCE_CONFIG

Path to a YAML (or JSON) file containing the configuration for MongoDB SourceConfig.

The JSON file is formatted as follows, with snake_case field naming:

{"includeObjects":{},"excludeObjects":{"databases":[{"database":"sampleDb","collections":[{"collection":"sampleCollection","fields":[{"field":"SAMPLE_FIELD",}]}]}]}}

--mysql-source-config=MYSQL_SOURCE_CONFIG

Path to a YAML (or JSON) file containing the configuration for MySQL SourceConfig.

The JSON file is formatted as follows, with snake_case field naming:

{"allowlist":{},"rejectlist":{"mysql_databases":[{"database_name":"sample_database","mysql_tables":[{"table_name":"sample_table","mysql_columns":[{"column_name":"sample_column",}]}]}]}}

--oracle-source-config=ORACLE_SOURCE_CONFIG

Path to a YAML (or JSON) file containing the configuration for Oracle SourceConfig.

The JSON file is formatted as follows, with snake_case field naming:

{"allowlist":{},"rejectlist":{"oracle_schemas":[{"schema_name":"SAMPLE","oracle_tables":[{"table_name":"SAMPLE_TABLE","oracle_columns":[{"column_name":"COL",}]}]}]}}

--postgresql-source-config=POSTGRESQL_SOURCE_CONFIG

Path to a YAML (or JSON) file containing the configuration for PostgreSQL SourceConfig.

The JSON file is formatted as follows, with camelCase field naming:

{"includeObjects":{},"excludeObjects":{"postgresqlSchemas":[{"schema":"SAMPLE","postgresqlTables":[{"table":"SAMPLE_TABLE","postgresqlColumns":[{"column":"COL",}]}]}]},"replicationSlot":"SAMPLE_REPLICATION_SLOT","publication":"SAMPLE_PUBLICATION"}

--salesforce-source-config=SALESFORCE_SOURCE_CONFIG

Path to a YAML (or JSON) file containing the configuration for Salesforce SourceConfig.

The JSON file is formatted as follows, with camelCase field naming:

{"pollingInterval":"3000s","includeObjects":{},"excludeObjects":{"objects":[{"objectName":"SAMPLE","fields":[{"fieldName":"SAMPLE_FIELD",}]}]}}

--spanner-source-config=SPANNER_SOURCE_CONFIG

Path to a YAML (or JSON) file containing the configuration for Spanner SourceConfig.

The JSON file is formatted as follows, with camelCase field naming:

"changeStreamName":"SAMPLE_CHANGE_STREAM_NAME",{"includeObjects":{},"excludeObjects":{"schemas":[{"schema":"SAMPLE","tables":[{"table":"SAMPLE_TABLE","columns":[{"column":"COL",}]}]}]},"maxConcurrentCdcTasks":1000,"maxConcurrentBackfillTasks":10,"backfillDataBoostEnabled":false,"fgacRole":"SAMPLE_FGAC_ROLE","spannerRpcPriority":"MEDIUM"}

--sqlserver-source-config=SQLSERVER_SOURCE_CONFIG

Path to a YAML (or JSON) file containing the configuration for SQL Server SourceConfig.

The JSON file is formatted as follows, with camelCase field naming:

{"includeObjects":{},"excludeObjects":{"schemas":[{"schema":"SAMPLE","tables":[{"table":"SAMPLE_TABLE","columns":[{"column":"COL",}]}]}]},"maxConcurrentCdcTasks":2,"maxConcurrentBackfillTasks":10,"transactionLogs":{}# Or changeTables}

--labels=[KEY=VALUE,…]

List of label KEY=VALUE pairs to add.

Keys must start with a lowercase character and contain only hyphens(-), underscores (_), lowercase characters, andnumbers. Values must contain only hyphens (-), underscores(_), lowercase characters, and numbers.

--rule-sets=RULE_SETS

Path to a JSON file containing a list of rule sets to be applied to the stream.

TheJSONfileisformattedasfollows,withcamelCasefieldnaming:

[{"objectFilter":{"sourceObjectIdentifier":{"oracleIdentifier":{"schema":"schema1","table":"table1"}}},"customizationRules":[{"bigqueryClustering":{"columns":["COL_A"]}}]},{"objectFilter":{"sourceObjectIdentifier":{"oracleIdentifier":{"schema":"schema2","table":"table2"}}},"customizationRules":[{"bigqueryPartitioning":{"timeUnitPartition":{"column":"TIME_COL","partitioningTimeGranularity":"PARTITIONING_TIME_GRANULARITY_DAY"}}}]}]

At most one of these can be specified:

--force

Create the stream without validating it.

--validate-only

Only validate the stream, but do not create any resources. The default is false.

Run$gcloud help for details.