Create and manage protobuf schemas

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

This document describes how to create and perform operations on schema bundles.

In Bigtable, you can useprotocol buffer (protobuf) schemas to query individual fieldswithin protobuf messages stored as bytes in your columns.You do this by uploading your schemas in aschema bundle, a table-levelresource that contains one or more of your protobuf schemas.

Using schema bundles offers the following benefits:

• Saves time and effort: With protocol buffers, you define your datastructure once in a proto file, and then use the generated source code towrite and read your data.
• Improves data consistency: By using a proto file as a single source oftruth, you can ensure that all applications and services are using the samedata model.
• Eliminates data duplication: You can use protocol buffers acrossprojects by defining message types in proto files that reside outside of aspecific project's codebase.

The process of using schemas in Bigtable starts with your protofiles. Aproto file is a text file where you define the structure of yourdata. You use the protobuf compiler tool, also referred to asprotoc, togenerate aprotobuf file descriptor set, which is a machine-readable schema ofyour proto file. You then use this descriptor set to create a schema bundle.

For examples of proto files and their corresponding descriptor sets, seeExample data.

The following diagram shows the process of using schemas inBigtable:

You can create schema bundles using the Google Cloud CLI. After you upload aschema bundle to Bigtable, you can query your data usingthe Bigtable Studio query builder, GoogleSQL forBigtable, or Bigtableexternal tables in BigQuery.

Before you begin

Take the following steps if you plan to use the gcloud CLI:

1. Install theGoogle Cloud CLI.

2. Initialize the gcloud CLI:

   gcloudinit

Required roles

To get the permissions that you need to create and manage schema bundles, askyour administrator to grant you theBigtable Admin (roles/bigtable.admin)Identity and Access Management (IAM) role on the table.

This predefined role contains the permissions that Bigtablerequires to work with schema bundles. To see the exact permissions that arerequired, expand theRequired permissions section:

You might also be able to get these permissions withcustom roles or otherpredefined roles.

For more information about Bigtable roles and permissions, seeAccess control with IAM.

Generate a protobuf file descriptor set

Before you can create a schema bundle, you need to generate a descriptor setfrom your proto files with the protobuf compiler tool.

1. To install the compiler,download the package and follow theinstructions in the README file.

2. Run the compiler:

   protoc--proto_path=IMPORT_PATH--include_imports\--descriptor_set_out=DESCRIPTOR_OUTPUT_LOCATIONPATH_TO_PROTO

   Replace the following:

   • IMPORT_PATH: the directory where the protoc compilersearches for proto files.
   • DESCRIPTOR_OUTPUT_LOCATION: the directory where theprotoc compiler saves the generated descriptor set.
   • PATH_TO_PROTO: the path to your proto file.

For example, to create a descriptor set namedlibrary.pb for thelibrary.proto file in the current directory, you can use the following command:

protoc--include_imports--descriptor_set_out=library.pblibrary.proto

Create a schema bundle

gcloudbigtableschema-bundlescreateSCHEMA_BUNDLE_ID\--instance=INSTANCE_ID\--table=TABLE_ID\--proto-descriptors-file=PROTO_DESCRIPTORS_FILE

try{InputStreamin=getClass().getClassLoader().getResourceAsStream(PROTO_FILE_PATH);CreateSchemaBundleRequestrequest=CreateSchemaBundleRequest.of(tableId,schemaBundleId).setProtoSchema(ByteString.readFrom(in));SchemaBundleschemaBundle=adminClient.createSchemaBundle(request);System.out.printf("Schema bundle: %s created successfully%n",schemaBundle.getId());}catch(NotFoundExceptione){System.err.println("Failed to create a schema bundle from a non-existent table: "+e.getMessage());}catch(IOExceptione){thrownewRuntimeException(e);}

View information about schema bundles

Before you can view information about schema bundles, you need to have aBigtable table with at least one schema bundle. You can getinformation about schema bundles in a table by retrieving a single schemabundle's definition or by listing all schema bundles in a table.

Get schema bundle definition

gcloudbigtableschema-bundlesdescribeSCHEMA_BUNDLE_ID\--instance=INSTANCE_ID\--table=TABLE_ID

SchemaBundleschemaBundle=null;try{schemaBundle=adminClient.getSchemaBundle(tableId,schemaBundleId);// Deserialize and print the FileDescriptorSetDescriptorProtos.FileDescriptorSetfileDescriptorSet=DescriptorProtos.FileDescriptorSet.parseFrom(schemaBundle.getProtoSchema());System.out.println("--------- Deserialized FileDescriptorSet ---------");for(DescriptorProtos.FileDescriptorProtofileDescriptorProto:fileDescriptorSet.getFileList()){System.out.println("File: "+fileDescriptorProto.getName());System.out.println("  Package: "+fileDescriptorProto.getPackage());for(DescriptorProtos.DescriptorProtomessageType:fileDescriptorProto.getMessageTypeList()){System.out.println("  Message: "+messageType.getName());}}System.out.println("--------------------------------------------------");}catch(InvalidProtocolBufferExceptione){System.err.println("Failed to parse FileDescriptorSet: "+e.getMessage());}catch(NotFoundExceptione){System.err.println("Failed to retrieve metadata from a non-existent schema bundle: "+e.getMessage());}

--------- Deserialized FileDescriptorSet ---------File: my_schema.protoPackage: my_packageMessage: MyMessage--------------------------------------------------

List schema bundles in a table

gcloudbigtableschema-bundleslist\--instance=INSTANCE_ID\--table=TABLE_ID

List<String>schemaBundleIds=newArrayList<>();try{schemaBundleIds=adminClient.listSchemaBundles(tableId);for(StringschemaBundleId:schemaBundleIds){System.out.println(schemaBundleId);}}catch(NotFoundExceptione){System.err.println("Failed to list schema bundles from a non-existent table: "+e.getMessage());}

my-schema-bundle-1my-schema-bundle-2

Update a schema bundle

When you update a schema bundle, Bigtable checks if the newdescriptor set is backward compatible with the existing one. If it is incompatible,the update fails with aFailedPrecondition error. We recommend that youreserve deleted field numbers to prevent their reuse. For more information, seeProto Best Practices in the protobuf documentation.

If you're sure that the incompatible changes are safe and want to force anupdate, you can use the--ignore-warnings flag with gcloud CLI.

gcloudbigtableschema-bundlesupdateSCHEMA_BUNDLE_ID\--instance=INSTANCE_ID\--table=TABLE_ID\--proto-descriptors-file=PROTO_DESCRIPTORS_FILE

try{InputStreamin=getClass().getClassLoader().getResourceAsStream(PROTO_FILE_PATH);UpdateSchemaBundleRequestrequest=UpdateSchemaBundleRequest.of(tableId,schemaBundleId).setProtoSchema(ByteString.readFrom(in));SchemaBundleschemaBundle=adminClient.updateSchemaBundle(request);System.out.printf("Schema bundle: %s updated successfully%n",schemaBundle.getId());}catch(NotFoundExceptione){System.err.println("Failed to modify a non-existent schema bundle: "+e.getMessage());}catch(IOExceptione){thrownewRuntimeException(e);}

Delete a schema bundle

gcloudbigtableschema-bundlesdeleteSCHEMA_BUNDLE_ID\--instance=INSTANCE_ID\--table=TABLE_ID

try{adminClient.deleteSchemaBundle(tableId,schemaBundleId);System.out.printf("SchemaBundle: %s deleted successfully%n",schemaBundleId);}catch(NotFoundExceptione){System.err.println("Failed to delete a non-existent schema bundle: "+e.getMessage());}

Limitations

Schema bundles have the following limitations:

• You can create a maximum of 10 schema bundles per table.
• The total size of the serialized protocol buffer descriptors within a schemabundle can't exceed 4 MB. There is no direct limit on the number ofindividual schemas you can include in a bundle, as long as the total size ofthe bundle doesn't exceed this limit.

What's next

• Learn how toquery protobuf data.
• Read aboutchanging or uncertain queries.
• Refer toGoogleSQL for Bigtable overview.