Introduction

We are pleased to announce the release of a major revision of the Go API forprotocol buffers,Google’s language-neutral data interchange format.

Motivations for a new API

The first protocol buffer bindings for Go wereannounced by Rob Pikein March of 2010. Go 1 would not be released for another two years.

In the decade since that first release, the package has grown anddeveloped along with Go. Its users’ requirements have grown too.

Many people want to write programs that use reflection to examine protocolbuffer messages. Thereflectpackage provides a view of Go types andvalues, but omits information from the protocol buffer type system. Forexample, we might want to write a function that traverses a log entry andclears any field annotated as containing sensitive data. The annotationsare not part of the Go type system.

Another common desire is to use data structures other than the onesgenerated by the protocol buffer compiler, such as a dynamic message typecapable of representing messages whose type is not known at compile time.

We also observed that a frequent source of problems was that theproto.Messageinterface, which identifies values of generated message types, does verylittle to describe the behavior of those types. When users create typesthat implement that interface (often inadvertently by embedding a messagein another struct) and pass values of those types to functions expectinga generated message value, programs crash or behave unpredictably.

All three of these problems have a common cause, and a common solution:TheMessage interface should fully specify the behavior of a message,and functions operating onMessage values should freely accept anytype that correctly implements the interface.

Since it is not possible to change the existing definition of theMessage type while keeping the package API compatible, we decided thatit was time to begin work on a new, incompatible major version of theprotobuf module.

Today, we’re pleased to release that new module. We hope you like it.

Reflection

Reflection is the flagship feature of the new implementation. Similarto how thereflect package provides a view of Go types and values, thegoogle.golang.org/protobuf/reflect/protoreflectpackage provides a view of values according to the protocol buffertype system.

A complete description of theprotoreflect package would run too longfor this post, but let’s look at how we might write the log-scrubbingfunction we mentioned previously.

First, we’ll write a.proto file defining an extension of thegoogle.protobuf.FieldOptionstype so we can annotate fields as containingsensitive information or not.

syntax = "proto3";import "google/protobuf/descriptor.proto";package golang.example.policy;extend google.protobuf.FieldOptions {    bool non_sensitive = 50000;}

We can use this option to mark certain fields as non-sensitive.

message MyMessage {    string public_name = 1 [(golang.example.policy.non_sensitive) = true];}

Next, we will write a Go function which accepts an arbitrary messagevalue and removes all the sensitive fields.

// Redact clears every sensitive field in pb.func Redact(pb proto.Message) {   // ...}

This function accepts aproto.Message,an interface type implemented by all generated message types. This typeis an alias for one defined in theprotoreflect package:

type ProtoMessage interface{    ProtoReflect() Message}

To avoid filling up the namespace of generatedmessages, the interface contains only a single method returning aprotoreflect.Message,which provides access to the message contents.

(Why an alias? Becauseprotoreflect.Message has a correspondingmethod returning the originalproto.Message, and we need to avoid animport cycle between the two packages.)

Theprotoreflect.Message.Rangemethod calls a function for every populated field in a message.

m := pb.ProtoReflect()m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {    // ...    return true})

The range function is called with aprotoreflect.FieldDescriptordescribing the protocol buffer type of the field, and aprotoreflect.Valuecontaining the field value.

Theprotoreflect.FieldDescriptor.Optionsmethod returns the field options as agoogle.protobuf.FieldOptionsmessage.

opts := fd.Options().(*descriptorpb.FieldOptions)

(Why the type assertion? Since the generateddescriptorpb packagedepends onprotoreflect, theprotoreflect package can’t return theconcrete options type without causing an import cycle.)

We can then check the options to see the value of our extension boolean:

if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {    return true // don't redact non-sensitive fields}

Note that we are looking at the fielddescriptor here, not the fieldvalue. The information we’re interested in lies in the protocolbuffer type system, not the Go one.

This is also an example of an area where wehave simplified theproto package API. The originalproto.GetExtensionreturned both a value and an error. The newproto.GetExtensionreturns just a value, returning the default value for the field if it isnot present. Extension decoding errors are reported atUnmarshal time.

Once we have identified a field that needs redaction, clearing it is simple:

m.Clear(fd)

Putting all the above together, our complete redaction function is:

// Redact clears every sensitive field in pb.func Redact(pb proto.Message) {    m := pb.ProtoReflect()    m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {        opts := fd.Options().(*descriptorpb.FieldOptions)        if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {            return true        }        m.Clear(fd)        return true    })}

A more complete implementation might recursively descend intomessage-valued fields. We hope that this simple example gives ataste of protocol buffer reflection and its uses.

Versions

We call the original version of Go protocol buffers APIv1, and thenew one APIv2. Because APIv2 is not backwards compatible with APIv1,we need to use different module paths for each.

(These API versions are not the same as the versions of the protocolbuffer language:proto1,proto2, andproto3. APIv1 and APIv2are concrete implementations in Go that both support theproto2 andproto3 language versions.)

Thegithub.com/golang/protobufmodule is APIv1.

Thegoogle.golang.org/protobufmodule is APIv2. We have taken advantage of the need to change theimport path to switch to one that is not tied to a specific hostingprovider. (We consideredgoogle.golang.org/protobuf/v2, to make itclear that this is the second major version of the API, but settled onthe shorter path as being the better choice in the long term.)

We know that not all users will move to a new major version of a packageat the same rate. Some will switch quickly; others may remain on the oldversion indefinitely. Even within a single program, some parts may useone API while others use another. It is essential, therefore, that wecontinue to support programs that use APIv1.

• github.com/golang/protobuf@v1.3.4 is the most recent pre-APIv2 version of APIv1.

• github.com/golang/protobuf@v1.4.0 is a version of APIv1 implemented in terms of APIv2.The API is the same, but the underlying implementation is backed by the new one.This version contains functions to convert between the APIv1 and APIv2proto.Messageinterfaces to ease the transition between the two.

• google.golang.org/protobuf@v1.20.0 is APIv2.This module depends upongithub.com/golang/protobuf@v1.4.0,so any program which uses APIv2 will automatically pick a version of APIv1which integrates with it.

(Why start at versionv1.20.0? To provide clarity.We do not anticipate APIv1 to ever reachv1.20.0,so the version number alone should be enough to unambiguously differentiatebetween APIv1 and APIv2.)

We intend to maintain support for APIv1 indefinitely.

This organization ensures that any given program will use only a singleprotocol buffer implementation, regardless of which API version it uses.It permits programs to adopt the new API gradually, or not at all, whilestill gaining the advantages of the new implementation. The principle ofminimum version selection means that programs may remain on the oldimplementation until the maintainers choose to update to the new one(either directly, or by updating a dependency).

Additional features of note

Thegoogle.golang.org/protobuf/encoding/protojsonpackage converts protocol buffer messages to and from JSON using thecanonical JSON mapping,and fixes a number of issues with the oldjsonpb packagethat were difficult to change without causing problems for existing users.

Thegoogle.golang.org/protobuf/types/dynamicpbpackage provides an implementation ofproto.Message for messages whoseprotocol buffer type is derived at runtime.

Thegoogle.golang.org/protobuf/testing/protocmppackage provides functions to compare protocol buffer messages with thegithub.com/google/cmppackage.

Thegoogle.golang.org/protobuf/compiler/protogenpackage provides support for writing protocol compiler plugins.

Conclusion

Thegoogle.golang.org/protobuf module is a major overhaul ofGo’s support for protocol buffers, providing first-class supportfor reflection, custom message implementations, and a cleaned up APIsurface. We intend to maintain the previous API indefinitely as a wrapperof the new one, allowing users to adopt the new API incrementally attheir own pace.

Our goal in this update is to improve upon the benefits of the oldAPI while addressing its shortcomings. As we completed each component ofthe new implementation, we put it into use within Google’s codebase. Thisincremental rollout has given us confidence in both the usability of the newAPI and the performance and correctness of the new implementation. We believeit is production ready.

We are excited about this release and hope that it will serve the Goecosystem for the next ten years and beyond!