Go support for Protocol Buffers

This project hosts the Go implementation forprotocol buffers, which is alanguage-neutral, platform-neutral, extensible mechanism for serializingstructured data. The protocol buffer language is a language for specifying theschema for structured data. This schema is compiled into language specificbindings. This project provides both a tool to generate Go code for theprotocol buffer language, and also the runtime implementation to handleserialization of messages in Go. See theprotocol buffer developer guidefor more information about protocol buffers themselves.

This project is comprised of two components:

• Code generator: Theprotoc-gen-gotool is a compiler plugin toprotoc, the protocol buffer compiler. Itaugments theprotoc compiler so that it knows how togenerate Go specific code for a given.proto file.

• Runtime library: Theprotobuf modulecontains a set of Go packages that form the runtime implementation ofprotobufs in Go. This provides the set of interfaces thatdefine what a message isand functionality to serialize message in various formats (e.g.,wire,JSON,andtext).

See thedeveloper guide for protocol buffers in Gofor a general guide for how to get started using protobufs in Go.

This project is the second major revision of the Go protocol buffer APIimplemented by thegoogle.golang.org/protobufmodule. The first major version is implemented by thegithub.com/golang/protobufmodule.

Package index

Summary of the packages provided by this module:

• proto: Packageproto provides functions operating on protobuf messages such as cloning,merging, and checking equality, as well as binary serialization.
• encoding/protojson:Packageprotojson serializes protobuf messages as JSON.
• encoding/prototext:Packageprototext serializes protobuf messages as the text format.
• encoding/protowire:Packageprotowire parses and formats the low-level raw wire encoding. Mostusers should use packageproto to serialize messages in the wire format.
• reflect/protoreflect:Packageprotoreflect provides interfaces to dynamically manipulateprotobuf messages.
• reflect/protoregistry:Packageprotoregistry provides data structures to register and lookupprotobuf descriptor types.
• reflect/protodesc:Packageprotodesc provides functionality for convertingdescriptorpb.FileDescriptorProto messages to/from the reflectiveprotoreflect.FileDescriptor.
• reflect/protopath:Packageprotopath provides a representation of a sequence ofprotobuf reflection operations on a message.
• reflect/protorange:Packageprotorange provides functionality to traverse a protobuf message.
• testing/protocmp:Packageprotocmp provides protobuf specific options for thecmp package.
• testing/protopack:Packageprotopack aids manual encoding and decoding of the wire format.
• testing/prototest:Packageprototest exercises the protobuf reflection implementation forconcrete message types.
• types/dynamicpb:Packagedynamicpb creates protobuf messages at runtime from protobufdescriptors.
• types/known/anypb:Packageanypb is the generated package forgoogle/protobuf/any.proto.
• types/known/timestamppb:Packagetimestamppb is the generated package forgoogle/protobuf/timestamp.proto.
• types/known/durationpb:Packagedurationpb is the generated package forgoogle/protobuf/duration.proto.
• types/known/wrapperspb:Packagewrapperspb is the generated package forgoogle/protobuf/wrappers.proto.
• types/known/structpb:Packagestructpb is the generated package forgoogle/protobuf/struct.proto.
• types/known/fieldmaskpb:Packagefieldmaskpb is the generated package forgoogle/protobuf/field_mask.proto.
• types/known/apipb:Packageapipb is the generated package forgoogle/protobuf/api.proto.
• types/known/typepb:Packagetypepb is the generated package forgoogle/protobuf/type.proto.
• types/known/sourcecontextpb:Packagesourcecontextpb is the generated package forgoogle/protobuf/source_context.proto.
• types/known/emptypb:Packageemptypb is the generated package forgoogle/protobuf/empty.proto.
• types/descriptorpb:Packagedescriptorpb is the generated package forgoogle/protobuf/descriptor.proto.
• types/pluginpb:Packagepluginpb is the generated package forgoogle/protobuf/compiler/plugin.proto.
• compiler/protogen:Packageprotogen provides support for writing protoc plugins.
• cmd/protoc-gen-go:Theprotoc-gen-go binary is a protoc plugin to generate a Go protocolbuffer package.

Reporting issues

The issue tracker for this project is currently located atgolang/protobuf.

Please report any issues there with a sufficient description of the bug orfeature request. Bug reports should ideally be accompanied by a minimalreproduction of the issue. Irreproducible bugs are difficult to diagnose and fix(and likely to be closed after some period of time). Bug reports must specifythe version of theGo protocol buffer moduleand also the version of theprotocol buffer toolchainbeing used.

Contributing

This project is open-source and accepts contributions. See thecontribution guidefor more information.

Compatibility

This module and the generated code are expected to be stable over time. However,we reserve the right to make breaking changes without notice for the followingreasons:

• Security: A security issue in the specification or implementation maycome to light whose resolution requires breaking compatibility. We reservethe right to address such issues.
• Unspecified behavior: There are some aspects of the protocol bufferspecification that are undefined. Programs that depend on unspecifiedbehavior may break in future releases.
• Specification changes: It may become necessary to address aninconsistency, incompleteness, or change in the protocol bufferspecification, which may affect the behavior of existing programs. Wereserve the right to address such changes.
• Bugs: If a package has a bug that violates correctness, a programdepending on the buggy behavior may break if the bug is fixed. We reservethe right to fix such bugs.
• Generated additions: We reserve the right to add new declarations togenerated Go packages of.proto files. This includes declared constants,variables, functions, types, fields in structs, and methods on types. Thismay break attempts at injecting additional code on top of what is generatedbyprotoc-gen-go. Such practice is not supported by this project.
• Internal changes: We reserve the right to add, modify, and removeinternal code, which includes all unexported declarations, theprotoc-gen-go/internal_gengopackage, theruntime/protoimplpackage, and all packages underinternal.

Any breaking changes outside of these will be announced 6 months in advance toprotobuf@googlegroups.com.

Users should use generated code produced by a version ofprotoc-gen-gothat is identical to the runtime version provided by theprotobuf module. Thisproject promises that the runtime remains compatible with code produced by aversion of the generator that is no older than 1 year from the version of theruntime used, according to the release dates of the minor version. Generatedcode is expected to use a runtime version that is at least as new as thegenerator used to produce it. Generated code contains references toprotoimpl.EnforceVersionto statically ensure that the generated code and runtime do not driftsufficiently far apart.

Historical legacy

This project is the second major revision(released in 2020)of the Go protocol buffer API implemented by thegoogle.golang.org/protobufmodule. The first major version(released publicly in 2010)is implemented by thegithub.com/golang/protobufmodule.

The first version predates the release of Go 1 by several years. It has a longhistory as one of the first core pieces of infrastructure software ever writtenin Go. As such, the Go protobuf project was one of many pioneers for determiningwhat the Go language should even look like and what would eventually beconsidered good design patterns and “idiomatic” Go (by simultaneously beingboth positive and negative examples of it).

Consider the changing signature of theproto.Unmarshal function as an exampleof Go language and library evolution throughout the life of this project:

// 2007/09/25 - Conception of Go// 2008/11/12export func UnMarshal(r io.Read, pb_e reflect.Empty) *os.Error// 2008/11/13export func UnMarshal(buf *[]byte, pb_e reflect.Empty) *os.Error// 2008/11/24export func UnMarshal(buf *[]byte, pb_e interface{}) *os.Error// 2008/12/18export func UnMarshal(buf []byte, pb_e interface{}) *os.Error// 2009/01/20func UnMarshal(buf []byte, pb_e interface{}) *os.Error// 2009/04/17func UnMarshal(buf []byte, pb_e interface{}) os.Error// 2009/05/22func Unmarshal(buf []byte, pb_e interface{}) os.Error// 2011/11/03func Unmarshal(buf []byte, pb_e interface{}) error// 2012/03/28 - Release of Go 1// 2012/06/12func Unmarshal(buf []byte, pb Message) error

These changes demonstrate the difficulty of determining what the right API isfor any new technology. It takes time multiplied by many users to determine whatis best; even then, “best” is often still somewhere over the horizon.

The change on June 6th, 2012 added a degree of type-safety to Go protobufs bydeclaring a new interface that all protobuf messages were required to implement:

type Message interface {   Reset()   String() string   ProtoMessage()}

This interface reduced the set of types that can be passed toproto.Unmarshalfrom the universal set of all possible Go types to those with a specialProtoMessage marker method. The intention of this change is to limit theprotobuf API to only operate on protobuf data types (i.e., protobuf messages).For example, there is no sensible operation if a Go channel were passed to theprotobuf API as a channel cannot be serialized. The restricted interface wouldprevent that.

This interface does not behaviorally describe what a protobuf message is, butacts as a marker with an undocumented expectation that protobuf messages must bea Go struct with a specific layout of fields with formatted tags. Thisexpectation is not statically enforced by the Go language, for it is animplementation detail checked dynamically at runtime using Go reflection. Backin 2012, the only types with this marker were those generated byprotoc-gen-go. Sinceprotoc-gen-go would always generate messages with theproper layout of fields, this was deemed an acceptable and dramatic improvementoverinterface{}.

Over the next 10 years,use of Go would skyrocket and use ofprotobufs in Go would skyrocket as well. With increased popularity also camemore diverse usages and requirements for Go protobufs and an increased number ofcustomproto.Message implementations that were not generated byprotoc-gen-go.

The increasingly diverse ecosystem of Go types implementing theproto.Messageinterface led to incompatibilities, which often occurred when:

• Passing customproto.Message types to the protobuf APIs: A concretemessage implementation might work with some top-level functions (e.g.,proto.Marshal), but cause others (e.g.,proto.Equal) to choke and panic.This occurs because the type only had partial support for being an actualmessage by only implementing theproto.Marshaler interface or havingmalformed struct field tags that happened to work with one function, but notanother.

• Using Go reflection on anyproto.Message types: A common desire is towrite general-purpose code that operates on any protobuf message. Forexample, a microservice might want to populate atrace_id field if it ispresent in a message. To accomplish this, one would use Go reflection tointrospect the message type, and assume it were a pointer to a Go structwith a field namedTraceId (as would be commonly produced byprotoc-gen-go). If the concrete message type did not match thisexpectation, it either failed to work or even resulted in a panic. Such wasthe case for concrete message types that might be backed by a Go map insteadof a Go struct.

Both of these issues are solved by following the idiom thatinterfaces shoulddescribe behavior, not data. This means that the interface itself shouldprovide sufficient functionality through its methods that users can introspectand interact with all aspects of a protobuf message through a principled API.This feature is calledprotobuf reflection. Just as how Go reflection providesan API for programmatically interacting with any arbitrary Go value, protobufreflection provides an API for programmatically interacting with any arbitraryprotobuf message.

Since an interface cannot be extended in a backwards compatible way, thissuggested the need for a new major version that defines a newproto.Messageinterface:

type Message interface {    ProtoReflect() protoreflect.Message}

The newproto.Messageinterface contains a singleProtoReflect method that returns aprotoreflect.Message,which is a reflective view over a protobuf message. In addition to making abreaking change to theproto.Message interface, we took this opportunity tocleanup the supporting functionality that operate on aproto.Message, split upcomplicated functionality apart into manageable packages, and to hideimplementation details away from the public API.

The goal for this major revision is to improve upon all the benefits of, whileaddressing all the shortcomings of the old API. We hope that it will serve theGo ecosystem well for the next 10 years and beyond.