Repository files navigation

Zanzibar is an extensible framework to build configuration driven web applications. The goal of Zanzibar is to simplify application development into two steps:

1. write configurations for the application and its components;
2. write code to implement and test business logic.

Based on the configurations, Zanzibar generates boilerplates and glue code, wires them up with your business domain code and the runtime components Zanzibar provides to create a deployable binary.

The builtin components of Zanzibar makes it easy to develop microservices and gateway services that proxy or orchestrate microservices. It is also simple to extend Zanzibar with custom plugins to ease the development of applications that suit your specific needs.

• Concepts
  • Module
    • ModuleClass
    • Type
      • Client
      • Endpoint
      • Middleware
      • Service
    • Dependency Injection
      • Dependency Injection
      • Module Initialization
      • Dependency Rules
  • Config
    • Config Layout
    • Module Config
      • General Layout
      • Non-Config Content
        • Custom Client
        • Circuit Breaker
        • Custom Workflow
      • Grouping
      • Config Schema
    • Application Config
  • Code Generation
    • Go Structs and (de)serializers
    • Zanzibar-generated Code
• How to Use
  • Install
  • Code Gen
  • Testing
    • Entry Points
      • Service
      • Endpoint
    • Fixture
  • Extend Zanzibar
    • New ModuleClass or Type
    • PostGenHook
• Development
  • Installation
  • Running make generate
  • Running the tests
  • Running the benchmarks
  • Running the end-to-end benchmarks
  • Running the server
  • Adding new dependencies

Zanzibar is built on three pillars: module, config, code generation.

Modules are the components that a Zanzibar application is made of. A module belongs to aModuleClass, has atype and can have dependencies on other modules.

ModuleClass abstracts the functionality of a specific class of components. Zanzibar predefines a few module classes, i.e.,client,endpoint,middleware andservice. Each represents a corresponding abstraction:

The moduletype differentiates module instances of the sameModuleClass with further classification. Types are somewhat arbitrary as they are not necessarily abstractions but indications about how Zanzibar should treat the modules.

A client module could be of typehttp,tchannel orcustom, wherehttp ortchannel means Zanzibar will generate a client with given configuration that speaks that protocol whilecustom means the client is fully provided and Zanzibar will use it as is without code generation. In other words,http andtchannel clients are configuration driven (no user code) whereascustom clients are user-defined and can be "smart clients".

Anendpoint module could also be of typehttp ortchannel, which determines the protocol that the endpoint will be made available to invoke externally via the Zanzibar router. Whileendpoint modules do not havecustom type, each method of anendpoint module has aworkflowType that indicates the type of workflow the endpoint method fulfills. The builtin workflow type ishttpClient,tchannelClient andcustom, wherehttpClient andtchannelClient means the endpoint method workflow is to proxy to a client, andcustom means the workflow is fulfilled by user code, see more inCustom Workflow.

Note that workflow type is likely to be deprecated in the future so that proxy to a client will be no longer a builtin option.

The builtin type of middleware module isdefault.

The builtin service type isgateway (it is likely to change in the future, becausedefault is probably a better name).

Note Zanzibar has support for user-defined module classes and module types in case the builtin types are not sufficient. The preferred way of extending Zanzibar is through user defined module classes and module types.

Module dependencies describe the relationships among various modules. The dependency relationship is critical to correctly assemble the modules to a full application.

A module is expected to define its immediate or direct dependencies. Zanzibar generates a module constructor with dependent modules as parameters, and passes the dependencies to the constructor during initilizaiton.

Zanzibar also constructs a full global dependency graph for a given set of modules. This graph is used to initialize modules in the correct order, e.g. leaf modules are initialized first and then passed to the constructors of parent modules for initialization.

To establish and enforce abstraction boundaries, dependency rules atModuleClass level are necessary. Zanzibar predefines the following dependency rules for built module classes:

This table exhausts the possible immediate or direct dependency relationships among builtin module classes. Take endpoint module class for example, an endpoint module can depend on client or middleware modules but not endpoint or service modules. The reasoning for such rules aligns with the abstractions the module classes represent.

TheModuleClass struct hasDependsOn andDependedBy public fields, which makes it simple to extend the dependency rules with custom module class, e.g., we can define a custom module classtask that abstracts common business workflow by setting itsDependsOn field to client andDependedBy field to endpoint.

Configurations are the interface that developers interact with when using the Zanzibar framework, they make up most of Zazibar's API. Various configurarions contain essential meta information of a Zanzibar application and its components. They are source of truth of the application.

Because configurations are the core of a Zanzibar application, we create a root directory to host configuration files when starting a Zanzibar application. There are a few typical directories and files under the root directory. Takeexample-gateway for example:

example-gateway                 # root directory├── bin                         # directory for generated application binaries│   └── example-gateway         # generated example-gateway binary├── build                       # directory for all generated code│   ├── clients                 # generated mocks and module initializers for clients│   ├── endpoints               # generated mocks and module initializers for endpoints│   ├── gen-code                # generated structs and (de)serializers by Thrift compiler│   ├── middlewares             # generated module initializers for middlewares│   │   └── default             # generated module initializers for default middlewares│   └── services                # generated mocks and module intialziers for services├── build.yaml                  # config file for Zanzibar code generation, see below for details├── clients                     # config directory for modules of client module class│   └── bar                     # config directory for a client named 'bar'├── config                      # config directory for application runtime properties│   ├── production.yaml         # config file for production environment│   └── test.yaml               # config file for test environment├── copyright_header.txt        # optional copyright header for open source application├── endpoints                   # config directory for modules of endpoint module class│   └── bar                     # config directory for an endpoint named 'bar'├── idl                         # idl directory for all thrift files│   ├── clients                 # idl directory for client thrift files│   └── endpoints               # idl directory for endpoint thrift files├── middlewares                 # config directory for modules of middleware module class│   ├── transform-response      # config directory for a middleware named 'transform-response'│   ├── default                 # directory for all default middlewares│   │   └── log-publisher       # config directory for a default middleware named 'log-publisher'│   └── default.yaml            # config file describing default middlewares and their execution order   └── services                    # config directory for modules of service module class    └── example-gateway         # config directory for a service named 'example-gateway'

Each module must have a config file so that it can be recognized by Zanzibar. This section explains how the module config files are organized and what goes into them.

Under the application root directory, there should be a corresponding top level config directory for each module class. For Zanzibar builtin module classes, the name of the directory is the plural of the module class name, e.g., aclients directory forclient module class. The directory name is used when registering generator for a module class (example). While it is not required, the same directory naming convention should be followed when defining custom module classes.

Under a module class directory, there should be a corresponding config directory for each module, e.g., theclients directory has a few subdirectories and each of them corresponds to a module.

Under a module directory, there should be a YAML file that contains the meta information of that module. It is required that the file is named of{$ModuleClass}-config.yaml, e.g. the path to the YAML config file ofbar client module isclients/bar/client-config.yaml, similarly the path to the YAML config file ofbar endpoint module isendpoints/bar/endpoint-config.yaml.

Besides the YAML config file, the module directory also contains other necessary directories/files. For example, thequux client is a custom (non-generated) client, its module config directory has following layout:

quxx                        # client module config directory├── client-config.yaml      # client module config file├── fixture                 # directory for fixtures used for testing│   └── fixure.go           # fixtures that can be used by a generated mock client for testing└── quux.go                 # custom client implementation, package is imported by generated code

For client and endpoint modules of builtin typecustom, Zanzibar expects user code to be placed in the module directory. This is important because Zaznibar-generated code refers to user code by importing the package of the module directory path. Furthermore, user code of custom client and endpoint modules must also define and implement necessarypublic types and interfaces so that Zanzibar can wire up the modules.

For client module of custom type, user code must define aClient interface and aNewClient constructor that returns theClient interface. Below is the example codesnippet for thequux custom client:

package quuximport"github.com/uber/zanzibar/examples/example-gateway/build/clients/quux/module"typeClientinterface {Echo(string)string}funcNewClient(deps*module.Dependencies)Client {return&quux{}}typequuxstruct{}func (c*quux)Echo(sstring)string {returns }

Note the type ofdeps parameter passed toNewClient constructor function is generated by Zanzibar, as indicated by the import path. Zanzibar takes care of initializing and passing in the acutaldeps argument, as mentioned inDependency Injection.

For increasing overall system resiliency, zanzibar uses aCircuit Breaker which avoids calling client when there is an increase in failure rate beyond a setthreshold. After a sleepWindowInMilliseconds, client calls are attempted recovery by going in half-open and then close state.

circuitBreakerDisabled: Default false. To disable the circuit-breaker:

    "clients.<clientID>.circuitBreakerDisabled" : true

maxConcurrentRequests: Default 50. To set how many requests can be run at the same time, beyond which requests arerejected:

   "clients.<clientID>.maxConcurrentRequests": 50

errorPercentThreshold: Default 20. To set error percent threshold beyond which to trip the circuit open:

    "clients.<clientID>.errorPercentThreshold": 20

requestVolumeThreshold: Default 20. To set minimum number of requests that will trip the circuit in a rolling windowof 10 (For example, if the value is 20, then if only 19 requests are received in the rolling window of 10 seconds thecircuit will not trip open even if all 19 failed):

    "clients.<clientID>.requestVolumeThreshold" : true

sleepWindowInMilliseconds: Default 5000. To set the amount of time, after tripping the circuit, to reject requestsbefore allowing attempts again to determine if the circuit should again be closed:

    "clients.<clientID>.sleepWindowInMilliseconds" : true

For endpoint module of custom workflow type, user code must define aNew{$endpoint}{$method}Workflow constructor that returns the Zanzibar-generated{$endpoint}{$method}Workflow interface which has a soleHandle method. Below is the example codesnippet for thecontacts custom endpoint:

package contactsimport ("context""github.com/uber/zanzibar/examples/example-gateway/build/endpoints/contacts/module""github.com/uber/zanzibar/examples/example-gateway/build/endpoints/contacts/workflow"contacts"github.com/uber/zanzibar/examples/example-gateway/build/gen-code/endpoints-idl/endpoints/contacts/contacts"zanzibar"github.com/uber/zanzibar/runtime""go.uber.org/zap")funcNewContactsSaveContactsWorkflow(c*module.ClientDependencies,l*zap.Logger,) workflow.ContactsSaveContactsWorkflow {return&saveContacts{... } }typesaveContactsstruct {... }func (w*saveContacts)Handle(ctx context.Context,headers zanzibar.Header,r*contacts.SaveContactsRequest,) (*contacts.SaveContactsResponse, zanzibar.Header,error) {... }

The idea of the workflow constructor is similar to the client constructor, with a couple of differences:

• the first parameter is specificallyClientDependencies and there is an additional logger parameter, this will be changed in the future so that the dependency parameter is generalized;
• the return value is an interface generated by Zanzibar, the parameter and return value of theHandle method refers to structs generated by Thrift compiler based on the endpoint thrift file configured in the endpoint-config.yaml, see more inConfig Schema.

Zanzibar allows nesting module config directories in the sub-directories of a module class config directory. This is useful to group related modules under a sub-directory. For example, thetchannel directory groups allTChannel endpoint modules:

endpoints├── ...└── tchannel                    # this directory does not correspond to a module, it represents a group    └── baz                     # module config directory under the 'tchannel' group        ├── baz_call.go        ├── baz_call_test.go        ├── call.yaml        └── endpoint-config.yaml

Modules of differentModuleClass andtype are likely to have different fields in their config files. Developers are expected to write module config files according to theschemas.

Note: fields are absent in config schemas but present in examples are experimental.

The endpoint module config is different from other module classes as it has multiple YAML files, where each endpoint method corresponds to a YAML file and the endpoint-config.yaml file refers to them.

endpoints/multi├── endpoint-config.yaml    # has a field 'endpoints' that is a list and contains helloA and helloB├── helloA.yaml             # config file for method helloA└── helloB.yaml             # config file for method helloB

The reason for such layout is to avoid a large endpoint-config.yaml file when an endpoint has many methods.

Besides the module configs, Zanzibar also expects a YAML file that configures necessary properties to boostrap the code generation process of a Zanzibar application. The schema for application config is definedhere.

Unlike the module configs, there is no restriction on how this config file should be named. It can be named{$appName}.yaml orbuild.yaml as it is inexample-gateway, as long as it is passed correctly as an argument to the code generationrunner.

In this config file, you can specify the paths from which to discover modules. You can also specifydefault dependencies.

Default Dependencies allow module classes to include instances of other module classes as default dependencies. This means that no explicit configurations are required for certain module instances to be included as a dependency. e.g., we can includeclients/logger as a default dependency forendpoint, and every endpoint will haveclients/logger as a dependency in itsmodule/dependencies.go file, even if the endpoint'sendpoint-config.yaml file does not listclients/logger as a dependency.

Note that these paths supportGlob patterns.

Zanzibar provides HTTP and TChannel runtime components for both clients and servers. Once all the configs are properly defined, Zanzibar is able to parse the config files and generate code and wire it up with the runime components to produce a full application. All generated code is placed in thebuild directory.

Zanzibar expects non-custom clients and endpoints to define their interfaces using Thrift (Zanzibar Thrift file semantics). For example, thebar endpoint defines its interfaces using thebar.thrift as specified inhello.yaml. The data types in such thrift files must have their equivalents in Go.

• For tchannel clients/endpoints, network communication is Thrift over TChannel. Zanzibar usesthriftrw to generate Go structs and thrift (de)serializers;
• For http clients/endpoints, network communication is JSON over HTTP. Zanzibar usesthriftrw to generate Go structs and then useseasyjson to generate JSON (de)serializers.

Thepre-steps.sh script takes care of this part of the code generation, and places the generated code underbuild/gen-code directory.

Everything exceptgen-code underbuild directory is generated by Zanzibar. Zanzibar parses config files for each module to gathers meta information and then executing varioustemplates by applying them to the meta data. Here is what is generated for each builtin module class:

• client: dependency type, client interface and constructor if non-custom, mock client constructor
• middleware: dependency type, middleware type and constructor (unstable)
• endpoint: dependency type, endpoint type and constructor, workflow interface, workflow if non-custom, mock workflow constructor if custom
• service: dependency type and initializer, main.go, mock service constructor, service constructor

Assuming you are using a vendor package management tool like Glide, then the minimal glide.yaml file would look like:

-package:go.uber.org/thriftrwversion:^1.8.0-package:github.com/mailru/easyjsonversion:master-package:github.com/uber/zanzibarversion:master

After installing the packages, create your module configs and application config in your application root directory. Then you are ready to run the following script to kick off code generation:

# put this script in application root directoryCONFIG_DIR="."BUILD_DIR="$CONFIG_DIR/build"THRIFTRW_SRCS=""# find all thrift files specified in the config filesconfig_files=$(find"." -name"*-config.yaml"! -path"*/build/*"! -path"*/vendor/*"| sort)forconfig_filein${config_files};dodir=$(dirname"$config_file")yaml_files=$(find"$dir" -name"*.yaml")foryaml_filein${yaml_files};dothrift_file=$(yq -r'.. | .idlFile? | select(strings | endswith(".thrift"))'"$yaml_file")[[-z${thrift_file} ]]&&continue[[${THRIFTRW_SRCS}==*${thrift_file}* ]]&&continue        THRIFTRW_SRCS+="$CONFIG_DIR/idl/$thrift_file"donedonebash ./vendor/github.com/uber/zanzibar/codegen/runner/pre-steps.sh"$BUILD_DIR""$CONFIG_DIR""zanzibar""$THRIFTRW_SRCS"go run ./vendor/github.com/uber/zanzibar/codegen/runner/runner.go --config="$CONFIG_DIR/build.yaml"

Note the above script will be abstracted for easier usage in the future.

Zanzibar comes with builtin integration testing frameworks to help test business logic with ease. SettinggenMock to true will trigger Zanzibar to generate mock client, workflow and service constructors. The mock clients, being the leaf nodes in the dependency graph, are wired with the rest modules to create a testing application, which you can test against by setting expectations of the mock clients. The generated test helpers make writing tests straightforward and concise.

Currently Zanzibar provides two entry points to write integration tests: service and endpoint.

Service level integration testing treats your application as a black box. Zanzibar starts a local server for your application and you write tests by sending requests to the server and verify the response is expected.

funcTestSaveContacts(t*testing.T) {ms:=ms.MustCreateTestService(t)ms.Start()deferms.Stop()ms.MockClients().Contacts.ExpectSaveContacts().Success()endpointReqeust:=&endpointContacts.SaveContactsRequest{Contacts: []*endpointContacts.Contact{},}rawBody,_:=endpointReqeust.MarshalJSON()res,err:=ms.MakeHTTPRequest("POST","/contacts/foo/contacts",nil,bytes.NewReader(rawBody),)if!assert.NoError(t,err,"got http error") {return}assert.Equal(t,"202 Accepted",res.Status)}

Endpoint level integration testing allows focusing on testing the business logic without a full server setup. It is lightweighted and feels more like unit tests.

funcTestSaveContactsCallWorkflow(t*testing.T) {mh,mc:=mockcontactsworkflow.NewContactsSaveContactsWorkflowMock(t)mc.Contacts.ExpectSaveContacts().Success()endpointReqeust:=&endpointContacts.SaveContactsRequest{UserUUID:"foo",Contacts: []*endpointContacts.Contact{},}res,resHeaders,err:=mh.Handle(context.Background(),nil,endpointReqeust)if!assert.NoError(t,err,"got error") {return}assert.Nil(t,resHeaders)assert.Equal(t,&endpointContacts.SaveContactsResponse{},res)}

The above snippets can be found insave_contacts_test.go.

Zanzibar usesgomock to generate client mocks. To avoid manually setting the same fixture expectations again and again, Zanzibar augments gomock-generated mocks with fixture support. For example, the client-config.yaml file ofcontacts client has afixturefield:

fixture:importPath:github.com/uber/zanzibar/examples/example-gateway/clients/contacts/fixturescenarios:SaveContacts:    -success

This basically says thesaveContacts method has asuccess scenario which is defined in thefixture package indicated by theimportPath. The fixture package is provided by users and here is what it looks like:

package fixtureimport (mc"github.com/uber/zanzibar/examples/example-gateway/build/clients/contacts/mock-client"gen"github.com/uber/zanzibar/examples/example-gateway/build/gen-code/clients-idl/clients/contacts/contacts")varsaveContactsFixtures=&mc.SaveContactsScenarios{Success:&mc.SaveContactsFixture{Arg0Any:true,Arg1Any:true,Arg2:&gen.SaveContactsRequest{UserUUID:"foo",},Ret0:&gen.SaveContactsResponse{},},}// Fixture ...varFixture=&mc.ClientFixture{SaveContacts:saveContactsFixtures,}

With that, in your tests you will be able to write

mc.Contacts.ExpectSaveContacts().Success()

rather than

s.mockClient.EXPECT().SaveContacts(arg0,arg1,arg2).Return(ret0,ret1,ret2)

Check outfixture abstraction to see how it works.

Once the concepts of module, config and code generation are clear, extending Zanzibar becomes straightforward. There are two ways to extend Zanzibar.

To extend Zanzibar with new module class or type is simply to extend each of its three pillars. For example, we want to add a newtask module class to abstract common business workflow, here is what we need to do for each pillar:

• module: understand what meta information is needed for each task module;
• config: add atasks directory under the application root directory, define proper schema for task module class;
• code generation: add templates for task if necessary, create a code generator that implements theBuildGenerator interface andregister it onto the module system for the task module class.

The same idea applies for adding new types of an existing module class.

Zanzibar provides post-generationhooks which has access to the meta information of all modules. You can do whatever (mutating the input is probably not a good idea) suits your needs within a post-generation hook. Zanzibar invokes post-generation hooks as the very last step of code generation. In fact, mocks are all generated via post-generationhooks.

mkdir -p $GOPATH/src/github.com/ubergit clone git@github.com:uber/zanzibar $GOPATH/src/github.com/uber/zanzibarcd $GOPATH/src/github.com/uber/zanzibarGO111MODULE=off make install

make generate

make test

for i in `seq 5`; do make bench; done

First fetchwrk

git clone https://github.com/wg/wrk ~/wrkcd ~/wrkmakesudo ln -s $HOME/wrk/wrk /usr/local/bin/wrk

Then you can run the benchmark comparison script

# Assume you are on feature branch ABC./benchmarks/compare_to.sh master

First create log dir...

sudo mkdir -p /var/log/my-gatewaysudo chown $USER /var/log/my-gatewaychmod 755 /var/log/my-gatewaysudo mkdir -p /var/log/example-gatewaysudo chown $USER /var/log/example-gatewaychmod 755 /var/log/example-gateway

make run# Logs are in /var/log/example-gateway/example-gateway.log

We use glide @ 0.12.3 to add dependencies.

Downloadglide @ 0.12.3and make sure it's available in your path

If we want to add a dependency:

• Add a new section to the glide.yaml with your package and version
• runglide up --quick
• check in theglide.yaml andglide.lock

If you want to update a dependency:

• Change theversion field in theglide.yaml
• runglide up --quick
• check in theglide.yaml andglide.lock