Repository files navigation

	Deis Workflow is no longer maintained. Pleaseread the announcement for more detail.
09/07/2017	Deis Workflowv2.18 final release before entering maintenance mode
03/01/2018	End of Workflow maintenance: critical patches no longer merged
	Hephy is a fork of Workflow that is actively developed and accepts code contributions.

Deis (pronounced DAY-iss) Workflow is an open source Platform as a Service (PaaS) that adds a developer-friendly layer to anyKubernetes cluster, making it easy to deploy and manage applications on your own servers.

We welcome your input! If you have feedback, please submit anissue. If you'd like to participate in development, please read the "Development" section below and submit apull request.

The Deis router handles ingress and routing of HTTP/S traffic bound for the Deis Workflow controller (API) and for your own applications. This component is 100% Kubernetes native and, while it's intended for use with the Deis WorkflowPaaS, it's flexible enough to be used standalone inside any Kubernetes cluster.

The Deis project welcomes contributions from all developers. The high level process for development matches many other open source projects. See below for an outline.

• Fork this repository
• Make your changes
• Submit apull request (PR) to this repository with your changes, and unit tests whenever possible.
  • If your PR fixes anyissues, make sure you write Fixes #1234 in your PR description (where #1234 is the number of the issue you're closing)
• The Deis core contributors will review your code. After each of them sign off on your code, they'll label your PR withLGTM1 andLGTM2 (respectively). Once that happens, they'll merge it.

This section documents simple procedures for installing the Deis Router for evaluation or use. Those wishing to contribute to Deis Router development might consider the more developer-oriented instructions in theHacking Router section.

Deis Router can be installed with or without the rest of the Deis Workflow platform. In either case, begin with a healthy Kubernetes cluster. Kubernetes getting started documentation is availablehere.

Next, install theHelm package manager, then use the commands below to initialize that tool and install all of Deis Workflow or the Deis Router by itself.

$ helm init

To install all of Deis Workflow:

$ helm repo add workflow https://charts.deis.com/workflow$ helm install workflow/workflow --namespace deis

Or to install the Deis Router by itself:

$ helm repo add router https://charts.deis.com/router$ helm install router/router --namespace deis

For next steps, skip ahead to theHow it Works andConfiguration Guide sections.

The only dependencies for hacking on / contributing to this component are:

• git
• make
• docker
• kubectl, properly configured to manipulate a healthy Kubernetes cluster that you presumably use for development
• Your favorite text editor

Although the router is written in Go, you donot need Go or any other development tools installed. Any parts of the developer workflow requiring tools not listed above are delegated to a containerized Go development environment.

The following sections setup, build, deploy, and test the Deis Router. You'll need a configured Docker registry to push changed images to so that they can be deployed to your Kubernetes cluster. You can easily make use of a public registry such ashub.docker.com, provided you have an account. To do so:

$ export DEIS_REGISTRY=registry.hub.docker.com/$ export IMAGE_PREFIX=your-username

The entire developer workflow for anyone hacking on the router is implemented as a set ofmake targets. They are simple and easy to use, and collectively provide a workflow that should feel familiar to anyone who has hacked on Deis v1.x in the past.

To "bootstrap" the development environment:

$ make bootstrap

In router's case, this step carries out some extensive dependency management using glide within the containerized development environment. Because the router leverages the Kubernetes API, which in turn has in excess of one hundred dependencies, this step can take quite some time.Be patient, and allow up to 20 minutes. You generally only ever have to do this once.

$ make build

Make sure to have defined the variableDEIS_REGISTRY previous to this step, as your image tags will be prefixed according to this.

$ make deploy

The deploy target will implicitly build first, then push the built image to your development registry (i.e. that specified byDEIS_REGISTRY). The router'sexisting KubernetesDeployment (installed via Helm) will be updated to use the newly built image.

To see that the router is running, you can look for its pod(s):

$ kubectl get pods --namespace=deis

To deploy some sample routable applications:

$ helm install charts/examples --namespace router-examples

This will deploy Nginx and Apache to your Kubernetes cluster as if they were user applications.

To test, first modify your/etc/hosts such that the following four hostnames are resolvable to the IP of the Kubernetes node that is hosting the router:

• nginx.example.com
• apache.example.com
• httpd.example.com
• unknown.example.com

By requesting the following three URLs from your browser, you should find that one is routed to a pod running Nginx, while the other two are routed to a pod running Apache:

• http://nginx.example.com
• http://apache.example.com
• http://httpd.example.com

Requestinghttp://unknown.example.com should result in a 404 from the router since no route exists for that domain name.

The router is implemented as a simple Go program that manages Nginx and Nginx configuration. It regularly queries the Kubernetes API for services labeled withrouter.deis.io/routable: "true". Such services are compared to known services resident in memory. If there are differences, new Nginx configuration is generated and Nginx is reloaded.

Routable services must expose port 80. The target port in underlying pods may be anything, but the service itself must expose port 80. For example:

apiVersion: v1kind: Servicemetadata:  name: foo  labels:  router.deis.io/routable: "true"  namespace: router-examples  annotations:    router.deis.io/domains: www.foobar.com  spec:    selector:      app: foo    ports:    - port: 80      targetPort: 3000# ...

When generating configuration, the program reads all annotations of each service prefixed withrouter.deis.io. These annotations describe all the configuration options that allow the program to dynamically construct Nginx configuration, including virtual hosts for all the domain names associated with each routable application.

Similarly, the router watches the annotations on itsown deployment object to dynamically construct global Nginx configuration.

Router configuration is driven almost entirely by annotations on the router's deployment object and the services of all routable applications-- those labeled withrouter.deis.io/routable: "true".

One exception to this, however, is that in order for the router to discover its own annotations, the router must be configured via environment variable with some awareness of its own namespace. (It cannot query the API for information about itself without knowing this.)

ThePOD_NAMESPACE environment variable is required by the router and it should be configured to match the Kubernetes namespace that the router is deployed into. If no value is provided, the router will assume a value ofdefault.

For example, consider the following Kubernetes manifest. Given a manifest containing the following metadata:

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: deis-router  namespace: deis# ...

The corresponding template must inject aPOD_NAMESPACE=deis environment variable into router containers. The most elegant way to achieve this is by means of the Kubernetes "downward API," as in this snippet from the same manifest:

# ...spec:  # ...  template:    # ...    spec:      containers:      - name: deis-router        # ...        env:        - name: POD_NAMESPACE          valueFrom:            fieldRef:              fieldPath: metadata.namespace# ...

Altering the value of thePOD_NAMESPACE environment variable requires the router to be restarted for changes to take effect.

All remaining options are configured through annotations. Any of the following three Kubernetes resources can be configured:

The table below details the configuration options that are available for each of the above.

Note that Kubernetes annotation maps are all of Go typemap[string]string. As such, all configuration values must also be strings. To avoid Kubernetes attempting to populate themap[string]string with non-string values, all numeric and boolean configuration values should be enclosed in double quotes to help avoid confusion.

apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: deis-router  namespace: deis  # ...  annotations:    router.deis.io/nginx.platformDomain: example.com    router.deis.io/nginx.useProxyProtocol: "true"# ...

apiVersion: v1kind: Servicemetadata:  name: deis-builder  namespace: deis  # ...  annotations:    router.deis.io/nginx.connectTimeout: "20000"    router.deis.io/nginx.tcpTimeout: "2400000"# ...

apiVersion: v1kind: Servicemetadata:  name: foo  labels:  router.deis.io/routable: "true"  namespace: router-examples  # ...  annotations:    router.deis.io/domains: foo,bar,www.foobar.com# ...

Router has support for HTTPS with the ability to perform SSL termination using certificates supplied via Kubernetes secrets. Just as router utilizes the Kubernetes API to discover routable services, router also uses the API to discover cert-bearing secrets. This allows the router to dynamically refresh and reload configuration whenever such a certificate is added, updated, or removed. There is never a need to explicitly restart the router.

A certificate may be supplied in the manner described above and can be used to provide a secure virtual host (in addition to the insecure virtual host) for anyfully-qualified domain name associated with a routable service.

Here is an example of a Kubernetes secret bearing a certificate for use with a specific fully-qualified domain name. The following criteria must be met:

• Secret name must be for the form<arbitrary name>-cert
  • This must be associated to the domain using therouter.deis.io/certificates annotation.
• Must be in the same namespace as the routable service
• Certificate must be supplied as the value of the keytls.crt
• Certificate private key must be supplied as the value of the keytls.key
• Both the certificate and private key must be base64 encoded

For example, assuming a routable service exists in the namespacecheery-yardbird and is configured withwww.example.com among its domains, like so:

apiVersion: v1kind: Servicemetadata:  namespace: cheery-yardbird  annotations:    router.deis.io/domains: cheery-yardbird,www.example.com    router.deis.io/certificates: www.example.com:www-example-com"# ...

The corresponding cert-bearing secret would appear as follows:

apiVersion: v1kind: Secretmetadata:  name: www-example-com-cert  namespace: cheery-yardbirdtype: Opaquedata:  tls.crt: MT1...uDh==  tls.key: MT1...MRp=

A wildcard certificate may be supplied in a manner similar to that described above and can be used as a platform certificate to provide a secure virtual host (in addition to the insecure virtual host) forevery "domain" of a routable service that is not a fully-qualified domain name.

For instance, if a routable service exists having a "domain"frozen-wookie and the router's platform domain isexample.com, a supplied wildcard certificate for*.example.com will be used to secure afrozen-wookie.example.com virtual host. Similarly, if no platform domain is defined, the supplied wildcard certificate will be used to secure a virtual host matching the expression~^frozen-wookie\.(?<domain>.+)$. (The latter is almost certainly guaranteed to result in certificate warnings in an end user's browser, so it is advisable to always define the router's platform domain.)

If the same routable service also had a domainwww.frozen-wookie.com, the*.example.com wildcard certificate plays no role in securing thewww.frozen-wookie.com virtual host.

Here is an example of a Kubernetes secret bearing a wildcard certificate for use by the router. The following criteria must be met:

• Namespace must be the same namespace as the router
• Namemust bedeis-router-platform-cert
• Certificate must be supplied as the value of the keytls.crt
• Certificate private key must be supplied as the value of the keytls.key
• Both the certificate and private key must be base64 encoded

For example:

apiVersion: v1kind: Secretmetadata:  name: deis-router-platform-cert  namespace: deistype: Opaquedata:  tls.crt: LS0...tCg==  tls.key: LS0...LQo=

When combined with a good certificate, the router'sdefault SSL options are sufficient to earn an A grade fromQualys SSL Labs.

Earning an A+ is as easy as simply enabling HTTP Strict Transport Security (see therouter.deis.io/nginx.ssl.hsts.enabled option), but be aware that this will implicitly trigger therouter.deis.io/nginx.ssl.enforce option and cause your applications to permanently use HTTPS forall requests.

Depending on what distribution of Kubernetes you use and where you host it, installation of the routermay automatically include an external (to Kubernetes) load balancer or similar mechanism for routing inbound traffic from beyond the cluster into the cluster to the router(s). For example,kube-aws andGoogle Container Engine both do this. On some other platforms-- Vagrant or bare metal, for instance-- this must either be accomplished manually or does not apply at all.

If a load balancer such as the one described above does exist (whether created automatically or manually)and if you intend on handling any long-running requests, the load balancer (or similar)may require some manual configuration to increase the idle connection timeout. Typically, this is most applicable to AWS and Elastic Load Balancers, but may apply in other cases as well. It doesnot apply to Google Container Engine, as the idle connection timeout cannot be configured there, but also works fine as-is.

If, for instance, router were installed on kube-aws, in conjunction with the rest of the Deis Workflow platform, this timeout should be increased to a recommended value of 1200 seconds. This will ensure the load balancer does not hang up on the client during long-running operations like an application deployment. Directions for this can be foundhere.

If using a Kubernetes distribution or underlying infrastructure that does not support the automated provisioning of a front-facing load balancer, operators will wish to manually configure a load balancer (or use other tricks) to route inbound traffic from beyond the clusterinto the cluster to the platform's own router(s). There are many ways to accomplish this. The remainder of this section discusses three general options for accomplishing this.

This manually replicates the configuration that would be achieved automatically with some distributions on some infrastructure providers, as discussed above.

First, determine the "node ports" for thedeis-router service:

$ kubectl describe service deis-router --namespace=deis

This will yield output similar to the following:

...Port:http80/TCPNodePort:http32477/TCPEndpoints:10.2.80.11:80Port:https443/TCPNodePort:https32389/TCPEndpoints:10.2.80.11:443Port:builder2222/TCPNodePort:builder30729/TCPEndpoints:10.2.80.11:2222Port:healthz9090/TCPNodePort:healthz31061/TCPEndpoints:10.2.80.11:9090...

The node ports shown above are high-numbered ports that are allocated onevery Kubernetes worker node for use by the router service. The kube-proxy component onevery Kubernetes node will listen on these ports and proxy traffic through to the corresponding port within an "endpoint--" that is, a pod running the Deis router.

If manually creating a load balancer, configure the load balancer to haveall Kubernetes worker nodes in the back-end pool, and listen on ports 80, 443, and 2222 (port 9090 can be ignored). Each of these listeners should proxy inbound traffic to the corresponding node ports on the worker nodes. Ports 80 and 443 may use either HTTP/S or TCP as protocols. Port 2222 must use TCP.

With this configuration, the path a request takes from the end-user to an application pod is as follows:

user agent (browser) --> front-facing load balancer --> kube-proxy on _any_ Kubernetes worker node --> _any_ Deis router pod --> kube-proxy on that same node --> _any_ application pod

Option 2 differs only slightly from option 1, but is more efficient. As such, even operators who had a front-facing load balancer automatically provisioned on their infrastructure by Kubernetes might consider manually reconfiguring that load balancer as follows.

Deis router pods will listen onhost ports 80, 443, 2222, and 9090 wherever they run. (They will not run on any worker nodes where all of these four ports are not available.) Taking advantage of this, an operator may completely dismiss the node ports discussed in option 1. The load balancer can be configured to haveall Kubernetes worker nodes in the back-end pool, and listen on ports 80, 443, and 2222. Each of these listeners should proxy inbound traffic to thesame ports on the worker nodes. Ports 80 and 443 may use either HTTP/S or TCP as protocols. Port 2222 must use TCP.

Additionally, a health checkmust be configured using the HTTP protocol, port 9090, and the/healthz endpoint. With such a health check in place,only nodes that are actually hosting a router pods will pass and be included in the load balancer's pool of active back end instances.

With this configuration, the path a request takes from the end-user to an application pod is as follows:

user agent (browser) --> front-facing load balancer --> a Deis router pod --> kube-proxy on that same node --> _any_ application pod

Option 3 is similar to option 2, but does not actually utilize a load balancer at all. Instead, a DNS A record may be created that lists the public IP addresses ofall Kubernetes worker nodes. This will leverage DNS round-robining to direct requests to all nodes. To guaranteeall nodes can adequately route incoming traffic, the Deis router component should be scaled out by increasing the number of replicas specified in the deployment object to match the number of worker nodes. Anti-affinity should ensure exactly one router pod runs per worker node.

This configuration is not suitable for production. The primary use case for this configuration is demonstrating or evaluating Deis Workflow on bare metal Kubernetes clusters without incurring the effort to configure anactual front-facing load balancer.

The Helmchart available for installing router (either with or without the rest of Deis Workflow) is intended to get users up and running as quickly as possible. As such, the chart does not strictly require any editing prior to installation in order to successfully bootstrap a cluster. However, there are some useful customizations that should be applied for use in production environments:

• Specify aplatform domain. Without a platform domain specified, any routable service specifying one or more non-fully-qualified domain names (not containing any. character) among itsrouter.deis.io/domains will be matched using a regular expression of the form^{{ $domain }}\.(?<domain>.+)$ where{{ $domain }} resolves to the non-fully-qualified domain name. By way of example, the idiosyncrasy that this exposes is that traffic bound for thefoo subdomain ofany domain would be routed to an application that lists the non-fully-qualified domain namefoo among itsrouter.deis.io/domains. While this behavior is not innately wrong, it may not be desirable. To circumvent this, specify aplatform domain. This will cause routable services specifying one or more non-fully-qualified domain names to be matched, explicitly, as subdomains of the platform domain. Apart from remediating this minor idiosyncrasy, this is required in order to properly utilize a wildcard SSL certificate and may also result in a very modest performance improvement.

• Do you need to use SSL tosecure the platform domain?

• If using SSL, generate and provide your own dhparam. A dhparam is a secret key used inDiffie Hellman key exchange during the SSL handshake in order to help ensureperfect forward secrecy. The Helmchart available for installing router (either with or without the rest of Deis Workflow) already includes a dhparam, but recall that dhparams are intended to be secret. The dhparam included in the chart is marginally preferable to using Nginx's default dhparam only because it is lesser-known, but it isstill publicly available in thechart. As such, users wishing to run the router in productionand use SSL are best off generating their own dhparam. After being generated, it should be base64 encoded and included as the value of thedhparam key in a Kubernetes secret nameddeis-router-dhparam in the same namespace as the router itself.

  For example, to generate and base64 encode the dhparam on a Mac:

  $ openssl dhparam -out dhparam.pem 1024$ base64 dhparam.pem

  To generate an even stronger key, use 2048 bits, but note that generating such a key will take a very long time-- possibly hours.

  Include the base64 encoded dhparam in a secret:

  apiVersion: v1kind: Secretmetadata:    name: deis-router-dhparam    namespace: deis    labels:      heritage: deistype: Opaquedata:    dhparam: <base64 encoded dhparam>

• If using SSL, do you need toenforce the use of SSL?

• If using SSL, do you need toenable strict transport security?

• If using SSL, what grade doesQualys SSL Labs give you?

• Should your routerdefine and enforce a default whitelist? This may be advisable for routers governing ingress to a cluster that hosts applications intended for a limited audience-- e.g. applications for internal use within an organization.

• Do you need to scale the router? For greater availability, it's desirable to run more than one instance of the router.How many can only be informed by stress/performance testing the applications in your cluster. To increase the number of router instances from the default of one, increase the number of replicas specified by thedeis-router deployment object. Do not specify a number of replicas greater than the number of worker nodes in your Kubernetes cluster.