Repository files navigation

Hassle-free minimal CI/CD for git repos for docker-based projects.

Features:

• zero configuration for repos by default
• automatic encrypted backup and recover via different providers including plain files or S3
• optional automatic TLS by Let's Encrypt
• optional automatic domain registration by supported providers
• minimal additional overhead
• multiple repos at once without ports conflicts

git-pipe does for you:

1. Clone/fetch remote repository
2. Detectpackaging type
3. Build package
4. Restorebackup (if applicable)
5. Starts container(s)
6. Createsproxy router
7. (optional) RegistersDNS
8. (optional) GeneratesTLS certificates by Let's Encrypt HTTP-01 ACME
9. (background) Regularly createsbackup
10. Starts from (1) in case something changes in repo

For installation from binaries:

git-pipe run https://github.com/kassambara/wordpress-docker-compose.git

Or for docker installation:

docker run -p 127.0.0.1:8080:80 -v /var/run/docker.sock:/var/run/docker.sock reddec/git-pipe run https://github.com/kassambara/wordpress-docker-compose.git

Where:

• -p 127.0.0.1:8080:80 - docker instruction to expose port 8080 to localhost
• -v /var/run/docker.sock:/var/run/docker.sock - expose docker control socket to git-pipe
• https://github.com/kassambara/wordpress-docker-compose.git - repo to pull and build (literally I picked just randomone. Could be several repos)

Checkusage section for details.

Wait a bit to finish building and go to

• http://wordpress.wordpress-docker-compose.localhost:8080 - wordpress app
• http://phpmyadmin.wordpress-docker-compose.localhost:8080 - for phpmyadmin app

Index page automatically generated for unknown domain. Ex:http://localhost:8080

• linux - high priority
• darwin - (i-wish-i-had-a-mac priority) should work...
• windows - (community support) maybe works, never tested but compiled

• zero-deps: replace OpenSSL, git, ssh and docker-compose to Go-native variants
• file config: support file-based per repo configurations
• authorization:
  • by JWT
  • OIDC
• support dynamic reconfiguration (over API/by file watch + signal)
• support GitHub-like webhooks
• lazy initialization (ie: bring up service on request only)
• path routing as alternative to domain-based

• docker
• docker-compose
• git
• openssl - for backup en(de)cryption

During the first deployment, the following images will be downloaded automatically from docker repository

• busybox

Download binary for your OS and arch fromgithub releases.

Versions

• reddec/git-pipe:<version> - all-in-one image, Alpine based
• reddec/git-pipe:<version>-light - without docker-compose

To download the latest version use:

docker pull reddec/git-pipe:latest

Download and install required .deb file fromgithub releases.

It is highly recommended to installdocker anddocker-compose from the official Docker repository instead of APT. APT repos could be very outdated.

Requires docker-compose.yaml or docker-compose.yaml file in the root directory.Seespecific configuration details;

Flow:

• build equal todocker-compose build
• start equal todocker-compose up

Requires Dockerfile in the root directory. Will be executed as-is.

Flow:

• build equal todocker build
• start equal todocker run

tested on docker-compose 1.27

• Deploys all services.
• All ports inports directive will be linked as sub-domains
• Each service with at least one port supports an optionaldomainname attribute which overrides sub-domain. Default isservice name.
• First services with attributex-root: yes or with namewww,web,gateway will be additionally exposed withoutsub-domain.
• All exposed ports will be additionally exposed as sub-sub-domain with port name as the name.
• Volumes automatically backup-ed and restored (see Backup)
• Root port for service picked by the same rules as fordocker

Domains will be generated as><port?>.<x-domain|service>.<x-domain|project>.<root-domain>and<x-domain|project>.<root-domain> points to<first x-root: true|www|web|gateway>

version:'3'services:web:image:nginxports:      -8080:80      -8081:9000api:image:hashicorp/http-echocommand:-listen :80 -text "web"ports:      -8082:80

Repo name: github.com/example/mini

Generated mapping (root domain (-d,--domain,$DOMAIN) islocalhost):

• web.mini.localhost - points toweb service to internal port80 (the first port in array)
• 80.web.mini.localhost - same
• 9000.web.mini.localhost - points toweb service to internal port9000
• api.mini.localhost - pointsapi service to internal port80
• 80.api.mini.localhost - same

Root domain:mini.localhost points toweb service to internal port80 (the first service with nameweb, firstport in array)

version:'3'services:web:domainname:indeximage:nginxports:      -8080:80      -8081:9000api:domainname:echox-root:yesimage:hashicorp/http-echocommand:-listen :80 -text "web"ports:      -8082:80

Repo name: github.com/example/mini

Generated mapping (root domain (-d,--domain,$DOMAIN) islocalhost):

• index.super.localhost - points toweb service to internal port80 (the first port in array)
• 80.index.super.localhost - same
• 9000.index.super.localhost - points toweb service to internal port9000
• echo.super.localhost - pointsapi service to internal port80
• 80.echo.super.localhost - same

Root domain:super.localhost points toapi service to internal port80 (the first service withx-root: yes,first port in array)

For the single Dockerfile setup:

• All definedVOLUME section in Dockerfile will be added to the archive.

For docker-compose setup:

• All non-external, local (driverlocal or empty) volumes defined involumes: section in full notation will be addedto archive.

Backup interval defined by-I,--backup-interval,$BACKUP_INTERVAL and by default equal to1h (every 1 hour).

The default encryption is symmetric AES-256 CBC done by OpenSSL. Encryption key defined in--backup-key,-K,$BACKUP_KEY andby-default equal togit-pipe-change-me.

Restore will be doneautomatically before the first run.

Defined by-B,--backup,$BACKUP. Default isfile://backups

• file://<directory> - archive in directory. Creates temp (.!tmp suffix) during backup.
• s3://<id>:<secret>@<endpoint>/<bucket>[?params] - upload/download to/from S3-like storage
• <empty> ornone - disable backup

S3 query params:

The bucket should be created by an administrator.

• force_path=true|false, defaultfalse - force use path style for buckets. Required for Minio
• region=<name>, defaultus-west-1 - region
• disable_ssl=true|false, defaultfalse, disable SSL for endpoint

Example for local Minio:

Launch minio:docker run -p 9000:9000 minio/minio server /data

Backup URL:s3://minioadmin:minioadmin@127.0.0.1:9000/backups?force_path=true&disable_ssl=true

Example for BackBlaze (B2):

• Obtain id and secret fromadmin panel
• Copy information about region and bucket namein dashboard

Backup URL:s3://<id>:<secret>@s3.<region>.backblazeb2.com/<bucket name>

(B2) There is some lag between backup and availability to download. Usually, it's around 2-5 minutes for me.

git-pipe usesgit executable so all configuration from~/.git is supported.

It is a good idea to generate deployment SSH keys with read-only access for production usage, however, it is notmandatory.

git-pipe [flags..] <repo, ...>

Seeusage for a list of all available flags.

Localhost example:

git-pipe run https://github.com/kassambara/wordpress-docker-compose.git

Expose to the public:

git-pipe run -b 0.0.0.0:8080 https://github.com/kassambara/wordpress-docker-compose.git

Public and with Let's Encrypt certificates:

git-pipe run --auto-tls https://github.com/kassambara/wordpress-docker-compose.git

--auto-tls implies binding to0.0.0.0:443 and automatic certificates by HTTP-01 ACME protocol.

The node should be accessible from the public internet by 443 port and routed by the domain name. Generally, there aretwo universal methods of how to route traffic from the unknown amount of domains to the machine:

1. Route wildcard* sub-domain to the node and use the sub-domain as root domain in git-pipe. For example: forwildcard domain*.apps.mydomain.com, git-pipe should be launched with flag-d apps.mydomain.com
2. Use automatic DNS registration fromproviders

Version:

• reddec/git-pipe:<version> - all-in-one image, Alpine based
• reddec/git-pipe:<version>-light - without docker-compose

Basic

docker run -p 80:80 -v /var/run/docker.sock:/var/run/docker.sock reddec/git-pipe run <flags same as for bin>

Expose to the public with TLS

It's better to havewildcard certificate.

In./certs should be fileserver.key andserver.crt.

docker run -p 443:443 -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/certs:/app/ssl reddec/git-pipe run --tls <flags same as for bin>

Automatic TLS

Uses Let's Encrypt ACME HTTP-01 protocol.

docker run -p 443:443 -v /var/run/docker.sock:/var/run/docker.sock reddec/git-pipe run --auto-tls <flags same as for bin>

Private repos

Feel free to mount SSH socket:

docker run -p 80:80 -v /var/run/docker.sock:/var/run/docker.sock -v $SSH_AUTH_SOCK:/ssh-agent -e SSH_AUTH_SOCK=/ssh-agent reddec/git-pipe run ...

By default, SSH will be used without strict host checking. To harden pulling you may mount your own configto/root/.ssh/config.

/app/backups - default directory for backups. Will not be used in case of non-file (ex: S3) backup. Without S3 itmakes sense to persist this volume.

/app/repos - default directory for cloned repository. It is not critical to persist this volume because git-pipe canre-download repos anytime.

/app/ssl - default directory for certificates. In the case ofauto-tls it will be used to cache keys and certs, so Ihighly recommended to persist this volume to prevent hitting rate-limit from Let's Encrypt.

In case you are using your certificates, you should them asserver.key andserver.crt and you may mount them inread-only mode.

git-pipe will pass the environment to the packs by prefix: where prefix is repo name (simple or FQDN - depends on setup)in upper case with dash replaced to underscore. Passed keys will be trimmed from suffix:TINC_BOOT_X_Y_Z will bepassed asX_Y_Z.

Environment variables can be passed by system-level and/or from file-e, --env-file path/to/file. Env files can be defined several times.Each next file overwrites the previous value with the same key. Latest goes system environment, which means that system'senvironment variables have the highest priority.

Basic example:

• repo:https://example.com/example/my-example.git

Let's guess that the application needs a database URL which you don't want to expose in the repo. App needs variableDB_URL.

By default, we need to pass it asMY_EXAMPLE_DB_URL=something because

• Repo name ismy-example which converted toMY_EXAMPLE_ prefix
• Variable nameDB_URL

In case you used--fqdn you should specify the full name of repo:MY_EXAMPLE.EXAMPLE.EXAMPLE.COM_DB_URL.

Trivial: just use environment variables as-is.

To use env variables in composeusevariables substitution:

version:'3'services:app:image:my-app:latestenvironment:DB_URL:"${DB_URL:-localhost}"

Router (proxy) provides reverse-proxy concept.

-D, --dummy, $DUMMY disables router completely. Could be useful for services deployed without HTTP services.

By-default, each repo and service deployed as separated domain. Root domain can be defined in-d,--domain,$DOMAIN.

In case multiple domains is not an option the path-based routing can be useful. Enabled byflag-P,--path-routing,$PATH_ROUTING which means that services will be under the same domain, but under different pathprefixes. In this mode,--domain flag will not be used for services name,however, it still required for automatic TLS.

git-pipe uses domain-based routing system which means that all exposed deployed containers will be externally accessibleby unique domain.

To support automatic TLS certificates and DNS routing allocated domains should be routed by the DNS provider. It couldbe done in several ways:

1. Wildcard* sub-domain pointed to the git-pipe node with the sub-domain as root domain in git-pipe. For example: forwildcard domain*.apps.mydomain.com, git-pipe should be launched with flag-d apps.mydomain.com
2. With selected DNS providers it's possible to register domains automatically:use flag-p, --provider, $PROVIDER and provider-specific flags.

Provider name:cloudflare

Requires API-Token for the zone in which you want to register sub-domains.

Enable by:

-d MYDOMAIN -p cloudflare --cloudflare.api-token XXXXX

WhereMYDOMAIN is your root domain which will be added to allapps;XXXXXCloudflare API token

-d, --domain, $DOMAIN <root domain name> is theoretically optional in case you hard-coded root domains inmanifest, but I guess it's not a common situation and should be avoided in most setups.

Options:

• --cloudflare.ip <IP> ($CLOUDFLARE_IP) - Public IP address for DNS record. If not defined - will be detectedautomatically by myexternalip.com
• --cloudflare.proxy ($CLOUDFLARE_PROXY) - Let Cloudflare proxy traffic. Implies some level of protection andautomatic SSL between client and Cloudflare
• --cloudflare.api-token <TOKEN> ($CLOUDFLARE_API_TOKEN) -Cloudflare API token

It's possible to secure exposed endpoints by JWT.

Important: index endpoint not secured by authorization, however, it can expose only list of deployed applications.Use--no-index flag to disable index page.

Supported claims:

• nbf - not before
• exp - expiration time. Permanent if not defined
• sub - restrict to single group (repo) by name (or FQDN if--fqdn set)
• aud - (required) client name (will be used for statistics and probably for lock)
• methods - list of allowed methods: GET, POST, etc.. Case-insensitive, all methods allowed if list is empty orundefined.

To enable JWT authorization use--jwt <key> flag, where<key> shared key used for signing tokens.

Tokens can be (sorted by priority):

• in headerAuthorization withBearer (case-insensitive) kind
• in query paramtoken

For example you have tokeneyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJjbGllbnQxIn0.tj2xpg4u-IHzqXtjfpmI8QUFKQIQUrxPdCQY4JSfCWIso cURL requests may be:

• in header:curl -H 'Authorization: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJjbGllbnQxIn0.tj2xpg4u-IHzqXtjfpmI8QUFKQIQUrxPdCQY4JSfCWI' http://app.example.com/
• in query:curl http://app.example.com/?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJjbGllbnQxIn0.tj2xpg4u-IHzqXtjfpmI8QUFKQIQUrxPdCQY4JSfCWI

Checkgit-pipe jwt command inusage.

Example:

git-pipe jwt -s changeme my-client-1

will generate and print token labeled asmy-client-1 without expiration, for all groups (repos), for all methods,assuming that shared key ischangeme.

Example with restriction to single group (repo):

git-pipe jwt -s changeme -g my-app my-client-1

Example with restriction to single group (repo) and only for GET and HEAD methods:

git-pipe jwt -s changeme -g my-app -m GET -m HEAD my-client-1

Example with expiration after 3 hours 20 minutes and 5 seconds:

git-pipe jwt -s changeme -e 3h20m5s my-client-1

Example how to generate for multiple same clients:

git-pipe jwt -s changeme my-client-1 my-client-2 my-client-3 my-client-4

git-pipe supports health checks in single Dockerfile repositories. It will not route traffic to the service untilcontainer will become healthy.

Seehow to define health check in Dockerfile.

Example for common HTTP service:

FROM my-serviceEXPOSE 80## ....HEALTHCHECK --interval=3s CMD curl -f http://localhost:80/health || exit 1## ...

• jwt - helper to generate JWT
• run - (default) run git-pipe and serve repos

Usage:  git-pipe [OPTIONS] jwt [jwt-OPTIONS] [name...]Help Options:  -h, --help            Show this help message[jwt command options]      -s, --secret=     Shared JWT secret [$SECRET]      -g, --group=      Allowed group (repo name) [$GROUP]      -e, --expiration= Expiration time [$EXPIRATION]      -m, --methods=    Allowed HTTP methods [$METHODS][jwt command arguments]  name:                 Client names for each token will be generated

Usage:  git-pipe [OPTIONS] run [run-OPTIONS] [git-url...]Help Options:  -h, --help                      Show this help message[run command options]      -d, --domain=               Root domain, default is hostname (default:                                  localhost) [$DOMAIN]      -D, --dummy                 Dummy mode disables HTTP router [$DUMMY]      -b, --bind=                 Address to where bind HTTP server (default:                                  127.0.0.1:8080) [$BIND]      -T, --auto-tls              Automatic TLS (Let's Encrypt), ignores bind                                  address and uses 0.0.0.0:443 port [$AUTO_TLS]          --tls                   Enable HTTPS serving with TLS. TLS files                                  should support multiple domains, otherwise                                  path-routing should be enabled. Ignored with                                  --auto-tls' [$TLS]          --ssl-dir=              Directory for SSL certificates and keys.                                  Should contain server.{crt,key} files unless                                  auto-tls enabled. For auto-tls it is used as                                  cache dir (default: ssl) [$SSL_DIR]          --no-index              Disable index page [$NO_INDEX]      -P, --path-routing          Enable path routing instead of domain-based.                                  Implicitly disables --domain [$PATH_ROUTING]          --jwt=                  Define JWT secret and enable JWT-based                                  authorization [$JWT]      -n, --network=              Network name for internal communication                                  (default: git-pipe) [$NETWORK]      -i, --interval=             Interval to poll repositories (default: 30s)                                  [$INTERVAL]      -o, --output=               Output directory for clone (default: repos)                                  [$OUTPUT]      -B, --backup=               Backup location (default: file://backups)                                  [$BACKUP]      -K, --backup-key=           Backup key (default: git-pipe-change-me)                                  [$BACKUP_KEY]      -I, --backup-interval=      Backup interval (default: 1h)                                  [$BACKUP_INTERVAL]      -F, --fqdn                  Construct from URL unique FQDN based on path                                  and domain [$FQDN]          --graceful-shutdown=    Interval before server shutdown (default:                                  15s) [$GRACEFUL_SHUTDOWN]      -e, --env-file=             Environment variables files [$ENV_FILE]      -p, --provider=[cloudflare] DNS provider for auto registration [$PROVIDER]    Cloudflare config:          --cloudflare.ip=        Public IP address for DNS record. If not                                  defined - will be detected automatically by                                  myexternalip.com [$CLOUDFLARE_IP]          --cloudflare.proxy      Let Cloudflare proxy traffic. Implies some                                  level of protection and automatic SSL between                                  client and Cloudflare [$CLOUDFLARE_PROXY]          --cloudflare.api-token= API token [$CLOUDFLARE_API_TOKEN][run command arguments]  git-url:                        remote git URL to poll with optional                                  branch/tag name after hash