Repository files navigation

This repo contains micros services including:

• Payout service (WIP)
• Payin service (WIP)
• Ledger service (WIP)
• Webhook service (WIP)

• Architecture and Tech Stack
• K8S access control
• Deployment
• Monitoring
• Infrastructure
• Development
  • Environment Setup
  • Development Guide
    • Running server locally
    • Running server with docker
  • Update Dependencies
  • Work with secret
  • Live debugging
  • Make commands reference

Thepayment-service is a Kubernetes pod with two containers:

• Theweb container running gunicorn-uviron-fastapi asyncio web service where
  • fastapi running asyncio supported web application
  • uvicorn running on top offastapi web app as an ASGI web server and providing event loop implementation
  • gunicorn running as process manager spawning up multiple uvicorn workers for parallelism
• Theruntime sidecar, used for configuration management

The following technologies/frameworks are used across the entire stack:

• helm for deployments and rollbacks
• docker as container engine
• kubernetes as container orchestrator
• python3.7 for application code
• fastapi as web framework for building asyncio web service
• uvicorn as ASGI web server / interface for contained fastapi service
• gunicorn as process manager of uvicorn worker
• pipenv for dependency management anddeterministic builds
• pytest for unit testing
• mypy for static type checking
• flake8 for linting
• black for code formatting
• pre-commit for pre-commit hooks
• DoorDash doorctl for images tagging/pushing
• DoorDash Pulse for semantic monitoring and system testing
• DoorDash Pressure for load testing
• DoorDash Runtime for configuration management
• Artifactory to store docker images
• Jenkins/Groovy for CICD

• OnCall Playbook
• Wavefront Dashboards
• Sentry Issues
• New Relic

• README-infra.md

Only authorized personnel can access payment-service k8s deployment (i.e. execute kubectl command)

In order to access staging payment-service deployment, you need to request permission inpayments-team@ google group

1. Differently than staging, you need to be inthis list
2. After you are authorized in step 1:

   $ ssh<YOUR USERNAME>@bastion.doordash.com$ kubeswitch payments# switch kube context to payment-service namespace

Payment-service usesddops to control releases and k8s deployment:

• Promote pipeline
• Release build pipeline
• CI pipeline
• Migration pipeline
• Dev CD pipeline (Always deploy merged code to staging only)

• All PRs running through CI steps defined inCI pipeline
• Approved PRs are merged to master branch,without being deployed to k8s cluster.

• Payment-servicedeploy pilots are authorized to perform regular deployment, hotfix, rollback, migration and bounce byddops commands through deploymentslack channel
• Deployment procedures
  • Regular deployment:
    1. /ddops cut-release payment-service -> output:releaseTag
    2. /ddops build payment-service <releaseTag>
    3. /ddops migrate payment-service <releaseTag> --env=staging # Migratestaging ONLY
    4. /ddops migrate payment-service <releaseTag> # Migrateprod ONLY
    5. /ddops promote payment-service <releaseTag> --env=staging # Promotestaging ONLY
    6. /ddops promote payment-service <releaseTag> # Promoteprod ONLY
  • Hotfix:
    1. /ddops add-to-release-branch payment-service <hotfix-sha> payment-service <deployed-release-branch> <hotfix-reason> -> output:releaseTag (example)
    2. /ddops hotfix payment-service <releaseTag> (example)
  • Rollback:
    1. /ddops rollback payment-service <rollback-to-release-tag> <rollback reason> (example)
  • Bounce:
    1. /ddops bounce payment-service # Rolling upgrade to bounce payment-service-web and payment-service-webhook with current live release

As of 1/21/2020 Payment service adopted blue-green deployment for web and webhook k8s services. These are the facts to keep in mind

• As part of blue-green deployment, a k8s rollout resource was created on top of existing k8s service instead of k8s deployment resource. Therefore any blue-green enabled k8s service, use following command to get current rollout:kubectl -n payment-service get rollouts
• Each blue-green deployment will set green (new replica set) to directly serve traffic besides blue (old replica set). After certain amount ofdelay, blue will be deleted.
• In case the green service behave unexpectedly during deployment, use/ddops rollback payment-service <blue release tag> will immediately remove green services and keep existing blue service unchanged.
• Follow thisInstall Argo Rollouts Kubectl Plugin to install local argo-rollout kubectl plugin to get a friendly view of blue-green deployment (argo rollout) process. Example:

kubectl-argo-rollouts get rollout payment-service-web -n payment-serviceName:            payment-service-webNamespace:       payment-serviceStatus:          ◌ ProgressingStrategy:        BlueGreenImages:          611706558220.dkr.ecr.us-west-2.amazonaws.com/payment-service:2af851a235cc327f4de02a89a72f76c2acffb5c2                 611706558220.dkr.ecr.us-west-2.amazonaws.com/payment-service:f5a565c05dc7756a9f0ac2f8b7e7372c17a6d2db (active)                 ddartifacts-docker.jfrog.io/runtime:latest (active)Replicas:  Desired:       2  Current:       4  Updated:       2  Ready:         4  Available:     2NAME                                             KIND        STATUS         AGE   INFO⟳ payment-service-web                            Rollout     ◌ Progressing  121m├──# revision:3│  └──⧉ payment-service-web-5b5bdcc678           ReplicaSet  ✔ Healthy      75s   active│     ├──□ payment-service-web-5b5bdcc678-gzr85  Pod         ✔ Running      75s   ready:2/2│     └──□ payment-service-web-5b5bdcc678-q826k  Pod         ✔ Running      75s   ready:2/2,restarts:1├──# revision:2│  └──⧉ payment-service-web-7c855dcb5b           ReplicaSet  ✔ Healthy      59m│     ├──□ payment-service-web-7c855dcb5b-pbzlk  Pod         ✔ Running      59m   ready:2/2│     └──□ payment-service-web-7c855dcb5b-vvvvj  Pod         ✔ Running      59m   ready:2/2└──# revision:1   └──⧉ payment-service-web-5547bd48cf           ReplicaSet  • ScaledDown   121m

1. Make sure you have followedNew eng setup guide to properly setup your PIP install indices by envionment variablesARTIFACTORY_URL,ARTIFACTORY_USERNAME,ARTIFACTORY_PASSWORD andPIP_EXTRA_INDEX_URL
2. After step #1, also includeFURY_TOKEN in your~/.bash_profile by running:

   echo"export FURY_TOKEN=$(echo$PIP_EXTRA_INDEX_URL| sed -E -e's/.*\/([^/]+):@repo.fury.io\/doordash.*/\1/')">>~/.bash_profilesource~/.bash_profile

1. Install python specified inPipfile.lock (v3.7) throughpyenv andpipenv into a newly created python virtual environment.

   brew install pyenv pipenvbrew upgrade pyenv pipenv# install all dependencies needed for development, including the ones installed with the --dev argument.make sync-pipenv

2. After step #1, a python virtual envionment should be created.

   1. To find where does environment locate, run$ pipenv --venv
   2. To start a sub-shell within create python virtual environment, run:

      pipenv shell# Try `pipenv shell --fancy` if you want to preserve your customized shell configuration from ~/.bashrc

   3. To go back to your original shell and deactivate python virtual env, simply run$ exit

3. To test if everything works, you can try:

   1. Activate python virtual env:$ pipenv shell
   2. running unit tests, linter and mypy:$ make test

1. Installpre-commit and initialize the pre-commit hooks; these will run before every commit

   brew install pre-commitpre-commit install

2. Run Pre-Commit Hooks

   If you want to manually run all pre-commit hooks on a repository, runpre-commit run --all-files. To run individual hooks usepre-commit run <hook_id>.

   Pre-commit<hook_id> are defined in.pre-commit-config.yaml. Example commands:

   pre-commit run --all-filespre-commit run black --all-files

pipenv shellmake local-server

• Local service will be running in development and debug mode.Any code changes will lead to a service reload and therefore be reflected in real-time.
• Confirming local server is up running:

  curl localhost:8000/health# port is defined in Makefile::local-server

1. Activate python virtual env:$ pipenv shell
2. Available test targets:

   maketest# runs unit tests, linter, mypy (not pre-commit hooks)make test-unit# runs unit tests onlymake test-typing# runs mypy onlymake test-lint# runs linter onlymake test-hooks# runs pre-commit hooks only

make start-local-docker-server

• Similarly aslocal-server, ./app directory is binded as a volume under web docker container.Therefore live reload is also available here.
• Confirming server is up running in docker:

  curl localhost:8001/health# docker port mapping defined in docker-compose.yml web container

Refers torun tests locally

We categorize tests cases into 3 groups with in payment-service repo. (Pulse test will be covered separately).

1. Unit tests: Test cases should focus on test unit functionalitywithout any real connections to remote dependencies like database, redis or stripe. Any remote dependencies should be mocked out in unit test case.

   • Make target:make test-unit
   • Directory: within each top level components (e.g. commons, payin, payout, ledger, middleware...), there is atest_unit folder where orresponding unit test files should be created.

2. Integration tests: Test cases should focus on validate integration contract and santity with remote dependencies. Alternatively, if you really have to wire up dependency components to test some integration behavior, you can create test cases as integration test.

   • Make target:
     • make test-integration: run all integration tests including test cases depending on remote dependencies we owned (e.g. DB) and external dependencies like stripe.
     • make test-external: only run test cases marked withpytest.mark.external, which usually are tests depending on external dependencies like stripe.
   • Directory: within each top level components (e.g. commons, payin, payout, ledger, middleware...), there is atest_integration folder where orresponding integration test files should be created.
   • Note:DO NOT place integrate tests outside of atest_integration folder, otherwise our test scripts won't startup corresponding dependencies for the test.

Thepayment-service usespipenv to ensure deterministic builds.To add or update a dependency, you can do the following:

1. Add or update dependency inPipefile via your text editor
2. Run following command to updatePipefile.lock and install from updated lock file

   make update-pipenv

3. After you are done, remember to open a PR to checkin the changes!

Payment-service integrated withNinoxas source of all secret configurations, such as DB credentials and Stripe private keys.

1. Make sure you are in google groupeng-payment@,otherwise please ask one of payment team member to add you in.

2. Fetchokta-prod-payment-eng-user Okta-aws profile via:

   okta-aws init

   In case, you don't haveokta-aws cli installed, followhere

3. Verify you have successfully fetched aws profile by:

   grep okta-prod-payment-eng-user~/.aws/credentials# Expected output>> [okta-prod-payment-eng-user]

4. Install Ninox:

   brew install Ninox

   If fails, ensure you have tapped intodoordash homebrew taps

5. Verify Ninox user role working:

   cd YOUR_PAYMENT_SERVICE_REPOninox -s staging_user config

   You should see output similar to following without errors:

   Loading team staging_user{'backend':'dbd','ignore_entropy_check': False,'kms_key_alias':'alias/ninox/payment-service','prefix':'/ninox/payment-service/','profile':'okta-prod-payment-eng-user','region':'us-west-2','role':'arn:aws:iam::016116557778:role/ninox_payment-service_xacct_user_staging','session': Session(region_name='us-west-2'),'table':'ninox-payment-service-staging'}

6. Ninox secret create, update, retrieve from cli

   1. As a payment engineer, I want to create or update secret

      cd PAYMENT_REPOninox -s [staging_user| prod_user] [create| update]<secret_name_all_lower_case>

      Note: as of 07/25/2019 Ninox cli hasn't supportls for user role yet.

   2. As a payment engineer, I want to see if the secret is created / updated as expected.

      Really not a good practice!!, but if you really want you need to login to one of the staging or prod payment-service-web pod and do:

      ninox -s [staging| prod] get<secret_name_all_lower_case>

      Note: Once we have a better way to validate, will update.

1. Setup your pycharm remote debugger byRun ->Edit Configurations ->+ ->Python Remote Debug
2. Name it asdebug-server and listen tolocalhost:9001 as following:
3. Start up your debugger from menu asRun ->Debug... ->debug-server
4. Same remote debugger can be used to live debug one of local server and test
   1. For local-server:make local-server DEBUGGER=enabled
   2. For tests:make test-unit DEBUGGER=enabled

Here's a reference to all availablemake commands:

make docker-build# use docker to build the service imagemake build-ci-container# build the docker container for CI, using docker-compose.ci.ymlmake run-ci-container# start the docker container for CI in daemon modemake local-server# run service on your local host within python virtual envmake local-docker-server# run service with docker-composemaketest# runs unit tests, linter, mypy (not pre-commit hooks)make test-unit# runs unit tests onlymake test-integration# runs integration tests (incl. database tests) onlymake test-external# runs external tests only (ie. that talk to an external dependency)make test-typing# runs mypy onlymake test-lint# runs linter onlymake test-install-hooks# installs pre-commit hooksmake test-hooks# runs pre-commit hooks only