Movatterモバイル変換

[0]ホーム
URL:

Skip to content
DEV Community
Log in Create account

DEV Community

Cover image for How to deploy django app to ECS Fargate Part2
Jinwook Baek
Jinwook Baek

Posted on

     

How to deploy django app to ECS Fargate Part2

This post was originally published on notion. Clickhere if you prefer to read in notion page which has better readability.

Introduction

This blog post illustrate development cycle using django app container. I assume readers are already somewhat familar with docker and docker-compose. Although I used django for app development, it is language-agnostic since the post is about containerized application deployment.

Walkthough is devided into three parts consisting three different environemt respectively. First part, describes the architecture of the app(api and async worker) and how they are deployed onlocal enviroment. Second part is how to deploy the docker containers on cloud using single ec2 instance with onstaging environment. Third part, illustrate how to convert traditional ec2 deployment into ECS using fargate with github actions onprod environment.

local - run docker containers on desktop/laptop with sqlite and redis server using docker-compose

stating - run docker containers on single ec2 instance with mysql RDS and ElasticCache

prod - convert stagning setup to ECS Fargate

For part1, click here.

You can skip to part3 if you are already familiar with AWS architectures.

Staging Infra Setup

Before going straight to ECS deployment, we will setup up astaging env to test the application using ec2 compute instance and other AWS cloud services. If you are familar with AWS infra and confident with ecs setup, you can skip this part. However I will be using same vpc, redis and mysql forproduction env so it would be worth while to take a look at the setup. (you should use different vpc, redis and mysql for your actualproduction deployment)

Thestaging cloud architecture will consist following AWS services.

  • VPC with public and private subnet
  • EC2 instance (in public subnet)
  • ALB in front of ec2 instance
  • RDS mysql (in private subnet)
  • ElasticCache - Redis (private subnet)

staging infra

Refer to the cloudformation for detailed configurations.

kokospapa8/ecs-fargate-sample-app

  • Preview

    • Network - VPC

      "VpcEcsSample":{"Type":"AWS::EC2::VPC","Properties":{"CidrBlock":"172.10.0.0/16","InstanceTenancy":"default","EnableDnsSupport":"true","EnableDnsHostnames":"true","Tags":[{"Key":"Name","Value":"ecs-sample"}]}},"EcsSamplePrivate1":{"Type":"AWS::EC2::Subnet","Properties":{"CidrBlock":"172.10.11.0/24","AvailabilityZone":"us-west-2a","VpcId":{"Ref":"VpcEcsSample"},"Tags":[{"Key":"Name","Value":"ecs-sample-private 1"}]},"DependsOn":"VpcEcsSample"},"EcsSamplePublic1":{"Type":"AWS::EC2::Subnet","Properties":{"CidrBlock":"172.10.1.0/24","AvailabilityZone":"us-west-2a","VpcId":{"Ref":"VpcEcsSample"},"Tags":[{"Key":"Name","Value":"ecs-sample-public 1"}]},"DependsOn":"VpcEcsSample"},"IgwEcsSample":{"Type":"AWS::EC2::InternetGateway","Properties":{"Tags":[{"Key":"Name","Value":"ecs-sample IGW"}]},"DependsOn":"VpcEcsSample"},"NATGateway":{"Type":"AWS::EC2::NatGateway","Properties":{"AllocationId":{"Fn::GetAtt":["EipNat","AllocationId"]},"SubnetId":{"Ref":"EcsSamplePublic1"}},"DependsOn":["EcsSamplePublic1","EipNat","IgwEcsSample","GatewayAttachment"]},

    • RDS

      "RDSCluster":{"Type":"AWS::RDS::DBCluster","Properties":{"MasterUsername":{"Ref":"DBUsername"},"MasterUserPassword":{"Ref":"DBPassword"},"DBClusterIdentifier":"ecs-sample","Engine":"aurora","EngineVersion":"5.6.10a","EngineMode":"serverless","ScalingConfiguration":{"AutoPause":true,"MinCapacity":4,"MaxCapacity":8,"SecondsUntilAutoPause":1000}}},

    • Redis

      cacheecssample001": {      "Type": "AWS::ElastiCache::CacheCluster",      "Properties": {        "AutoMinorVersionUpgrade": "true",        "AZMode": "single-az",        "CacheNodeType": "cache.t2.micro",        "Engine": "redis",        "EngineVersion": "5.0.6",        "NumCacheNodes": "1",        "PreferredAvailabilityZone": "us-west-1b",        "PreferredMaintenanceWindow": "thu:02:30-thu:03:30",        "CacheSubnetGroupName": {          "Ref": "cachesubnetecssampleredissubnetgroup"        },        "VpcSecurityGroupIds": [          {            "Fn::GetAtt": [              "sgecssampleredis",              "GroupId"            ]          }        ]      }    },

This template is preconfigured tous-west-2 region.

Migrate docker images to ECR

Before setting up the infrastructure I will move docker images toelastic container registry. If you want to still use docker hub or other private docker image repository, you can skip this part

  • ecr setup

    Getting started with Amazon ECR using the AWS CLI

    log into ecr

    $eval$(awsecrget-login-password--regionus-east-2--no-include-email)>LoginSucceeded

    Create repository with cli

    $awsecrcreate-repository--repository-nameecs-sample-nginx--regionus-east-2$awsecrcreate-repository--repository-nameecs-sample-api--regionus-east-2$awsecrcreate-repository--repository-nameecs-sample-worker--regionus-east-2

    or create repository on the console

ecr creation

Build docker image if you have not build the image from previoud post
```bash$ docker build -f config/app/Docker_base -t ecs-fargate-sample_base:latest .$ docker build -f config/app/Docker_app -t ecs-fargate-sample_app:latest .$ docker build -f config/app/Docker_worker -f ecs-fargate-sample_app:latest .```
Tag and push images
```bash$ docker tag ecs-fargate-sample_app:base <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-api:base$ docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-api:base$ docker tag ecs-fargate-sample_app:latest <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-api:latest$ docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-api:latest$ docker tag ecs-fargate-sample_worker:base <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-worker:latest$ docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecs-sample-worker:latest```
Check the console for the images

ecr image pushed

IAM role

You need access to ECR from your ec2 instances, instead of embedding AWS accesskey in the instance I will attach an iam role.

Network

VPC

I have created VPC with CIDR 172.10.0.0/16. refer to the images attached for reference. For production I advise you to make more subnets in multi-AZ for better availability. For the sake of the post, I have just used two available zones (serverless mysql needs at least two AZs)

  • Public subnet

public subnet

CIDR - 172.10.1.0/24, 172.10.2.0/24Components- ec2 instance for `api`- NAT GATEWAY for `worker` instance
  • Private subnet

private subnet

CIDR - 172.10.11.0/24, 172.10.12.0/24Components- mysql- elastic cache- ec2 instance for `worker`

Security group

I have added minial security measure for the sake of the post. This is just sample you should not use this for production.

  • Setup

    ecs-sample-VPC

    • Opened all ports to public (not safe)

    ecs-sample-loadbalancer

    • 80 and 443 to public
    • 80 with ec2-api instance

    ecs-sample-mysql

    • 3306 with ec2

    ecs-sample-ec2-api andecs-sample-ec2-worker (they should be different but I just used single sg for both ec2, should block 80 access for particular ec2 instance)

    • 3306 mysql
    • 6379 redis,
    • 80 for ALB and public ip access
    • 22 for ssh

    ecs-sample-redis

    • 6379 with ec2### ALB

Created Alb with simple setup attached to public subnet. We don't have instance yet so just created target group with no instance attached yet.

  • configuration
    • name: ecs-sanple-lb
    • Listener: port 80
    • Security group: ecs-sample-lb
    • VPC and subnet - public subnet created above

alb config 1

alb result 1

Database

RDS - Mysql

I will use serverless mysql instead of traditional mysql instance to minimize cost. (serverless mysql only support mysql engine version up to 5.6)

  • configuration
    • database -sample
    • username -admin
    • network - VPC and private subnet created above

mysql db

Take a note on your mysql endpoint and password, will be using it later for env variable setup.

Elasticcache - Redis

Take a note on your primary endpoint, will be using it later for env variable setup.

Compute instances

We will start two ec2 instances. One for api in public subnet and another for worker in private sunet.

EC2 instance - API

  • configurations
    • AMI : Ubuntu image 20.04 LTS
    • Network: public subnet
    • auto-assign ip : enabled

ec2 conf1

ec2 conf2

ec2 conf3

Ec2 instance - worker

This instance is basically same withapi instance but it needs to in private subnet. I advise making AMI fromapi instance and fire up on private instance. Since there will be no public ip attached, you need to tunnel throughapi orbastion instance to connect the instance with private ip. (You can usefollowing method to connect directly since we have NAT gateway deployed)

Deployment

Once the instance is up and running, we are ready to deploy docker containers.

Let's ssh into to ec2 instance.

ssh-i"/path/to/keyfile/" ubuntu@<public_ip>

Docker setup

Next, you need to install docker engine and docker-compose. refer to followinglink

  • install steps

    Install Docker Engine on Ubuntu

    Install Docker Compose

    Post-installation steps for Linux

    # docker engine$sudoapt-get update$sudoapt-getinstall\    apt-transport-https\    ca-certificates\    curl\    gnupg-agent\    software-properties-common$curl-fsSL https://download.docker.com/linux/ubuntu/gpg |sudoapt-key add -$sudoadd-apt-repository\"deb [arch=amd64] https://download.docker.com/linux/ubuntu\$(lsb_release-cs)\   stable"$sudoapt-get update$sudoapt-getinstalldocker-ce docker-ce-cli containerd.io# docker-compose$sudocurl-L\"https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname-s)-$(uname-m)"\-o /usr/local/bin/docker-compose

Environment variables

You need to set couple env for staging since we are using redis and mysql.

Please take a closer look atsettings/staging.py forDB_USER andDB_NAME. If they are different, update the file accordingly.

exportSECRET_KEY=""exportDB_HOST=""exportDB_PASSWORD=""exportREDIS_HOST=""

Run containers

Check out sources from git and run the docker-compose

$git clone https://github.com/kokospapa8/ecs-fargate-sample-app.git$cdecs-fargate-sample-app$docker-compose-f docker-compose-staging-api.yml up--build# add -d option for detach mode I explicitly left it out to see access logs# if you get error running docker-compose or docker, you probably don't have previlige to run docker as user,# refer to https://docs.docker.com/engine/install/linux-postinstall/# instead of building the whole images you can pull the image from ECR - https://docs.docker.com/compose/reference/pull/# in order to pull the image from repository make sure you add images filed to each docker-compose file - https://docs.docker.com/compose/compose-file/#image#$docker-compose-f docker-compose-staging-api.yml pull$docker-compose-f docker-compose-staging-api.yml up

Once docker containers are up and running and no error shown, access public ip of your app instance.

asciicast

Attach instace behind ALB

Attach your instance to ALB's target group then you will start seeing ALB's health check on the log.

target group

For worker instance

Follow same step as previous step on docker setup or create another AMI image from api instance and start the instance in private subnet.

$git clone https://github.com/kokospapa8/ecs-fargate-sample-app.git$cdecs-fargate-sample-app$docker-compose-f docker-compose-staging-worker.yml up--build

Caveats

Everything checked out; in order to use this setup in production level, there are couple problems we need to address.

  • Scaling services
  • Updating sources for further development

Scaling services

We can create launch config and attach it to autoscaling group for automatic scaling.

What Is AWS Auto Scaling?

We need to prepare couple things for this to work

  • Each instances must have environment variable injected on startup. There are couple ways to do this
  • Pull new source
  • Run docker-compose on start

There are various way to accomplish this, you can search for other methods

Aws Ec2 run script program at startup

Updating sources for further development

  • Configuration changes - environment
  • pulling new src from git

Brute force

Login into each instance and update the source and build image manualy

  • ssh into each ec2 instance
> ssh-i"/path/to/keyfile/" ubuntu@<public_ip>$git pull$docker-compose-f docker-compose-staging-api.yml up--build

Obviously this is not a great idea if you have multiple instances of ec2 running. Also ip address change if instances are running behind autoscaling group.

Fabric

You can usefabric to alleviate such painful process. It's library designed to execute shell commands remotely over SSH.

Fabric

I have provided sample fab file for accessing ec2 instances in public subnets.

def_get_ec2_instances():instances=[]connection=connect_to_region(region_name="ap-northeast-2",#TODO beta envaws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"))try:reservations=connection.get_all_reservations(filters={'tag:Name':'ecs-sample-api','tag:env':'staging'})forrinreservations:forinstanceinr.instances:instances.append(instance)exceptboto.exception.EC2ResponseErrorase:print(e)returninstances=filter(None,instances)returninstances

kokospapa8/ecs-fargate-sample-app

When you run fabfile, it uses boto library to traverse ec2 instance with tag named asecs-sample-api. Therefore you need to addecs-sample-api for Name tag for each ec2 instances.

# you need to install fab and boto library$pipinstallfabric, boto#edit fabfile.py# SSH_KEY= "PATH/TO/YOUR/SSH_KEY"# set env for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with ec2 describe policies$fab--list$fab gitpull$fab docker-restart

Wrap up

We are done for staging environment setup. We now have a docker containers running on cloud evironment. However there are couple concerns I woud like to address.

  • Accessing each instances manually to update the source is not secure job
  • I would like to address immutability to container deployment.
  • I would like to fully utilize my compute node for containers.

Let's move to part3

Reference

Save yourself a lot of pain (and money) by choosing your AWS Region wisely

VPC with public and private subnets (NAT)

Fabric

Securely Connect to Linux Instances Running in a Private Amazon VPC | Amazon Web Services

What Is Amazon EC2 Auto Scaling?

Top comments(0)

Subscribe
pic
Create template

Templates let you quickly answer FAQs or store snippets for re-use.

Dismiss

Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment'spermalink.

For further actions, you may consider blocking this person and/orreporting abuse

  • Location
    Seoul
  • Work
    Engineer at self-employed
  • Joined

More fromJinwook Baek

DEV Community

We're a place where coders share, stay up-to-date and grow their careers.

Log in Create account
[8]ページ先頭
©2009-2025 Movatter.jp