Repository files navigation

Turbo Deploy is an enterprise-grade deployment platform that automates the entire process of deploying React/Vite applications. With a single Git URL, it orchestrates building, hosting, and serving your application through a sophisticated AWS infrastructure pipeline. This project is built using Go, Node.js, PostgreSQL, and several AWS services, including RDS, S3, ElasticCache, SES, SQS, ECS, and EC2.

• Features
• Architecture
• Prerequisites
• Project Structure
• Installation
• Configuration
• Usage
• API Reference
• Infrastructure
• Monitoring
• Security
• Contributing
• License

• Automated Deployments
  • Zero-configuration deployment pipeline
  • Automatic build optimization
  • Smart caching strategies
  • Rolling updates with zero downtime

• Isolated Build Environment
  • Containerized builds using AWS ECS
  • Concurrent build support
  • Custom build parameter configuration
  • Build artifact caching

• S3-Powered Static Hosting
  • Automatic asset optimization
  • Content compression
  • Cache control headers
  • Custom domain support

• Comprehensive Observability
  • Real-time deployment logs
  • Performance metrics
  • Error tracking
  • Resource utilization monitoring

• Enterprise-Grade Security
  • SSL/TLS encryption
  • Sanitization of Build Commands

• React/Vite application hosting
• Custom domain management
• SSL/TLS termination
• Static asset serving

1. Build Server (Nodejs)

   • Handles build requests
   • Manages ECS task lifecycle
   • Coordinates with S3 for artifact storage
   • Implements build caching

2. Reverse Proxy (Go, Node.js)

   • Routes requests to appropriate S3 buckets
   • Handles custom domain mapping
   • Manages SSL/TLS certificates
   • Implements caching strategies

3. Log Consumer Service (Go)

   • Processes SQS messages
   • Aggregates build logs
   • Handles log retention
   • Provides real-time log streaming

4. Email Service (Go)

   • Manages notification templates
   • Handles deployment notifications
   • Processes email queues
   • Tracks delivery status

1. Database Layer (PostgreSQL)

   • Deployment records
   • User management
   • Configuration storage
   • Audit logs

2. Cache Layer (Redis)

   • Session management
   • Rate limiting
   • Build cache
   • Temporary data storage

3. Message Queue (SQS)

   • Service communication
   • Event processing
   • Log aggregation
   • Email notifications

4. Storage Layer (S3)

   • Static file hosting
   • Build artifacts
   • Log archives
   • Backup storage

• Docker Engine 20.10+
• Docker Compose 2.0+
• Node.js 16.x+
• Go 1.19+
• PostgreSQL 13+
• Redis 6+

• AWS Account with admin access
• Configured AWS CLI
• Required AWS services enabled:
  • EC2
  • ECS
  • S3
  • RDS
  • ElastiCache
  • SQS
  • SES
  • IAM

• Public IP address
• Domain name
• SSL certificate

The Turbo-Deploy project is built with a modular structure to ensure scalability, maintainability, and ease of development. Below is a breakdown of the key directories and their roles:

Contains the logic for handling incoming requests and orchestrating business processes.

• prometheus: Custom Prometheus metrics for real-time monitoring and observability.
• user: Handles user-specific operations like authentication, user profile management, and access control.

Houses project documentation, API references, and technical guides.

Centralized error-handling mechanism used throughout the application to ensure consistency and structured error responses.

A collection of utility modules to handle common tasks across services:

• general: Generic utilities used in various modules.
• logger: Custom logging utilities for structured and level-based logging.
• retry: Helper functions for implementing retry logic for failed operations.
• tracer: OpenTelemetry tracing utilities for distributed system observability.
• uploader: Tools for handling file uploads to S3.
• validator: Input validation functions to enforce data integrity.

Infrastructure-specific modules that abstract interactions with external services:

• db: PostgreSQL database connection and query abstraction.
• redis: Integration with AWS ElasticCache for caching.
• s3: S3 bucket management and operations for file storage.
• ses: Email sending functionalities via AWS SES.
• sqs: Integration with AWS SQS for event queue management.

Scripts for managing database schema migrations. Ensures smooth transitions when modifying the database schema.

Defines the data models and business logic for various entities:

• deployment: Manages deployment processes and state.
• deployment_log: Tracks deployment history and logs.
• error: Standardized error model for the system.
• project: Handles project-related data and logic.
• user: User-specific data and operations.

Defines API routes and their mapping to controller functions. Acts as the entry point for incoming requests.

Contains microservices, each performing specific roles:

• build-server: Executes ECS tasks to clone repositories, build projects, and upload artifacts to S3.
• email-consumer: Listens to SQS queues and sends email notifications using AWS SES.
• logs-sqs-consumer: Processes logs/events from the SQS queues and updates deployment logs or metrics.
• reverse-proxy: Dynamically serves files from S3 based on subdomains.

1. Clone the Repository

git clone https://github.com/swarajkumarsingh/turbo-deploy.gitcd turbo-deploy

3. Env Setup

# edit the run.local.sh files accordingly to your confignano ./run.local.sh

2. Run .sh file

./run.local.sh

3. Run all services(consumers)

# migrate to root/servicesmake start

DB_HOST=http://127.0.0.1/DB_PORT=5432DB_USER=userDB_PASSWORD=postgresDB_NAME=turbo-deployREDIS_HOST=REDIS_PORT=REDIS_USER=REDIS_PASSWORD=SENTRY_DSN=DD_AGENT_HOST=S3_BUCKET=STAGE=devAWS_SQS_URL=AWS_REGION=AWS_ACCESS_KEY=AWS_SECRET_ACCESS_KEY=DB_URL=

• check .postman folder postman.json --> import into Postman Client

1. Compute

   • ECS Cluster
   • EB elastic beanstalk
   • EC2 Auto Scaling Group
   • Application Load Balancer
   • ECR - Registry for code storage

2. Storage

   • S3 Buckets
   • RDS Instance
   • ElastiCache Cluster

3. Networking

   • VPC
   • Subnets
   • Security Groups

4. Services

   • SES Email Service
   • SQS Queue Service

{"taskDefinitionArn":"arn:aws:ecs:ap-north-99:xxxx:task-definition/builder-task:1","containerDefinitions": [    {"name":"my-image","image":"xxxx.dkr.ecr.ap-south-1.amazonaws.com/build-server:latest","cpu":0,"portMappings": [],"essential":true,"environment": [],"environmentFiles": [],"mountPoints": [],"volumesFrom": [],"ulimits": [],"logConfiguration": {"logDriver":"awslogs","options": {"awslogs-group":"/ecs/thor-flask","mode":"non-blocking","awslogs-create-group":"true","max-buffer-size":"25m","awslogs-region":"ap-south-1","awslogs-stream-prefix":"ecs"        },"secretOptions": []      },"systemControls": []    }  ],"family":"thor-flask","executionRoleArn":"arn:aws:iam::xxxx:role/ecsTaskExecutionRole","networkMode":"awsvpc","revision":1,"volumes": [],"status":"ACTIVE","requiresAttributes": [    {"name":"com.amazonaws.ecs.capability.logging-driver.awslogs"    },    {"name":"ecs.capability.execution-role-awslogs"    },    {"name":"com.amazonaws.ecs.capability.ecr-auth"    },    {"name":"com.amazonaws.ecs.capability.docker-remote-api.1.19"    },    {"name":"com.amazonaws.ecs.capability.docker-remote-api.1.28"    },    {"name":"ecs.capability.execution-role-ecr-pull"    },    {"name":"com.amazonaws.ecs.capability.docker-remote-api.1.18"    },    {"name":"ecs.capability.task-eni"    },    {"name":"com.amazonaws.ecs.capability.docker-remote-api.1.29"    }  ],"placementConstraints": [],"compatibilities": ["EC2","FARGATE"],"requiresCompatibilities": ["FARGATE"],"cpu":"1024","memory":"3072","runtimePlatform": {"cpuArchitecture":"XxxXX","operatingSystemFamily":"LINUX"  },"registeredAt":"2024-12-xxxx:36.078Z","registeredBy":"arn:aws:iam::xxxx:root","enableFaultInjection":false,"tags": []}

# ECS Task Definitionresource"aws_ecs_task_definition""build" {family="build"requires_compatibilities=["FARGATE"]network_mode="awsvpc"cpu=1024memory=2048container_definitions=jsonencode([    {      name="build"      image="${aws_ecr_repository.build.repository_url}:latest"# ... additional configuration    }  ])}

1. System Metrics

   • CPU Usage
   • Memory Usage
   • Network I/O
   • Disk Usage

2. Application Metrics

   • Build Success Rate
   • Build Duration
   • Cache Hit Rate
   • Error Rate

3. Business Metrics(yet to be built)

   • Active Deployments
   • Total Users
   • Resource Consumption
   • Cost Analysis

global:scrape_interval:15sscrape_configs:  -job_name:'turbo-deploy'static_configs:      -targets:['localhost:9090']

• JWT-based authentication
• Rate limiting
• IP whitelisting

• npm build sanitization for security

1. Fork the repository
2. Create a feature branch
3. Implement changes
4. Add tests
5. Submit pull request

• Follow Go style guide
• Use ESLint for JavaScript
• Write unit tests
• Document changes

MIT License

Copyright (c) 2024 Swaraj Kumar Singh

Permission is hereby granted, free of charge, to any person obtaining a copyof this software and associated documentation files (the "Software"), to dealin the Software without restriction, including without limitation the rightsto use, copy, modify, merge, publish, distribute, sublicense, and/or sellcopies of the Software, and to permit persons to whom the Software isfurnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in allcopies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS ORIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THEAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHERLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THESOFTWARE.

• Email:sswaraj169@gmail.com
• Medium:Join our community
• Documentation:Read the docs
• Issues:GitHub Issues

Made with ❤️ by Swaraj Kumar Singh