Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up

Video transcoder system based on microservices with RabbitMQ, Redis and FFmpeg.

NotificationsYou must be signed in to change notification settings

cieslarmichal/transcoder

Repository files navigation

Introduction

The video transcoder system is a distributed system that allows users to upload videos and transcode them to different formats.One video will generate following artifacts:

  • Original video
  • Encoded videos in different formats with HLS playlist(1080p, 720p, 480p, 360p)
  • Preview videos for each format (first 4 seconds)
  • Master HLS playlist
  • Thumbnails image - grid of images from the video taken every 10 seconds

Technologies

  • Node.js
  • Typescript
  • RabbitMQ
  • Redis
  • S3
  • FFmpeg

Transcoding process

  1. User uploads a video
  2. API service uploads the video to S3
  3. Downloader service downloads the video from S3 to shared filesystem storage
  4. Encoding director service decides the encoding profiles for the video
  5. Encoder service encodes the video to the desired formats
  6. Uploader service uploads the video artifacts to S3
  7. Playlist sticher service creates a master playlist for the encoded videos

All services communicate with each other via RabbitMQ.

Architecture

flowchart TB  API[API Service]:::service  DOWNLOADER[Downloader Service]:::service  ENCODING_DIRECTOR[Encoding Director Service]:::service  ENCODER[Encoder Service]:::service  UPLOADER[Uploader Service]:::service  PLAYLIST_STICHER[Playlist Sticher Service]:::service  USERS[Users]:::external  RABBITMQ[RabbitMQ]:::external  S3[S3 Storage]:::external  REDIS[Redis]:::db  USERS -->|send video| API  USERS -->|get progress| API  USERS -->|get artifacts| API  API -->|video upload| S3  API -->|done| RABBITMQ  API -->|get progress| REDIS  RABBITMQ -->|url to download| DOWNLOADER  DOWNLOADER -->|video download| S3  DOWNLOADER -->|done| RABBITMQ  RABBITMQ -->|video id| ENCODING_DIRECTOR  ENCODING_DIRECTOR -->|video encoding spec| RABBITMQ  RABBITMQ -->|video path with encoding spec| ENCODER  ENCODER -->|save progress| REDIS  ENCODER -->|done| RABBITMQ  RABBITMQ -->|encoded file path| UPLOADER  UPLOADER -->|file upload| S3  UPLOADER -->|done| RABBITMQ  RABBITMQ -->|encoding id| PLAYLIST_STICHER  PLAYLIST_STICHER -->|master playlist upload| S3  classDef db color:#fff,fill:#ff9655,stroke:#ffa764,stroke-width:2px;  classDef external color:#fff,fill:#9b84d0,stroke:#9676d7,stroke-width:2px;  classDef service color:#fff,fill:#3b5dae,stroke:#97a9d3,stroke-width:2px;
Loading

RabbitMQ Architecture

flowchart LR  API[API Service]:::service  DOWNLOADER[Downloader Service]:::service  ENCODING_DIRECTOR[Encoding Director Service]:::service  ENCODER[Encoder Service]:::service  UPLOADER[Uploader Service]:::service  PLAYLIST_STICHER[Playlist Sticher Service]:::service  INGESTED_VIDEOS[ingested-videos queue]:::queue  DOWNLOADED_VIDEOS[downloaded-videos queue]:::queue  ENCODING_REQUESTS[encoding-requests queue]:::queue  ENCODED_VIDEOS[encoded-videos queue]:::queue  UPLOADED_ARTIFACTS[uploaded-artifacts queue]:::queue  EXCHANGE[transcoder exchange]:::exchange  API -->|video.ingested| EXCHANGE  EXCHANGE --> INGESTED_VIDEOS  INGESTED_VIDEOS --> DOWNLOADER  DOWNLOADER -->|video.downloaded| EXCHANGE  EXCHANGE --> DOWNLOADED_VIDEOS  DOWNLOADED_VIDEOS --> ENCODING_DIRECTOR  ENCODING_DIRECTOR -->|video.encoding.requested| EXCHANGE  EXCHANGE --> ENCODING_REQUESTS  ENCODING_REQUESTS --> ENCODER    ENCODER -->|video.encoded| EXCHANGE  EXCHANGE --> ENCODED_VIDEOS  ENCODED_VIDEOS --> UPLOADER  UPLOADER -->|video.artifact.uploaded| EXCHANGE  EXCHANGE --> UPLOADED_ARTIFACTS  UPLOADED_ARTIFACTS --> PLAYLIST_STICHER  classDef queue color:#fff,fill:#ff9655,stroke:#ffa764,stroke-width:2px;  classDef exchange color:#fff,fill:#9b84d0,stroke:#9676d7,stroke-width:2px;  classDef service color:#fff,fill:#3b5dae,stroke:#97a9d3,stroke-width:2px;
Loading

Services

API Service

  • Accepts a video from a user by HTTP
  • Uploads a video to S3
  • Sends a message with video id and download to RabbitMQ
  • Checks the encoding progress in Redis
  • Checks the encoding artifacts in S3

Downloader Service

  • Consumes messages with video URL to download from RabbitMQ
  • Downloads a video from S3 and saves it to the shared volume
  • Sends a downloading done message to RabbitMQ

Encoding Director Service

  • Consumes messages with video id from RabbitMQ
  • Decides the encoding profile for the video
  • Sends encoding request messages to RabbitMQ (one for each encoding profile)

Encoder Service

  • Consumes messages with encoding format and video path from RabbitMQ
  • Encodes a video to the desired format
  • Saves the encoding progress to Redis
  • Sends an encoding done message to RabbitMQ

Uploader Service

  • Consumes messages about downloading done from RabbitMQ
  • Uploads encoding artifacts from shared volume to S3
  • Sends an uploading done message to RabbitMQ

Playlist Sticher Service

  • Consumes messages with encoding id from RabbitMQ
  • Creates a master playlist for the encoded videos
  • Uploads the master playlist to S3

About

Video transcoder system based on microservices with RabbitMQ, Redis and FFmpeg.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

[8]ページ先頭

©2009-2025 Movatter.jp