Core

The cloud-native audio/video processing API.

datarhei Core is management for FFmpeg processes without development effort. It is a central interface for mapping AV processes, is responsible for design and management, and provides all necessary interfaces to access the video content. The included control for FFmpeg can keep all used functions reliable and executable without the need for software developers to take care of it. In addition, process and resource limitation for all FFmpeg processes protects the host system from application overload. The overall system gives access to current process values (CPU, RAM) and complete control of system resources and loads with statistical access to process data and current and historical logs.

Features

• Unrestricted FFmpeg process management
• Optimized for long-running tasks
• In-Memory- and Disk-Filesystem for media assets
• HTTP and RTMP services
• Let's Encrypt for HTTPS and RTMPS
• HLS/DASH Session tracking with bandwidth and current viewer limiters
• Multiple resource limiters and monitoring
• FFmpeg progress data
• Metrics incl. Prometheus support
• Logging and debugging for FFmpeg processes with history
• Multiple auth. by JWT and Auth0
• 100% JSON REST API (Swagger documented)
• GraphQL for metrics, process, and progress data

Quick start

1. Run the Docker image

docker run --name core -d \    -e CORE_API_AUTH_USERNAME=admin \    -e CORE_API_AUTH_PASSWORD=secret \    -p 8080:8080 \    -v ${HOME}/core/config:/core/config \    -v ${HOME}/core/data:/core/data \    datarhei/core:latest

1. Open Swaggerhttp://host-ip:8080/api/swagger/index.html

2. Log in with SwaggerAuthorize > Basic authorization > Username: admin, Password: secret

Docker images

Native (linux/amd64,linux/arm64,linux/arm/v7)

• datarhei/base:alpine-core-latest
• datarhei/base:ubuntu-core-latest

Bundle with FFmpeg (linux/amd64,linux/arm64,linux/arm/v7)

• datarhei/core:latest

Bundle with FFmpeg for Raspberry Pi (linux/arm/v7)

• datarhei/core:rpi-latest

Bundle with FFmpeg for Nvidia Cuda (linux/amd64)

• datarhei/core:cuda-latest

Bundle with FFmpeg for Intel VAAPI (linux/amd64)

• datarhei/core:vaapi-latest

Documentation

Environment variables

The environment variables can be set in the file.env, e.g.

CORE_API_AUTH_USERNAME=adminCORE_API_AUTH_PASSWORD=datarhei...

You can also provide them on the command line, whatever you prefer. If the same environment variable is setin the.env file and on the command line, the one set on the command line will overrule the one from the.env file.

The currently known environment variables (but not all will be respected) are:

Config

The minimum config file has to look like this:

{    "version": 1}

All other values will be filled with default values and persisted on disk. The entire default config file:

{    "version": 1,    "id": "[will be generated if not given]",    "name": "[will be generated if not given]",    "address": ":8080",    "log": {        "level": "info",        "topics": [],        "max_lines": 1000    },    "db": {        "dir": "./config"    },    "host": {        "name": [],        "auto": true    },    "api": {        "read_only": false,        "access": {            "http": {                "allow": [],                "block": []            },            "https": {                "allow": [],                "block": []            }        },        "auth": {            "enable": true,            "disable_localhost": false,            "username": "",            "password": "",            "jwt": {                "secret": ""            },            "auth0": {                "enable": false,                "tenants": []            }        }    },    "tls": {        "address": ":8181",        "enable": false,        "auto": false,        "cert_file": "",        "key_file": ""    },    "storage": {        "disk": {            "dir": "./data",            "max_size_mbytes": 0,            "cache": {                "enable": true,                "max_size_mbytes": 0,                "ttl_seconds": 300,                "max_file_size_mbytes": 1,                "types": []            }        },        "memory": {            "auth": {                "enable": true,                "username": "admin",                "password": "vxbx0ViqfA75P1KCyw"            },            "max_size_mbytes": 0,            "purge": false        },        "cors": {            "origins": [                "*"            ]        },        "mimetypes_file": "mime.types"    },    "rtmp": {        "enable": false,        "enable_tls": false,        "address": ":1935",        "app": "/",        "token": ""    },    "ffmpeg": {        "binary": "ffmpeg",        "max_processes": 0,        "access": {            "input": {                "allow": [],                "block": []            },            "output": {                "allow": [],                "block": []            }        },        "log": {            "max_lines": 50,            "max_history": 3        }    },    "playout": {        "enable": false,        "min_port": 0,        "max_port": 0    },    "debug": {        "profiling": false,        "force_gc": 0    },    "stats": {        "enable": true,        "ip_ignorelist": [            "127.0.0.1/32",            "::1/128"        ],        "session_timeout_sec": 30,        "persist": false,        "persist_interval_sec": 300,        "max_bitrate_mbit": 0,        "max_sessions": 0    },    "service": {        "enable": false,        "token": "",        "url": "https://service.datarhei.com"    },    "router": {        "blocked_prefixes": [            "/api"        ],        "routes": {}    }}

If you don't provide a path to a config file, the default config will be used, and nothing will be persisted to the disk. Default values can be overruled by environment variables.

TLS / HTTPS

Enable TLS / HTTPS support by settingCORE_TLS_ENABLE=true and provide the certificate file and key file in PEM format by setting the environment variablesCORE_TLS_CERTFILE andCORE_TLS_KEYFILE accordingly. If a certificate authority signs the certificate, the certificate file should be the concatenation of the server's certificate, any intermediates, and the CA's certificate.

If TLS with given certificates is enabled, an HTTP server listening onCORE_ADDRESS (address) will be additionally started. This server provides access to the same memory filesystem as the HTTPS server (including limits and authorization), but its access is restricted to localhost only.

Let's Encrypt

If you want to use automatic certificates from Let's Encrypt, set the environment variableCORE_TLS_AUTO totrue. To work, theenvironment variablesCORE_TLS_ENABLE have to betrue, andCORE_HOST_NAME has to be set to the host this host will be reachable. Otherwise, the ACME challenge will not work. The environment variablesCORE_TLS_CERTFILE andCORE_TLS_KEYFILE will be ignored.

If automatic TLS is enabled, the HTTP server (CORE_ADDRESS, resp. address) must listen on port 80. It is required to automatically acquire the certificate (serving theHTTP-01 challenge). As a further requirement,CORE_HOST_NAME (host.name) must be set because it is used a the canonical name for the certificate.

The obtained certificates will be stored inCORE_DB_DIR/cert to be available after a restart.

The obtained certificates will be stored inCORE_DB_DIR/cert to be available after a restart.

Self-Signed certificates

To create a self-signed certificate and key file pair, run this command and provide a reasonable value for the Common Name (CN). The CN is the fully qualified name of the host the instance is running on (e.g.,localhost). You can also use an IP address or a wildcard name, e.g.,*.example.com.

RSA SSL certificate

openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out cert.pem -subj '/CN=localhost'

ECDSA SSL certificate

openssl ecparam -name secp521r1 -genkey -out key.pemopenssl req -new -x509 -key key.pem -out cert.pem -days 365 -subj '/CN=localhost'

Callopenssl ecparam -list_curves to see all available supported curves listed.

Access Control

To control who has access to the API, a list of allowed IPs can be defined. This list is provided at startup with the environment variablesCORE_API_ACCESS_HTTP_BLOCK andCORE_API_ACCESS_HTTP_ALLOW. This is a comma-separated list of IPs in CIDR notation,e.g.127.0.0.1/32,::1/128. If the list is empty, then all IPs are allowed. If the list contains any invalid IP range, the serverwill refuse to start. This can be separately defined for the HTTP and HTTPS server if you have TLS enabled with the environment variablesCORE_API_ACCESS_HTTPS_BLOCK andCORE_API_ACCESS_HTTPS_ALLOW.

Input/Output Control

To control where FFmpeg can read and where FFmpeg can write, you can define a pattern that matches theinput addresses or the output addresses. These patterns are regular expressions that can be provided at startup with theenvironment variablesCORE_FFMPEG_ACCESS_INPUT andCORE_FFMPEG_ACCESS_OUTPUT. The expressions need to be space-separated, e.g.HTTPS?:// RTSP:// RTMP://. If one of the lists is empty, then no restriction on input, resp. The output will be applied.

Independently of the value ofCORE_FFMPEG_ACCESS_OUTPUT there's a check that verifies that output can only be written to the specifiedCORE_STORAGE_DISK_DIR and works as follows: If the address has a protocol specifier other thanfile:, then no further checks will be applied. If the protocol isfile: or no protocol specifier is given, the address is assumed to be a path that is checked against the path shown inCORE_STORAGE_DISK_DIR.

It will be rejected if the address is outside theCORE_STORAGE_DISK_DIR directory. Otherwise, the protocolfile: will be prepended. If you give some expressions forCORE_FFMPEG_ACCESS_OUTPUT, you should also allowfile:.

Special cases are the output addresses- (which will be rewritten topipe:), and/dev/null (which will be allowed even though it's outside ofCORE_STORAGE_DISK_DIR).

If you set a value forCORE_STORAGE_DISK_CACHE_MAXSIZEMBYTES, which is larger than0, it will be interpreted as max—allowed megabytes for theCORE_STORAGE_DISK_DIR. As soon as the limit is reached, all processes that have outputs writing toCORE_STORAGE_DISK_DIR will be stopped. You are responsible for cleaning up the directory and restarting these processes.

RTMP

The datarhei Core includes a simple RTMP server for publishing and playing streams. Set the environment variableCORE_RTMP_ENABLE totrue to enable the RTMP server. It is listening onCORE_RTMP_ADDRESS. UseCORE_RTMP_APP to limit the app a stream can be published on, e.g./live to require URLs to start with/live. To prevent anybody can publish streams, setCORE_RTMP_TOKEN to a secret only known to the publishers. The token has to be put in the query of the stream URL, e.g./live/stream?token=....

Playout

FFmpeg processes with aavstream: (orplayout:) input stream can expose an HTTP API to control the playout of that stream. WithCORE_PLAYOUT_ENABLE you enable exposing this API. The API is only exposed tolocalhost and is transparently connected to the datarhei Core API. You have to provide a port range (CORE_PLAYOUT_MINPORT andCORE_PLAYOUT_MAXPORT) where datarhei/core can use ports to assign it to the playout API.

MIME Types

The file with the MIME types has one MIME type per line followed by a list of file extensions (including the ".").

text/plain  .txttext/html   .htm .html...

Memory Filesystem

AA very simple in-memory filesystem is available. The uploaded data is stored in a map, where the path used to upload the fileis used as the key. Use thePOST orPUT method with the proper direction for uploading a file. The body of the request contains the contents of the file. No particular encoding orContent-Type is required. The file can then be downloaded from the same path.

Use these endpoints to, e.g., store HLS chunks and .m3u8 files (in contrast to an actual disk or a ramdisk):

ffmpeg -f lavfi -re -i testsrc2=size=640x480:rate=25 -c:v libx264 -preset:v ultrafast -r 25 -g 50 -f hls -start_number 0 -hls_time 2 -hls_list_size 6 -hls_flags delete_segments+temp_file+append_list -method PUT -hls_segment_filename http://localhost:8080/memfs/foobar_%04d.ts -y http://localhost:8080/memfs/foobar.m3u8

Then you can play it generally with, e.g.,ffplay http://localhost:3000/memfs/foobar.m3u8.

Use the environment variablesCORE_STORAGE_MEMORY_AUTH_USERNAME andCORE_STORAGE_MEMORY_AUTH_PASSWORD to protect the/memfs with Basic-Auth. Basic-Auth will only be enabledif both environment variables are set to non-empty values. TheGET /memfs/:path will not be protected with Basic-Auth.

Use the environment variableCORE_STORAGE_MEMORY_MAXSIZEMBYTES to limit the amount of data that is allowed to be stored. The value is interpreted as megabytes. Use a value equal to or smaller than0 not to impose any limits. A507 Insufficient Storage will be returned if you hit the limit.

Listing all currently stored files is done by calling/v3/memfs with the credentials set by the environment variablesCORE_API_AUTH_USERNAME andCORE_API_AUTH_PASSWORD.It also accepts the query parametersort (name,size, orlastmod) andorder (asc ordesc). If a valid value forsort is given, the results are sorted in ascending order.

Routes

All contents inCORE_STORAGE_DISK_DIR are served from/. If you want to redirect some paths to an existing file, you can add static routes inrouter.routes by providing a direct mapping, e.g.

router: {    routes: {        "/foo.txt": "/bar.txt",    }}

The paths have to start with a/. Alternatively, you can serve whole directories from another root thanCORE_STORAGE_DISK_DIR. Use a/* at the end of a path as key and a path on the filesystem as the target, e.g.

router: {    routes: {        "/ui/*": "/path/to/ui",    }}

If you use a relative path as target, then it will be added to the current working directory.

API

Check the detailed API description on/api/swagger/index.html.

Login / Auth

With auth enabled, you have to retrieve a JWT/OAuth token before you can access the/v3/ API calls.

For the login you have to send

{    "username": "...",    "password": "..."}

Theusername and thepassword are set by the environment variablesCORE_API_AUTH_USERNAME andCORE_API_AUTH_PASSWORD.

On successful login, the response looks like this:

{    "expire": "2019-01-18T19:55:55+01:00",    "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NDc4Mzc3NTUsImlkIjpudWxsLCJvcmlnX2lhdCI6MTU0NzgzNDE1NX0.ZcrpD4oRBqG3wUrfnh1DOVpXdUT7dvUnvetKFEVRKKc"}

Use thetoken in all subsequent calls to the/api/v3/ endpoints, e.g.

http http://localhost:8080/api/v3/process "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NDc4Mzc3NTUsImlkIjpudWxsLCJvcmlnX2lhdCI6MTU0NzgzNDE1NX0.ZcrpD4oRBqG3wUrfnh1DOVpXdUT7dvUnvetKFEVRKKc"

Config

When retrieving the config via the API, critical values (such as passwords) will be disguised if not required otherwise.

Process

With the process API call, you can manage different FFmpeg processes. A process is defined as:

{    "id": "SomeId",    "reference": "SomeId",    "type": "ffmpeg",    "input": [        {            "id": "inputid",            "address": "rtsp://1.2.3.4/stream.sdp",            "options": [                ... list of input options ...            ]        },        ... list of inputs ...    ],    "output": [        {            "id": "outputid",            "address": "rtmp://rtmp.youtube.com/live2/...",            "options": [                ... list of output options ...            ],            "cleanup": [{                "pattern": "(memfs|diskfs):...",                "max_files: "number,                "max_file_age_seconds": "number",                "purge_on_delete: "(true|false)"            }]        },        ... list of outputs ...    ],    "options": [        ... list of global options ...    ],    "reconnect": (true|false),    "reconnect_delay_seconds": 10,    "autostart": (true|false),    "stale_timeout_seconds": 30}

The input, output, and global options are interpreted as command-line options for FFmpeg.

Process Cleanup

With the optional array of cleanup rules for each output, it is possible to define rules for removing files from thememory filesystem or disk. Each rule consists of a glob pattern and a max. allowed number of files matching that pattern orpermitted maximum age for the files matching that pattern. The pattern starts with eithermemfs: ordiskfs: depending onwhich filesystem this rule is designated to. Then aglob pattern follows toidentify the files. Ifmax_files is set to a number > 0, then the oldest files from the matching files will be deleted ifthe list of matching files is longer than that number. Ifmax_file_age_seconds is set to a number > 0, then all filesthat are older than this number of seconds from the matching files will be deleted. Ifpurge_on_delete is set totrue,then all matching files will be deleted when the process is deleted.

The API calls are

Commands

A command is defined as:

{    "command": ("start"|"stop"|"restart"|"reload")}

Placeholder

Currently supported placeholders are:

Before replacing the placeholder in the process, all references will be resolved, i.e., you can put the placeholder also in the params for anoption.

References

The input address of a process may contain a reference to the output of another process. It has the form#[processid]:output=[id].

A reference starts with a# followed by the process ID it refers to, followed by a:. Then comesoutput, followed by a=.and the ID of the output.

FFmpeg

Statistics

This repository contains a patch for the FFmpeg program to provide detailed progress information. With this patch, FFmpeg will outputthe progress information in a JSON string that contains the data for each input and output stream individually. The JSON output is enabledby default. It can be enabled with the global-jsonstats switch on the command line. Use the-stats switchon the command line for the standard progress output.

The Docker image that you can build with the provided Dockerfile includes the patched version of FFmpeg for your convenience.

Example output with-stats:

frame=  143 fps= 25 q=-1.0 Lsize=     941kB time=00:00:05.68 bitrate=1357.0kbits/s speed=0.995x

Example output with-jsonstats:

JSONProgress:{"inputs":[{"id":0, "stream":0, "type":"video", "codec":"rawvideo", "coder":"rawvideo", "pix_fmt":"rgb24", "frame":188, "fps":24.95, "width":1280, "height":720, "size_kb":507600, "bitrate_kbps":552960.0},{"id":1, "stream":0, "type":"audio", "codec":"pcm_u8", "coder":"pcm_u8", "frame":314, "sampling_hz":44100, "layout":"stereo", "size_kb":628, "bitrate_kbps":705.6}], "outputs":[{"id":0, "stream":0, "type":"video", "codec":"h264", "coder":"libx264", "pix_fmt":"yuv420p", "frame":188, "fps":24.95, "q":-1.0, "width":1280, "height":720, "size_kb":1247, "bitrate_kbps":1365.6},{"id":0, "stream":1, "type":"audio", "codec":"aac", "coder":"aac", "frame":315, "sampling_hz":44100, "layout":"stereo", "size_kb":2, "bitrate_kbps":2.1}], "frame":188, "fps":24.95, "q":-1.0, "size_kb":1249, "bitrate_kbps":1367.7, "time":"0h0m7.48s", "speed":0.993, "dup":0, "drop":0}

The same output but nicely formatted:

{    "bitrate_kbps": 1367.7,    "drop": 0,    "dup": 0,    "fps": 24.95,    "frame": 188,    "inputs": [        {            "bitrate_kbps": 552960.0,            "codec": "rawvideo",            "coder": "rawvideo",            "fps": 24.95,            "frame": 188,            "height": 720,            "id": 0,            "pix_fmt": "rgb24",            "size_kb": 507600,            "stream": 0,            "type": "video",            "width": 1280        },        {            "bitrate_kbps": 705.6,            "codec": "pcm_u8",            "coder": "pcm_u8",            "frame": 314,            "id": 1,            "layout": "stereo",            "sampling_hz": 44100,            "size_kb": 628,            "stream": 0,            "type": "audio"        }    ],    "outputs": [        {            "bitrate_kbps": 1365.6,            "codec": "h264",            "coder": "libx264",            "fps": 24.95,            "frame": 188,            "height": 720,            "id": 0,            "pix_fmt": "yuv420p",            "q": -1.0,            "size_kb": 1247,            "stream": 0,            "type": "video",            "width": 1280        },        {            "bitrate_kbps": 2.1,            "codec": "aac",            "coder": "aac",            "frame": 315,            "id": 0,            "layout": "stereo",            "sampling_hz": 44100,            "size_kb": 2,            "stream": 1,            "type": "audio"        }    ],    "q": -1.0,    "size_kb": 1249,    "speed": 0.993,    "time": "0h0m7.48s"}

Resilient Streaming

Prepend the input source withavstream:, e.g.... -i avstream:rtsp://1.2.3.4/stream.sdp .... It will reconnect to the stream if it breaks and repeats the last known intraframe until new data from the input stream is available.

Example

Startcore with the proper environment variables. Create a.env file or provide them on the command line. For this example, please use the following command line:

env CORE_API_AUTH_USERNAME=admin CORE_API_AUTH_PASSWORD=datarhei CORE_LOGLEVEL=debug CORE_STORAGE_DISK_DIR=./data ./core

Also, make sure that the directory./data exists. Otherwise, the state will not be stored and will be lost after a restart ofdatarhei/core and the FFmpeg process will not be able to write the files.

In this example, we will add a fake video and audio source. The video will be encoded with H264, and the audio will be encoded with AAC. The output will be an m3u8 stream.

To talk to the API, we use the programhttpie.

First, we create a JSON file with the process definition (e.g.testsrc.json):

{    "id": "testsrc",    "type": "ffmpeg",    "options": ["-loglevel", "info", "-err_detect", "ignore_err"],    "input": [        {            "address": "testsrc=size=1280x720:rate=25",            "id": "video",            "options": ["-f", "lavfi", "-re"]        },        {            "address": "anullsrc=r=44100:cl=stereo",            "id": "audio",            "options": ["-f", "lavfi"]        }    ],    "output": [        {            "address": "http://127.0.0.1:8080/memfs/{processid}_{outputid}.m3u8",            "id": "hls",            "options": [                "-codec:v",                "libx264",                "-preset:v",                "ultrafast",                "-r",                "25",                "-g",                "50",                "-pix_fmt",                "yuv420p",                "-b:v",                "1024k",                "-codec:a",                "aac",                "-b:a",                "64k",                "-hls_time",                "2",                "-hls_list_size",                "10",                "-hls_flags",                "delete_segments+temp_file+append_list",                "-hls_segment_filename",                "http://127.0.0.1:8080/memfs/{processid}_{outputid}_%04d.ts"            ]        }    ],    "reconnect": true,    "reconnect_delay_seconds": 10,    "stale_timeout_seconds": 10}

and POST it to the API:

http POST http://localhost:8080/v3/process < testsrc.json

Then check if it is there (as provided)

http http://localhost:8080/v3/process/testsrc

For the advanced, create another JSON file (e.g.dump.json):

{    "id": "dump",    "type": "ffmpeg",    "options": ["-loglevel", "info", "-err_detect", "ignore_err"],    "input": [        {            "address": "#testsrc:output=hls",            "id": "video",            "options": []        }    ],    "output": [        {            "address": "{diskfs}/{processid}.mp4",            "id": "hls",            "options": ["-codec", "copy", "-y"]        }    ],    "reconnect": true,    "reconnect_delay_seconds": 10,    "stale_timeout_seconds": 10}

and POST it to the API:

http POST http://localhost:8080/v3/process < dump.json

Then check if it is there (as provided)

http http://localhost:8080/v3/process/dump

Let's start thetestsrc process

http PUT http://localhost:8080/v3/process/testsrc/command command=start

Now we can observe the progress of the process

http http://localhost:8080/v3/process/testsrc

or the log of the process

http http://localhost:8080/v3/process/testsrc/log

If you want to change the video bitrate, edit thetestsrc.json file accordingly and replace the process:

http PUT http://localhost:8080/v3/process/testsrc < testsrc.json

It will stop the process, replace the config, and restart it.

Now open, e.g., VLC, and load the streamhttp://localhost:8080/memfs/testsrc_hls.m3u8.

This is enough; let's stop it

http PUT http://localhost:8080/v3/process/testsrc/command command=stop

and check its progress again

HTTP http://localhost:8080/v3/process/testsrc

Delete the process

HTTP DELETE http://localhost:8080/v3/process/testsrc

Metrics

Metrics for the processes and other aspects are provided for a Prometheus scraper on/metrics.

Currently, available metrics are:

Profiling

Profiling information is available under/profiling. Set the environment variableCORE_DEBUG_PROFILING=true to make this endpointavailable. If authentication is enabled, you have to provide the token in the header.

Development

Requirement

• Go v1.16+ (Download here)

Build

Clone the repository and build the binary

git clone git@github.com:datarhei/core.gitcd coremake

After the build process, the binary is available ascore

For more build options, runmake help.

Cross Compile

If you want to run the binary on a different operating system and/or architecture, you create the appropriate binary by simply setting someenvironment variables, e.g.

env GOOS=linux GOARCH=arm go build -o core-linux-armenv GOOS=linux GOARCH=arm64 go build -o core-linux-arm64env GOOS=freebsd GOARCH=amd64 go build -o core-freebsd-amd64env GOOS=windows GOARCH=amd64 go build -o core-windows-amd64env GOOS=macos GOARCH=amd64 go build -o core-macos-amd64...

Docker

Build the Docker image and run it to try out the API

docker build -t core .docker run -it --rm -v ${PWD}/data:/core/data -p 8080:8080 core

API Documentation

The documentation of the API is available on/api/swagger/index.html.

To generate the API documentation from the code, useswag.

go install github.com/swaggo/swag (unless already installed)make swagger

Code style

The source code is formatted withgo fmt, or simply runmake fmt. Static analysis of the source code is done withstaticcheck(seestaticcheck), or simply runmake lint.

Before committing changes, you should runmake commit to ensure that the source code is in shape.

License

datarhei/core is licensed under the Apache License 2.0

There is no documentation for this package.