Folders and files

Name		Name	Last commit message	Last commit date
Latest commit History 132 Commits
.github		.github
scripts		scripts
.gitignore		.gitignore
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
s3_storage_provider.py		s3_storage_provider.py
setup.cfg		setup.cfg
setup.py		setup.py
test_s3.py		test_s3.py
tox.ini		tox.ini

Repository files navigation

Synapse S3 Storage Provider

This module can be used by synapse as a storage provider, allowing it to fetchand store media in Amazon S3.

Usage

Thes3_storage_provider.py should be on the PYTHONPATH when startingsynapse.

Example of entry in synapse config:

media_storage_providers:-module:s3_storage_provider.S3StorageProviderBackendstore_local:Truestore_remote:Truestore_synchronous:Trueconfig:bucket:<S3_BUCKET_NAME># All of the below options are optional, for use with non-AWS S3-like# services, or to specify access tokens here instead of some external method.region_name:<S3_REGION_NAME>endpoint_url:<S3_LIKE_SERVICE_ENDPOINT_URL>access_key_id:<S3_ACCESS_KEY_ID>secret_access_key:<S3_SECRET_ACCESS_KEY>session_token:<S3_SESSION_TOKEN># Server Side Encryption for Customer-provided keys#sse_customer_key: <S3_SSEC_KEY># Your SSE-C algorithm is very likely AES256# Default is AES256.#sse_customer_algo: <S3_SSEC_ALGO># The object storage class used when uploading files to the bucket.# Default is STANDARD.#storage_class: "STANDARD_IA"# Prefix for all media in bucket, can't be changed once media has been uploaded# Useful if sharing the bucket between Synapses# Blank if not provided#prefix: "prefix/to/files/in/bucket"# The maximum number of concurrent threads which will be used to connect# to S3. Each thread manages a single connection. Default is 40.##threadpool_size: 20# Set request_checksum_calculation or response_checksum_validation.# Depending on your S3 provider you may need to set these values,# especially if you are using a self-hosted system that does not# support the functionality provided by AWS.# Default is 'when_required'# request_checksum_calculation: "when_supported" | "when_required"# response_checksum_validation: "when_supported" | "when_required"

This module usesboto3, and so the credentials should be specified asdescribedhere.

Regular cleanup job

There is additionally a script atscripts/s3_media_upload which can be usedin a regular job to upload content to s3, then delete that from local disk.This script can be used in combination with configuration for the storageprovider to pull media from s3, but upload it asynchronously.

Once the package is installed, the script should be run somewhat like thefollowing. We suggest usingtmux orscreen as these can take a long timeon larger servers.

database.yaml should contain the keys that would be passed to psycopg2 toconnect to your database. They can be found in the contents of thedatabase.args parameter in your homeserver.yaml.

More options are available in the command help.

> cd s3_media_upload# cache.db will be created if absent. database.yaml is required to# contain PG credentials> lscache.db database.yaml# Update cache from /path/to/media/store looking for files not used# within 2 months> s3_media_upload update /path/to/media/store 2mSyncing files that haven't been accessed since: 2018-10-18 11:06:21.520602Synced 0 new rows100%|█████████████████████████████████████████████████████████████| 1074/1074 [00:33<00:00, 25.97files/s]Updated 0 as deleted> s3_media_upload upload /path/to/media/store matrix_s3_bucket_name --storage-class STANDARD_IA --delete# prepare to wait a long time

Packaging and release

For maintainers:

Update the__version__ in setup.py. Commit. Push.
Create a signed tag and push that:
```
git tag -s vX.Y.Zgit push origin vX.Y.Z
```
Create a release on GitHub for this version.
When published, aGitHub action workflow will build the package and upload toPyPI.