Repository files navigation

A Secure File Upload Proxy

eldim is a web server that accepts file uploads from a particular set ofhosts, and its job is to encrypt them, and then store them in an ObjectStorage backend system.

It has a preconfigured ACL that only allows specific IP Addresses, or tokenbearers to access the file upload service. After a file is uploaded, it isencrypted, and then uploaded to a configured provider.

It has been designed to work as a standalone application, which means it mustnot sit behind a proxy, but instead be exposed directly to the Internet.

Currently the project has two mailing lists, in Google Groups, that are usedfor communication:

Theeldim-announcegroup isrecommended for all users of eldim. It includes announcementsfor new versions, a changelog, as well as breaking changes that may occurin the future. Moreover, it is the place that will be used for securityannouncements in the future, if and when they come.

This is a very low volume list, and it is read-only. That is, only eldimupdates are posted there, and you cannot send e-mails to other members.

Theeldim-dev grouptries to address that final point above, and it is the techincal mailinglist of the eldim project.

This group can be used to report problems, share news, exchange ideas, etc.Basically it exists for communication about technical matters related toeldim, between the users, the contributors, or the developers.

The design of eldim is data agnostic, and tries to push the relevant logicof all operations to the proper server. For example, the service itself doesnot care what types of files are uploaded, or when they're uploaded, or whatthey are. It simply receives a file and a file name, and then encrypts anduploads this file under a specific name to the Object Storage.

In eldim's configuration file you can add a list of hosts, as well as their(host)names, and eldim makes sure that all files uploaded from a particularhost will always have that host's name in their name. For example, files fromthe hostmail.example.com, will always have a file name starting withmail.example.com/.

The data collection part is left to the servers sending data to it. It isthem who decide what to send, when to send it, and what operations, such ascompression for example, must be applied to the file.

In order for every server to be able to upload logs or backups to a centralobject storage bucket, they need to have some secrets stored in them. Forexample, in Swift, each server needs to have a username and an API key. Thisis something that is not really secure, as compromising any server would givefull access to the backup repository. An attacker could download files, deletefiles, change them, etc.

In eldim, the servers do not have any stored information, and instead justupload the files to a single server. This server is the one with the access,and can control what operations are being performed, and by whom.

The way eldim works, no server is allowed to mess with another server's files.Servermail.example.com cannot upload files asftp.example.com, even ifthey upload to the very same bucket. eldim automatically prepends all fileuploads with the server hostname, which is inside its configuration file, andnot sent by the servers themselves.

Moreover, eldim will reject files that already exist. If the filemail.example.com/2018-01-01/mail.log.tgz already exists in the object store,it will not allow for it to be overwritten. This check is in place to preventa hacked server from overwritting all previous log entries with empty data,effectively deleting everything.

Finally, eldim works only over HTTPS. This decision is hard coded inside theserver itself, and cannot be changed by the configuration file. A code changeis required. It is configured to only work with at least TLSv1.2, the onlycurrently secure versions of TLS, but currently it may accept some more weakciphers and not only the most secure ones.

Since version v0.6.0, eldim usesage for fileencryption. It is a well defined protocol, with multiple implementations, avery good CLI tool, and is already part of some operating system distributions.More importantly, it is modern, well-designed, and opinionated, with one andonly one purpose in mind: encrypt files. It uses state of the art practicesand algorithms, and is also very flexible.

age is using asymmetric encryption, which means that eldim only needs to knowabout thepublic keys in its configuration file, and never needs or hasaccess to theprivate keys. This vastly reduces the risk of a compromisedeldim server, as files uploaded cannot be decrypted by the attacker.

With age, eldim supports multiple public keys, so you can use more than one,and have the files encrypted with all of them. That means that files can bedecrypted withany of the keys. You can use this functionality to havebackup keys, or give access to multiple people, each one holding their ownkey pair. Unfortunately, eldim currently does not supportM of N so youneed to keep this in mind while threat modelling.

To generate an age keypair, you can use theage-keygen CLI tool. However,a very nice feature is that eldim also supports SSH keys! You can use yourRSA or Ed25519 SSH keys in addition to the age keys. A single eldim serversupports multiple keys, of different types.

eldim runs as a daemon, since it has to listen for HTTPS requestscontinuously. For this reason, you need to ensure that the binary isrunning all the time. The recommended way of achieving this is through youroperating system's startup / init system. If you are usingsystemd, a basicunit file is provided in this repository for you to use.

As with any software, it isnot recommended to run eldim asroot. Forthis reason, you should create aneldim user. The includedsystemd unitfile assumes theeldim user exists in the system.

You can create such user by running:

sudo useradd -s /usr/sbin/nologin -r -M eldim

When executed, eldim has two command line flags that you can use to configureit before it even reads the configuration file. They are:

• -j: When set, it will output all logs in JSON format, instead of plaintext
• -c: The path to the configuration file

As ofeldim v0.2.0, eldim supports metrics exporting usingPrometheus. You can find more information about themetrics currently supported and exportedhere.

In order to read the full documentation on how to configureeldim, clickhere.

You can find the full specification of the HTTP API ofeldim by clickinghere.

You can basically upload files to eldim in any way you like, as long as youfollow the above API, but here are some examples. This code can be for examplein a daily or weekly cron job:

# Compress nginx' access.logtar -zcf /tmp/nginx.access.log.tgz /var/log/nginx/access.log /var/log/nginx/access.log.1# Upload to eldimcurl -F filename=$(date +%F-%H-%M)/access.log -F file=@/tmp/nginx.access.log.tgz https://eldim.example.com/api/v1/file/upload/

The$(date +%F-%H-%M) part will automatically print the date in the2018-01-01-13-37 format (YYYY-MM-DD-HH-MM).

If you are testing eldim, you may use-k incurl, to skip certificatechecks, as you may be using a self-signed certificate. However, deployingthis to production without a trusted certificate isnot recommended.

For production workloads, you may want to use the--retry N flag ofcurl,to retry the request up toN times, if it fails. It is recommended to alsoset the--retry-connrefused flag as well. You can combine the above with--retry-delay X, socurl will sleepX seconds between retries. Goodvalues forX are eldim's domain TTL * 2, or something similar.

eldim is designed to work without placing trust on the file upload servers.If, however, you want to not have to trust the eldim server either, you canoptionally encrypt all data sent to eldim withage (orgpg). That wayeldim won't be able to decrypt them, but neither will the sender alone.

To encrypt files withage, use:

cat file.tgz| age -r"AgeID"> out.tgz.enc

Of course, you need to replace "AgeID" with an age recipient address.

Currently eldim logs a lot of information in detail. This is done on purposeand is not a debugging leftover. Since it is a tool that is related tosecurity, it is always good to have a lot of information to be able to go backto in case something happens.

It is totally normal for eldim to log up to 20 lines per successful uploadrequest, or even more, depending on the configuration.

During service startup, all information logged is related to actions andthe configuration file, and is in plain text. After the service is started,all logs start with a UUID. This is called the Request ID. During thearrival of every request, eldim generates a unique identifier for thisrequest. This identifier is included in every future log file entry thatis related to this request.

By default eldim logs tostdout andstderr, so if you are using theprovidedsystemd unit file, all its logs will be available insyslog.