Setting Up Storage

services:archivebox:...volumes:# your index db, config, logs, etc. should be stored on a local SSD (usually <10Gb)            -./data:/data# but bulk archive/ content can be located on an HDD or remote filesystem            -/mnt/archivebox-s3/data/archive:/data/archive

Related Docs

Supported Local Filesystems

EXT4 (default on Linux),APFS (default on macOS)

Tip

These default filesystems are fully supported by ArchiveBox on Linux and macOS (w/wo Docker).

ZFS (recommended for best experience on Linux/BSD) ⭐️

Tip

This is the recommended filesystem for ArchiveBox on Linux, macOS, and BSD (w/wo Docker).
apt install zfsutils-linux
_{Provides RAID, compression, encryption, deduping, 0-cost point-in-time backups, remote sync, integrity verification, and more...}

# create a new archivebox pool to hold your datasetzpool create -f \    -O mountpoint=/mnt/archivebox \    -O sync=standard \    -O compression=lz4 \    -O recordsize=128K \    -O dnodesize=auto \    -O atime=off \    -O xattr=sa \    -O acltype=posixacl \    -O aclinherit=passthrough \    -O utf8only=on \    -O normalization=formD \    -O casesensitivity=sensitive \    archivebox /dev/disk/by-uuid/disk1... /dev/disk/by-uuid/disk2...# create the archivebox/data ZFS datasetzfs create \    -o mountpoint=/mnt/archivebox/data \    archivebox/data# optional: add encryption    -o encryption=on \    -o keysource=passphrase,prompt \

NTFS,HFS+,BTRFS

Warning

These filesystems are likely supported, but are not officially tested.

EXT2,EXT3,FAT32,exFAT

Caution

Not recommended. Cannot store files >4GB or more than 31k ~ 65k Snapshot entries due to directory entry limits.

Supported Remote Filesystems

Important

data/index.sqlite3 is your main archive DB,it must be on a fast, reliable, local filesystem which supportsFSYNC (SSD/NVMe recommended for best experience).

Tip

If you use a remote filesystem, you should switch ArchiveBox's search backend fromripgrep tosonic (orFTS5).
_{(ripgrep scans over every byte in the archive to do each search, which isslow and potentially costly on remote cloud storage)}

NFS (Docker Driver)

services:archivebox:volumes:            -./data:/data            -archivebox-archive:/data/archivevolumes:archivebox-archive:driver_opts:type:"nfs"o:"addr=some-remote-server.example.com,nolock,soft,rw,nfsvers=4"device:":/archivebox-archive"

SMB /Ceph (Docker CIFS Driver)

services:archivebox:volumes:            -./data:/data            -archivebox-archive:/data/archivevolumes:archivebox-archive:driver:localdriver_opts:type:cifsdevice:"//some-remote-server.example.com/archivebox-archive"o:"username=XXX,password=YYY,uid=911,gid=911"

Amazon S3 / Backblaze B2 / Google Drive / etc. (RClone)

# install the RClone and FUSE packages on your hostapt install rclone fuse# or brew install# IMPORTANT: needed to allow FUSE drives to be shared with Dockerecho'user_allow_other'>> /etc/fuse.conf

Tip

You can also createrclone.conf using the RClone Web GUI:rclone rcd --rc-web-gui

# Example rclone.conf using Amazon S3 for storage:[archivebox-s3]type = s3provider = AWSaccess_key_id = XXXsecret_access_key = YYYregion = us-east-1

RClone Config Examples

Option A: Running RClone on Bare Metal host

rclone sync --fast-list --transfers 20 --progress /opt/archivebox/data/archive/ archivebox-s3:/data/archivemv /opt/archivebox/data/archive /opt/archivebox/data/archive.localbackup

rclone mount    --allow-other \                # essential, allows Docker to access FUSE mounts    --uid 911 --gid 911 \          # 911 is the default used by ArchiveBox    --vfs-cache-mode=full \        # cache both file metadata and contents    --transfers=16 --checkers=4 \  # use 16 threads for transfers & 4 for checking    archivebox-s3/data/archive:/opt/archivebox/data/archive         # remote:local

Tip

You can use any RClone FUSE mounts as a normal volumes (bind mount) for Docker ArchiveBox, typically no storage plugin is needed as long asallow-other is setup properly.

services:archivebox:...volumes:            -./data:/data            -/opt/archivebox/data/archive:/data/archive

Option B: Running RClone with Docker Storage Plugin

docker plugin install rclone/docker-volume-rclone:amd64 --grant-all-permissions --alias rcloneln -sf~/.config/rclone/rclone.conf /var/lib/docker-plugins/rclone/config/rclone.conf

services:archivebox:volumes:            -./data:/data            -archivebox-s3:/data/archivevolumes:archivebox-s3:driver:rclonedriver_opts:remote:'archivebox-s3/data/archive'allow_other:'true'vfs_cache_mode:fullpoll_interval:0uid:911gid:911transfers:16checkers:4

docker compose run archivebox /bin/bash'ls -lah /data/archive/ | tee /data/archive/.write_test.txt'

More Docker Storage Plugins