- Notifications
You must be signed in to change notification settings - Fork68
Rsync-based OSX-like time machine for Linux, MacOS and BSD for atomic and resumable local and remote backups
License
cytopia/linux-timemachine
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Install |Uninstall |TL;DR |Features |How does it work |Restore |Retention |Usage |FAQ |Disclaimer |License
timemachine is a tiny and stableKISS driven andPOSIX compliant script that mimics the behavior of OSX's timemachine.It usesrsync to incrementally back up your data to a differentdirectory, hard disk or remote server via SSH. All operations are incremental, atomic and automatically resumable.
By default it uses the rsync options:--recursive,--perms,--owner,--group,--times and--links.In case your target filesystem does not support any of those options or you cannot use them dueto missing permission, you can explicitly disable them via--no-perms,--no-owner,--no-group,--no-times, and--copy-links.SeeFAQ for examples.
Motivation
The goal of this project is to have a cross-operating system and minimal as possible backup scriptthat can be easily reviewed by anyone without great effort.Additionally it should provide one task only and do it well withoutthe need of external requirements and only rely on default installed tools.
sudo make install
sudo make uninstall
UsingPOSIX.1-2008 argument syntax:
# Recursive, incremental and atomic backup (locally)$ timemachine /source/dir /target/dir# Recursive, incremental and atomic backup (via ssh)$ timemachine /source/dir user@host:target/dir# Recursive, incremental and atomic backup (via ssh with non-standard port)$ timemachine --port 10000 /source/dir user@host:target/dir# Append rsync options$ timemachine /source/dir /target/dir -- --specials --progress$ timemachine /source/dir /target/dir -- --specials --no-perms$ timemachine /source/dir /target/dir -- --archive --progress# Make the timemachine script be more verbose$ timemachine -v /source/dir /target/dir$ timemachine --verbose /source/dir /target/dir# Make the timemachine script be even more verbose$ timemachine -d /source/dir /target/dir$ timemachine --debug /source/dir /target/dir# Make the timemachine script and rsync more verbose$ timemachine -v /source/dir /target/dir -- --verbose$ timemachine --verbose /source/dir /target/dir -- --verbose
| Feature | Description |
|---|---|
| SSH or local | Local backups as well as backups via SSH are supported. |
| Incremental | Backups are always done incrementally using rsync's ability to hardlink to previous backup directories. You can nevertheless always see the full backup on the file system of any incrementally made backup without having to generate it. This will also be true when deleting any of the previously created backup directories. See theBackups section for how this is achieved via rsync. Incremental Backups also mean that only the changes on your source, compared to what is already on the target, have to be backed up. This will save you time as well as disk space on the target disk. |
| Partial | When backing up, files are transmitted partially, so in case a 2GB movie file backup is interrupted the next run will pick up exactly where it left off at that file and will not start to copy it from scratch. |
| Resumable | Not only is this script keeping partial files, but also the whole backup run is also resumable. Whenever there is an unfinished backup and you starttimemachine again, it will automatically resume it. It will resume any previously failed backup as long as it finally succeeds. |
| Atomic[1] | The whole backup procedure is atomic. Only if and when the backup procedure succeeds, it is then properly named and symlinked. Any non-successful backup directory is either waiting to be resumed or to be deleted. |
- [1] The backup process is atomic, but not the backup itself.
rsynccopies files as it finds them and in the meantime there could already be changes on the source. To achieve an atomic backup, either back up from a read-only volume or from a snapshot.
The following directory structure will be created:
$ tree -L 1 /my/backup/folder.├── 2018-01-06__18-43-30/├── 2018-01-06__18-44-23/├── 2018-01-06__18-50-44/├── 2018-01-06__18-50-52/└── current -> 2018-01-06__18-50-52/
current will always link to the latest created backup.All backups are incremental except the first created one.You can nevertheless safely remove all previous folders and the remaining folders will still have all of their content.
Except for the first one, backups are always and automatically doneincrementally,so the least amount of space is consumed.Due torsync's ability, every directory will still contain all files, even though they are justincremental backups. This is archived via hardlinks.
$ du -hd1.497M ./2018-01-06__18-43-3024K ./2018-01-06__18-44-2324K ./2018-01-06__18-50-4424K ./2018-01-06__18-50-52497M.
You can also safely delete the initial full backup directory without having to worry about losingany of your full backup data:
$ rm -rf ./2018-01-06__18-43-30$ du -hd1.497M ./2018-01-06__18-44-2324K ./2018-01-06__18-50-4424K ./2018-01-06__18-50-52497M.
rsync andhardlinks are magic :-)
In case thetimemachine script aborts (self-triggered, disk unavailable or for any other reason)you can simply run it again to automaticallyresume the last failed run.
This is due to the fact that the backup process isatomic. During a non-complete run,all data will be stored in a directory named.inprogress/. This will hold all alreadytransferred data and will be picked up during the next run.Once the backup is complete, it will be renamed and symlinked tocurrent.
$ tree -a -L 1 /my/backup/folder.├── .inprogress/├── 2018-01-06__18-43-30/├── 2018-01-06__18-44-23/├── 2018-01-06__18-50-44/├── 2018-01-06__18-50-52/└── current -> 2018-01-06__18-50-52/
No special software is required to restore your data. Backed up files can be easily browsed andthus copied back to where you need them. Recall the backup directory structure:
$ tree -L 1 /my/backup/folder.├── 2018-01-06__18-43-30/├── 2018-01-06__18-44-23/├── 2018-01-06__18-50-44/├── 2018-01-06__18-50-52/└── current -> 2018-01-06__18-50-52/
Chose a backup directory and simply copy them to where you need it:
# Test it out in dry run mode before applying$ rsync --archive --progress --dry-run /my/backup/folder/2018-01-06__18-50-52/ /src/# Apply restoration$ rsync --archive --progress /my/backup/folder/2018-01-06__18-50-52/ /src/
As decribed above this project isKISS driven and only tries to do one job:back up your data.
Retention is a delicate topic as you want to be sure that data is removed as intended. For this there are already well-established tools that do an excellent job and have proven themselves over time:tmpreaper andtmpwatch.
The script is written and maintained with maximum care.In order to retain a reliable and stable backup solution, a lot of effort goes into a vast amount ofintegration and regression tests.These tests not only give you measurable confidence, but also help new contributors to notaccidentally introduce new or old bugs.
$ timemachine -hUsage: timemachine [-vd] <source> <dest> -- [rsync opts] timemachine [-vdpi] <source> <host>:<dest> -- [rsync opts] timemachine [-vdpi] <source> <user>@<host>:<dest> -- [rsync opts] timemachine [-vdpi] <source> <ssh-alias>:<dest> -- [rsync opts] timemachine [-vdpi] <host>:<source> <dest> -- [rsync opts] timemachine [-vdpi] <user>@<host>:<source> <dest> -- [rsync opts] timemachine [-vdpi] <ssh-alias>:<source> <dest> -- [rsync opts] timemachine -V, --version timemachine -h, --helpThis shell script mimics the behavior of OSX's timemachine.It uses rsync to incrementally back up your data to a different directory or remote server via SSH.All operations are incremental, atomic and automatically resumable.By default it uses --recursive --perms --owner --group --times --links.In case your target filesystem does not support any of those options, you can explicitlydisable those options via --no-perms --no-owner --no-group --no-times and --copy-links.Required arguments: <source> Local source directory <dest> Local destination directory. <host>:<dest> SSH host and source/destination directory on server <user>@<host>:<dest> SSH user, SSH host and source/destination directory on server <ssh-alias>:<dest> SSH alias (defined in ~/.ssh/config) and source/destination directory on serverOptions: -p, --port Specify alternative SSH port for remote backups if it is not 22. -i, --identity Specify path to SSH key. -v, --verbose Be verbose. -d, --debug Be even more verbose.Misc Options: -V, --version Print version information and exit -h, --help Show this help screenExamples: Simply back up one directory recursively timemachine /home/user /data Do the same, but be verbose timemachine -v /home/user /data Append rsync options and be very verbose timemachine -d /home/user /data -- --progress --verbose Log to file timemachine -v /home/user /data > /var/log/timemachine.log 2> /var/log/timemachine.errDocumentation: View more examples at: https://github.com/cytopia/linux-timemachineThe following can be used as an example crontab entry. It assumes you have an external disk (NFS, USB, etc..) that mounts at/backup. Before adding the crontab entry, ensure the filesystem in/backup is mounted and use:
$ touch /backup/mounted
This guards against accidentally backing up to an unmounted directory
Next, add the following to crontab usingcrontab -e as whichever user you intend to run the backup script as. You may need to place this in the root crontab if you are backing up sensitive files that only root can read
0 2***if [[-e /backup/mounted ]];then /usr/local/bin/timemachine /home/someuser /backup;fi
This will causetimemachine to run at 2AM once per day. Sincetimemachine keeps track of backups with granularity up to the hour, minute and second, you could have it run more than once per day if you want backups to run more often.
Should I add trailing directory slashes (/)?
Trailing directory slashes only matter for the source directory and will not make a differenceif added to the destination directory.
# The following backs up the contents of the src directory$ timemachine src/ dst/$ tree -L 2 /dst.├── 2018-01-06__18-43-30/│ └── file.txt└── current -> 2018-01-06__18-43-30/
# The following backs up the the src directory itself$ timemachine src dst/$ tree -L 3 /dst.├── 2018-01-06__18-43-30/│ └── src│ └── file.txt└── current -> 2018-01-06__18-43-30/
How to dry-run the backup?
$ timemachine src/ dst/ -- --dry-run
How to use a non-standard SSH port?
$ timemachine --port 1337 src/ user@host:dst/
How to use SSH aliases from~/.ssh/config?
$ cat~/.ssh/configHost my-ssh-alias HostName 192.168.0.1 Port 1234 User john IdentityFile~/.ssh/id_rsa__alternative_key
$ timemachine src/ my-ssh-alias:dst/
How to speed up remote backups?
# With this option, rsync compresses the file data as it is sent to the des‐# tination machine, which reduces the amount of data being transmitted$ timemachine src/ user@host:dst/ --compress
How to preserve ACLs?
$ timemachine src/ dst/ -- --acls
How to preserve extended attributes?
$ timemachine src/ dst/ -- --xattrs
How to disable preserving file and directory permissions?
$ timemachine src/ dst/ -- --no-owner --no-perms
How to disable preserving modification time?
$ timemachine src/ dst/ -- --no-times
How to copy the content instead of a symlink?
# This is useful in case your file system does not support symlinks.# It is recommended to read rsync man page about symlinks to be sure# about what you are doing$ timemachine src/ dst/ -- --copy-links --safe-links$ timemachine src/ dst/ -- --copy-links --safe-links --keep-dirlinks
How to ensure all files in the back up have the ownership of current user?
# Regardless of who owns the files, ensure the backup has uid and gid of current user# This will only work if you have read-permission on all files.$ timemachine src/ dst/ -- --no-owner --no-group# If you do not have permission to read all files, you require sudo or root permission.# The following will instruct rsync to ensure the backed up data has the uid and gid# of the desired user.$ sudo timemachine src/ dst/ -- --chown=<user>:<group>
Backups are one of the most important things. We all care about our data and want it to be safe,so do not blindly trust scripts when it comes to backups!Doreview the code,it is not too complex and kept as short as possible. Have a look at how much effort goes into theintegration tests to provide measurablestability.
Learn aboutrsync it is a very powerful tool and you mighteven be able to just use this for backups.
There are many other backup solutions out there that might be a better fit for your needs. Do yourown research, look at GitHub issues, source code, integration tests and try them out as well.
ContributingSeeContributing guidelines to help to improve this project.
Copyright (c) 2017cytopia
About
Rsync-based OSX-like time machine for Linux, MacOS and BSD for atomic and resumable local and remote backups