Time Machine toAPFS: Evolution
For anyone who uses Time Machine, one of the major improvements in Big Sur is that it can now make its backups to APFS volumes. In this series, I’m going to explore what this means, how it works, and how it can go wrong. To distinguish between what are effectively two different systems, I’m going to refer to Time Machine backing up to APFS asTMA, and when backing up to HFS+ asTMH. One other caution: although I’ve been testing and using TMA since last summer, knowledge of and experience with it remain limited. If you spot any errors, please don’t hesitate to correct me.
The Past
It’s easy to forget, but TMH was released in Mac OS X 10.5 not just as an enhancement for all users, but to support new hardware products, Time Capsules, which Apple introduced in January 2008, just three months after we first started using TMH.
From its release, TMH has been dependent on features of the HFS+ file system to create its Finder illusion. Every hour the backup service examined the record of changes made to the file system since the last backup was made, using its FSEvents database. It worked out what had changed and needed to be copied into the backup. During the backup phase itself, it only copied across those files which had been created or changed since the last backup was made.
It still does this in TMH by usinghard links in the backup, and Apple added a new feature to its HFS+ file system to support this:directory hard links. Where an entire folder has remained unchanged since the last backup, TMH simply creates a hard link to the existing folder in that backup. Where an existing file has been changed, though, the new file is written to the backup inside a changed folder, which in turn can contain hard links to unchanged contents.
This preserves the illusion that each backup consists of the complete contents of the source, but only requires the copying of changed files, and creation of a great many hard links to files and folders. It’s also completely dependent on the backup volume using the HFS+ file system, to support those directory hard links.
Without the directory hard link, backups would quickly be overwhelmed by hard links to files. If you had a million files and folders on the backup source volume, every hourly backup would have to create a total of a million copied files or hard links for those which remained unchanged. Directory hard links enable the efficiency needed for this scheme to work.
Apple later introduced what it called Mobile Time Machine, intended for laptops which could be away from their normal backup destination for some time. In around 10,000 lines of code, Mac OS X came to create something like a primitive snapshot, but on HFS+.
When Apple released the first version of APFS on Mac OS X in High Sierra, its new snapshot feature was incorporated into TMH. They were initially used instead of the FSEvents database to determine what should be backed up.
Since then, making each backup of an APFS volume has involved creating a snapshot, but that is stored locally on the APFS volume being backed up. The structure of backups themselves didn’t change: they still required an HFS+ volume and used directory hard links.
Catalina introduced a more complicated scheme to replace snapshots as the normal means for determining what to back up. This was presumably because computing a snapshot delta proved slow, and the introduction of the Volume Group, with specialist types of APFS volume for which snapshot deltas would be inappropriate or impossible.
Determining what gets backed up in Catalina can thus depend on snapshots, and snapshots are made during each backup, but because the backup destination remains in HFS+ format it can’t use snapshots itself, and instead still has to rely on directory hard links.
TMH
Big Sur retains the option to continue using TMH to backups stored on HFS+ volumes, where it fixes one bug which for many made TMH unusable in Catalina: in Big Sur, TMH should no longer choke when trying to back up large hidden folders containing a volume’s version database, as those are now excluded from backups. They were in any case of limited value, as they could only be restored when a whole volume was being restored, and not when restoring only part of the volume.
TMA
TMA interestingly reverses the design of TMH in High Sierra: instead of using snapshots to determine what needs to be backed up before creating a backup using traditional hard links, most of the time TMA determines what’s changed using the traditional method with FSEvents, then creates its backup as a snapshot on the backup volume. The latter is essential, as without directory hard links, there’s no way of using the TMH method to make backups to an APFS volume.
TMA backup volumes are special: you can’t just use any convenient APFS volume to store backups. They use the case-sensitive variant of APFS, which normally isn’t used in macOS but is the standard for iOS/iPadOS, volume ownership is enabled, and the volume is assigned the role of Backup.
Backups are called using the same timer system involving DAS and CTS. TMA (at least) might now bring forward a backup when there have been many events recorded in FSEvents: I believe that I have precipitated this when writing over 130 GB of test files during SSD benchmarking, for example.
As with TMH, several strategies are available to determine which files need to be backed up from each volume. Local snapshots are made and stored on each APFS volume to be backed up, then the decision is made as to which strategy to use. On most occasions, when backing up a Data volume or similar, this prefers the use of FSEvents, as used prior to High Sierra. Note that, unlike previous versions of Time Machine, in Big Sur neither TMH nor TMA can back up the System volume.
A detailed assessment of the items to be backed up is then made, thus the total size to be backed up is forecast. Once that is complete, the local snapshot is copied to an .inprogress folder on the backup volume, and backup copying proceeds. Where possible, only changed blocks of files are copied, rather than having to copy the whole of every file’s data, an option termeddelta-copying, which can lead to significant savings. Old backups are removed both according to age, and to maintain sufficient free space on the backup volume, in what TMA refers to asage-based andspace-based thinning.
Data copied to assemble the backup on the backup volume is formed into a snapshot, which is then used to present the contents of that backup both in the Time Machine app and the Finder. Those snapshots appear in /Volumes/.timemachine/ although they’re still stored on the backup volume.
One puzzling process which appears periodically is what’s shown as backup activity on the backup volume, when TMA isn’t making a backup. This appears to be housekeeping performed on the backup volume, including Spotlight indexing of the snapshots stored there.
TMA appears to be considerably quicker and more efficient than TMH, and doesn’t seem prone to choking problems which have troubled recent versions of TMH. However, its snapshot system is less flexible that the hard links of TMH. In Big Sur, TMA’s snapshots aren’t whole-volume, only including the folders which are being backed up. Unlike in TMH, though, at present it doesn’t appear possible to delete any of the contents of an existing backup, only the whole of the backup stored in any given snapshot. These new part-volume snapshots don’t yet appear to be available to users.
The end result is, in almost every respect, superior to TMH, and Apple rightly recommends users to prefer it.
Share this:
- Click to share on X (Opens in new window)X
- Click to share on Facebook (Opens in new window)Facebook
- Click to share on Reddit (Opens in new window)Reddit
- Click to share on Pinterest (Opens in new window)Pinterest
- Click to share on Threads (Opens in new window)Threads
- Click to share on Mastodon (Opens in new window)Mastodon
- Click to share on Bluesky (Opens in new window)Bluesky
- Click to email a link to a friend (Opens in new window)Email
- Click to print (Opens in new window)Print
28Comments
Add yours- 1
zebrum Regarding TMA, do you mean the backup volume shows the latest backup as browsable files? Then when a backup starts it first snapshots the backup volume so that version is saved? Presumably with the wrong date? Since the date will be the current date new backup is happening rather than date old one finished. Then do you mean that it now copies the diff to the backup volume bringing it up to date with source volume? Adding, deleting and modifying files?
Isn’t there a utility to view the backup volume at any snapshot time at the command line?
- 2
hoakley Thank you.
All the backups are shown in the Finder and the TM app as browsable files – that’s the same illusion as created by TMH which TMA makes using its new method of snapshots.
TMA doesn’t make snapshots of the volume on which its backups are stored, but makes snapshots of the volumes which it is backing up. Unlike TMH, those snapshots (the file system metadata) are copied across to the backup volume, where they’re used to create the snapshot which forms the backup.
When FSEvents are being used to determine what to back up, no diff is performed between snapshots – that’s a method which apparently fell out of favour after its was introduced in High Sierra.
I believe that you can browse backups from the command line in the normal way that you browse any snapshot, as that’s what they are, although their mountpoint is via /Volumes/.timemachine/.
Howard.- 3
Duncan With TMH I usually used the TimeMachine app to locate an old file, but often used the Finder to copy it back to the desired volume. (Sometimes I want it in a different folder than where it was originally backed up.) Or, if I knew where the older file was located on the backup disk, I’d use the Finder itself to browse the TM file system hierarchy and then drag the file back to the original disk. No TimeMachine app used at all in this latter case.
Is this functionality still available with TMA? (I’m not particularly keep on the TimeMachine app, and would rather use the Finder for some recovery tasks.)
- 3
- 2
- 5
EcleX Many thanks for the interesting review. Do you know if Time Machine of macOS Big Sur on APFS disks saves just the bits modified when a virtual machine is opened and closed, or does it backup the full virtual machine disk image, as Time Machine for HFS+ does?
I ask because such disk images may be big (eg., 10 to 30 GB or more). For instance the ones of VMware Fusion are stored as “Name.vmwarevm” in “/Users/~/Documents/Virtual Machines.localized”.
- 6
hoakley Thank you.
I don’t know. However if you refer tomy previous article specifically about backing up VMs, this is only part of the problem which needs to be solved. Time Machine, whether TMA or TMH, isn’t designed to back up VMs, and you’re far better using other techniques, which are discussed in the comments to that article.
Howard.LikeLike
- 10
Dariusz Bogdanski (@Wendish) I use VMs extensively (VMware among them) and I can confirm that TMA indeed only backups modified bits. Sometimes on my system it says during backup that 100GB has to be copied and only few GB are copied in reality. I have double checked it by comparing backup destination disk space before and after.
- 11
kapitainsky TMA is very space efficient for VM images but you have to make sure that you only backup stable images (not running VM). It is not doable with how default TM works unless you do this manually which is not ideal for any backup.
I solved this by controlling TM via my own script and launchd – then you can not only run TM with schedule of your choice but also include additional conditions e.g. in my case backup does not run when any VM is active.
- 6
- 13
Dariusz Bogdanski (@Wendish) There is one shortcoming of TMA compared to TMH – it looks like Apple stopped using files checksumming and from what I can see so far there is no way to verify backup consistency. There is “verify backup” for sparsebundle based backups but what it does is only checking file system consistency (fsck_apfs). This is unfortunately also only option for local backup.
It is massive issue for any serious backup solution if backups cant be verified.
- 14
hoakley Thank you.
I think that very few users use this feature. It may yet return. In the meantime, I’ll look at how you might be able to use my Fintch/Dintch/cintch.
Howard.- 15
kapitainsky Six of one and half a dozen of the other:) Every workflow is different. Personally I prefer to have a bit more granular control over TM behaviour. Defaults are good – and kudos to Apple for such easy to use backup solution for desktop/laptop workstation. But it is painfully limited when user has a bit more specific needs.
Thank you for your quest to shed some light on inners working of macOS. Us mortals can try to use it to do better what Apple forgot.
- 15
- 14
- 17
Dariusz Bogdanski (@Wendish) Another problem I can see is total lack of documentation regarding APFS backup volumes. How I can migrate such backup to another disk (e.g. to new bigger disk). How to migrate backup from local drive to sparsebundle or vice versa. It was not straightforward by possible with TMH.
It’s great that TMA is faster and more space efficient. But as overall backup solution it is one step forward, two steps back.
- 21
Simon When now using TMA with Big Sur I’ve noticed that the root of the TM disk has become read-only even to an admin user. You can’t use that disk for anything but TM. Much unlike previous TMH where all TM backups resided in a folder and next to it you could store other items on the root of that disk. Is this read-only change a BigSur thing or a TMA thing?
- 22
kapitainsky You can’t use volume where time machine snapshots live. But you can always add (and delete) other volumes in the same container and use them as use like – including even for time machine for another mac. This is what I do. Please note that all volumes share the same disk space and time machine will eventually eat all it can find so it is wise to use “Reserved Size” parameter during APFS volume creation.
IMHO it is much nicer solution than before with HFS+ filesystem.
- 22
- 24
Simon And one more question regarding TMA vs. TMH. Back in the day, when your Mac crashed and you had to reboot, TMH would on its next invocation write to log that it had to do a deep traversal (supposedly because it’s usual fsevents logging was interrupted during the crash) and that meant it was going to take much longer. That no longer shows up in logs and I suppose vanished either with TMA (or with Mojave/Catalina?). Could you indicate where in your TMH diagrams that step occurs?
- 25
hoakley Thank you.
Deep traversals occurred because TMH had no other option when there had been a problem with FSEvents.
Since High Sierra, TMH hasn’t relied just on FSEvents. In High Sierra, they were ignored in favour of doing snapshot diffs, and later TMH was had a range of strategies to choose from. It’s at that stage the decision is made: in most cases in TMA, FSevents is the default; then that’s not available, the another strategy is used instead.
Howard.LikeLike
- 25
- 26
marksealeyfor3 Thanks for this, Howard!
I was particularly interested in: ‘…One puzzling process which appears periodically is what’s shown as backup activity on the backup volume…’
When I first started using Big Sur that activity was almost constant… could go on for hours and hours. I worried for the durability of my hardware.
It does seem, though, as if such constant spinning has decreased significantly after what has been three months. I hope I’m right, and right to pay little attention to it now, assuming that it’s keeping Spotlight content.
- 28
·Comments are closed.
The Eclectic Light Company