This is just half of what Time Machine does. What people are constantly missing is that Apple Time Machine is fast, as it does not need to walk through the whole filesystem to find changed files. Thanks to FSEvents, introduced in Mac OS X Leopard, it knows which directories actually contain changed files and hence usually only needs to check a small fraction of the filesystem. (Not sure if it still works that way after the switch to APFS).
This would of course also be possible on Linux (using *notify), and there are some projects which try to do this, but it's really hard to do it reliably. You might argue that this feature is less important nowadays because NVME SSDs are so fast, but still, I remember very well how astonished I was that creating a new time machine snapshot on OS X Leopard took mere seconds.
I lost all my files to Time Machine in 2008. I don't remember exactly what happened. But since then I'll take a slightly slower, observable command-line copy over sparkly magic.
Yes, I do not trust TM. That's why I have both a backup with TM for convenience and also to have all the files (including system files), and a mirror of the important files (basically my home directory) with `rsync`.
No, the right way to do this on Linux and FreeBSD is to use zfs with zfs send/receive. Creating snapshots and sending them is efficient enough to use it as the underlying storage for moderately loaded databases and VMs.
They are atomic and require zero downtime. They can be encrypted and resent to other machines. Cloning whole machines from them is easy and efficient.
Yeah, give borg a look. It's just faster to back up, faster to delete old backups, and just easier to do restores because so long as you have the appropriate credentials you can list the archive from any machine.
I think there's still a place/use for --link-dst and hardlinks, but as a backup system I think borg does it better.
I’m sure others will chime in that they used hard links like this before then, however as noted in that page, it’s the one that made it popular enough that rsync was updated to support the idea natively.
Yes, it is; among a few other great cross-platform FOSS tools that are built just for backups, and they do it really well. But most of them do periodic scans (as opposed to file change trigger-based backup runs, which I guess is what you might be looking for, I assume by your second sentence).
FSEvents can be used for triggering a backup on change, but in the case of emulating their use in Time Machine, the goal is to use them to list all directories which have been changed since the last backup to avoid unnecessary rescanning.
I remember doing my own backups with rsync a super long time ago! It worked pretty well, but it didn't have compression, encryption, desktop integration, or deduplication, and adding anything would be one more script to maintain.
Now I use Kopia, no real complaints there. I used to use Borg but until recently it needed some fussy extra community package to work on Synology.
I use bontmia since forever, based on the same rsync feature (link-dest for creating hard links to the unmodified files since the last backup). It also supports backup rotation and I think it's quite solid/reliable after all these years: https://github.com/hcartiaux/bontmia
If we spoke about a Linux box, one that prudently ran ZFS, or XFS on top of LVM, it would be possible to make a snapshot before the diffing and sending, so that the snapshot would be indeed point-in-time. IDK if whatever macOS uses for the filesystem supports snapshots.
Otherwise, I think, restic or kopia are better for proper backups, and Syncthing for keeping a mirror copy. But the simplicity of this script in charming.
Have you ever faced a case when restic would not restore a backup? Or would not mount a remote backup?
I have a mirror that Syncthing maintains, with some staggered versioning; it seems similar, and should help me immediately in a case of clobbering something. Restic backups helped me several times in cases of catastrophic failures, and are now the default way of moving my stuff to a new machine.
"borg" has basically solved backups permanently. It's deduplicated, even across snapshots, compressed, and end-to-end encrypted. I'm surprised it's not more well known.
> "borg" has basically solved backups permanently. It's deduplicated, even across snapshots, compressed, and end-to-end encrypted.
Deduplication helps minimize space, but isn't it a major liability in backups? I mean, what happens when you try to restore your backups but a lone sector holding a file from way back in the past happens to not be recoverable? Doesn't it mean that no matter how frequent your backups are, your data is lost?
If you are storing your data in a filesystem like ZFS then the risk is lower since you'll know you have a bad sector long before you attempt to deduplicate. But I otherwise share your same concerns and I'm eager to hear from others how to mitigate this. I've lost a volume due to poor practice while encrypting so I'm understandably reluctant to overcomplicate my backup strategies in the name of space compression or privacy.
This is why Proxmox Backup Server has scheduled "verify" tasks which look at the sha256 of every stored hunk to make sure it hasn't rotted on disk. (If it has, it makes sure to re-acquire that data on the next backup - hopefully it's still available!)
It doesn't really hurt in practice because it's only one part of the full backup procedure. Deduplicate to save space; re-duplicate the (smaller) backups to separate media for redundancy; scrub regularly for bit rot and rotate your media. A proper backup system requires all but the first of those anyway.
the backups in borg and restic are not classical copies, each snapshot is not independent, it is a collection of hashed blocks. In a sense, time machine and hard links do the same: they reference the same bits across snapshots, saving space. for the bit rot / faulty sector or failed drive, you use redundancy with RAID or similar in your file system. if all else fails and your backup system burns in a fire, then you reach for your other backup. deduplication and redundancy solve different problems.
What? You actually think that the mere idea of a backup getting corrupted is something that is "bordering on paranoia"? Have you ever heard of redundancy or even RAID?
> If the bad sector contains a critical part of the filesystem, you're going to lose everything anyway.
Do you honestly failed to understand that the problem is actually the "lose everything" part?
I’m currently trying to decide between Borg and Restic myself. Borg has worked very well for me in the past, and I’m a bit excited by the new Vorta GUI frontend. But I already have a BackBlaze B2 setup for backing up my Mac, and Restic would let me continue to use that as a backup store. I’m interested in hearing about differentiating points between them.
Primarily it's the reason you already know: restic and borg are the same model, but restic doesn't need it to be an ssh-accessible filesystem on the remote end. Restic can send backups almost anywhere, including object storage like your Backblaze B2 (that's what I use with restic, too). I agree with OP: restic is strictly better. There's no reason to use borg today; restic is a superset of its functionality.
I don't know specifically, but it's a self-contained single file Go executable. It doesn't need much from a Linux system beyond its kernel. Chances are good that it'll work.
Used this technique (albeit counter-based, not time-based) successfully for backing up the 2006 World Cup site newsfeed[0] configurations/data. Had 6 rotating backups on two hosts. Never had to use them (as best I remember) but it was definitely comforting to know they were there in case of urgent rollback.
[0] Not strictly the right name but I've forgotten. Y!uk people of that era know what I mean.
And then once all references to the inode are removed (by rotating out backups) it's freed. So there's no maintenance of the deduping needed, it's all just part of how the filesystem and --link-dest work together.
The hard links are to the most recent backup before the one happening now in the script, so that you aren't storing full copies of files that haven't changed between backups.
It does make an actual copy but then it builds a directory structure that you can browse by date (like Time Machine). That directory contains hard links so only one copy of a file is ever backed up but you see contents that match the date of the backup.
Once you have btrfs you don't really need rsync anymore, its snapshot + send/receive functionality are all you need for convenient and efficient backups, using a tool like btrbk.
This is just half of what Time Machine does. What people are constantly missing is that Apple Time Machine is fast, as it does not need to walk through the whole filesystem to find changed files. Thanks to FSEvents, introduced in Mac OS X Leopard, it knows which directories actually contain changed files and hence usually only needs to check a small fraction of the filesystem. (Not sure if it still works that way after the switch to APFS).
This would of course also be possible on Linux (using *notify), and there are some projects which try to do this, but it's really hard to do it reliably. You might argue that this feature is less important nowadays because NVME SSDs are so fast, but still, I remember very well how astonished I was that creating a new time machine snapshot on OS X Leopard took mere seconds.
I lost all my files to Time Machine in 2008. I don't remember exactly what happened. But since then I'll take a slightly slower, observable command-line copy over sparkly magic.
Yes, I do not trust TM. That's why I have both a backup with TM for convenience and also to have all the files (including system files), and a mirror of the important files (basically my home directory) with `rsync`.
No, the right way to do this on Linux and FreeBSD is to use zfs with zfs send/receive. Creating snapshots and sending them is efficient enough to use it as the underlying storage for moderately loaded databases and VMs.
They are atomic and require zero downtime. They can be encrypted and resent to other machines. Cloning whole machines from them is easy and efficient.
Another thing Time Machine (hopefully) does is append-only backups.
Ha. That's a throwback.
I did the same thing, but with a more detailed writeup, in 2009: https://nuxx.net/blog/2009/12/06/time-machine-for-freebsd/
It was really handy, but I now use borg as it just works better.
HA! It read your post I read back then and I have been using it ever since. So thank you. I guess I need to check out borg ;)
Small world!
You're welcome.
Yeah, give borg a look. It's just faster to back up, faster to delete old backups, and just easier to do restores because so long as you have the appropriate credentials you can list the archive from any machine.
I think there's still a place/use for --link-dst and hardlinks, but as a backup system I think borg does it better.
For reference: https://nuxx.net/blog/2019/11/10/using-borg-for-backing-up-n...
The original post that introduced this idea into general public: http://www.mikerubel.org/computers/rsync_snapshots/
I’m sure others will chime in that they used hard links like this before then, however as noted in that page, it’s the one that made it popular enough that rsync was updated to support the idea natively.
Seems similar to https://rsnapshot.org/
This is the more robust way to go. Uses rsync under the hood.
Does rsnapshot do any of the hard thinking? Or just configures, then hands off to rsync?
I've been using snapshot for backups since so many years that I forgot when I started. I keep the last 14 snapshots.
Isn’t restic better for backups overall?
Anyone have a good script for macOS triggered by launchd, ideally something that uses FSEvents to check for directory changes?
Yes, it is; among a few other great cross-platform FOSS tools that are built just for backups, and they do it really well. But most of them do periodic scans (as opposed to file change trigger-based backup runs, which I guess is what you might be looking for, I assume by your second sentence).
FSEvents can be used for triggering a backup on change, but in the case of emulating their use in Time Machine, the goal is to use them to list all directories which have been changed since the last backup to avoid unnecessary rescanning.
I use restic, too, and I am very pleased with it, although I run it on windows, which works, but isn't perfect. So, you use it regularly?
I remember doing my own backups with rsync a super long time ago! It worked pretty well, but it didn't have compression, encryption, desktop integration, or deduplication, and adding anything would be one more script to maintain.
Now I use Kopia, no real complaints there. I used to use Borg but until recently it needed some fussy extra community package to work on Synology.
I use bontmia since forever, based on the same rsync feature (link-dest for creating hard links to the unmodified files since the last backup). It also supports backup rotation and I think it's quite solid/reliable after all these years: https://github.com/hcartiaux/bontmia
If we spoke about a Linux box, one that prudently ran ZFS, or XFS on top of LVM, it would be possible to make a snapshot before the diffing and sending, so that the snapshot would be indeed point-in-time. IDK if whatever macOS uses for the filesystem supports snapshots.
Otherwise, I think, restic or kopia are better for proper backups, and Syncthing for keeping a mirror copy. But the simplicity of this script in charming.
Yes, APFS supports snapshots. [0]
[0] https://support.bombich.com/hc/en-us/articles/20686443871383...
> But the simplicity of this script in charming.
An important feature of backups is the ability to restore them. As much as I love restic, I have at least one backup target with hard links.
Have you ever faced a case when restic would not restore a backup? Or would not mount a remote backup?
I have a mirror that Syncthing maintains, with some staggered versioning; it seems similar, and should help me immediately in a case of clobbering something. Restic backups helped me several times in cases of catastrophic failures, and are now the default way of moving my stuff to a new machine.
"borg" has basically solved backups permanently. It's deduplicated, even across snapshots, compressed, and end-to-end encrypted. I'm surprised it's not more well known.
> "borg" has basically solved backups permanently. It's deduplicated, even across snapshots, compressed, and end-to-end encrypted.
Deduplication helps minimize space, but isn't it a major liability in backups? I mean, what happens when you try to restore your backups but a lone sector holding a file from way back in the past happens to not be recoverable? Doesn't it mean that no matter how frequent your backups are, your data is lost?
In that case you are supposed to use your /other/ backup. Which you have.
Defeating the point of deduplication...
You don't have backups if you only have one "backup". Look up sysadmin's 3-2-1.
On a single drive, deduplicate all you want. But you need to have two drives, at least, anyway.
If you are storing your data in a filesystem like ZFS then the risk is lower since you'll know you have a bad sector long before you attempt to deduplicate. But I otherwise share your same concerns and I'm eager to hear from others how to mitigate this. I've lost a volume due to poor practice while encrypting so I'm understandably reluctant to overcomplicate my backup strategies in the name of space compression or privacy.
This is why Proxmox Backup Server has scheduled "verify" tasks which look at the sha256 of every stored hunk to make sure it hasn't rotted on disk. (If it has, it makes sure to re-acquire that data on the next backup - hopefully it's still available!)
It doesn't really hurt in practice because it's only one part of the full backup procedure. Deduplicate to save space; re-duplicate the (smaller) backups to separate media for redundancy; scrub regularly for bit rot and rotate your media. A proper backup system requires all but the first of those anyway.
the backups in borg and restic are not classical copies, each snapshot is not independent, it is a collection of hashed blocks. In a sense, time machine and hard links do the same: they reference the same bits across snapshots, saving space. for the bit rot / faulty sector or failed drive, you use redundancy with RAID or similar in your file system. if all else fails and your backup system burns in a fire, then you reach for your other backup. deduplication and redundancy solve different problems.
This is bordering on paranoia. If the bad sector contains a critical part of the filesystem, you're going to lose everything anyway.
Do modern disks even have physical "sectors" anymore? Isn't it all virtual?
Without dedup you're just going to backup less stuff, which is far worse.
> This is bordering on paranoia.
What? You actually think that the mere idea of a backup getting corrupted is something that is "bordering on paranoia"? Have you ever heard of redundancy or even RAID?
> If the bad sector contains a critical part of the filesystem, you're going to lose everything anyway.
Do you honestly failed to understand that the problem is actually the "lose everything" part?
I used to use Borg, but Restic (and it's Rust clone Rustic) are better.
Why?
I’m currently trying to decide between Borg and Restic myself. Borg has worked very well for me in the past, and I’m a bit excited by the new Vorta GUI frontend. But I already have a BackBlaze B2 setup for backing up my Mac, and Restic would let me continue to use that as a backup store. I’m interested in hearing about differentiating points between them.
Primarily it's the reason you already know: restic and borg are the same model, but restic doesn't need it to be an ssh-accessible filesystem on the remote end. Restic can send backups almost anywhere, including object storage like your Backblaze B2 (that's what I use with restic, too). I agree with OP: restic is strictly better. There's no reason to use borg today; restic is a superset of its functionality.
Thanks! Then I’ll look more at Restic :)
Does restic work well with truenas?
I don't know specifically, but it's a self-contained single file Go executable. It doesn't need much from a Linux system beyond its kernel. Chances are good that it'll work.
Reminds me of a tool I used to use that did a similar thing: https://github.com/jeremywohl/glastree
Isn't it also how timeshift [1] backups are working?
[1] https://github.com/linuxmint/timeshift
I've been using a bash encapsulation of this for a long time, works great.
https://github.com/laurent22/rsync-time-backup
Used this technique (albeit counter-based, not time-based) successfully for backing up the 2006 World Cup site newsfeed[0] configurations/data. Had 6 rotating backups on two hosts. Never had to use them (as best I remember) but it was definitely comforting to know they were there in case of urgent rollback.
[0] Not strictly the right name but I've forgotten. Y!uk people of that era know what I mean.
Why make hard links when you can use rsync or syncthing to just make an actual copy on an external hard drive eg via wifi or just remotely?
Hard links are file level dedupe.
And then once all references to the inode are removed (by rotating out backups) it's freed. So there's no maintenance of the deduping needed, it's all just part of how the filesystem and --link-dest work together.
The hard links are to the most recent backup before the one happening now in the script, so that you aren't storing full copies of files that haven't changed between backups.
It does make an actual copy but then it builds a directory structure that you can browse by date (like Time Machine). That directory contains hard links so only one copy of a file is ever backed up but you see contents that match the date of the backup.
sounds similar to rdiff-backup ( https://rdiff-backup.net ).
I know some folks that have been using that for a very long time as well.
What happened to tarsnap? Anybody using it?
It's still there; and yes, people are using it.
One big drawback of these solutions is that they cannot archive the system apps properly. Nor can they be used during the OS setup.
TM integration is just so convenient...
Need to combine this with LVM or BTRFS or similar to be a true snapshot. Rsnapshot supports LVM snapshot pretty good.
Once you have btrfs you don't really need rsync anymore, its snapshot + send/receive functionality are all you need for convenient and efficient backups, using a tool like btrbk.
restic to s3 has been very reliable - haven't tried borg.