"encrypted, chunk-level deduplicated, GFS-rotated, point-in-time archived, cloud, 3-2-1 backup solution" is now my newest password, no commas. (Don't tell anyone!)
One of our customers decided to go for off site backups with us - they had been a customer for 15 odd years for other services. Backups are not our core business, its one of the services and I got around to setting them up sooner rather than later.
Two weeks later they had a fire in one of their huge printing machines which caused quite a lot of direct damage due to heat and rendered a lot of equipment broken due to smoke/fumes. It was also, as it turns out, a week before the first COVID 19 lockdown in the UK.
Their backups landed on a XFS file system with reflinks enabled. I cloned their repo and fired up NFS with their VMs running on our gear. It took me another hour to spin up another OpenVPN server (CA etc) for them to use and sort out a few other details (coffee doesn't brew itself).
They ran via VPN out of our data centre for most of the pandemic.
That was an utter triumph but I also have some rather less triumphant stories about backups and lack thereof. Let's skip over those 8)
>Motivated by our success in developing this solution, we try to use it to backup the homelab with its 10 Docker containers. But later we find out from logs on the individual machines that backups are failing. The reason being that many Docker containers like to create root-owned files, and if you’re not careful you can create a cronjob running as the default user.
Try running a business on that basis - where do the restores come from? 8)
We charge our customers for backup storage by volume and that includes an annual demonstration of recovery. We fire up some or all of their systems on our gear in isolation and show that they are reasonably functional.
Your teacher's comment is riffing on the well trod lines of: "heights don't kill people, it's depths that kill people", which is all about perspective. This is is not related to "guns don't kill people ..." which is about agency.
It is true that very few people proactively care about backups. It’s like flossing. People know they’re supposed to do it, but in the moment it feels like a waste of time and energy, when there are more important things to do.
When data is lost, or decay takes over, the value can skyrocket. Depending on what is lost, someone may be willing to pay 100x when they would have spent on backup in order to get their data back, this is why data recovery services can basically charge whatever they want.
My assumption is the Venn diagram between people who pass the marshmallow test, and those who proactively backup their systems, has a lot of overlap.
Running a business on that basis would look like the white(ish)-hat version of ransomware. "Oh, your data is gone? How much is it really worth to you?"
It is a special sensation when you go to your hand rolled backup "solution" to make some simple recoveries and start finding problems. It is a complex mix of not dread but disgust, sick to your stomach, self-anger, and loss. I can only image what that's like with like all of your children's baby pictures being the lost material. Borg and restic are fantastic, it still seems like some sort of dummy proof mode might be desirable, all the features in the world and they have some some reasonably sane policy or something that includes verifications.
On the flip side of that though, really and truly, I've restored a laptop from a timemachine backup and it came back damn nearly identical and that was magical.
I needed to temporarily move off 20TB from my NAS and bought a Hetzner storage box.
I wanted to encrypt the data before sending it so i went with the borg ssh mount. Long story short, borg failed a couple of times due to https://github.com/borgbackup/borg/issues/7672, i resumed it per documentation and got 2gb of silently corrupted data. There was no way to do data integrity checks due to broken pipes and there was no option to resume the job from the previous point either. I couldn't afford to go with the s3, backblaze wasn't an option due to placing trust into their proprietary client doing the encryption and i got majorly screwed in the end.
Accepts stuff from pipe too so you can just pipe mysqldump or pg_dumpall without intermediate files
Very decent options for checking repo integrity, personally (well, at work too, we use it on few hundred user machines and servers) I also added "roll a dice for backup and try to restore it" test script to make sure it is working
Decent deduplication too so "store year's worth of weekly snapshot" is very sensible strategy once you exclude the "constantly changing and useless" (caches etc.) files out of it
can mount FUSE directory with all backups on sensible OSes.
https://kopia.io/ does the same +GUI but a bit worse on CLI front (it insists on keeping local config which makes scripting a bit more involved, but not much)
IMHO, the easiest way to perform a restore test is to use your production backups to refresh a lesser environment on a weekly cadence. Naturally this has to be appropriate to the type of data you're restoring, e.g., our E2E (end to end) testing environment has been built to the same risk tolerances of production so it's suitable for production data and the PII (Personally Identifiable Information) it hosts. If this isn't in your own risk tolerances, you can still perform the same test but ensure the data from prod is anonymised or alternatively destroyed, and perform a second restore after production restore with the dataset that preceeded it that has no PII data or similar.
With this the E2E tests confirm the backups are expected and you can tick a box that when shit does eventually hit the fan, data corruption isn't a problem. Normal caveats apply, you must make sure your backups are immutable so they cannot be modified or changed _after_ the tests have been ran.
importantly, does your die have a side that corresponds to a multi-continental cataclysmic natural disaster of magnitude only known unknown to pre-recorded-history
> I don't want backups. I want synced duplicated data in geographically separated by shared credentials.
If you accidentally delete something, how long does it take for the delete to propagate to all of your geographically separated datacenters? You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption.
Two weeks later they had a fire in one of their huge printing machines which caused quite a lot of direct damage due to heat and rendered a lot of equipment broken due to smoke/fumes. It was also, as it turns out, a week before the first COVID 19 lockdown in the UK.
Their backups landed on a XFS file system with reflinks enabled. I cloned their repo and fired up NFS with their VMs running on our gear. It took me another hour to spin up another OpenVPN server (CA etc) for them to use and sort out a few other details (coffee doesn't brew itself).
They ran via VPN out of our data centre for most of the pandemic.
That was an utter triumph but I also have some rather less triumphant stories about backups and lack thereof. Let's skip over those 8)
and a privesc also.
https://archive.is/Ay7RJ
One of my teachers claimed that contrary to common behavior, no one actually valued backups; it was restores that were worth paying for.
We charge our customers for backup storage by volume and that includes an annual demonstration of recovery. We fire up some or all of their systems on our gear in isolation and show that they are reasonably functional.
Your teacher's comment is riffing on the well trod lines of: "heights don't kill people, it's depths that kill people", which is all about perspective. This is is not related to "guns don't kill people ..." which is about agency.
When data is lost, or decay takes over, the value can skyrocket. Depending on what is lost, someone may be willing to pay 100x when they would have spent on backup in order to get their data back, this is why data recovery services can basically charge whatever they want.
My assumption is the Venn diagram between people who pass the marshmallow test, and those who proactively backup their systems, has a lot of overlap.
On the flip side of that though, really and truly, I've restored a laptop from a timemachine backup and it came back damn nearly identical and that was magical.
I needed to temporarily move off 20TB from my NAS and bought a Hetzner storage box.
I wanted to encrypt the data before sending it so i went with the borg ssh mount. Long story short, borg failed a couple of times due to https://github.com/borgbackup/borg/issues/7672, i resumed it per documentation and got 2gb of silently corrupted data. There was no way to do data integrity checks due to broken pipes and there was no option to resume the job from the previous point either. I couldn't afford to go with the s3, backblaze wasn't an option due to placing trust into their proprietary client doing the encryption and i got majorly screwed in the end.
Software like https://restic.net/ does a good job. Few choices of backend
Accepts stuff from pipe too so you can just pipe mysqldump or pg_dumpall without intermediate files
Very decent options for checking repo integrity, personally (well, at work too, we use it on few hundred user machines and servers) I also added "roll a dice for backup and try to restore it" test script to make sure it is working
Decent deduplication too so "store year's worth of weekly snapshot" is very sensible strategy once you exclude the "constantly changing and useless" (caches etc.) files out of it
can mount FUSE directory with all backups on sensible OSes.
https://kopia.io/ does the same +GUI but a bit worse on CLI front (it insists on keeping local config which makes scripting a bit more involved, but not much)
IMHO, the easiest way to perform a restore test is to use your production backups to refresh a lesser environment on a weekly cadence. Naturally this has to be appropriate to the type of data you're restoring, e.g., our E2E (end to end) testing environment has been built to the same risk tolerances of production so it's suitable for production data and the PII (Personally Identifiable Information) it hosts. If this isn't in your own risk tolerances, you can still perform the same test but ensure the data from prod is anonymised or alternatively destroyed, and perform a second restore after production restore with the dataset that preceeded it that has no PII data or similar.
With this the E2E tests confirm the backups are expected and you can tick a box that when shit does eventually hit the fan, data corruption isn't a problem. Normal caveats apply, you must make sure your backups are immutable so they cannot be modified or changed _after_ the tests have been ran.
If you accidentally delete something, how long does it take for the delete to propagate to all of your geographically separated datacenters? You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption.