Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backups Aren't Simple

filipovski.net

Post reply on HN
Print view View original

21–30 of 206 posts

Re: Backups Aren't Simple

#21

Post by cvhc »

innocent_name wrote: post #11

>Motivated by our success in developing this solution, we try to use it to backup the homelab with its 10 Docker containers. But later we find out from logs on the individual machines that backups are failing. The reason being that many Docker containers like to create root-owned files, and if you’re not careful you can create a cronjob running as the default user. and a privesc also.

Yeah it feels it can be risky to "make the backup also dump the databases, and give it full filesystem permissions on our Docker volumes", depending on what binaries get the root privileges.

What I do is having a minimal systemd timer under root that only calls restic for backing up files (with some additional systemd seucurity restrictions). App-dependent backup logics, like dumping databases, are done by user/container-level cronjobs separately.

Re: Backups Aren't Simple

#22

Post by Forgeties79 »

publlus_enigma wrote: post #20

There are four times in my life I have suffered regrettable data loss incidents. The first was when the telephone pole outside our house was struck directly by lightning. Not only was it the loudest thing I have ever heard, the current surged through the telephone line, into the internal fax modem, and fries everything within its vicinity. I was 10. I did have backuos, but only only floppy and they didn't cover every…

>I now shoot 2 cards simultaneously

A very lovely change over the last 8 years or so lol. I came up in film during the DSLR revolution. 5D2’s/7D’s/Rebels (i series) years.

8bit 420 nasty aliasing recording on single cards and praying baby. Magic lantern booted on those same SD’s!

Re: Backups Aren't Simple

#23

Post by ebrahimh »

I’m setting up 3-2-1-ish backups for my infra of 3 hosts, and definitely leaning towards Restic + Backrest.

All my hosts run the same CoreOS setup (https://github.com/ebrahim37/infra-template), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup.

I plan to implement it like this:

  vps1:
  - restic container with custom sh entrypoint that will backup volumes/ to homelab every 24 hours
  homelab:
  - backrest container, to back up volumes/, do prune/check, replicate repo to offsite
  - rest-server container, will store backups from vps1, homelab, offsite
  offsite:
  - restic container, backs up volumes/ to homelab every 24 hours
  - rest-server container, store copy of backups from homelab
Only caveat is backing up databases, will either have to do: stop container, backup volume/database-data, start container; or use pg dump etc.

The deduplication is nice, you can have a snapshot for each week of the past year without crazy storage cost

Re: Backups Aren't Simple

#24

Post by cvhc »

publlus_enigma wrote: post #20

There are four times in my life I have suffered regrettable data loss incidents. The first was when the telephone pole outside our house was struck directly by lightning. Not only was it the loudest thing I have ever heard, the current surged through the telephone line, into the internal fax modem, and fries everything within its vicinity. I was 10. I did have backuos, but only only floppy and they didn't cover every…

> a change to their terms surrounding "lifetime" unlikely noted storage

I had a similar experience with a Chinese cloud storage provider. They didn't even give a convenient way to export the data. And the client throttled download to like 100KB/s. I luckily was able to parallel download by running the client on several VMs...

> An errant script deleted the source content, but I'd also deleted the existing backup to free up space for the new backup.

Sounds like the backup workflow is fundamentally flawed... But I also have the fear that the backup program with root privileges can go off the rails... So I keep my backup job as simple as just running restic with some systemd restrictions.

Re: Backups Aren't Simple

#25

Post by keithnz »

pizzafeelsright wrote: post #10

I don't want backups. I want synced duplicated data in geographically separated by shared credentials.

that's not really the same thing, what you are describing is redundancy so failure in one (or more) place doesn't bring the system down. Backups are about preventing data loss. Data loss can happen in many different ways, not just a system crashing and burning. It can be deliberately/accidently done such that it would be sync'd everywhere.

Re: Backups Aren't Simple

#27

Post by AdieuToLogic »

A friend of mine used to work at Veritas[0] making enterprise data retention solutions. When I spoke about their product as being "making backups", he corrected me by saying:

  We are not in the backup business.  We are in the restoration
  business.
0 - https://en.wikipedia.org/wiki/Backup_Exec

Re: Backups Aren't Simple

#28

Post by codys »

ebrahimh wrote: post #23

I’m setting up 3-2-1-ish backups for my infra of 3 hosts, and definitely leaning towards Restic + Backrest. All my hosts run the same CoreOS setup ( https://github.com/ebrahim37/infra-template ), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup. I plan to implement it like this: vps1: - restic container with custom sh entrypoint that will backup volumes/ to…

If your system has a way to take consistent snapshots in the filesystem (btrfs, zfs) or volume manager (lvm2, perhaps with a bit of filesystem support to obtain fs consistency), that can be used to avoid database downtime, if desired.

https://www.postgresql.org/docs/current/backup-file.html

Re: Backups Aren't Simple

#29

Post by bob1029 »

offmycloud wrote: post #16
pizzafeelsright wrote: post #10

I don't want backups. I want synced duplicated data in geographically separated by shared credentials.

> I don't want backups. I want synced duplicated data in geographically separated by shared credentials. If you accidentally delete something, how long does it take for the delete to propagate to all of your geographically separated datacenters? You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption.

> You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/versio...

Re: Backups Aren't Simple

#30

Post by snizovtsev »

I'm polishing my backup system right now, and found that `tar | zstd | gpg` plus a small vibe-coded wrapper is pretty much all I need.

GNU tar has its own incremental index via `--listed-incremental=FILE`. Unlike Borg or Restic, which have their own more complicated repository formats, this leaves me with just one additional file (the `.snar`) alongside a dumb, portable full-disk tar archive.

The nice part is that, unlike repositories that require both read/write access patterns, tar can compute deltas using only the small `.snar` file, while the main `full-disk.tar` can remain buried in write-only Glacier storage. This makes it a much better fit for Glacier's write-once model and 180-day minimum retention.

My current plan is to upload a full-disk tar to an external HDD + Glacier on a weekly cadence, and the `.snar`-based deltas daily. That gives me a pretty simple cloud backup solution for a few bucks a month (after burning through $100+ of free credits).

Post reply on HN