Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Restic 0.13.0

restic.net

Post reply on HN
Print view View original

21–30 of 68 posts

Re: Restic 0.13.0

#21

Post by timmg »

I have a hand-coded backup system for my photo library that writes to S3. It runs every night at 2AM.

The one feature I have that's important to me is this: it will figure out what files need to be uploaded and then upload as many as possible for an hour then stop.

That means that it runs for at-most an hour a night.

The reason I need/wanted this feature is that I might come home from a trip with (eg) 30G worth of photos. My (cable) internet will upload at around 1G an hour. I don't want this thing to saturate my internet for 30 hours straight. Instead, it backs up a small amount every night for 30 days.

Am I the only one that wants a feature like this? I've never seen it in any other backup system. (At alternative might be to have configurable bandwidth for uploads.)

Re: Restic 0.13.0

#23

Post by Mo3 »

lucgommans wrote: post #20
Mo3 wrote: post #16

Earlier quoted context omitted.

Excellent. One more question. Assuming someone gains access to a node - will they be able to access the backups or screw with them in any way?

Depends which back-end you use and which configuration. (Assuming "node" here means a backup client and not a backup-hosting server.) If you just put stuff on some standard storage (FTP, B2, etc.) without any permissions set up, then no backup client could stop the attacker from using the credentials you've deployed to the backup client to login to your storage and delete your backup files. This is not really specifi…

Got it. So I guess the best solution to go forward here would be to deploy Minio and then backup the backup directory itself in regular intervals onto different storage not accessible from anywhere.

Re: Restic 0.13.0

#25

Post by photon-torpedo »

CameronNemo wrote: post #7
crecker wrote: post #6

Where can I read about the best solutions for backups? For nerdy people

Restic and BorgBackup really seem to be the favored solutions out there. Restic for encryption, Borg for deduplication and compression. Or maybe bacula if you want pull based backups instead of push based. https://restic.readthedocs.io/en/stable/ https://www.borgbackup.org/ https://www.bacula.org/documentation/documentation/

BorgBackup can be made to operate in pull mode too, see: https://borgbackup.readthedocs.io/en/stable/deployment/pull-...

At work I've implemented a variant of the ssh-agent method described there. Admittedly, it requires some scripting :)

Re: Restic 0.13.0

#26

Post by sigio »

cma wrote: post #12
mekster wrote: post #3

Unless it's for experimenting, I've stopped caring for backup solutions other than borg and zfs as the only way to prove their stability is to have them exist for a while without big complaints and new ones all seem to have complaints. Just having no data loss isn't enough which is the absolute base point but huge memory consumption and other operational issues are also showstoppers.

Borg being single-threaded is painful in the era of consumer 12 and 16 core CPUs, and even prosumer 64-core.

Don't notice this much.... disk is usually the bottleneck, and otherwise it will be the network to the remote backup location. Still backups complete in seconds:

Repository: ssh://backup/./backups/mungedhostname.borg Archive name: 20220327-2201 Archive fingerprint: 8b710144579c8d531e7c4a0192304323081b14a71445557608d859494bbe84b6 Time (start): Sun, 2022-03-27 22:01:35 Time (end): Sun, 2022-03-27 22:01:52 Duration: 17.28 seconds Number of files: 45678

Original size Compressed size Deduplicated size This archive: 24.22 GB 9.35 GB 47.78 MB All archives: 363.60 GB 140.91 GB 7.81 GB

Unique chunks Total chunks Chunk index: 59857 852257

Re: Restic 0.13.0

#27

Post by crecker »

timmg wrote: post #21

I have a hand-coded backup system for my photo library that writes to S3. It runs every night at 2AM. The one feature I have that's important to me is this: it will figure out what files need to be uploaded and then upload as many as possible for an hour then stop. That means that it runs for at-most an hour a night. The reason I need/wanted this feature is that I might come home from a trip with (eg) 30G worth of ph…

I wonder if you can obtain that behavior with a bash script, but it's boring to write scripts and I do not know if SIGTERM can exit Restic gracely.

Re: Restic 0.13.0

#28

Post by lucgommans »

timmg wrote: post #21

I have a hand-coded backup system for my photo library that writes to S3. It runs every night at 2AM. The one feature I have that's important to me is this: it will figure out what files need to be uploaded and then upload as many as possible for an hour then stop. That means that it runs for at-most an hour a night. The reason I need/wanted this feature is that I might come home from a trip with (eg) 30G worth of ph…

Makes perfect sense. Restic kind-of supports this because you can just kill the client after an hour and, tomorrow, it'll see which objects are there already.

I'm not deep enough into the project to know whether this is like an officially supported use-case, but restic was of course made with the idea that interruptions can happen (your computer can crash) and should be handled safely, and for the deduplication it'll cut files up in a deterministic way and thus (as I understand it) store those chunks in a deterministic place.

Re: Restic 0.13.0

#29

Post by lucgommans »

crecker wrote: post #27
timmg wrote: post #21

I have a hand-coded backup system for my photo library that writes to S3. It runs every night at 2AM. The one feature I have that's important to me is this: it will figure out what files need to be uploaded and then upload as many as possible for an hour then stop. That means that it runs for at-most an hour a night. The reason I need/wanted this feature is that I might come home from a trip with (eg) 30G worth of ph…

I wonder if you can obtain that behavior with a bash script, but it's boring to write scripts and I do not know if SIGTERM can exit Restic gracely.

If you've ever ctrl+c'd restic, you'll know the message

> signal interrupt received, cleaning up

Re: Restic 0.13.0

#30

Post by lykron »

timmg wrote: post #21

I have a hand-coded backup system for my photo library that writes to S3. It runs every night at 2AM. The one feature I have that's important to me is this: it will figure out what files need to be uploaded and then upload as many as possible for an hour then stop. That means that it runs for at-most an hour a night. The reason I need/wanted this feature is that I might come home from a trip with (eg) 30G worth of ph…

You could do this with the `timeout` command.

`timeout -s SIGINT 1h restic...`

That would let restic run for one hour, and then once the hour elapses send a SIGINT which will stop the process (see https://github.com/restic/restic/blob/a29777f46794ea4e35548f...)

Post reply on HN