Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Borgbase backups have been unavailable for 3 days

status.borgbase.com

Post reply on HN
Print view View original

81–88 of 88 posts

Re: Borgbase backups have been unavailable for 3 days

#81

Post by rsync »

countula wrote: post #39

TIL Borgbase. The UI looks great for snapshot-oriented backups, and it's good to see them fund borgbackup development. rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653 Unavailability is occasionally expected. Don't rely on a single provider to be your sole backup. Any one have any experience with Hetzner Storage Boxes for this?

"rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653"

This is correct - we believed we had a working recipe that properly sandboxed the 'rclone mount' and 'rclone serve' directives but we couldn't make it work in a way that allowed us to sleep at night.

But now that is all changing...

It appears that we can run rclone serve restic --stdio in a way that doesn't actually serve anything or create sockets, etc., and as soon as we finish our tests we will have it in place and people can lock their accounts with something like this:

restrict,command="rclone serve restic --stdio --append-only backups/my-restic-repo" ssh-rsa ...

... in authorized_keys and achieve proper append-only mode at long last.

Stand by ...

Re: Borgbase backups have been unavailable for 3 days

#82

Post by aborsy »

tjoff wrote: post #80
aborsy wrote: post #64

Earlier quoted context omitted.

That would also be a major security concern: people gaining access to accounts by social engineering .

Well, you do at least encrypt your backups yourself, right?

That doesn’t prevent the deletion of backups and snapshots, encrypting the encrypted backups for a ransom, and using your account for distribution of the illegal content or malware.

Re: Borgbase backups have been unavailable for 3 days

#83

Post by rsync »

colechristensen wrote: post #32
shrubble wrote: post #17

Earlier quoted context omitted.

Are they using big disks and having to resilver or re-do their RAID setup across the added drives, requiring everything on the disks to be read and re-written?

It would appear to be something like that. > However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. This is embarrassing. Somebody made a big mistake of the “mistakes like this shouldn’t be made” category. Like not an accident or fat finger or a bug, but engineers not understanding fundamentals of how a system worked and having none of the processes in pl…

"This is embarrassing. Somebody made a big mistake of the “mistakes like this shouldn’t be made” category."

You have no idea what happened.

Protecting against "something like this" often introduces complexity and failure cascades that are much more harmful than a simple system being offline.

In fact, I would consider it a feature if uptime was deliberately sacrificed for simplicity and data integrity.

I wish Manu the very best and will consider it not a failure but a success if this array/subsystem/whatever comes through without data loss.

Re: Borgbase backups have been unavailable for 3 days

#84

Post by tjoff »

aborsy wrote: post #82
tjoff wrote: post #80

Earlier quoted context omitted.

Well, you do at least encrypt your backups yourself, right?

That doesn’t prevent the deletion of backups and snapshots, encrypting the encrypted backups for a ransom, and using your account for distribution of the illegal content or malware.

Sure, but lets try and be a little more realistic and weigh the scenarios accordingly.

Re: Borgbase backups have been unavailable for 3 days

#85

Post by btschaegg »

rsync wrote: post #81
countula wrote: post #39

TIL Borgbase. The UI looks great for snapshot-oriented backups, and it's good to see them fund borgbackup development. rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653 Unavailability is occasionally expected. Don't rely on a single provider to be your sole backup. Any one have any experience with Hetzner Storage Boxes for this?

"rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653 " This is correct - we believed we had a working recipe that properly sandboxed the 'rclone mount' and 'rclone serve' directives but we couldn't make it work in a way that allowed us to sleep at night. But now that is all changing... It appears that we can run rclone serve restic --stdio in a way that d…

As a customer, that is great to hear (well, read). I was reading about how a restore scenario with Borg would look like if "append only" really matters (so in a ransomware scenario), and what I've seen makes me rather uncomfortable [1]. I don't want to have to do that in a stressful situation.

Is there a particular channel where you intend to post your findings? :)

[1]: https://borgbackup.readthedocs.io/en/stable/usage/notes.html...

Re: Borgbase backups have been unavailable for 3 days

#86

Post by trallnag »

bloopernova wrote: post #75
trallnag wrote: post #71

Earlier quoted context omitted.

What happens next? Shouldn't the array continue working, albeit with significantly degraded performance?

Sorry, I didn't quite answer your question! Yes, the expected behaviour would be for the disks to be initialized. Beyond that, it'd be a configuration setting for what to do with new disks. It could be used to extend the existing LUNs, or added as a new pair in a RAID 10 style setup, or added as new members of a different level RAID. Then it would be up to the sysadmin to extend the LUN, or divide the newly added spa…

Thanks for the insightful answer!

Re: Borgbase backups have been unavailable for 3 days

#87

Post by wtallis »

mananaysiempre wrote: post #67
colechristensen wrote: post #33

Earlier quoted context omitted.

> However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. One, it is obviously some flavor of raid rebalancing, there’s not a different kind of thing it could be. Two, they’re admitting they didn’t expect this, that’s not possible if it’s a tested procedure.

By the way, is this a plausible scenario? (1) Add more drives, start online rebuild (2) Have one drive fail/URE mid-rebuild (3) Be forced to switch to offline rebuild because the array was in a strange not-entirely-consistent state even before the failure, even though without the new drives you’d be in a routine online-rebuild situation If so, possibility (2) could in principle not be detectable in staging, as it cou…

> and having to reread everything—as a rebuild does—is a rather abnormal load.

Far from being an abnormal load, it should be a cron job. Backups need to be verified. Data at rest needs to be periodically checked for bitrot. Reading everything on the array and verifying parity and any checksums is important preventive maintenance and an early warning system.

Re: Borgbase backups have been unavailable for 3 days

#88

Post by kiwicopple »

ryan29 wrote: post #51
chanandler_bong wrote: post #14

The fact that as a customer that is impacted by this, I only found out about it when my backups and automated test restore failed is worrying. I get that things happen, but not realising that this was going to be a service impacting event does not inspire continued confidence. Sure, this situation probably won't happen again, but what else don't they understand about their infrastructure?

I got an email about it right away and I’ve also been getting warnings (that I configured in the dashboard) for inactivity on a repo that’s affected.

(off topic) I promised to reply to you in another thread, but I can't because replies are locked. feel free to reach out to me if you want - my contact is in my profile
Post reply on HN