Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Borgbase backups have been unavailable for 3 days

status.borgbase.com

Post reply on HN
Print view View original

41–50 of 88 posts

Re: Borgbase backups have been unavailable for 3 days

#41

Post by viraptor »

colechristensen wrote: post #33
viraptor wrote: post #30

Earlier quoted context omitted.

They messed up here in some way, but assuming you know what events and complaining about their skills is just making stuff up at this point. Unless you can tell us exactly what happened, maybe hold off with that kind of criticism.

> However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. One, it is obviously some flavor of raid rebalancing, there’s not a different kind of thing it could be. Two, they’re admitting they didn’t expect this, that’s not possible if it’s a tested procedure.

Possible, but that's still guessing. Also possible: extra load while adding storage caused multiple drive failures, so suddenly they're without spares and want to finish the migration as soon as possible, so they stopped the customer traffic.

"did not anticipate" can be interpreted in different ways too. It may be "didn't know it could happen", or just "estimated it to be so unlikely that emergency downtime was acceptable as a response".

I've seen outside speculations as an inside person before and they're often very likely to be BS. Let's stick to facts and skip the "obviously" and "not possible" until we learn more. What's happening right now may be their prepared and tested procedure. (Or it may not)

Re: Borgbase backups have been unavailable for 3 days

#42

Post by exitheone »

countula wrote: post #39

TIL Borgbase. The UI looks great for snapshot-oriented backups, and it's good to see them fund borgbackup development. rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653 Unavailability is occasionally expected. Don't rely on a single provider to be your sole backup. Any one have any experience with Hetzner Storage Boxes for this?

Been using hetzner storage boxes for borg backup for >5 years. Works without issue for multiple TB of backups. Every maintenance downtime so far has been announced in advance.

Re: Borgbase backups have been unavailable for 3 days

#43

Post by greenie_beans »

istjohn wrote: post #25
TylerE wrote: post #16

Earlier quoted context omitted.

This is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.

If you have multiple backups, you don't have to trust anyone.

except yourself !

Re: Borgbase backups have been unavailable for 3 days

#44

Post by chillfox »

crossroadsguy wrote: post #34
TylerE wrote: post #16

Earlier quoted context omitted.

This is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.

Actually if I must entrust my data to companies like these where reliability and competence is of utmost importance - I’d rather trust Google and Facebook (much much before Apple). If we are talking about privacy (or rather privacy theatre) then yeah Apple, why not.

I pulled down my Google Drive data some time ago with rclone and a bunch of the older files were corrupted.

Re: Borgbase backups have been unavailable for 3 days

#45

Post by colechristensen »

wmf wrote: post #38
x0x0 wrote: post #36

Earlier quoted context omitted.

And three, they're admitting they don't back anything up (backup defined as not just slinging bytes somewhere but the availability to restore those bytes and bring a service back up on a timeline that is certainly well lower than 3 days. With regular testing to understand the timeline and ensure everything actually works.) ps -- it's not about being bashing these folks. But these type of multiple operational incompet…

It's a backup company. I can understand that they don't back up the backups.

It’s a backup company so I would expect the store 1 copy plus the minimum redundancy to make drive failures a predictable amount of unlikely to cause data loss. More than that would be additional features.

Re: Borgbase backups have been unavailable for 3 days

#46

Post by lifeofguenter »

countula wrote: post #39

TIL Borgbase. The UI looks great for snapshot-oriented backups, and it's good to see them fund borgbackup development. rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653 Unavailability is occasionally expected. Don't rely on a single provider to be your sole backup. Any one have any experience with Hetzner Storage Boxes for this?

Hetzner Storage just lost a couple of snapshots a while ago…

If you want “perfect architecture” and reliability/availability you just need to pay for it (aws/s3). Or make use of multiple providers.

Re: Borgbase backups have been unavailable for 3 days

#47

Post by justinclift »

wmf wrote: post #38
x0x0 wrote: post #36

Earlier quoted context omitted.

And three, they're admitting they don't back anything up (backup defined as not just slinging bytes somewhere but the availability to restore those bytes and bring a service back up on a timeline that is certainly well lower than 3 days. With regular testing to understand the timeline and ensure everything actually works.) ps -- it's not about being bashing these folks. But these type of multiple operational incompet…

It's a backup company. I can understand that they don't back up the backups.

If there's one thing you'd expect from "a backup company", it'd be that they're really good at doing backups. :)

That does include their own of course.

Re: Borgbase backups have been unavailable for 3 days

#48

Post by Wowfunhappy »

ksec wrote: post #35
TylerE wrote: post #16

Earlier quoted context omitted.

This is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.

I wish Apple made a Time Capsule that requires iCloud 2TB Subscription to work. They could have priced it at $299 to include the initial 12 months iCloud subscription.

Sorry, can you expand on that? Why require the sub?

Re: Borgbase backups have been unavailable for 3 days

#49

Post by seba_dos1 »

TheNewsIsHere wrote: post #6
smcleod wrote: post #4

Earlier quoted context omitted.

Personally I appreciate the honesty and look for it in vendors, there are too many that cover up the truth (MS/Github etc...). However I agree that it doesn't look good for a storage company.

I agree with you. I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? Expanding a storage environment can certainly be a mine-filled exercise for even the most experienced. But why was this expansion unexpectedly an offline operation? I’m not throwing shade here. I’m genuinely curious because I’ve been considering Borg and Borgbase for an upcoming proj…

> I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion?

People who can give solid explanations are probably busy right now, and those who aren't can only write what you already seen.

Re: Borgbase backups have been unavailable for 3 days

#50

Post by wtallis »

noAnswer wrote: post #29
SteveNuts wrote: post #21

Earlier quoted context omitted.

I've been in this exact situation before in a storage expansion. With enough 22TB spinning drives nowadays you can get in to a scenario where, with the new data being written into the array and the expansion process going on at the same time the rebuild essentially will never complete. This is especially true with lower end CPU servers and without dedicated RAID cards. It is dangerous because the rebuild stresses the…

> I've been in this exact situation before in a storage expansion. Can you tell us what solution that was? Something with ZFS or BTRFS? My experience with classic RAID system is, you can't expand them without reinitializing them. (But that comes with obvious warnings about data loss.)

Traditional hardware RAID requires you to start over from scratch to expand an array. ZFS has some limited ability to expand an array, but there are still a lot of scenarios where the recommended solution is to start over from scratch. BTRFS has full flexibility for online resizing of arrays and changing between RAID modes and rebalancing the array, but its parity RAID modes aren't really trustworthy enough yet for a backup provider to be relying on.
Post reply on HN