Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Borgbase backups have been unavailable for 3 days

status.borgbase.com

Post reply on HN
Print view View original

61–70 of 88 posts

Re: Borgbase backups have been unavailable for 3 days

#61

Post by jskrablin »

js4ever wrote: post #59
colechristensen wrote: post #32

Earlier quoted context omitted.

It would appear to be something like that. > However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. This is embarrassing. Somebody made a big mistake of the “mistakes like this shouldn’t be made” category. Like not an accident or fat finger or a bug, but engineers not understanding fundamentals of how a system worked and having none of the processes in pl…

It's not an org but a one man show. I'm using borgbase myself, really happy with it until now and I have to admit I'm now reconsidering this choice...

One man show has a bus factor of 1. Which is kinda bad for anything critical...

Re: Borgbase backups have been unavailable for 3 days

#62

Post by alex3305 »

I used their trial for a bit to test it out with Vorta [1] in a container. Vorta (and Borg) seemed to work fine, until I wanted to restore an archive and I noticed that my recent snapshots were completely empty. Probably because of a misconfiguration on my end though. But it made me look elsewhere. For me backups should be a fire, test and forget solution.

Recently I made the switch to Kopia [2] which seems to have feature parity with Borg (and Restic [3]). It also has a web UI which is way easier to work with than Vorta. And I can easily view, extract and restore individual files or folders from there. This gave me way more confidence about this solution. The only thing I really miss is that I cannot chose different targets for different paths. For instance, with Borg I was able to backup a partial of my Docker appdata to an external source. And I haven't found a way to do this with Kopia. Besides that I'm pretty happy with this solution and I would recommend it.

1. https://vorta.borgbase.com/

2. https://kopia.io/

3. https://restic.net/

Re: Borgbase backups have been unavailable for 3 days

#63

Post by mananaysiempre »

viraptor wrote: post #30
x0x0 wrote: post #11

Earlier quoted context omitted.

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

They messed up here in some way, but assuming you know what events and complaining about their skills is just making stuff up at this point. Unless you can tell us exactly what happened, maybe hold off with that kind of criticism.

That a decision needs to be made under uncertainty doesn’t relieve one from the necessity of making the decision. That implies estimating an uncertain state of the world and drawing inferences from that, and whether those are framed as “criticism” is irrelevant.

In situations where you could plausibly get more data, it’s usually OK to admonish people to just go and get some, because whatever the difficulty is doing that, judging the current degree of uncertaintly is probably even harder. In situations where you couldn’t, though (trade or state secrets), there’s nothing to do but to use what data you do have and point out your uncertainty carefully.

(This is a very general argument because what you’ve said is a fully general counterargument to any criticism of anyone from anybody but an insider. As for this particular case, GP’s allegation of lacking ops knowledge seems to be on the charitable side to me: an uncharitable one, as stated elsethread, would be that they had one drive too many fail on them, whether due to incompetence or misfortune, and are forced to do an offline rebuild while choosing to lie about it. It’s the lie part that I’d have the greatest problem with, if it’s actually there.)

Re: Borgbase backups have been unavailable for 3 days

#64

Post by aborsy »

adobrawy wrote: post #54
TylerE wrote: post #16

Earlier quoted context omitted.

This is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.

Large corporations have the technical skills in place, but organizationally they can lock your account away leaving you with no access and no real avenue to evict your data. There are many posts on HN and on the Internet where corporate support had no procedure to give back access to data when they blocked the account. Account bans are sometimes for unclear or out of control reasons (e.g. someone stole your credit ca…

That would also be a major security concern: people gaining access to accounts by social engineering .

Re: Borgbase backups have been unavailable for 3 days

#65

Post by jskrablin »

jm4 wrote: post #22
x0x0 wrote: post #11

Earlier quoted context omitted.

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

They messed up and either ran out of disk space or had an array fail. Everything in that statement reads like they are trying to frame a critical production failure as a glitch that occurred during maintenance. This wasn’t about a routine storage expansion. It was an emergency caused by some operational screwup and the only way to fix it is to rebuild the array with new disks.

I had a raid6 array going almost bad when 2 HDDs called it a day at the same time years ago. Had to send the entire office of 30 people to unscheduled holiday for two days before the array was rebuilt with new drives. Third drive failing during the rebuild would likely result in a week off as the storage would need a full rebuild.

I can imagine something similar happening here, however there should be a fail over storage setup in case they're selling a service.

Re: Borgbase backups have been unavailable for 3 days

#66

Post by chanandler_bong »

exhilaration wrote: post #37
chanandler_bong wrote: post #14

The fact that as a customer that is impacted by this, I only found out about it when my backups and automated test restore failed is worrying. I get that things happen, but not realising that this was going to be a service impacting event does not inspire continued confidence. Sure, this situation probably won't happen again, but what else don't they understand about their infrastructure?

> automated test restore failed You automatically test restore? That makes sense but I've never heard of that before, can you describe the process?

Pretty simple, really.

Pick a couple random files that should be in the repo, restore them from a random archive, check the md5sums against the source. If the md5sums don't match (or the file can't be found), something is wrong. I am mainly backing up RAW image files, so they should never change.

Basically...

$TEST_FILE=$(ls -p /source_dir | grep -v / | shuf -n1)

$TEST_ARCHIVE=$(borgmatic -c config.file list | shuf -n1)

borgmatic extract yada yada yada

md5sum $TEST_FILE restored_file

Re: Borgbase backups have been unavailable for 3 days

#67

Post by mananaysiempre »

colechristensen wrote: post #33
viraptor wrote: post #30

Earlier quoted context omitted.

They messed up here in some way, but assuming you know what events and complaining about their skills is just making stuff up at this point. Unless you can tell us exactly what happened, maybe hold off with that kind of criticism.

> However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. One, it is obviously some flavor of raid rebalancing, there’s not a different kind of thing it could be. Two, they’re admitting they didn’t expect this, that’s not possible if it’s a tested procedure.

By the way, is this a plausible scenario?

(1) Add more drives, start online rebuild

(2) Have one drive fail/URE mid-rebuild

(3) Be forced to switch to offline rebuild because the array was in a strange not-entirely-consistent state even before the failure, even though without the new drives you’d be in a routine online-rebuild situation

If so, possibility (2) could in principle not be detectable in staging, as it could depend on the drive’s age, and having to reread everything—as a rebuild does—is a rather abnormal load. The failure could even predate (1), if it happened to data nobody’d looked at for a long time.

This is less a question about these guys and more in general—is a RAID/ZFS/etc. array/pool/etc. in a vulnerable state after you’ve expanded it?

Re: Borgbase backups have been unavailable for 3 days

#68

Post by jnsaff2 »

chanandler_bong wrote: post #14

The fact that as a customer that is impacted by this, I only found out about it when my backups and automated test restore failed is worrying. I get that things happen, but not realising that this was going to be a service impacting event does not inspire continued confidence. Sure, this situation probably won't happen again, but what else don't they understand about their infrastructure?

It looks to me that their US10 is not like an AZ but an actual server with bunch of HBA and disks. So very much pets and not only in a single location but possibly in a single rack or even box.

You are (maybe) protected against a few disk failures but that's about it.

This FAQ entry seems to confirm this: https://docs.borgbase.com/faq/#which-storage-backend-are-you...

Re: Borgbase backups have been unavailable for 3 days

#69

Post by AnonHP »

exitheone wrote: post #42
countula wrote: post #39

TIL Borgbase. The UI looks great for snapshot-oriented backups, and it's good to see them fund borgbackup development. rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653 Unavailability is occasionally expected. Don't rely on a single provider to be your sole backup. Any one have any experience with Hetzner Storage Boxes for this?

Been using hetzner storage boxes for borg backup for >5 years. Works without issue for multiple TB of backups. Every maintenance downtime so far has been announced in advance.

The main reason I’ve avoided Hetzner storage box is because they don’t accept loading money into the account as a prepaid credit for a longer period. It’s always a month-by-month payment. Ok, so they do allow it, but only through a bank transfer, which is either not easy or possible for those outside the EU. If they allowed this through credit/debit cards or PayPal or some other mechanism, I’d surely try it. Hetzner in general seems to be very conservative in handling (and avoiding) risks.

I’d like to prepay for a year or longer for critical services so that in case something goes wrong (card expired, didn’t notice reminder emails, temporarily off the grid, temporarily incapacitated), things just don’t go poof.

Re: Borgbase backups have been unavailable for 3 days

#70

Post by ksec »

Wowfunhappy wrote: post #48
ksec wrote: post #35

Earlier quoted context omitted.

I wish Apple made a Time Capsule that requires iCloud 2TB Subscription to work. They could have priced it at $299 to include the initial 12 months iCloud subscription.

Sorry, can you expand on that? Why require the sub?

Old Time Capsule isn't really a backup. You either need two HDD with BTRFS or ZFS set up as minimum. And then You will need an offsite backup option. iCloud , Google storage etc.

If you only go via the iCloud route, you have zero local copy of your data. If you only only go via local, it isn't safe enough option. So ending up with a combination of the two.

And Apple really wanted services revenue, if it was only just a Time Capsule I doubt Apple would ever make it again.

Post reply on HN