Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Borgbase backups have been unavailable for 3 days

status.borgbase.com

Post reply on HN
Print view View original

11–20 of 88 posts

Re: Borgbase backups have been unavailable for 3 days

#11

Post by x0x0 »

smcleod wrote: post #2

Their current ETA for service restoration is another 2-4 days. Despite the warning banner the uptime for customers on us10 instance which has been unavailable for almost a week now is showing at 100%. They are compensating affected customers by crediting 4 weeks subscription but I must say this does make me wonder about their redundancy and recovery architecture. Opshugs to the folks at Borgbase. -- > box-us10 offlin…

> me wonder about their redundancy and recovery architecture

You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

Re: Borgbase backups have been unavailable for 3 days

#12

Post by willbush »

ddtaylor wrote: post #5

I guess now is a fun time to describe my cheap backup process. All of this is related to Linux and the backups are fully encrypted client-side. Under the hood `borgbackup` is used, but I use the PikaBackup app (see Flathub) to make it easier to interface with. First I have a local copy of my data living on my actual hard drive. Next I use PikaBackup (via borg) to encrypt and sync that to a cloud server I run that has…

I do something similar, but with restic. I wrote about it here: https://willbush.dev/blog/synology-nas-backup/ if anyone is interested.

Re: Borgbase backups have been unavailable for 3 days

#13

Post by sparrish »

smcleod wrote: post #4
kiririn wrote: post #3

A vague description or white lie might have been better here - a storage company with a fleet of servers shouldn’t ever be in this situation for the reason given

Personally I appreciate the honesty and look for it in vendors, there are too many that cover up the truth (MS/Github etc...). However I agree that it doesn't look good for a storage company.

And it's a practical guarantee that something like this will never happen again. The wise man learns from his mistakes.

Re: Borgbase backups have been unavailable for 3 days

#14

Post by chanandler_bong »

The fact that as a customer that is impacted by this, I only found out about it when my backups and automated test restore failed is worrying.

I get that things happen, but not realising that this was going to be a service impacting event does not inspire continued confidence. Sure, this situation probably won't happen again, but what else don't they understand about their infrastructure?

Re: Borgbase backups have been unavailable for 3 days

#15

Post by dikei »

x0x0 wrote: post #11
smcleod wrote: post #2

Their current ETA for service restoration is another 2-4 days. Despite the warning banner the uptime for customers on us10 instance which has been unavailable for almost a week now is showing at 100%. They are compensating affected customers by crediting 4 weeks subscription but I must say this does make me wonder about their redundancy and recovery architecture. Opshugs to the folks at Borgbase. -- > box-us10 offlin…

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

Yeah, I wouldn't trust their process at all, if they'd rather have the server down for 1 week instead of aborting and performing a rollback to the last known good state.

Re: Borgbase backups have been unavailable for 3 days

#16

Post by TylerE »

x0x0 wrote: post #11
smcleod wrote: post #2

Their current ETA for service restoration is another 2-4 days. Despite the warning banner the uptime for customers on us10 instance which has been unavailable for almost a week now is showing at 100%. They are compensating affected customers by crediting 4 weeks subscription but I must say this does make me wonder about their redundancy and recovery architecture. Opshugs to the folks at Borgbase. -- > box-us10 offlin…

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

This is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.

Re: Borgbase backups have been unavailable for 3 days

#17

Post by shrubble »

smcleod wrote: post #2

Their current ETA for service restoration is another 2-4 days. Despite the warning banner the uptime for customers on us10 instance which has been unavailable for almost a week now is showing at 100%. They are compensating affected customers by crediting 4 weeks subscription but I must say this does make me wonder about their redundancy and recovery architecture. Opshugs to the folks at Borgbase. -- > box-us10 offlin…

Are they using big disks and having to resilver or re-do their RAID setup across the added drives, requiring everything on the disks to be read and re-written?

Re: Borgbase backups have been unavailable for 3 days

#18

Post by dikei »

ddtaylor wrote: post #5

I guess now is a fun time to describe my cheap backup process. All of this is related to Linux and the backups are fully encrypted client-side. Under the hood `borgbackup` is used, but I use the PikaBackup app (see Flathub) to make it easier to interface with. First I have a local copy of my data living on my actual hard drive. Next I use PikaBackup (via borg) to encrypt and sync that to a cloud server I run that has…

Similarly, for my Linux devices I use `borg` to backup to a local NAS, and use `kopia` to have another backup to Google Cloud Storage, as B2 too slow where I live.

I used to have back up on local external drives too, but stopped doing that, since the process was manual and I often forgot about it.

Re: Borgbase backups have been unavailable for 3 days

#19

Post by x0x0 »

dikei wrote: post #15
x0x0 wrote: post #11

Earlier quoted context omitted.

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

Yeah, I wouldn't trust their process at all, if they'd rather have the server down for 1 week instead of aborting and performing a rollback to the last known good state.

I suspect they can't. Abort and rollback, that is. Evidence: it's not rational to eat a week of downtime if you have an alternative.

edit: Which tells you something about their backups, and it ain't great. Or they'd restore onto another box.

Re: Borgbase backups have been unavailable for 3 days

#20

Post by mkoryak »

sparrish wrote: post #13
smcleod wrote: post #4

Earlier quoted context omitted.

Personally I appreciate the honesty and look for it in vendors, there are too many that cover up the truth (MS/Github etc...). However I agree that it doesn't look good for a storage company.

And it's a practical guarantee that something like this will never happen again. The wise man learns from his mistakes.

That man was not very wise to begin with. I wonder what else he needs to learn about _in prod_.
Post reply on HN