Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Borgbase backups have been unavailable for 3 days

status.borgbase.com

Post reply on HN
Print view View original

21–30 of 88 posts

Re: Borgbase backups have been unavailable for 3 days

#21

Post by SteveNuts »

TheNewsIsHere wrote: post #6
smcleod wrote: post #4

Earlier quoted context omitted.

Personally I appreciate the honesty and look for it in vendors, there are too many that cover up the truth (MS/Github etc...). However I agree that it doesn't look good for a storage company.

I agree with you. I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? Expanding a storage environment can certainly be a mine-filled exercise for even the most experienced. But why was this expansion unexpectedly an offline operation? I’m not throwing shade here. I’m genuinely curious because I’ve been considering Borg and Borgbase for an upcoming proj…

I've been in this exact situation before in a storage expansion.

With enough 22TB spinning drives nowadays you can get in to a scenario where, with the new data being written into the array and the expansion process going on at the same time the rebuild essentially will never complete. This is especially true with lower end CPU servers and without dedicated RAID cards.

It is dangerous because the rebuild stresses the drive which makes a failure more likely, and a failure during the rebuild process is not great, on top of that you've got a months long ETA...

No fun

Re: Borgbase backups have been unavailable for 3 days

#22

Post by jm4 »

x0x0 wrote: post #11
smcleod wrote: post #2

Their current ETA for service restoration is another 2-4 days. Despite the warning banner the uptime for customers on us10 instance which has been unavailable for almost a week now is showing at 100%. They are compensating affected customers by crediting 4 weeks subscription but I must say this does make me wonder about their redundancy and recovery architecture. Opshugs to the folks at Borgbase. -- > box-us10 offlin…

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

They messed up and either ran out of disk space or had an array fail. Everything in that statement reads like they are trying to frame a critical production failure as a glitch that occurred during maintenance. This wasn’t about a routine storage expansion. It was an emergency caused by some operational screwup and the only way to fix it is to rebuild the array with new disks.

Re: Borgbase backups have been unavailable for 3 days

#23

Post by rexreed »

ddtaylor wrote: post #5

I guess now is a fun time to describe my cheap backup process. All of this is related to Linux and the backups are fully encrypted client-side. Under the hood `borgbackup` is used, but I use the PikaBackup app (see Flathub) to make it easier to interface with. First I have a local copy of my data living on my actual hard drive. Next I use PikaBackup (via borg) to encrypt and sync that to a cloud server I run that has…

Which cloud server do you use for your backups for the $1/mo in added costs?

Re: Borgbase backups have been unavailable for 3 days

#24

Post by muppetman »

I've loved BorgBase up until this point. I mean at least they emailed me quickly to tell me, but now I don't have backups. Well that's not true, I backup to a number of locations and BorgBase is just one of them. But it's a chink of my redundancy lost and a huge outage like this for a week makes me wonder what I'm paying for. What if BorgBase WAS my only backup solution and I needed to roll back?

Everyone makes mistakes and I appreciate they were very upfront about it, but yea. Questioning if I'll be an ongoing customer now.

Re: Borgbase backups have been unavailable for 3 days

#25

Post by istjohn »

TylerE wrote: post #16
x0x0 wrote: post #11

Earlier quoted context omitted.

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

This is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.

If you have multiple backups, you don't have to trust anyone.

Re: Borgbase backups have been unavailable for 3 days

#27

Post by kiwijamo »

TheNewsIsHere wrote: post #6
smcleod wrote: post #4

Earlier quoted context omitted.

Personally I appreciate the honesty and look for it in vendors, there are too many that cover up the truth (MS/Github etc...). However I agree that it doesn't look good for a storage company.

I agree with you. I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? Expanding a storage environment can certainly be a mine-filled exercise for even the most experienced. But why was this expansion unexpectedly an offline operation? I’m not throwing shade here. I’m genuinely curious because I’ve been considering Borg and Borgbase for an upcoming proj…

I've used borg with rsync.net for several years for my primary off-site backup and have had fantastic reliability and support from them. They cover the usage of borg on their website[1] and support a range of other backup tools as well (including even just simply copying data over SFTP and relying on automatic ZFS snapshots on their end). I pay something in the region of $100 USD a year for this service.

My secondary backup is borg as well, this one hosted on a Hetzner Storage Box[2]. Having seen some bad practice in this thread, I want to make it clear that I am not duplicating the rsync instance to Hetzner but have created a separate instance entirely to avoid issues with the primary borg instance being replicated. From what I understand this is best practice (although it does mean I have to run the backup process twice--once to rsync.net and then again to Hetzner). Hetzner also provides info[3] on using their service with borg. I pay around 3.50 EUR a month for this service.

I chose rsync.net due to their CEO being active on HN and also because they're based in North America (away from me in the Pacific). Hetzner , I chose because they are based in Europe giving me both operational diversity (operated by two different unrelated companies) and locational diversity (Western US for rsync.net and Northern Europe for Hetzner. Hetzner sharp pricing also helped as well.

I'm sure I looked at BorgBase at one point for my EU-based backup location but something must have put me in Hetzner's direction instead.

I also store photos (not my general backup, for now) in AWS S3 (largely because that makes it easier to deploy my static photo album site which it itself deployed via S3+CloudFront). I use S3 standard storage (for the time being) for a bucket in Sydney (nearest to my location) and then a glacier storage in Ireland for a bucket which is automatically replicated from my Sydney bucket.

Hopefully all this is enough redundancy! But all up I only spend ~ USD20 or so a month for this peace of mind.

[1]: https://www.rsync.net/products/borg.html [2]: https://www.hetzner.com/storage/storage-box [3]: https://community.hetzner.com/tutorials/install-and-configur...

Re: Borgbase backups have been unavailable for 3 days

#28

Post by fjfuvucucuc »

ddtaylor wrote: post #5

I guess now is a fun time to describe my cheap backup process. All of this is related to Linux and the backups are fully encrypted client-side. Under the hood `borgbackup` is used, but I use the PikaBackup app (see Flathub) to make it easier to interface with. First I have a local copy of my data living on my actual hard drive. Next I use PikaBackup (via borg) to encrypt and sync that to a cloud server I run that has…

[dead]

Re: Borgbase backups have been unavailable for 3 days

#29

Post by noAnswer »

SteveNuts wrote: post #21
TheNewsIsHere wrote: post #6

Earlier quoted context omitted.

I agree with you. I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? Expanding a storage environment can certainly be a mine-filled exercise for even the most experienced. But why was this expansion unexpectedly an offline operation? I’m not throwing shade here. I’m genuinely curious because I’ve been considering Borg and Borgbase for an upcoming proj…

I've been in this exact situation before in a storage expansion. With enough 22TB spinning drives nowadays you can get in to a scenario where, with the new data being written into the array and the expansion process going on at the same time the rebuild essentially will never complete. This is especially true with lower end CPU servers and without dedicated RAID cards. It is dangerous because the rebuild stresses the…

> I've been in this exact situation before in a storage expansion.

Can you tell us what solution that was? Something with ZFS or BTRFS? My experience with classic RAID system is, you can't expand them without reinitializing them. (But that comes with obvious warnings about data loss.)

Re: Borgbase backups have been unavailable for 3 days

#30

Post by viraptor »

x0x0 wrote: post #11
smcleod wrote: post #2

Their current ETA for service restoration is another 2-4 days. Despite the warning banner the uptime for customers on us10 instance which has been unavailable for almost a week now is showing at 100%. They are compensating affected customers by crediting 4 weeks subscription but I must say this does make me wonder about their redundancy and recovery architecture. Opshugs to the folks at Borgbase. -- > box-us10 offlin…

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

They messed up here in some way, but assuming you know what events and complaining about their skills is just making stuff up at this point. Unless you can tell us exactly what happened, maybe hold off with that kind of criticism.
Post reply on HN