Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Borgbase backups have been unavailable for 3 days

status.borgbase.com

Post reply on HN
Print view View original

31–40 of 88 posts

Re: Borgbase backups have been unavailable for 3 days

#31

Post by somishere »

ddtaylor wrote: post #5

I guess now is a fun time to describe my cheap backup process. All of this is related to Linux and the backups are fully encrypted client-side. Under the hood `borgbackup` is used, but I use the PikaBackup app (see Flathub) to make it easier to interface with. First I have a local copy of my data living on my actual hard drive. Next I use PikaBackup (via borg) to encrypt and sync that to a cloud server I run that has…

I've a similar local / S3 (compat) strategy but use Kopia. I currently only backup remotely to B2 (risk taker!), but could easily and cheaply add redundancy in here .. e.g S3 / R2 / random hosting. It's very cheap ~US$1. UI and default strategies are perfect for what's required. Completely automatic unless I need to browse some files in which case I just open the Kopia UI. I haven't used it, but understand Kopia also supports rsync.

Edit: previous comment on same (not a shill, I swear!): https://news.ycombinator.com/item?id=34152369

Re: Borgbase backups have been unavailable for 3 days

#32

Post by colechristensen »

shrubble wrote: post #17
smcleod wrote: post #2

Their current ETA for service restoration is another 2-4 days. Despite the warning banner the uptime for customers on us10 instance which has been unavailable for almost a week now is showing at 100%. They are compensating affected customers by crediting 4 weeks subscription but I must say this does make me wonder about their redundancy and recovery architecture. Opshugs to the folks at Borgbase. -- > box-us10 offlin…

Are they using big disks and having to resilver or re-do their RAID setup across the added drives, requiring everything on the disks to be read and re-written?

It would appear to be something like that.

> However, we did not anticipate that the expansion process would require the server storage to be temporarily offline.

This is embarrassing. Somebody made a big mistake of the “mistakes like this shouldn’t be made” category. Like not an accident or fat finger or a bug, but engineers not understanding fundamentals of how a system worked and having none of the processes in place to to dry runs in non production or any of the other things that would protect against something like this. Really questionable to trust an organization that makes a mistake like this.

Re: Borgbase backups have been unavailable for 3 days

#33

Post by colechristensen »

viraptor wrote: post #30
x0x0 wrote: post #11

Earlier quoted context omitted.

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

They messed up here in some way, but assuming you know what events and complaining about their skills is just making stuff up at this point. Unless you can tell us exactly what happened, maybe hold off with that kind of criticism.

> However, we did not anticipate that the expansion process would require the server storage to be temporarily offline.

One, it is obviously some flavor of raid rebalancing, there’s not a different kind of thing it could be.

Two, they’re admitting they didn’t expect this, that’s not possible if it’s a tested procedure.

Re: Borgbase backups have been unavailable for 3 days

#34

Post by crossroadsguy »

TylerE wrote: post #16
x0x0 wrote: post #11

Earlier quoted context omitted.

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

This is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.

Actually if I must entrust my data to companies like these where reliability and competence is of utmost importance - I’d rather trust Google and Facebook (much much before Apple).

If we are talking about privacy (or rather privacy theatre) then yeah Apple, why not.

Re: Borgbase backups have been unavailable for 3 days

#35

Post by ksec »

TylerE wrote: post #16
x0x0 wrote: post #11

Earlier quoted context omitted.

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

This is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.

I wish Apple made a Time Capsule that requires iCloud 2TB Subscription to work.

They could have priced it at $299 to include the initial 12 months iCloud subscription.

Re: Borgbase backups have been unavailable for 3 days

#36

Post by x0x0 »

colechristensen wrote: post #33
viraptor wrote: post #30

Earlier quoted context omitted.

They messed up here in some way, but assuming you know what events and complaining about their skills is just making stuff up at this point. Unless you can tell us exactly what happened, maybe hold off with that kind of criticism.

> However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. One, it is obviously some flavor of raid rebalancing, there’s not a different kind of thing it could be. Two, they’re admitting they didn’t expect this, that’s not possible if it’s a tested procedure.

And three, they're admitting they don't back anything up (backup defined as not just slinging bytes somewhere but the availability to restore those bytes and bring a service back up on a timeline that is certainly well lower than 3 days. With regular testing to understand the timeline and ensure everything actually works.)

ps -- it's not about being bashing these folks. But these type of multiple operational incompetencies strongly call into question their ability to safeguard your bytes and produce them when you need them. It's kinda the whole point of the thing.

Re: Borgbase backups have been unavailable for 3 days

#37

Post by exhilaration »

chanandler_bong wrote: post #14

The fact that as a customer that is impacted by this, I only found out about it when my backups and automated test restore failed is worrying. I get that things happen, but not realising that this was going to be a service impacting event does not inspire continued confidence. Sure, this situation probably won't happen again, but what else don't they understand about their infrastructure?

> automated test restore failed

You automatically test restore? That makes sense but I've never heard of that before, can you describe the process?

Re: Borgbase backups have been unavailable for 3 days

#38

Post by wmf »

x0x0 wrote: post #36
colechristensen wrote: post #33

Earlier quoted context omitted.

> However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. One, it is obviously some flavor of raid rebalancing, there’s not a different kind of thing it could be. Two, they’re admitting they didn’t expect this, that’s not possible if it’s a tested procedure.

And three, they're admitting they don't back anything up (backup defined as not just slinging bytes somewhere but the availability to restore those bytes and bring a service back up on a timeline that is certainly well lower than 3 days. With regular testing to understand the timeline and ensure everything actually works.) ps -- it's not about being bashing these folks. But these type of multiple operational incompet…

It's a backup company. I can understand that they don't back up the backups.

Re: Borgbase backups have been unavailable for 3 days

#39

Post by countula »

TIL Borgbase. The UI looks great for snapshot-oriented backups, and it's good to see them fund borgbackup development.

rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653

Unavailability is occasionally expected. Don't rely on a single provider to be your sole backup.

Any one have any experience with Hetzner Storage Boxes for this?

Re: Borgbase backups have been unavailable for 3 days

#40

Post by semi-extrinsic »

exhilaration wrote: post #37
chanandler_bong wrote: post #14

The fact that as a customer that is impacted by this, I only found out about it when my backups and automated test restore failed is worrying. I get that things happen, but not realising that this was going to be a service impacting event does not inspire continued confidence. Sure, this situation probably won't happen again, but what else don't they understand about their infrastructure?

> automated test restore failed You automatically test restore? That makes sense but I've never heard of that before, can you describe the process?

Not OP, but I would guess it's something like this:

  1. Make a e.g. 30MB file of random data  
  2. Copy it to "_reference" file  
  3. Upload the file to backup service  
  4. Restore the file from backup service  
  5. Diff restored file against reference
Post reply on HN