Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Borgbase backups have been unavailable for 3 days

status.borgbase.com

Post reply on HN
Print view View original

51–60 of 88 posts

Re: Borgbase backups have been unavailable for 3 days

#51

Post by ryan29 »

chanandler_bong wrote: post #14

The fact that as a customer that is impacted by this, I only found out about it when my backups and automated test restore failed is worrying. I get that things happen, but not realising that this was going to be a service impacting event does not inspire continued confidence. Sure, this situation probably won't happen again, but what else don't they understand about their infrastructure?

I got an email about it right away and I’ve also been getting warnings (that I configured in the dashboard) for inactivity on a repo that’s affected.

Re: Borgbase backups have been unavailable for 3 days

#52

Post by nieve »

kiwijamo wrote: post #27
TheNewsIsHere wrote: post #6

Earlier quoted context omitted.

I agree with you. I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? Expanding a storage environment can certainly be a mine-filled exercise for even the most experienced. But why was this expansion unexpectedly an offline operation? I’m not throwing shade here. I’m genuinely curious because I’ve been considering Borg and Borgbase for an upcoming proj…

I've used borg with rsync.net for several years for my primary off-site backup and have had fantastic reliability and support from them. They cover the usage of borg on their website[1] and support a range of other backup tools as well (including even just simply copying data over SFTP and relying on automatic ZFS snapshots on their end). I pay something in the region of $100 USD a year for this service. My secondary…

For me everything about rsync.net was great except for the throughput. It didn't matter which continent, isp, or operating system I tested, I couldn't get past single digit Mbps and sometimes had trouble reaching that. Support tried moving me to another server, but the problem persisted. Other than that I was pretty happy, but it was completely infeasible to store PostgreSQL backups there, much less server & laptop backups. Every now and then I consider going back in case they've fixed the root problem.

Re: Borgbase backups have been unavailable for 3 days

#53

Post by ryan29 »

countula wrote: post #39

TIL Borgbase. The UI looks great for snapshot-oriented backups, and it's good to see them fund borgbackup development. rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653 Unavailability is occasionally expected. Don't rely on a single provider to be your sole backup. Any one have any experience with Hetzner Storage Boxes for this?

Borgbase is pretty good. I’ve used it for 6 years.

The usage graph on each repo is great. I redid my server this year and was accidentally backing up a stale snapshot every night for one of my repos. It would succeed, but without changes. The flat line for usage tipped me off. I’m not sure I would have noticed otherwise since it was a VM image that I could restore and boot, but it wasn’t super obvious it was full of stale data.

This downtime isn’t ideal, but, if downtime is needed to preserve data integrity, I’ll take it over the risk of trying to maintain availability.

Re: Borgbase backups have been unavailable for 3 days

#54

Post by adobrawy »

TylerE wrote: post #16
x0x0 wrote: post #11

Earlier quoted context omitted.

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

This is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.

Large corporations have the technical skills in place, but organizationally they can lock your account away leaving you with no access and no real avenue to evict your data. There are many posts on HN and on the Internet where corporate support had no procedure to give back access to data when they blocked the account. Account bans are sometimes for unclear or out of control reasons (e.g. someone stole your credit card and then used that card with a another account, so the corporate banned all accounts indefinitely). In the case of a smaller organization, you have a better chance that if they have to say goodbye to you, they will do it without unnecessary harm to you.

Re: Borgbase backups have been unavailable for 3 days

#55

Post by adobrawy »

exhilaration wrote: post #37
chanandler_bong wrote: post #14

The fact that as a customer that is impacted by this, I only found out about it when my backups and automated test restore failed is worrying. I get that things happen, but not realising that this was going to be a service impacting event does not inspire continued confidence. Sure, this situation probably won't happen again, but what else don't they understand about their infrastructure?

> automated test restore failed You automatically test restore? That makes sense but I've never heard of that before, can you describe the process?

I don't use borg, but I used duplicity, which offers something like that. The verify operation simulates a backup restore to compare whether the restored file's checksum matches that expected from the metadata and optionally against the local file. I use this routinely, interesting to see that a local S3 provider can sometimes mess up your files silently.

Re: Borgbase backups have been unavailable for 3 days

#56

Post by boring_twenties »

x0x0 wrote: post #11
smcleod wrote: post #2

Their current ETA for service restoration is another 2-4 days. Despite the warning banner the uptime for customers on us10 instance which has been unavailable for almost a week now is showing at 100%. They are compensating affected customers by crediting 4 weeks subscription but I must say this does make me wonder about their redundancy and recovery architecture. Opshugs to the folks at Borgbase. -- > box-us10 offlin…

> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

In a storage business no less. They literally had one job: to know how to resize arrays, so the rest of us don't have to.

Re: Borgbase backups have been unavailable for 3 days

#57

Post by rvieira »

countula wrote: post #39

TIL Borgbase. The UI looks great for snapshot-oriented backups, and it's good to see them fund borgbackup development. rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653 Unavailability is occasionally expected. Don't rely on a single provider to be your sole backup. Any one have any experience with Hetzner Storage Boxes for this?

I've tried Borg but I settled with Kopia and a[0] cheap S3-like storage.

Everything is automated and runs on a schedule, but when I need some ad-hoc restore, say, of a single snapshot or folder I can always use KopiaUI.

Been quite happy with it.

[0] - actually two providers, for redundancy, plus local physical storage

Re: Borgbase backups have been unavailable for 3 days

#58

Post by HumanOstrich »

kiwijamo wrote: post #27
TheNewsIsHere wrote: post #6

Earlier quoted context omitted.

I agree with you. I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? Expanding a storage environment can certainly be a mine-filled exercise for even the most experienced. But why was this expansion unexpectedly an offline operation? I’m not throwing shade here. I’m genuinely curious because I’ve been considering Borg and Borgbase for an upcoming proj…

I've used borg with rsync.net for several years for my primary off-site backup and have had fantastic reliability and support from them. They cover the usage of borg on their website[1] and support a range of other backup tools as well (including even just simply copying data over SFTP and relying on automatic ZFS snapshots on their end). I pay something in the region of $100 USD a year for this service. My secondary…

I'm confused because none of this seems relevant to the comment thread. Thank you for the information anyway.

Re: Borgbase backups have been unavailable for 3 days

#59

Post by js4ever »

colechristensen wrote: post #32
shrubble wrote: post #17

Earlier quoted context omitted.

Are they using big disks and having to resilver or re-do their RAID setup across the added drives, requiring everything on the disks to be read and re-written?

It would appear to be something like that. > However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. This is embarrassing. Somebody made a big mistake of the “mistakes like this shouldn’t be made” category. Like not an accident or fat finger or a bug, but engineers not understanding fundamentals of how a system worked and having none of the processes in pl…

It's not an org but a one man show. I'm using borgbase myself, really happy with it until now and I have to admit I'm now reconsidering this choice...

Re: Borgbase backups have been unavailable for 3 days

#60

Post by jnsaff2 »

noAnswer wrote: post #29
SteveNuts wrote: post #21

Earlier quoted context omitted.

I've been in this exact situation before in a storage expansion. With enough 22TB spinning drives nowadays you can get in to a scenario where, with the new data being written into the array and the expansion process going on at the same time the rebuild essentially will never complete. This is especially true with lower end CPU servers and without dedicated RAID cards. It is dangerous because the rebuild stresses the…

> I've been in this exact situation before in a storage expansion. Can you tell us what solution that was? Something with ZFS or BTRFS? My experience with classic RAID system is, you can't expand them without reinitializing them. (But that comes with obvious warnings about data loss.)

This may not be the answer you are looking for but one option is Ceph.

Several cheap, low-power NAS boxes running Linux/Ceph and throw disk at them.

Depending on the data you store you can have 3-way replication of Erasure Coding at your preferred risk/performance level.

Disk failures are painless, box failures don't lose data, expansion and re-balancing scales with the number of disks and pretty quickly the limiting factor is your network (4-5 modern spinning disks can easily saturate a 10G network link).

Post reply on HN