71–80 of 88 posts
Earlier quoted context omitted.
> unplanned expansion process Sounds like an Elon Musk euphemism after a bad day at SpaceX.
Someone added disk to an array without understanding what would happen next.
Earlier quoted context omitted.
I agree with you. I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? Expanding a storage environment can certainly be a mine-filled exercise for even the most experienced. But why was this expansion unexpectedly an offline operation? I’m not throwing shade here. I’m genuinely curious because I’ve been considering Borg and Borgbase for an upcoming proj…
I've used borg with rsync.net for several years for my primary off-site backup and have had fantastic reliability and support from them. They cover the usage of borg on their website[1] and support a range of other backup tools as well (including even just simply copying data over SFTP and relying on automatic ZFS snapshots on their end). I pay something in the region of $100 USD a year for this service. My secondary…
Earlier quoted context omitted.
Someone added disk to an array without understanding what would happen next.
What happens next? Shouldn't the array continue working, albeit with significantly degraded performance?
But I don't know the details of what really happened, so I can't make a judgement really. If I could edit my previous comment I would, to be less emphatic that someone did something "wrong".
It could have been that this particular disk array had different firmware or was faulty. All we know is that something unexpected happened.
A vague description or white lie might have been better here - a storage company with a fleet of servers shouldn’t ever be in this situation for the reason given
You mean to downplay what actually happened?
Earlier quoted context omitted.
Someone added disk to an array without understanding what would happen next.
What happens next? Shouldn't the array continue working, albeit with significantly degraded performance?
Yes, the expected behaviour would be for the disks to be initialized. Beyond that, it'd be a configuration setting for what to do with new disks. It could be used to extend the existing LUNs, or added as a new pair in a RAID 10 style setup, or added as new members of a different level RAID.
Then it would be up to the sysadmin to extend the LUN, or divide the newly added space among multiple LUNs.
Different arrays have different behaviour with new disk. Some only add to offline LUNs. Some do everything online with lower performance, as you said. Even some super expensive enterprise kit will be very easy to break by adding new disk.
Earlier quoted context omitted.
> I've been in this exact situation before in a storage expansion. Can you tell us what solution that was? Something with ZFS or BTRFS? My experience with classic RAID system is, you can't expand them without reinitializing them. (But that comes with obvious warnings about data loss.)
Traditional hardware RAID requires you to start over from scratch to expand an array. ZFS has some limited ability to expand an array, but there are still a lot of scenarios where the recommended solution is to start over from scratch. BTRFS has full flexibility for online resizing of arrays and changing between RAID modes and rebalancing the array, but its parity RAID modes aren't really trustworthy enough yet for a…
Earlier quoted context omitted.
Been using hetzner storage boxes for borg backup for >5 years. Works without issue for multiple TB of backups. Every maintenance downtime so far has been announced in advance.
The main reason I’ve avoided Hetzner storage box is because they don’t accept loading money into the account as a prepaid credit for a longer period. It’s always a month-by-month payment. Ok, so they do allow it, but only through a bank transfer, which is either not easy or possible for those outside the EU. If they allowed this through credit/debit cards or PayPal or some other mechanism, I’d surely try it. Hetzner…
TIL Borgbase. The UI looks great for snapshot-oriented backups, and it's good to see them fund borgbackup development. rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653 Unavailability is occasionally expected. Don't rely on a single provider to be your sole backup. Any one have any experience with Hetzner Storage Boxes for this?
Though just this weekend there was some screwiness with sub accounts and ssh keys not working that did resolve itself after 24-48 hours. I didn’t notice anything communicated about it.
TIL Borgbase. The UI looks great for snapshot-oriented backups, and it's good to see them fund borgbackup development. rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653 Unavailability is occasionally expected. Don't rely on a single provider to be your sole backup. Any one have any experience with Hetzner Storage Boxes for this?
Hetzner Storage just lost a couple of snapshots a while ago… If you want “perfect architecture” and reliability/availability you just need to pay for it (aws/s3). Or make use of multiple providers.
I find that really funny. AWS has outages regularly. Whole data centers who go dark. Remember the S3 outage because an admin fat-fingerly deleted a large number of servers? ...no, people forget very quickly because of the "No one was ever fired for using AWS" mantra.
Earlier quoted context omitted.
Large corporations have the technical skills in place, but organizationally they can lock your account away leaving you with no access and no real avenue to evict your data. There are many posts on HN and on the Internet where corporate support had no procedure to give back access to data when they blocked the account. Account bans are sometimes for unclear or out of control reasons (e.g. someone stole your credit ca…
That would also be a major security concern: people gaining access to accounts by social engineering .