101–110 of 190 posts
Earlier quoted context omitted.
Multi-region redundancy was on the roadmap for their B2 offering 10 months ago, not sure if it has shipped yet. https://news.ycombinator.com/item?id=15125643
I poked around their pricing page a bit before I posted to see if I could find multi-site redundancy options, but couldn't find anything.
Earlier quoted context omitted.
I've posted the math here before but if we assume that an asteroid hits the earth every 65 million years and wipes out the dominant life forms, then this fact alone puts your yearly durability at a maximum of ~8 nines. The point about billing is better, though. My other concern is that a software bug, operator error, or malicious operator deletes your data.
That's why no sane entity would use one earth. Use two and you can quickly recover.
Earlier quoted context omitted.
Also, there are only 3 hard drive manufacturers left. If one of them have a bug that affects across their product line, that can take out 1/3 of all hard drives.
It's vanishingly unlikely that a bug would affect all their drives (across all recent models, and only after burn-in) simultaneously, unless the drives are managed by a remote server with a SPOF.
How many years between incidents are you talking, and when was the last time a manufacturer had a multigenerational bug? (and is quality improving, or decreasing? That is, are we more or less likely to see failures in the next 10 years than we did in the last 10?)
Earlier quoted context omitted.
Nope. They aren’t even in the same city. For example, each of the AZs in us-west-2 are separated by about 50-60 miles. This doesn’t list all the locations, but is a good map to get an idea: https://www.google.com/maps/d/u/0/viewer?ll=50.9584270000000...
That map is incorrect: us-west-2a, 2b and 2c are not static names for the AZs. Every user gets their own mapping of which physical location is a, which one is b and which one is c. My us-west-2a may be your us-west-2c. They are not the same.
[0] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-re...
One question since I know some of the Backblaze folks respond to these threads: In addition to calling (and possibly getting blocked/ignored), does your customer service staff send text messages? I suspect that a big percentage of the phone numbers you have are for cell phones these days, and I see a lot less SMS spam than I do telemarketing. SMS would also allow you to get a bit of info visible to recipients (e.g. "…
This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…
> Correlated failures are common in drives. This is why, when I was building DIY arrays for startups (around the same time Backblaze published their first pod design [1]), I went through the extra effort of sourcing disks from as many different vendors as possible. Although it was somewhat more time consuming and limited how good a price I could get and how fast the delivery could be, it meant that, for any given dis…
This is very good advice!
If you already built your array, consider advice: "replace a bad disk with a different brand, whenever possible".
Over time, you naturally migrate away from the bad vendors/models/batches. After following this practice, it seems ridiculous to me now to keep replacing the same bad disks with the same vendor+model.
This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…
I'm not sure how anyone, in their disaster recovery plans, ever expect anything less than 100% failure of a data center. The scenario is: a tornado hits the data center. Your power outage causing 3% of drives to fail is just a subset of that.
I've written in more detail before[0], but just to share the gotchas in case anyone here is thinking of switching to Backblaze:
1. They backup almost no file metadata.
2. The client is very slow (days or more) to add new files and there's no transparency (it claims everything is backed up when it's not).
3. There are still bugs in the client that can put your backup into an invalid state where it gets deleted.
4. Support is terrible, and won't be any help when you run into these bugs.
Earlier quoted context omitted.
I'm not sure how anyone, in their disaster recovery plans, ever expect anything less than 100% failure of a data center. The scenario is: a tornado hits the data center. Your power outage causing 3% of drives to fail is just a subset of that.
If it's your data center, you can plan the location to prevent most of this. There are locations where the natural hazards can be completely managed. (No tornados, fires, tsunamis, earthquakes, ...) So the power outage is the most likely thing to happen.
Earlier quoted context omitted.
> Correlated failures are common in drives. This is why, when I was building DIY arrays for startups (around the same time Backblaze published their first pod design [1]), I went through the extra effort of sourcing disks from as many different vendors as possible. Although it was somewhat more time consuming and limited how good a price I could get and how fast the delivery could be, it meant that, for any given dis…
> I went through the extra effort of sourcing disks from as many different vendors as possible. This is very good advice! If you already built your array, consider advice: "replace a bad disk with a different brand, whenever possible". Over time, you naturally migrate away from the bad vendors/models/batches. After following this practice, it seems ridiculous to me now to keep replacing the same bad disks with the sa…
Some of this can also be achieved ahead of time if one has multiple arrays with hot spares, by shuffling hot spares around, assuming there's some model diversity between the arrays but not within them.
I doubt I'll ever again have the luxury of being able to perform this kind of engineering, however. Even a minor increase in cost or cognitive/procedure complexity or a decrease in convenience just serves to encourage a "let's move everything to the cloud" reaction, so I keep my mouth shut.