201–210 of 353 posts
Earlier quoted context omitted.
Redundancy, I assume. Lots of unreliable facilities and nodes can be very durable in aggregate.
Sounds kinda like the people who thought that bundling a bunch of bad mortgage debt together in slices could get it rated AAA.
Earlier quoted context omitted.
Who are the customers for such unreliable systems?
Sia is very reliable from the customer's perspective. Its only individual systems and storage pools that have lax uptime requirements. Thanks to some clever network-level redundancy mechanisms (10-of-30 redundancy), 95% uptime at the storage pool level translates to 99.9999% uptime from the customer's perspective. See the "Uptime Math" section in the OP for details.
Earlier quoted context omitted.
> exabyte-scale storage system Somewhat of a random question, can you point me to some state of the art research?
Storage Systems of this scale are thesedays almost exclusively built around object based storage rather than "legacy" block or file backed solutions. I guess today, min.io is would be the way to go. (to "go".. little pun on the end:)
Earlier quoted context omitted.
I didn't say anything about speed of recovery because it's not relevant. Recovery can't happen at all if enough fragments aren't online. The maths says that with unncorelated fragment placement, and thus uncorrelated failures, and with enough data, you are basically guaranteed to lose data. Try doing the maths for an entire filesystem, where each file/block is individually erasure coded.
We stored hundreds of petabytes on cheap SATA drives with random fragment placement using reed solomon 6+3 coding (half the space of 3 replicas but same durability). Never lost a byte. Speed of recovery is crucial, because that’s your window of vulnerability to multiple failures. For example. try raid 5 on giant drives. The chances of losing a second drive during recovery is very likely.
What was the probability of failure of your drives? My guess is you just didn't hit the threshold for your failure rate. The maths checks out (PhD here). Seriously, do the calculation.
I wouldn't even buy new hardware. I would buy retired hardware for a fraction of the price.
I would imagine that managing such a pool of dated and highly heterogeneous hardware would incur a lot of overhead. The procurement effort may end up being rather high as well.
I worked on the design of Dropbox's exabyte-scale storage system, and from that experience I can say that these numbers are all extremely optimistic, even with their "you can do it cheaper if you only target 95% uptime" caveat. Networking is much more expensive, labor is much more expensive, space is much more expensive, depreciation is faster than they say, etc etc. I don't think the authors have ever done any actua…
Author here. A lot of these numbers are drawn from experience in the mining world, where people realized that when cost is the ultimate bottom line, a lot of corners can be cut. Sia systems don't need a ton of networking. I ran the networking buildout costs by some networking people, and again it comes down to cutting corners. If you only need 10 gbps per rack, if you don't mind having extra milliseconds added, etc,…
That puts the cost of 192 TB at more like $6240, not $4945. Could be less if you find a good deal on mini-SAS PCIE cards, but still going to be substantially higher than $4945.
I worked on the design of Dropbox's exabyte-scale storage system, and from that experience I can say that these numbers are all extremely optimistic, even with their "you can do it cheaper if you only target 95% uptime" caveat. Networking is much more expensive, labor is much more expensive, space is much more expensive, depreciation is faster than they say, etc etc. I don't think the authors have ever done any actua…
> I didn't read all their math but I expect their final result to be off by a factor of 2-5x. Can't be more than 2.5 because Backblaze B2 already gives you $5/TB/Mo.
Earlier quoted context omitted.
Storage Systems of this scale are thesedays almost exclusively built around object based storage rather than "legacy" block or file backed solutions. I guess today, min.io is would be the way to go. (to "go".. little pun on the end:)
What does object-based mean? How is an object different from a file (which I presumed was a collection of blocks)?
How did they go from 8 SATA ports to 32 SATA devices? The linked cable uses a SAS host/connector. Was a SAS card left out of the parts list?
Using "8x 4 way SATA data splitters". What i don't get is why they don't use 14TB HDDs, they are only 15% more expensive per TB. On the other hand they'd need 2.33x less PCs at $550 each, plus their power use. So instead of every 7 PCs with 6TB HDDs they'd need 3 with 14TB HDDs. PS: They could also use a mainboard with 10 SATA ports instead of 8. They are only $15 more than the chosen board. Adding one or more PCIe 8…
Earlier quoted context omitted.
Redundancy, I assume. Lots of unreliable facilities and nodes can be very durable in aggregate.
Sounds kinda like the people who thought that bundling a bunch of bad mortgage debt together in slices could get it rated AAA.