Cloud Storage for $2 per TB per month • Hacker News Board

Re: Cloud Storage for $2 per TB per month

#51

Post by breakyerself » Fri, Mar 06, 2020, 9:17 PM UTC

wmf wrote: post #44

walrus01 wrote: post #18

I work in telecom/datacenter infrastructure and this is fanciful. The whole way they take the wattage load of one machine and then hand wave away all of the rest of the costs of either building and running a datacenter, or paying ongoing monthly colocation costs... Is just scary. I truly don't mean to offend anyone but this looks like a bunch of enthusiastic dilettantes. Generators? UPS? Cooling costs? Square footage…

They're not talking about a real datacenter; they're talking about a deathtrap crypto mine with hard disks instead of GPUs/ASICs. Can't you run a million hard disks off a consumer cable modem? ;-) At the (slow) rate Sia is growing, I don't think there will ever be enough demand to justify this design anyway.

How much is a terabyte of dropbox storage?

Re: Cloud Storage for $2 per TB per month

#52

Post by ghostpepper » Fri, Mar 06, 2020, 9:17 PM UTC

vikiomega9 wrote: post #41

kmod wrote: post #15

I worked on the design of Dropbox's exabyte-scale storage system, and from that experience I can say that these numbers are all extremely optimistic, even with their "you can do it cheaper if you only target 95% uptime" caveat. Networking is much more expensive, labor is much more expensive, space is much more expensive, depreciation is faster than they say, etc etc. I don't think the authors have ever done any actua…

> exabyte-scale storage system Somewhat of a random question, can you point me to some state of the art research?

I am not the op nor an expert on this, but I think https://ceph.io/ targets clusters of this scale.

Re: Cloud Storage for $2 per TB per month

#53

Post by walrus01 » Fri, Mar 06, 2020, 9:17 PM UTC

atsmyles wrote: post #50

growt wrote: post #9

I feel like backblaze has already done most of this and has it in production [1]. Whereas this is just done back of the napkin calculation. [1] https://www.backblaze.com/b2/storage-pod.html

According to your link, backblaze hardware costs are around 10x cheaper than OP's estimate. $2.56 per TB (based on .05 per GB) vs $25 per TB.

Backblaze has not only purchase costs but also considerable ongoing operational costs for salaries, network, datacenter.

Re: Cloud Storage for $2 per TB per month

#54

Post by ajross » Fri, Mar 06, 2020, 9:20 PM UTC

walrus01 wrote: post #35

ajross wrote: post #34

Earlier quoted context omitted.

Per-site reliability (beyond a stated/assumed 95%, which is right around the "I put a computer on my desk" level) isn't a design goal, though. You can argue with their math or their assumptions, but you can't say they're wrong for not designing in the reliability features you list above.

Things on my desk are meeting three nines or better of uptime at present, 95% would be 18 days hard down per year.

I think it very much depends on the desk. I mean, power alone at my home just barely reaches three nines. Internet connectivity glitches out occasionally. I'll fat finger configurations and reinstall stuff for some more hours a year. It all adds up.

You're right that you can take the same hardware, add a $70 UPS and some thought and care, and do much, much better. But my point was only that "95%" is a trivially achievable goal even in the most naive setups, which makes me treat their analysis with a little more credence than most folks here.

Re: Cloud Storage for $2 per TB per month

#55

Post by notyourday » Fri, Mar 06, 2020, 9:20 PM UTC

kmod wrote: post #15

I worked on the design of Dropbox's exabyte-scale storage system, and from that experience I can say that these numbers are all extremely optimistic, even with their "you can do it cheaper if you only target 95% uptime" caveat. Networking is much more expensive, labor is much more expensive, space is much more expensive, depreciation is faster than they say, etc etc. I don't think the authors have ever done any actua…

We have done this calculation and even if you put your gear into Equinix/Digital Realty in the most expensive places and use Backblaze-type setup ( which is not optimized and buying retail) bringing 10Gbit to every 4U the price for double-writes at 5TB disks are $10/year per TB.

Re: Cloud Storage for $2 per TB per month

#56

Post by mtlynch » Fri, Mar 06, 2020, 9:20 PM UTC

In 2018, I spent about six weeks running a series of tests to measure Sia's real world costs. At that time, storage cost ~$4.50/TB on Sia to back up large real world files (backups of DVDs and Blu-Rays).[0] Community members have re-run my tests every few months, most recently in October 2019, when the cost was measured at $1.31/TB, though it's worth noting that recent tests use synthetic data optimized to minimize Sia's cost.[1] It's also unclear how much the market value of Sia's utility token affects these costs, as the price of Siacoin has fallen by ~80% since I conducted my original set of tests.

The calculations in today's blog post account for the labor cost of assembling hardware, but leave out major other labor costs:

1. You need an SRE to keep the servers online. Sia pushes out updates every few months, and the network penalizes you if you don't upgrade to the latest version. In addition, to optimize costs, you need to adjust your node's pricing in response to changes in the market.

2. You need a compliance officer to handle takedown requests. Since Sia allows anyone to upload data to your server without proving their identity, there's nothing stopping anyone from uploading illegal data to the network. If Sia reached the point where people are building $4k hosting rigs, then it's safe to assume clients would also be using Sia to store illegal data. When law enforcement identifies illegal data, they would send takedown notices to all hosts who are storing copies of it, and those hosts would need someone available to process those takedowns quickly.

[0] https://blog.spaceduck.io/load-test-wrapup/

[1] https://siastats.info/benchmarking

Re: Cloud Storage for $2 per TB per month

#57

Post by anonymousiam » Fri, Mar 06, 2020, 9:21 PM UTC

Taek wrote: post #36

kmod wrote: post #15

I worked on the design of Dropbox's exabyte-scale storage system, and from that experience I can say that these numbers are all extremely optimistic, even with their "you can do it cheaper if you only target 95% uptime" caveat. Networking is much more expensive, labor is much more expensive, space is much more expensive, depreciation is faster than they say, etc etc. I don't think the authors have ever done any actua…

Author here. A lot of these numbers are drawn from experience in the mining world, where people realized that when cost is the ultimate bottom line, a lot of corners can be cut. Sia systems don't need a ton of networking. I ran the networking buildout costs by some networking people, and again it comes down to cutting corners. If you only need 10 gbps per rack, if you don't mind having extra milliseconds added, etc,…

The third sentence of your Medium article says; "Despite this, the Sia network is able to achieve 99.9999% uptime for files." How do you achieve this in a "not a highly reliable facility"?

Re: Cloud Storage for $2 per TB per month

#58

Post by chx » Fri, Mar 06, 2020, 9:23 PM UTC

kmod wrote: post #15

I worked on the design of Dropbox's exabyte-scale storage system, and from that experience I can say that these numbers are all extremely optimistic, even with their "you can do it cheaper if you only target 95% uptime" caveat. Networking is much more expensive, labor is much more expensive, space is much more expensive, depreciation is faster than they say, etc etc. I don't think the authors have ever done any actua…

> I didn't read all their math but I expect their final result to be off by a factor of 2-5x.

Can't be more than 2.5 because Backblaze B2 already gives you $5/TB/Mo.

Re: Cloud Storage for $2 per TB per month

#59

Post by prepend » Fri, Mar 06, 2020, 9:23 PM UTC

anonymousiam wrote: post #57

Taek wrote: post #36

Earlier quoted context omitted.

Author here. A lot of these numbers are drawn from experience in the mining world, where people realized that when cost is the ultimate bottom line, a lot of corners can be cut. Sia systems don't need a ton of networking. I ran the networking buildout costs by some networking people, and again it comes down to cutting corners. If you only need 10 gbps per rack, if you don't mind having extra milliseconds added, etc,…

The third sentence of your Medium article says; "Despite this, the Sia network is able to achieve 99.9999% uptime for files." How do you achieve this in a "not a highly reliable facility"?

Redundancy, I assume. Lots of unreliable facilities and nodes can be very durable in aggregate.

Re: Cloud Storage for $2 per TB per month

#60

Post by wmf » Fri, Mar 06, 2020, 9:23 PM UTC

Taek wrote: post #47

mjb wrote: post #16

The IO operations amplification for 64 of 96 is pretty brutal, and particularly unfavorable in a world where capacity-per-IOPS keeps trending up. I wonder how they'll deal with that.

Each of the 64 pieces is fetched from a different host on the network, meaning all of the IOPS are happening in parallel at network speeds. You aren't going to be doing sub-millisecond updates for sure, but you can easily get under 100ms. The reed-solomon also isn't even the most computationally expensive part, the computationally expensive part is computing and verifying the Merkle roots. All parts of the system tho…

It's fine from the perspective of a single request but it seems like it reduces the overall throughput of the network. If Sia has, say, 1M hard disks and can do 100M raw IOPS then 10-of-30 gives 10M net IOPS but 64-of-96 gives only 1.5M net IOPS.