Ask HN: How would you store 10PB of data for your startup today?

Post by philippb » Fri, Apr 23, 2021, 9:25 PM UTC

joering2 wrote: post #117

It would be cool to actually have a "blockchain" for something like this. I know the huge amount of data to be store is a niche market, but hear me out: Everyone that wants to make extra money can join You join with your computer hooked up to internet, a piece of software running in background You share % of your hard-drive and limit speed that can be used to upload/download When someone needs to store 100PB of data…

Post by derefr » Fri, Apr 23, 2021, 9:29 PM UTC

lamontcg wrote: post #133

TechBro8615 wrote: post #108

Earlier quoted context omitted.

I’m not an AWS pricing expert, but you should be aware you’re still on the hook for S3 requests even if you can get out of paying for bandwidth. Is AWS direct connect a pure peering arrangement? I wonder what their requirements are for that. Guess I’ll read the link :) Idk what your team’s expertise is, but I’d advise avoiding the cloud as long as possible. If you can build out an on-premise infrastructure, it will b…

> as long as you’ve got a capable team looking after it who chooses standard and robust software. And cheap. If you put people in charge who are looking for ways of expanding their empire and budget through spending money on EMC/VMWare/Oracle/etc/etc then you can quickly wind up spending a lot more money. Simplistic network designs, simplistic server designs, simplistic storage designs with mostly open source softwar…

Post by canucker2016 » Fri, Apr 23, 2021, 9:31 PM UTC

nicoburns wrote: post #156

canucker2016 wrote: post #146

Earlier quoted context omitted.

According to https://aws.amazon.com/snowball/pricing/ , egress fees depends on the region, which can range from $0.03/GB (North America & parts of Europe) to $0.05/GB (parts of Asia and Africa). So US$300K to US$500K for egress fees + cost of Snowball devices. The major downside of Snowball in this export case is the size limit of 80TB per device - from https://aws.amazon.com/snowball/features/ : "Snowball Edge Stora…

There's also the snowmobile https://aws.amazon.com/snowmobile/

Post by royalresolved » Fri, Apr 23, 2021, 9:34 PM UTC

Post by user5994461 » Fri, Apr 23, 2021, 9:37 PM UTC

skynet-9000 wrote: post #141

spookthesunset wrote: post #64

Earlier quoted context omitted.

Don’t forget labor. You need to find talent to manage your little data center. And deal with it when it shits the bed at 4:12am on Christmas morning. So toss in at least one SRE type person. Say $200k/year. Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one.... Since redundancy is a thing, double that $350k. And 10pb is what they…

10 petabytes at AWS is $210,000 per month just for storage (even excluding AWS's very high egress and transaction pricing), so even $1M (which seems like a high estimate indeed) would be amortized in less than six months. Also, the hardware can be depreciated, which reduces its net (of taxes) cost dramatically over time. Five years (probably the useful life of the equipment in general) of $210,000 per month is $12.6M…