Earlier quoted context omitted.
He's the CTO of KeepSafe.
Ok, so it seems that they are indeed a data storage company.
We are.
Ask HN: How would you store 10PB of data for your startup today?
51–60 of 374 posts
Re: Ask HN: How would you store 10PB of data for your startup today?
#52Earlier quoted context omitted.
Excellent advice for a data backup startup.
Doesn't this imply that they started a company without actually having a plan for the most fundamental part of what they are selling? This is like an ISP asking how they can get hooked up to the internet.
No. They are already managing 10PB, planning for which would be very stupid when just starting up.
Re: Ask HN: How would you store 10PB of data for your startup today?
#53Earlier quoted context omitted.
Excellent advice for a data backup startup.
Doesn't this imply that they started a company without actually having a plan for the most fundamental part of what they are selling? This is like an ISP asking how they can get hooked up to the internet.
It's more like a company that has validated product fit now needing to figure out how to scale economically.
Apple didn't start manufacturing with mega Foxconn contracts. They had to figure that out along the way as their scale demanded.
However I share your sentiment: doing things the same way but cheaper is usually not the solution. Doing things differently (in-sourcing) might be the path forward.
Re: Ask HN: How would you store 10PB of data for your startup today?
#54Earlier quoted context omitted.
Lol. A stick of 256gb RAM costs ~$3000. 1TB needs 4. 1PB needs 4000. 10PB needs 40,000. So this would be an upfront cost of $120M. And this doesn't even cover how you'd fit 40,000 sticks of RAM together.
10PB of RAM being only $120M blows my mind, to be honest. I would have guessed that price was closer to the SSD cost for 10PB.
This made me curious about what the SSD cost would be. It looks like you can get a 2TB SSD for $200. So that's $100/TB = $1M for 10PB. Of course prices may be higher for enterprise SSDs and you may need redundancy. Then again, you could probably get a bulk discount at that scale.
Re: Ask HN: How would you store 10PB of data for your startup today?
#55Here's an unpopular answer - don't store 10PB of data. Find a way for your startup to work without needlessly having to store insane amounts of data that will likely never be needed.
If you can solve that for me without affecting revenue I have $1m in cash for you right there. We are a photo/video storage service.
So from a completely evil (well, capitalist) perspective, do you have data on how often people retrieve backups, and at what 'age' they do so?
Because there may be an inflection point that offering monetary compensation for data loss, rather than actually trying to store the data, would make more financial sense. I.e., "All data > than 2 years gets silently expunged, and anyone trying to retrieve it at that point gets $10 per gig in compensation for 'our mistake'".
Please don't actually consider that though.
Re: Ask HN: How would you store 10PB of data for your startup today?
#56It seems to me like you could save a ton of money by using your own hardware.
Perhaps buy a bunch of big Synology boxes?
At that scale you should also consider looking at technologies such as Ceph.
We've recently switched to a setup with several Synology boxes for around 1PB net storage.
Re: Ask HN: How would you store 10PB of data for your startup today?
#57If you have good sysadmin/devops types, this is a few racks of storage in a datacenter. Ceph is pretty good at managing something this size, and offers an S3 interface to the data (with a few quirks). We were mostly storing massive keys that were many gigabytes, so if you have smaller keys, so I'm not sure about performance/scalding limits with smaller keys and 10PB. I'd be sure to give your team a few months to build a test cluster then build and scale the full size cluster. And a few months to transfer the data...
But you'll need to balance the cost of finding people with that level of knowledge and adaptability with the cost of bundled storage packages. We were running super lean, got great deals on bandwidth, power, and has low performance requirements. When we ran the numbers for all in costs, it was less than we thought we could get from any other vendor. And if you commit to buying the severs racks it will take to fit 10PB, you can probably get somebody like Quanta to talk to you.
Re: Ask HN: How would you store 10PB of data for your startup today?
#58Earlier quoted context omitted.
10PB with their pricing calculator comes out to over $60,000/mo. Feels like a lot. edit: perhaps their RCS option would be cheaper if you know exactly how much data you need to store in advance.
To be fair, purchasing and hosting even the most basic mirrored RAID array of that scale comes to well over half a million for the disks alone. Then you need to manage them.
10x Supermicro SSG-6049P-E1CR60H servers (60 x 3,5" HDD in 4U enclosure) - $5k each
600x WESTERN DIGITAL Ultrastar DC HC550 18TB (10800PB in total) - $500 each
~$350k in hardware, up to 20kW energy consumption, should fit in two rack towers. You can host it for about $1.5k somewhere. All assuming no redundancy :)
Re: Ask HN: How would you store 10PB of data for your startup today?
#59Earlier quoted context omitted.
Doesn't this imply that they started a company without actually having a plan for the most fundamental part of what they are selling? This is like an ISP asking how they can get hooked up to the internet.
It's more like a company that has validated product fit now needing to figure out how to scale economically. Apple didn't start manufacturing with mega Foxconn contracts. They had to figure that out along the way as their scale demanded. However I share your sentiment: doing things the same way but cheaper is usually not the solution. Doing things differently (in-sourcing) might be the path forward.
Figuring out how to scale is not just a part of a storage startup, it's the whole thing.
Apple created something people wanted and sold at a price that would still make money if it was assembled by hand. They didn't form a company around a commodity like data storage.
Data storage is a commodity. Everyone already has some, online storage companies already exist. If you don't know how to store a lot of data and your company's whole purpose is to store a lot of data, it sounds like something that should have been worked out before making the company.
Re: Ask HN: How would you store 10PB of data for your startup today?
#60Earlier quoted context omitted.
Doesn't this imply that they started a company without actually having a plan for the most fundamental part of what they are selling? This is like an ISP asking how they can get hooked up to the internet.
No. They are already managing 10PB, planning for which would be very stupid when just starting up.
Why would planning to be able to execute the single focus of your startup be stupid?