Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

331–340 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#331

Post by Terretta »

walrus01 wrote: post #327
Terretta wrote: post #326

Earlier quoted context omitted.

Costs less to leave them alone, and go once a year for a trash run. Cattle not pets, no trips to the vet. Don’t waste money diagnosing / fixing.

'cattle not pets' is not a valid argument when you're the owner and operator of the bare metal hardware. Do you also recommend that ISPs not replace failed fans in core and edge routers and optical transport systems? Let things with dual power supplies run for six months on one failed power supply? Also you've clearly never interacted with cattle, sheep, goats, llamas or alpacas, which absolutely do get things like v…

I’ve been both farmer of black angus beef cattle (on 750 acres) right down to castrating steer, and founder/owner of the world’s largest VDN (14 international data centers) right down to pulling drives.

For what it’s worth, meat packers buy dead cattle and don’t ask questions. But this is a well known metaphor, and I’m pointing out by that metaphor, no trips to the vet. As a cattle farmer, I’d argue it holds true if you’re big enough they’ve got tags not names: the vet comes to you and only if you think you’ve got a herd problem instead of an individual problem.

As for the HN angle: these are contrarian and objection-inspiring policies that let us wholesale video delivery to/through other CDNs while making a profit.

Re: Ask HN: How would you store 10PB of data for your startup today?

#332

Post by cortesoft »

Terretta wrote: post #326
cortesoft wrote: post #282

Earlier quoted context omitted.

That's what remote hands are for. Yes, you batch the replacements up, but this is exactly what remote hands are for.

Costs less to leave them alone, and go once a year for a trash run. Cattle not pets, no trips to the vet. Don’t waste money diagnosing / fixing.

This doesn't make sense to me. I work for a CDN with tens of thousands of servers in over a hundred data centers. We are always working to improve our turnaround time on repairing servers, even though we have thousands. Hard drive failure is one of the leading causes of server failures. Dead servers means diminished capacity, and capacity is what pays our bills.

Farmers absolutely have a vet who takes care of the cattle. I am not sure what you are on about.

The whole point of cattle-vs-pets is you are supposed to treat all the servers the same, not that you have to treat all of them poorly.

Re: Ask HN: How would you store 10PB of data for your startup today?

#333

Post by jedberg »

dilyevsky wrote: post #290
jedberg wrote: post #216

Earlier quoted context omitted.

I've put 39U of drives in a rack before. You only need 1U for a network switch, and you can get power that attaches vertically to the back, so it doesn't take up any space. If you have a cabinet with rack in front and back and all the servers have rails, the weight shouldn't be an issue. The biggest issue will be cooling depending on how hot your servers run. Specifically, it was a rack full of Xserve RAIDs, which ar…

You run a single network switch for a rack full of drives to the brim?

You could easily run two switches, there would be enough room. But normally yes, I'd run one switch per rack. Switch failure is pretty rare, and when it does happen it's pretty easy to switch it out for a spare.

Re: Ask HN: How would you store 10PB of data for your startup today?

#334

Post by oneplane »

marcinzm wrote: post #330
oneplane wrote: post #328

Earlier quoted context omitted.

The author doesn't want a second site. And at that scale you do redundancy at that scale within the requested parameters. If you set your object store to be resilient to single-partition loss per object (within CAP) you effectively duplicate everything once. If you want more-than-one you get into sharding to spread the risk. We're not talking about RAID here, but about replicas or copies. Windows Storage Server doesn…

>We're not talking about RAID here, but about replicas or copies. Most distributed filesystems support some form of erasure coding. Ceph does, Minio does, HDFS does, etc. So no, you don't need to duplicate everything.

You're talking about data integrity, this is not the same as redundancy.

Re: Ask HN: How would you store 10PB of data for your startup today?

#336

Post by anij »

Disclaimer: *I work for Nutanix*

Consider looking at Nutanix - you can get the hardware from HPE (including Apollo).

Object storage from Nutanix doesn’t even break a sweat at 10PB of usable storage.

However the main reasons to look at Nutanix would be ease of use for

day 0 (bootstrapping) day 1 (administration operations, capacity management), fault tolerance and day n operations (upgrades, security patches etc)

Nutanix spends considerable time and resources on all this to make life of our customers easy.

Re: Ask HN: How would you store 10PB of data for your startup today?

#337

Post by webmaven »

Titan2189 wrote: post #259
Johnny555 wrote: post #221

Earlier quoted context omitted.

10PB costs more than $210,000 per month at S3, or more than $12M after five years. Your pricing is off by a 2X - he said he's ok with infrequent access, 1 zone, which is $0.01/GB, or $100K/month. If he rarely needs to read most of the data, he can cut the price by 1/10th by using deep archive, $0.00099 per GB, so $10K/month, or around $600K over 5 years, not including retrieval costs.

Nope, can't use Deep Archive as he specified max retrieval time of 1000ms. But you're correct with S3-IA

> Nope, can't use Deep Archive as he specified max retrieval time of 1000ms.

If accesses can be anticipated, pre-loading data from cold storage to something warmer might make it viable.

Re: Ask HN: How would you store 10PB of data for your startup today?

#338

Post by webmaven »

oneplane wrote: post #334
marcinzm wrote: post #330

Earlier quoted context omitted.

>We're not talking about RAID here, but about replicas or copies. Most distributed filesystems support some form of erasure coding. Ceph does, Minio does, HDFS does, etc. So no, you don't need to duplicate everything.

You're talking about data integrity, this is not the same as redundancy.

> You're talking about data integrity, this is not the same as redundancy.

To be clear, you're talking about mitigating the risk of data corruption (eg. bits will flip randomly due to cosmic rays or what have you) over time, vs. the risk of outright data loss, yes?

Isn't there some some overlap between the solutions?

Re: Ask HN: How would you store 10PB of data for your startup today?

#339

Post by webmaven »

derefr wrote: post #324
lamontcg wrote: post #269

Earlier quoted context omitted.

If you narrowly focus on the words outside of the context of what "SAN" has meant in the industry for decades now, yes it is. But no, it isn't.

Can you explain more? Because I honestly don't know enough about SANs to know the difference. To me, a "Storage Area Network" is 1. a cluster of disk-servers, serving the role of exposing logical block-storage over a protocol like iSCSI (whether directly to client machines, or managed and dynamically allocated by hypervisor software like vSphere), where 2. machines are connected to that storage cluster over a dedicat…

> Does "SAN" imply some specific internal architecture for the storage cluster or something?

It implies purchasing dedicated hardware. SANs are CAPEX heavy solutions.

> And, if so, then what do you call the type of thing that EBS is?

If you insist, you could call EBS a SAN-as-a-Service, I suppose.

Re: Ask HN: How would you store 10PB of data for your startup today?

#340

Post by oneplane »

webmaven wrote: post #338
oneplane wrote: post #334

Earlier quoted context omitted.

You're talking about data integrity, this is not the same as redundancy.

> You're talking about data integrity, this is not the same as redundancy. To be clear, you're talking about mitigating the risk of data corruption (eg. bits will flip randomly due to cosmic rays or what have you) over time, vs. the risk of outright data loss, yes? Isn't there some some overlap between the solutions?

No, I'm talking about mitigating system failure (be it a dead disk, PHY, entire server, single PDU, single rack or entire feed. I didn't even go down the level of individual object durability yet (or web access to those objects, consistent access control and the likes).

There is some overlap in the sense that having redundant copies makes it possible to replace a bad copy with a good copy if a checksum mismatches on one of them. That also allows for bringing the copy count back in spec if a single copy goes missing (regardless of the type of failure).

But no matter what methods are used, data is data and needs to be stored somewhere. It the bits constituting that data go missing, the data is is gone. To prevent that, you need to make sure those bits exist in more than one place. The specific places come with differences in cost, mitigations and effort:

- Two copies on the same disk mitigates bit flips in one copy but not disk failure - Two copies on two disks on the same HBA mitigates bit flips and disk failure but not HBA failure

The list goes on until you reach the requirement posted at the top of this Ask HN where it is stated that OneZone IA is used. That means it does not need multiple zones for zone-outage mitigation. Effectively that means the racks are allowed to be placed in the same datacenter. So that datacenter being unavailable or destroyed means the data is unavailable (temporarily or permanently), which appears to be the accepted risk.

But within that zone (or datacenter) you would still need all other mitigations offered by the durable object storage S3 provides (unless specified differently - if we just make up new requirements we can make it very cheap and just accept total system failure with 1 bit flip and be done with it).

Post reply on HN