Ask HN: How would you store 10PB of data for your startup today? • Hacker News Board

Re: Ask HN: How would you store 10PB of data for your startup today?

#81

Post by babelfish » Fri, Apr 23, 2021, 6:53 PM UTC

I assume you're already making use of most of S3s auto-archive features?[0] Really it seems like this comes down to how quickly any of your data /needs/ to be loaded. I'd probably investigate after how much time a file is only ~1-10% likely to be accessed in the next 30 days, then auto-archive files in S3 to Glacier after that threshold. If you want to be a bit 'smarter' about it, here's an article by Dropbox[1] on how they saved $1.7M/year by determining which file previews actually need to be generated, and their strategy seems like it could be applied to your use case. That said, it seems like you are more likely to save money by going colo than by staying in the cloud.

[0] https://aws.amazon.com/blogs/aws/archive-s3-to-glacier/ [1] https://dropbox.tech/machine-learning/cannes--how-ml-saves-u...

Re: Ask HN: How would you store 10PB of data for your startup today?

#82

Post by european321 » Fri, Apr 23, 2021, 6:56 PM UTC

lostcolony wrote: post #55

philippb wrote: post #41

Earlier quoted context omitted.

If you can solve that for me without affecting revenue I have $1m in cash for you right there. We are a photo/video storage service.

So from a completely evil (well, capitalist) perspective, do you have data on how often people retrieve backups, and at what 'age' they do so? Because there may be an inflection point that offering monetary compensation for data loss, rather than actually trying to store the data, would make more financial sense. I.e., "All data > than 2 years gets silently expunged, and anyone trying to retrieve it at that point get…

What about storing the older things in some slower backup service that's slow to access but cheap? If the user eventually accesses those some day you would kick of some background job to get them fresh again. Not super UX friendly of course, but could reduce costs. Or is this already the standard thing to do?

Re: Ask HN: How would you store 10PB of data for your startup today?

#83

Post by jeffrallen » Fri, Apr 23, 2021, 6:59 PM UTC

On tape.

Re: Ask HN: How would you store 10PB of data for your startup today?

#84

Post by ktpsns » Fri, Apr 23, 2021, 7:04 PM UTC

I have worked in HPC (academia) where the cluster storage size is measured in multiples of PB since a decade. Since latency and bandwidth is a killer requirement there, Infiniband (instead of Ethernet) is the defacto standard for connecting the storage pools to the computing nodes.

Maintaining such a (storage) cluster requires 1-2 people on site which replace a few hard disks every day.

Nevertheless, when I would continously need massive amount of data, I would opt in doing it myself anytime instead of cloud services. I just know how well these clusters run and there is little to no saving when outsourcing it.

Re: Ask HN: How would you store 10PB of data for your startup today?

#85

Post by segmondy » Fri, Apr 23, 2021, 7:04 PM UTC

nikisweeting wrote: post #6

Backblaze B2, ingress and egress are free through Cloudflare, and it's S3 compatible. It's peanuts by comparison but I've been storing ~22TB on there for years and love it. Wasabi and Glacier would be my 2nd choices.

Definitely not backblaze. If you get a signed URL it remains valid for 24hrs and can be used over and over. If they are going through a proxy, that would be different, but I imagine they don't want that as that doubles bandwidth cost. You definitely don't want your client to be able to upload all the data they can in your bucket.

Re: Ask HN: How would you store 10PB of data for your startup today?

#86

Post by hemmert » Fri, Apr 23, 2021, 7:09 PM UTC

I happen to own exa-byte.com, in case you need a domain for it ;-)

(In 1998, in school, I looked up in our math book what would come after mega, giga... 20 years later, just as fresh and useless as on day one ;))

Re: Ask HN: How would you store 10PB of data for your startup today?

#87

Post by segmondy » Fri, Apr 23, 2021, 7:10 PM UTC

howeyc wrote: post #75

If you want to stick with cloud, then stick with what you're doing or migrate to a cheaper alternative like wasabi, backblaze, etc. If you're not afraid of having a few operations people on staff and running a few racks in multiple data centers, then buy a bunch of drives and servers and install something to expose everything via S3 interface (Ceph, Minio, ...) so none of your tools have to change.

I think they either stick to S3 or run their own DC with Minio in front. BB as I mentioned in another comment will be a bad idea due to the poor S3 compatible interface, See - https://www.backblaze.com/b2/docs/b2_get_upload_url.html Wasabi might be fine, but don't know if they can handle 10PB.

Re: Ask HN: How would you store 10PB of data for your startup today?

#88

Post by mceachen » Fri, Apr 23, 2021, 7:11 PM UTC

philippb wrote: post #66

epistasis wrote: post #57

If you have good sysadmin/devops types, this is a few racks of storage in a datacenter. Ceph is pretty good at managing something this size, and offers an S3 interface to the data (with a few quirks). We were mostly storing massive keys that were many gigabytes, so if you have smaller keys, so I'm not sure about performance/scalding limits with smaller keys and 10PB. I'd be sure to give your team a few months to buil…

The is amazing. Thank you. I’ve been looking at Backblaze storage pods that seem to be designed for that use case. Never rented rack space. Do you remember somehow the math on how much cheaper it was or how you thought about upfront cost vs ongoing. Just order of magnitude would be great.

Roughly a decade ago S3 storage pricing had a ~10x premium over self-hosted. The convenience of not having to touch any hardware is expensive.

Re: Ask HN: How would you store 10PB of data for your startup today?

#89

Post by cookguyruffles » Fri, Apr 23, 2021, 7:13 PM UTC

CodesInChaos wrote: post #44

stlava wrote: post #26

If data storage isn't your startup's job then I would negotiate heavily on the AWS contract.

How much can you get the pricing reduced at AWS? At list price, 10PB of IA storage cost $1.5M/yr.

AWS will blow up your phone if they know you're interested in dealing. Various online forms smattered around the site will put you in this pipeline. Just ensure you have a competing quote for them to work against

Re: Ask HN: How would you store 10PB of data for your startup today?

#90

Post by pmlnr » Fri, Apr 23, 2021, 7:13 PM UTC

Non-cloud:

HPE sells their Apollo 4000[^1] line, which takes 60x3.5" drives - with 16TB drives, that's 960TB each machine, one rack of 10 of these is 9PB+ therefore, which nearly covers your 10PB needs. (We have some racks like this). They are not cheap. (Note: Quanta makes servers that can take 108x3.5" drive, but they need special deep racks.)

The problem here would be the "filesystem" (read: the distributed service): I don't have much experience with Ceph, and ZFS across multiple machines is nasty as far as I'm aware, but I could be wrong. HDFS would work, but the latency can be completely random there.

[^1]: https://www.hpe.com/uk/en/storage/apollo-4000.html

So unless you are desperate to save money in the long run, stick to the cloud, and let someone else sweat about the filesystem level issues :)

EDIT: btw, we let the dead drives "rot": replacing them would cost more, and the failure rate is not that bad, so they stay in the machine, and we disable them in fstabs, configs, etc.

EDIT2: at 10PB HDFS would be happy; buy 3 racks of those apollos, and you're done. We started struggling at 1000+ nodes first; now, with 2400 nodes, nearly 250PB raw capacity, and literally a billion filesystem objects, we are slow as f*, so plan carefully.