Ask HN: How would you store 10PB of data for your startup today?

#1

Post by philippb » Fri, Apr 23, 2021, 8:12 AM UTC

I'm running a startup and we're storing north of 10PB of data and growing. We're currently on AWS and our contract is up for renewal. I'm exploring other storage solutions.

Min requirements of AWS S3 One Zone IA (https://aws.amazon.com/s3/storage-classes/?nc=sn&loc=3)

How would you store >10PB if you'd be in my shoes? Thought experiment can be with and without data transfer cost our of current S3 buckets. Please mention also what your experience is based on. Ideally you store large amounts of data yourself and speak of first hand experience.

Thank you for your support!! I will post a thread once we got to a decision on what we ended up doing.

Update: Should have mentioned earlier, data needs to be accessible at all time. It’s user generated data that is downloaded in the background to a mobile phone, so super low latency is not important, but less than 1000ms required.

The data is all images and videos, and no queries need to be performed on the data.

Re: Ask HN: How would you store 10PB of data for your startup today?

#2

Post by verdverm » Fri, Apr 23, 2021, 9:16 AM UTC

It definitely depends on how you accumulate and the usage patterns. More clarity is needed there to make recommendations.

As an aside, you can often get nice credits for moving off of AWS to Azure or GCP. I recommend the later.

Re: Ask HN: How would you store 10PB of data for your startup today?

#3

Post by brudgers » Fri, Apr 23, 2021, 12:53 PM UTC

Is the storage of the data critical to the future growth of the business?

Re: Ask HN: How would you store 10PB of data for your startup today?

#4

Post by giantg2 » Fri, Apr 23, 2021, 1:41 PM UTC

Agree with someone else's comment questioning how is the data ingested and used.

10PB seems like a lot to store in S3 buckets. I assume much of that data is not accessed frequently or would be used in a big data scenario. Maybe some other services like Glacier or RedShift (I think).

Re: Ask HN: How would you store 10PB of data for your startup today?

#5

Post by glitchc » Fri, Apr 23, 2021, 3:25 PM UTC

Compression is always a good alternative, which is especially effective when modification is infrequent.

Re: Ask HN: How would you store 10PB of data for your startup today?

#6

Post by nikisweeting » Fri, Apr 23, 2021, 3:26 PM UTC

Backblaze B2, ingress and egress are free through Cloudflare, and it's S3 compatible. It's peanuts by comparison but I've been storing ~22TB on there for years and love it.

Wasabi and Glacier would be my 2nd choices.

Re: Ask HN: How would you store 10PB of data for your startup today?

#7

Post by PLenz » Fri, Apr 23, 2021, 3:26 PM UTC

I would consider moving to my own metal and using hadoop.

Re: Ask HN: How would you store 10PB of data for your startup today?

#8

Post by gruez » Fri, Apr 23, 2021, 3:27 PM UTC

nikisweeting wrote: post #6

Backblaze B2, ingress and egress are free through Cloudflare, and it's S3 compatible. It's peanuts by comparison but I've been storing ~22TB on there for years and love it. Wasabi and Glacier would be my 2nd choices.

>Backblaze B2, ingress and egress are free through cloudflare

AFAIK cloudflare ToS prohibits you from using it as a file hosting proxy. You might not run into issues if you're transferring a few gigabytes a month, but if you're transferring multiple terabytes it's just asking for trouble.

edit:

https://www.cloudflare.com/terms/ section 2.8 Limitation on Serving Non-HTML Content

Re: Ask HN: How would you store 10PB of data for your startup today?

#9

Post by garciasn » Fri, Apr 23, 2021, 3:28 PM UTC

In my opinion, knowing what you're planning to do w/the data once it's stored is the important piece to giving you some idea of where to put it.

Re: Ask HN: How would you store 10PB of data for your startup today?

#10

Post by ecesena » Fri, Apr 23, 2021, 3:28 PM UTC

S3 + Glacier. For data you're accessing via Spark/Presto/Hive I believe Parquet is a good format. At your scale AWS should prob provide discounts, worth connecting w/ an account rep.

I'd recommend reaching out to some data eng in the various Bigs, they certainly have more clear numbers. Happy to make an intro if you need, feel free to dm me.