Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#21

Post by lostcolony »

boringg wrote: post #14
garciasn wrote: post #9

In my opinion, knowing what you're planning to do w/the data once it's stored is the important piece to giving you some idea of where to put it.

Agreed - though I feel like every data use comes after the fact. Original software engineers/developpers rarely have the foresight that the data scientists need the information for (at least imho).

To be fair, the data scientists rarely have the foresight to know what the data scientists need the information for. The only time I've seen a data scientist correctly include all the data they needed (but still be wrong) was when they answered "All of it. We need all of the data".

Re: Ask HN: How would you store 10PB of data for your startup today?

#22

Post by rewq4321 »

gruez wrote: post #8
nikisweeting wrote: post #6

Backblaze B2, ingress and egress are free through Cloudflare, and it's S3 compatible. It's peanuts by comparison but I've been storing ~22TB on there for years and love it. Wasabi and Glacier would be my 2nd choices.

>Backblaze B2, ingress and egress are free through cloudflare AFAIK cloudflare ToS prohibits you from using it as a file hosting proxy. You might not run into issues if you're transferring a few gigabytes a month, but if you're transferring multiple terabytes it's just asking for trouble. edit: https://www.cloudflare.com/terms/ section 2.8 Limitation on Serving Non-HTML Content

You can definitely serve way more than a few GB per month through Cloudflare on the free plan. I serve tens of terabytes a month for free. If OP needs to serve hundreds of terabytes per month they may get an email asking to upgrade, but the backblaze/Cloudflare setup would probably still be the cheapest. BunnyCDN is great too.

Re: Ask HN: How would you store 10PB of data for your startup today?

#23

Post by lostcolony »

qeternity wrote: post #16

Are you fundamentally a data storage business or are you another business that happens to store a tremendous amount of data? If it's the former, then investing in-house might make sense (a la Dropbox's reverse course).

He's the CTO of KeepSafe.

Re: Ask HN: How would you store 10PB of data for your startup today?

#24

Post by reversengineer »

jagger27 wrote: post #19
byteshock wrote: post #12

Wasabi is a good option. They’re S3 compatible and don’t charge any egress or ingress fees. Been using them for a few years. Great speeds and customer support.

10PB with their pricing calculator comes out to over $60,000/mo. Feels like a lot. edit: perhaps their RCS option would be cheaper if you know exactly how much data you need to store in advance.

To be fair, purchasing and hosting even the most basic mirrored RAID array of that scale comes to well over half a million for the disks alone. Then you need to manage them.

Re: Ask HN: How would you store 10PB of data for your startup today?

#25

Post by ZeroCool2u »

Take any credits you can get from a provider switch and then thoroughly map out your access patterns, ingestion, and egress. Do whatever you can to segment data by your needs for availability and modification.

If it's all archival storage then it's pretty straight forward. If you're on GCP you take it all and dump it into archival single region DRA (Durable Reduced Availability) storage for the lowest costs.

Otherwise, identify your segments and figure out a strategy for "load balancing" between standard, nearline, coldline, and archive storage classes. If you can figure out a chronological pattern, you can write a small script that uses the gsutils built-in rsync feature to mirror over data from a higher grade storage class to a lower one at the right time.

The strategy will probably be similar in any of the other big 3 providers as well, but fair warning, some providers archival grade storage does not have immediate availability last I checked.

See: https://cloud.google.com/storage/docs/storage-classes

https://cloud.google.com/storage/docs/gsutil/commands/rsync

Re: Ask HN: How would you store 10PB of data for your startup today?

#27

Post by bushbaba »

ZeroCool2u wrote: post #25

Take any credits you can get from a provider switch and then thoroughly map out your access patterns, ingestion, and egress. Do whatever you can to segment data by your needs for availability and modification. If it's all archival storage then it's pretty straight forward. If you're on GCP you take it all and dump it into archival single region DRA (Durable Reduced Availability) storage for the lowest costs. Otherwis…

Flip side. How much time would that migration take. As a startup focusing that time on product would lead to more VC investment or more sales sooner. With the seed/series funding and sales being many multiples of the cost savings.
Post reply on HN