101–110 of 374 posts
Escherichia coli, for instance, has a storage density of about 10 to the 19 bits per cubic centimeter. At that density, all the world’s current storage needs for a year could be well met by a cube of DNA measuring about one meter on a side.
There are several companies doing it: https://www.scientificamerican.com/article/dna-data-storage-...
Earlier quoted context omitted.
Great question. I updated the original post. It’s user generated images and videos. We download those to the phones in the background. We don’t touch the data at all.
> Update: Should have mentioned earlier, data needs to be accessible at all time. It’s user generated data that is downloaded in the background to a mobile phone, so super low latency is not important, but less than 1000ms required. > The data is all images and videos, and no queries need to be performed on the data. OK, so this definitely helps a bit. At 10PB my assumption is that storage costs are the major thing t…
What are your access patterns? You say "no queries need to be performed," but are you accessing via key-value look-ups? Or ranged look-ups?
What do customers do with the pictures? Do customers browse through images and videos?
You mention it's "user generated data" - how many users (order of magnitude)? How often is new data generated? Does the dataset grow, or can you evict older images/videos (so you have a moving window of data through time)?
Besides your immediate needs, what other needs do you anticipate? (Will you need to do ML/Analytics work on the data in the future? Will you want to generate thumbnails from the existing data set?)
What my experience is based on: I was formerly Senior Software Engineer/Principal Engineer for a team that managed reporting tools for internal reporting of Amazon's Retail data. The team I was on provides tools for accessing several years worth of Amazon.com's order/shipment data.
Earlier quoted context omitted.
Don’t forget labor. You need to find talent to manage your little data center. And deal with it when it shits the bed at 4:12am on Christmas morning. So toss in at least one SRE type person. Say $200k/year. Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one.... Since redundancy is a thing, double that $350k. And 10pb is what they…
> Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one.... This person's entire job is managing a few racks of hard drives? How often do you think they're actually going to get called in? > Since redundancy is a thing, double that $350k. True, but you can do redundancy for cheaper with parity or tape. > And 10pb is what they have no…
Not often. But the server gods are a cruel mistress and it will definitely shit the bed when you are on your honeymoon, or maybe the day after your first kid is born.
surprised I didn't see Gluster already in this thread. maybe its not for such big scale?
edit: Wikipedia says " GlusterFS to scale up to several petabytes on commodity hardware"
10PB costs more than $210,000 per month at S3, or more than $12M after five years.
RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. (https://www.rackmountpro.com/product.php?pid=3154)
That means that you could fit all 15TB (for erasure encoding with Minio) in less than two racks for around $150k up-front.
Figure another $5k/mo for monthly opex as well (power, bandwidth, etc.)
Instead of $12M spent after five years, you'd be at less than $500k, including traffic (also far cheaper than AWS.) Even if you got AWS to cut their price in half (good luck with that), you'd still be saving more than $5 million.
Getting the data out of AWS won't be cheap, but check out the snowball options for that: https://aws.amazon.com/snowball/pricing/
I only read about it, but never used it.
It advertises itself as exabyte scalable and provides s3 and nfs access.
I would host in a datacenter of your choice and do a cross connect into AWS: https://aws.amazon.com/directconnect/pricing/ This allows you to read the data into AWS instances at no cost and process it as needed since there is 0 cost for ingress into AWS. I have some experience with this (hosting using Equinix)
Thanks for the pointer. Never thought about this as an option. Great stuff!!!
Idk what your team’s expertise is, but I’d advise avoiding the cloud as long as possible. If you can build out an on-premise infrastructure, it will be a huge competitive advantage for your company because it will allow you to offer features that your competitors can’t.
Examples of this:
- Cloudflare built up their own network and infrastructure and it’s always been their biggest asset. They set the standard for free tier of CDN pricing, and nobody who builds a CDN on top of an existing cloud provider will ever beat it.
- Zoom. By hosting their own servers and network, Zoom is similarly able to offer a free tier where they are not subject to variable costs from free customers losing them money on bandwidth charges.
- WhatsApp. They scaled to hundreds of millions of users with less than a dozen engineers, a few dozen (?) servers, and some Erlang code.
IMO defaulting to the cloud is one of the worst mistakes a young company can make. If your app is not business critical, you can probably afford up to a day of downtime or even some data loss. And that is unlikely to happen anyway, as long as you’ve got a capable team looking after it who chooses standard and robust software.
2. As a general-purpose alternative, I would use Backblaze. It's cheap and they know what they're doing. Here is a comparison of (non-personal) cloud vendor storage prices: https://gist.github.com/peterwwillis/83a4636476f01852dc2b670...
3. You need to know how the architecture impacts the storage costs. There are costs for incoming traffic, outgoing traffic, intra-zone traffic, storage costs, archive costs, 'access' costs (cost per GET | POST | etc). You may end up paying $500K a month just to serve files smaller than 1KB.
4. You need to match up availability and performance requirements against providers' guarantees, and then measure a real-world performance test over a month. Some providers enforce rate limits, with others you might be in a shared pool of rate limits.
5. You need to verify the logistics for backup and restore. For 10PB you're gonna need an option to mail physical drives/tapes. Ensure that process works if you want to keep the data around.
6. Don't become your own storage provider. Unless you have a ton of time and money and engineering talent to waste and don't want to ship a reliable product soon.
Earlier quoted context omitted.
10x Supermicro SSG-6049P-E1CR60H servers (60 x 3,5" HDD in 4U enclosure) - $5k each 600x WESTERN DIGITAL Ultrastar DC HC550 18TB (10800PB in total) - $500 each ~$350k in hardware, up to 20kW energy consumption, should fit in two rack towers. You can host it for about $1.5k somewhere. All assuming no redundancy :)
Don’t forget labor. You need to find talent to manage your little data center. And deal with it when it shits the bed at 4:12am on Christmas morning. So toss in at least one SRE type person. Say $200k/year. Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one.... Since redundancy is a thing, double that $350k. And 10pb is what they…
- network switch Juniper EX4600 (10Gbps ports) + 3rd party optics ~$11k
- cheap 1Gbps switch for management access - some router for VPN for management network - $500
- 1Gbps (not guaranteed) internet access with few IPs ~$350 / month
- 100Mbps low traffic internet access for the management/OOB network.
Time to get the hardware - 2 months. Time to rent and install hardware in rack - about 1 month. I don't count configuring the software.
This setup is full of single points of failure so I would consider it one "region" and use something like CEPH + some spare servers in each "region". That way you don't need to react immediately to hardware failures. Just send a box of hardware from time to time to the DC and use ~$20-40h/h remote hands service to replace the failed drives or whole servers. You could also buy on-site service from the hardware vendor for 1-3 years adding some cost.
I think the most important thing would be have a cleaver person who design a fault tolerant system, automatic failover, good monitoring and alerting so that any on-call and maintenance job is easy and based on procedures. That way you could outsource it. Only then it might have some sense.