71–80 of 374 posts
Compression is always a good alternative, which is especially effective when modification is infrequent.
It seems to me like you could save a ton of money by using your own hardware. Perhaps buy a bunch of big Synology boxes? At that scale you should also consider looking at technologies such as Ceph. We've recently switched to a setup with several Synology boxes for around 1PB net storage.
Plus, there were no backups so if one Synology were to blow up, all the data on it was lost.
Since they were a small startup it made some sense to start this way, but they had no plans on what to do about it as they got bigger.
It's going to depend entirely on a number of factors. How are you storing this data? Is it tons of small objects, or a smaller number of massive objects? If you can aggregate the small objects into larger ones, can you compress them? Is this 10PB compressed or not? If this is video or photo data, compression won't buy you nearly as much. If you have to access small bits of data, and this data isn't something like Par…
Great question. I updated the original post. It’s user generated images and videos. We download those to the phones in the background. We don’t touch the data at all.
> The data is all images and videos, and no queries need to be performed on the data.
OK, so this definitely helps a bit.
At 10PB my assumption is that storage costs are the major thing to optimize for. Compression is an obvious must, but as it's image and video you're going to have some trouble there.
Aggregation where you can is probably a good idea - like if a user has a photo album, it might make sense to store all of those photos together, compressed, and then store an index of photo ID to album. Deduplication is another thing to consider architecting for - if the user has the same photo, across N albums, you should ensure it's only stored the one time. Depending on what you expect to be more or less common this will change your approach a lot.
Of course, you want to avoid mutating objects in S3 too - so an external index to track all of this will be important. You don't want to have to pull from S3 just to determine that your data was never there. You can also store object metadata and query that first.
AFAIK S3 is the cheapest way to store a huge amount of data other than running your own custom hardware. I don't think you're at that scale yet.
Latency is probably an easy one. Just don't use Glacier, basically, or use it sparingly for data that is extremely rare to access ie: if you back up disabled user accounts in case they come back or something like that.
I think this'll be less of a "do we use S3 or XYZ" and more of a "how do we organize our data so that we can compress as much of it together, deduplicate as much of it as possible, and access the least bytes necessary".
If you're not afraid of having a few operations people on staff and running a few racks in multiple data centers, then buy a bunch of drives and servers and install something to expose everything via S3 interface (Ceph, Minio, ...) so none of your tools have to change.
I would host in a datacenter of your choice and do a cross connect into AWS: https://aws.amazon.com/directconnect/pricing/ This allows you to read the data into AWS instances at no cost and process it as needed since there is 0 cost for ingress into AWS. I have some experience with this (hosting using Equinix)
Thanks for the pointer. Never thought about this as an option. Great stuff!!!
We had about 25 Dell R730xd servers. When the cluster would start to fill up, we would just replace drives with larger drives. Upgrading drives with SwiftStack is a piece of cake. When I left we were upgrading to 10TB drives as that was the best pricing. We didn't buy the drives from Dell as they were crazy expensive. We just bought drives from Amazon/New Egg, and kept some spares onsite. We got a better warranty that way too. Dell only had a 1 year warranty, but the drives we were buying had a 5 year warranty.
> The data is all images and videos, and no queries need to be performed on the data.
Okay, this is a good start, but there are some other important factors.
For every PB of data, how much bandwidth is used in a month, and what percentage of the data is actually accessed?
Annoyingly, the services that have the best warm/"cold" storage offerings also tend to be the services that overcharge the most for bandwidth.
Earlier quoted context omitted.
No. They are already managing 10PB, planning for which would be very stupid when just starting up.
Why would planning to be able to execute the single focus of your startup be stupid?
It's a bit different nowadays that a lot of scaling tech is commoditized, but still means things like negotiating new contracts, finding & fixing the odd pieces that weren't stressed before, etc.
(congrats on hitting the new usage levels + good luck! we're at a much smaller scale, but trying to figure out some similar questions for stuff like web-scale publishing of data journalism without dying on egress $, so it's an interesting thread...)