Why we built a 40TB photo server in-house instead of using S3
1–10 of 78 posts
Re: Why we built a 40TB photo server in-house instead of using S3
#2From my experience building storage, you're better off buying an enclosure that has expanders (e.g. Supermicro); it really simplifies cabling.
Re: Why we built a 40TB photo server in-house instead of using S3
#3For starters, 40TB on S3 costs around $60,000 annually. The components to build the Box — about 1/10th of that
I wonder why no one ever factors the cost of having a knowledgeable person handling the system into their calculations. TBH 40TB doesn't sound as much, but once you start growing you'll want someone familiar enough with the subject to take care of it (especially if it's their most valuable asset).
Re: Why we built a 40TB photo server in-house instead of using S3
#4For starters, 40TB on S3 costs around $60,000 annually. The components to build the Box — about 1/10th of that I wonder why no one ever factors the cost of having a knowledgeable person handling the system into their calculations. TBH 40TB doesn't sound as much, but once you start growing you'll want someone familiar enough with the subject to take care of it (especially if it's their most valuable asset).
If your business depends on handling large amounts of data, you want that person on-staff anyway.
Re: Why we built a 40TB photo server in-house instead of using S3
#5How much more expensive is a clustered software solution (e.g., Hadoop FS) than this RAID box?
Re: Why we built a 40TB photo server in-house instead of using S3
#6S3 gives you multi-host, multi-region redundancy. Putting it all in one box is asking for trouble. What if the raid controller grows a bug and corrupts on write? It's happened. What if there's a fire in the building that has both your server and your back-up? Eggs, meet basket!
Re: Why we built a 40TB photo server in-house instead of using S3
#7Static image storage makes the most sense todo on S3 or similar. Building your own storage, does not provide the redundancy and reliability of S3. Additionally, you have the flexibility to enable CloudFront and distribute the images via CDN if you need.
Re: Why we built a 40TB photo server in-house instead of using S3
#8After reading that I'm afraid they're going to have downtime because of a 'shoddy backplane connection' sometime soon. Or that it's going to fall out of its 'delicate balance' or have a 'driver conflict' soon :(
I wonder if they subconsciously undervalue their photos, or if this is just a naive 'we can do better' moment.
I'm all for building your own, but there's a good reason enterprise server hardware costs more, and there's a good reason to go for enterprise hardware when you say 'The one most valuable asset at Oyster.com is our photo collection'.
For instance a Dell or HP storage server with 24 disks would be around 18K list - and be really engineered for that as opposed to hacked together.
Re: Why we built a 40TB photo server in-house instead of using S3
#9This seems like a half-solution - I can understand building a big box locally for day-to-day access to the images (if they indeed need that) but I didn't see any mention of an off-site backup. Even assuming no problems like file corruption, what happens if there's a fire or flood? At the least, there should be two copies of this big box, in different places.
Re: Why we built a 40TB photo server in-house instead of using S3
#10For starters, 40TB on S3 costs around $60,000 annually. The components to build the Box — about 1/10th of that I wonder why no one ever factors the cost of having a knowledgeable person handling the system into their calculations. TBH 40TB doesn't sound as much, but once you start growing you'll want someone familiar enough with the subject to take care of it (especially if it's their most valuable asset).
Amazon S3 price also takes into account having redundancy at three data centers. So you should multiply the cost by 3, just for that. Of course if you don't need the redundancy, you can build it for cheaper.
Also for some applications it really does make sense to move away from S3, and have a solution in house. For example The Broad Institute has about 6 petabytes of storage [1]. They in particular benefit from local storage, since all of their data is generated on-site. However, even at this scale, they don't build the boxes themselves [2].
[1] http://www.genome.gov/27538886
[2] http://www.isilon.com/press-release/isilon-iq-powers-data-st...