Yes S3 can get expensive, but imho this swings the pendulum too far in the other direction. Something like a riak cluster of four 2U/24-drive servers would get you the cost structure of good colo but the features/resilience/operational-flexibility of something more like s3.
Why we built a 40TB photo server in-house instead of using S3
11–20 of 78 posts
Re: Why we built a 40TB photo server in-house instead of using S3
#12"The first challenge in putting together the big box was getting internal SAS connectors properly seated into the backplane adaptor sockets"
Excuse me?
As a customer I'd feel slightly uncomfortable about my data by now. You make it sound like stuffing 24 disks into a box is rocket science to you, all the while SuperMicro and others sell plug&play chassis for up to 45 disks[1].
Also, you didn't mention it in the post, but you do have at least two of these in two physically distant racks, right?
[edit: deleted snarky comment about people running windows on a fileserver]
Re: Why we built a 40TB photo server in-house instead of using S3
#13Off site backup conveniently left out of the write up?
Re: Why we built a 40TB photo server in-house instead of using S3
#14For starters, 40TB on S3 costs around $60,000 annually. The components to build the Box — about 1/10th of that I wonder why no one ever factors the cost of having a knowledgeable person handling the system into their calculations. TBH 40TB doesn't sound as much, but once you start growing you'll want someone familiar enough with the subject to take care of it (especially if it's their most valuable asset).
Amazon S3 price also takes into account having redundancy at three data centers. So you should multiply the cost by 3, just for that. Of course if you don't need the redundancy, you can build it for cheaper. Also for some applications it really does make sense to move away from S3, and have a solution in house. For example The Broad Institute has about 6 petabytes of storage [1]. They in particular benefit from local…
He says it's 1/10 of the price, so assume 10 Colo.
enough with people defending s3. Go buy a IBM mainframe already :)
Re: Why we built a 40TB photo server in-house instead of using S3
#15This is honestly pretty scary. There are a lot of single points of failure in this solution.
1) Single Box
2) Single Location
3) Single 40TB RAID 6 Array on single RAID card with 22 Drives (assuming 24 2TB drives, 2 parity, 2 hot spare = 40TB)
4) Single bonded network link means single switch, no redundancy against switch / network device failure
Honestly this may have been cheap but your getting what your paying for: An unreliable backup solution.
Also: Norco? Really? I wouldn't be trusting anything they produce with your company's critical data. Supermicro isn't much more expensive all things considered.
Re: Why we built a 40TB photo server in-house instead of using S3
#16"The first challenge in putting together the big box was getting internal SAS connectors properly seated into the backplane adaptor sockets" Excuse me? As a customer I'd feel slightly uncomfortable about my data by now. You make it sound like stuffing 24 disks into a box is rocket science to you, all the while SuperMicro and others sell plug&play chassis for up to 45 disks[1]. Also, you didn't mention it in the post,…
At least for this website it isn't your data but it is their data.
They are backing up their curated photos (which seems to pretty much be the whole business as they say in the article).
It appears they are using Akamai services to server their images so hopefully their is some extra redundancy there already.
Other than that it seems like it would be a steal to use s3 as a backup system at this point because from this article it looks like they need to hire another employee to tell them why this backup solution seems a bit silly (I mean having trouble setting up the hardware).
Re: Why we built a 40TB photo server in-house instead of using S3
#17We had something similar but smaller (~8 TB) at a place I worked, and it was a nightmare. Migrating from that to S3 was one of the best things to happen to that project.
Being a single big box, it had a bunch of single points of failure, and boy did they fail; we probably had 5-10 hours per month of downtime due to the photo server falling over (flaky RAID controller firmware, mostly).
Also, since the big box was expensive, we only had one in production. There was code for taking a newly-uploaded photo and copying it over to the photo server that only executed in production, which meant the only way to functionally test it was to ship it and hope.
We switched to S3 about a year ago with different buckets for prod, staging, and dev; the production-only code paths went away, and there hasn't been any photo-related downtime since. Definitely worth it.
Re: Why we built a 40TB photo server in-house instead of using S3
#18Hey everyone, don't worry, be happy! Their VP of engineering used to work at 'a startup' and prior to that, he was a 'rocket scientist' because he worked at Raytheon on missile guidance systems. Oh, and before that, he worked at the Mothership, I mean Microsoft, in the 'user experience team'. As an added bonus, if you want to work at Oyster, you get your choice of such cutting edge technologies as 'Python, PostgreSQL, Nginx, Windows, CentOS, C++ and more'!
Nothing to worry about here, I think we are in good hands.
Re: Why we built a 40TB photo server in-house instead of using S3
#19For starters, 40TB on S3 costs around $60,000 annually. The components to build the Box — about 1/10th of that I wonder why no one ever factors the cost of having a knowledgeable person handling the system into their calculations. TBH 40TB doesn't sound as much, but once you start growing you'll want someone familiar enough with the subject to take care of it (especially if it's their most valuable asset).
Amazon S3 price also takes into account having redundancy at three data centers. So you should multiply the cost by 3, just for that. Of course if you don't need the redundancy, you can build it for cheaper. Also for some applications it really does make sense to move away from S3, and have a solution in house. For example The Broad Institute has about 6 petabytes of storage [1]. They in particular benefit from local…
Exactly. Plus power and rack space are not cheap. Also hiring the sys admins 24/7 won't be cheap.
Re: Why we built a 40TB photo server in-house instead of using S3
#20From my experience building storage, you're better off buying an enclosure that has expanders (e.g. Supermicro); it really simplifies cabling.
Supermicro is just such a fantastic company. Love their stuff.