Earlier quoted context omitted.
SANS are too expensive. They don't scale cost-effectively compared to commodity PC hardware.
I'd beg to differ. Go take a look at Adam Leventhal's work with the Fishworks stuff. Specifically, go check out the Sun Storage 7310. It will scale HUGE, and is nowhere near the stupid cost of NetApp or EMC or the other major vendors. I'd love to see how a bunch of commodity PC's will scale to 100TB, and still be manageable, and have anywhere near the same feature sets. Again, if you're railing against something as u…
100T is 666 spindles when you go RAID10 with 300G drives (common size in the SAS/FC area).
If you go S-ATA then it'll fit on 200 spindles using 1T drives. I'll stick to the latter variant for now because I'm too lazy to lookup Sun's pricing for SAS spindles, for my comparison below.
So, 200 spindles amounts to roughly 15 hosts (throwing in a few spares for good measure). The whole setup will comfortably fit into one rack, including the FibreChannel machinery and other fluff that you'll likely want.
Thus from the hardware side this is trivially managable, 100T is just not a lot of data nowadays.
On the software side it's up to your creativity and mostly depends on what you actually need to store. I've seen people setup commodity postgres clusters, as well as more fancy things like HDFS, GFS or homegrown storage layers that way. And it worked.
And, regardless of the indeed relatively sane pricing of the Sun (formerly StorageTek) products, the bottomline is what makes the difference.
In figures, for a 100T SAN on the 7310 you're looking at something like $50k for one head, plus around $75k for three trays. We're in the $125k ballpark, hardware-only. And I'm being rather optimistic here: This setup actually holds only 96T and the head is maxed out (3 trays max per controller/head). That means your next upgrade will incur another $25k markup for the next head, good thing you didn't ask for 150T...
Squeezing the same amount of storage into 15 min-spec supermicro pizzaboxes I arrive at roughly $3000 per node, including spindles. A good buyer will get them cheaper. This commodity cluster sets us back only ~$45k in hardware.
That's 1/3 the price of the 7310 solution, being optimistic on the Sun and pessimistic on the commodity side.
That kind of difference makes up for a lot of development effort for the custom solution - most of which is a one time investment anyways and yields better flexibility in the long run. It's also the reason why most of the big boys don't use off-the-shelf SANs for their primary storage.