Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQL Databases Don't Scale

adam.blog.heroku.com

Post reply on HN
Print view View original

91–100 of 112 posts

Re: SQL Databases Don't Scale

#91

Post by itgoon »

moe wrote: post #81
nettdata wrote: post #44

Earlier quoted context omitted.

I'd beg to differ. Go take a look at Adam Leventhal's work with the Fishworks stuff. Specifically, go check out the Sun Storage 7310. It will scale HUGE, and is nowhere near the stupid cost of NetApp or EMC or the other major vendors. I'd love to see how a bunch of commodity PC's will scale to 100TB, and still be manageable, and have anywhere near the same feature sets. Again, if you're railing against something as u…

I'd love to see how a bunch of commodity PC's will scale to 100TB, and still be manageable 100T is 666 spindles when you go RAID10 with 300G drives (common size in the SAS/FC area). If you go S-ATA then it'll fit on 200 spindles using 1T drives. I'll stick to the latter variant for now because I'm too lazy to lookup Sun's pricing for SAS spindles, for my comparison below. So, 200 spindles amounts to roughly 15 hosts…

FWIW, I've built one of those SuperMicro machines, with eight drives, around the beginning of the year.

Allowing for redundancy (RAID 5 on six disks for data, and mirrored the boot drives), we ended up with 6.5TB of storage at about $3500 (that includes i7 proc and 12Gb RAM, and a fancy controller). My labor added some more on top of the $3500.

Mostly, we've been happy. It is more sensitive to heat issues than our HP G5s, and of course, generates more of that heat. If I were to go with any kind of density, I'd need a much more robust cooling solution.

Re: SQL Databases Don't Scale

#92

Post by stcredzero »

derefr wrote: post #78
pj wrote: post #71

Earlier quoted context omitted.

I've seen the term artificial scarcity used a few times in this context. Could you elaborate on what you mean by that? When I think artifical scarcity, I think something like DeBeers who has warehouses full of diamonds they are hoarding. Whereas with software, there actually is a scarcity of great programmers. There are only so many of them. With software, you know... it takes time and energy to support it. It costs…

Artificial scarcity is not an ethical or legal principle, but an economic one. Because 1s and 0s can be "copied almost infinitely", in order to make money on it one has to enforce an artifical constraint on the number of copies that are allowed to be made. The distinction is simply that with its opposite, natural scarcity, no effort needs to be made to ensure that each object has value on its own. To put it another w…

Artificial scarcity is not an ethical or legal principle, but an economic one. Because 1s and 0s can be "copied almost infinitely", in order to make money on it one has to enforce an artifical constraint on the number of copies that are allowed to be made.

Good customer service and a deep understanding of specific technical issues are not trivially reproducible.

Re: SQL Databases Don't Scale

#93

Post by stcredzero »

pj wrote: post #79
derefr wrote: post #78

Earlier quoted context omitted.

Artificial scarcity is not an ethical or legal principle, but an economic one. Because 1s and 0s can be "copied almost infinitely", in order to make money on it one has to enforce an artifical constraint on the number of copies that are allowed to be made. The distinction is simply that with its opposite, natural scarcity, no effort needs to be made to ensure that each object has value on its own. To put it another w…

Okay, I think I understand better now. There's only physically so much gold on the planet. Barring alchemy, we can't make more of it, so it has a natural scarcity. From that, I gather that we can't use the traditional economics of Supply and Demand to allow a market to decide the value of software. Because the denominator in the equation demand divided by supply is infinite, essentially, it doesn't matter what value…

Perhaps we can value software based on how much additional revenue it helps you generate or how much savings it generates through optimizations or automations.

My former company tried to do this. People balk at this. "Why should I pay when there's 'free?'"

Re: SQL Databases Don't Scale

#94

Post by nettdata »

rythie wrote: post #83
nettdata wrote: post #23

I've been an Oracle database architect for almost 20 years. His whole concept of "SQL doesn't scale" is the typical crap I always hear from people that are either not database experts, are using the wrong database technologies, or don't know what they're doing. More than likely a combination of all three. And just because you can create an object model, and a simplistic data model, does not make you an architect of l…

Whilst I can believe that Oracle probably does scale, when it costs $17.5k per CPU in licensing then it's not for everyone. It would seem that problems of a bank with a large already established userbase and lucrative, stable business model is very different to a start-up with no tested business model, may never become popular enough to need to scale and may not survive.

Uhmmm... duh?

The original article mentions nothing about cost or the ability (or lack thereof) of any company to afford it.

He made a flat out statement that said SQL DOESN'T SCALE, which is wrong.

There are other methods for dealing with initial growth and the cost constraints... no need to blow your wad on Oracle out of the gate.

But then I imagine a lot of non-Oracle types aren't even aware that Oracle can be very flexible with their licensing for startups. You can, for example, lease/rent your Oracle licenses on a monthly basis to help get the most out of your cash flow.

I've been involved with a number of startups where we did our initial rollout on Postgres, but ensured that the application architecture allowed for us to fairly easily swap in a larger, more scalable solution if/when needed.

For that matter, in my opinion, too many people and startups tend to over-engineer their initial product offerings, making them too complex and worrying more about nonexistent problems (like Google-sized scaling) rather than on solid features and business process. But that's fodder for another thread, I'm sure.

Re: SQL Databases Don't Scale

#95

Post by gaius »

dflock wrote: post #89
gaius wrote: post #77

Earlier quoted context omitted.

That's a very curious statement, especially for HN. Why would you consider needing good, experienced people to be a disadvantage ?

Because that kind of hardware+DBA team is prohibitively expensive for a startup?

Hardware yes, but the basic premise of a startup is that your people are top-notch. Scalability doesn't imply building something up-front that can handle enormous loads - it means building something that can grow with you. As opposed to "oh crap, we've got some load now, better start again" a la Twitter.

Re: SQL Databases Don't Scale

#96

Post by moe »

itgoon wrote: post #91
moe wrote: post #81

Earlier quoted context omitted.

I'd love to see how a bunch of commodity PC's will scale to 100TB, and still be manageable 100T is 666 spindles when you go RAID10 with 300G drives (common size in the SAS/FC area). If you go S-ATA then it'll fit on 200 spindles using 1T drives. I'll stick to the latter variant for now because I'm too lazy to lookup Sun's pricing for SAS spindles, for my comparison below. So, 200 spindles amounts to roughly 15 hosts…

FWIW, I've built one of those SuperMicro machines, with eight drives, around the beginning of the year. Allowing for redundancy (RAID 5 on six disks for data, and mirrored the boot drives), we ended up with 6.5TB of storage at about $3500 (that includes i7 proc and 12Gb RAM, and a fancy controller). My labor added some more on top of the $3500. Mostly, we've been happy. It is more sensitive to heat issues than our HP…

Yep, that spec would be over the top for a storage node, though. A large chunk likely went into the controller and the i7 whereas in a storage-node most cheapo controllers (i.e. 2x8 is cheaper than 1x16) and whatever old xeon/core2 will do.

Wrt cooling, for actual servers (i.e. not storage nodes) we've had good success with Sun XFires, the 4100+ range. They are very well built and the markup over the SuperMicro junk is minimal (around 15% last time I checked). Among the niceties is a nice array of hot-swap fans - something that SuperMicro still doesn't seem to deliver in their popular chassis.

I'd also consider the xfires for storage nodes if they'd take 3,5" drives, but afaik all their low-end models only have max eight 2,5" slots. That's just no worthwhile density when compared to the larger SuperMicro tins (which go up to 30x3,5" now I think).

Re: SQL Databases Don't Scale

#97

Post by joshu »

nettdata wrote: post #23

I've been an Oracle database architect for almost 20 years. His whole concept of "SQL doesn't scale" is the typical crap I always hear from people that are either not database experts, are using the wrong database technologies, or don't know what they're doing. More than likely a combination of all three. And just because you can create an object model, and a simplistic data model, does not make you an architect of l…

Have you ever noticed that every time there's an article like this there's one 20-year DBA veteran who says the original author doesn't understand SQL?

Yeah.

Anyway, having built both transactional systems (trading algorithms) and big social systems (delicious) I think the main issue is: The right data structure and tool for the right job.

Banking is a radically different problem then internet-scale social software (which I assume is what they are talking about when they say "doesn't scale".) Access patterns are different, read and write loads are different, etc.

The main issue here is that a lot of social software wants something like a fast per-user data store, something like a distributed inverted index for globally finding things.

There's NO great reason for one user's data to be in the same table as someone else's. I guess it was handy for stuff like calculating the average number of tags, etc.

Instead, you want to be able to have better control over data locality and similar, given your access patterns.

Now, personally, I would use an actual SQL engine for a single-machine persistent store, and build a distribution layer on top of that. Concurrency is hard, etc.

But assuming RAC is the right solution to all problems is probably not a good one. I've seen this go terribly awry.

SQL wins at things that btrees and hash indexes are good at; but a lot of things are better with other organizations of data.

Re: SQL Databases Don't Scale

#98

Post by nettdata »

trapper wrote: post #63
nettdata wrote: post #23

I've been an Oracle database architect for almost 20 years. His whole concept of "SQL doesn't scale" is the typical crap I always hear from people that are either not database experts, are using the wrong database technologies, or don't know what they're doing. More than likely a combination of all three. And just because you can create an object model, and a simplistic data model, does not make you an architect of l…

I've just read all your comments. Do you have a blog?

Nope. Had one for a while (http://orageek.com) but found I didn't really have much to say all that often to warrant maintaining it.

Re: SQL Databases Don't Scale

#99

Post by jmtulloss »

gaius wrote: post #32
jmtulloss wrote: post #22

Earlier quoted context omitted.

[citation needed]

OK, try this: http://press.web.cern.ch/press/PressReleases/Releases2003/PR...

A press release isn't data. I'd be interested in seeing the comparative results of the trials CERN put Oracle through.

It's not that I don't believe that Oracle can scale better than MySQL, it's just that I haven't seen any convincing data that it can scale so much better that it's worth the cost. I'm no expert, so maybe the data exists. I just haven't seen it.

Re: SQL Databases Don't Scale

#100

Post by TheXenocide »

nettdata wrote: post #23

I've been an Oracle database architect for almost 20 years. His whole concept of "SQL doesn't scale" is the typical crap I always hear from people that are either not database experts, are using the wrong database technologies, or don't know what they're doing. More than likely a combination of all three. And just because you can create an object model, and a simplistic data model, does not make you an architect of l…

For future reference, cold hard facts are much more useful than posting your resume. I'm not trying to say you're unqualified or unintelligent in any way, just that rationalizing an opinion with a job history is far less compelling evidence than factual examples. It's also interesting that you would immediately associate a well-made and supported argument (for those of us who have any experience scaling with technologies other than Oracle) with someone who isn't a database expert and doesn't know what he's doing.

Now let's get to the facts: while perhaps Oracle might have nice features, that doesn't mean that SQL is the best we can come up with, which is one of the primary points of the article (and one that seems completely unaddressed here). In a sense he brings to attention a common occurrence throughout human history wherein we reject change for comfort and these comments are doing nothing more than supporting that. Oracle is, for the most part, entirely too expensive for most businesses and even the businesses that are large enough to adopt it aren't making the profit they could with more affordable technology. You should also make note that, while banks do have strict requirements on data handling, they are really responsible for serving very small user bases when compared to things like Google, Amazon, Ebay, Facebook and the like. Sure the requirements are different for each of these, but ultimately your argument hold no water against these infrastructures, which are inherently not SQL and, at the same time, seem to be much more accepted as innovators of scalable technologies for the future...

One big problem is that real innovation comes at the expense of backwards compatibility, which would involve making a lot of changes. I can relate, since massive changes in most case imply bugs, which is a very unsettling position for some of these, but it doesn't mean there isn't a problem. Sure banks and other largely established companies would rather shell out the cash to support their legacy ideals than innovate new solutions, but he's right in that we've been spending many many years of man-hours trying to tackle the problem of porting a dated philosophy to an age that requires more scalability at lower cost. Lets also say that banks haven't been proving their practices to be economically sounds lately, so what is their input worth in this matter? Other than large boatloads of cash for Oracle that is.

Post reply on HN