Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQL Databases Don't Scale

adam.blog.heroku.com

Post reply on HN
Print view View original

101–110 of 112 posts

Re: SQL Databases Don't Scale

#101

Post by TheXenocide »

TheXenocide wrote: post #100
nettdata wrote: post #23

I've been an Oracle database architect for almost 20 years. His whole concept of "SQL doesn't scale" is the typical crap I always hear from people that are either not database experts, are using the wrong database technologies, or don't know what they're doing. More than likely a combination of all three. And just because you can create an object model, and a simplistic data model, does not make you an architect of l…

For future reference, cold hard facts are much more useful than posting your resume. I'm not trying to say you're unqualified or unintelligent in any way, just that rationalizing an opinion with a job history is far less compelling evidence than factual examples. It's also interesting that you would immediately associate a well-made and supported argument (for those of us who have any experience scaling with technolo…

Oh, it may also be worth noting that engineers and administrators would have to change their way of thinking in all this too, which doesn't affect some too much, but may have a bit of an impact on somebody who's been sitting on the same technology platform for 20 years.

Re: SQL Databases Don't Scale

#102

Post by TheXenocide »

4buot wrote: post #28
staunch wrote: post #27

Earlier quoted context omitted.

Yeah, and if Facebook used Oracle instead of MySQL and memcached, they'd be looking at a $50 million/year+ bill no doubt. Those silly kids with their minor technologies should learn to use a real DB. If those 100 million users that login every day only knew...

For what its worth, Facebook does have a big Oracle install. I find it strange that everyone trotted out their preformed opinion for this piece, but no one comments on the cassandra article which actually proposes a solution...although it requires you to learn something, how horrible.

http://incubator.apache.org/cassandra/

Re: SQL Databases Don't Scale

#103

Post by TheXenocide »

trezor wrote: post #55

Summary of an article about how "SQL Databases Don't Scale": 1. Mentions RAID, not SAN 2. Mentions MySQL and only MySQL (with the exception of PostgreSQL once). 3. Mentions Master-slave replication as a killer scalability feature. I suggest he rename the piece to "A $500 server and MySQL Don't Scale" and then we can all agree and get along.

Most of the things he mentions apply to most DB engines, MySQL, MS-SQL, Postgres, Firebird and many editions of Oracle (all of which I've used in data-intensive applications) suffer the same woes. Ultimately it would appear that you went into it reading the article with a narrow minded view of "Yes they do!" instead of "Why don't they?" Also, the disk analogy was for relating information not discussing storage technology or to imply that was what he was using when trying to scale. The portion of vertical scalability clearly presents the option of (and pitfalls therein) using hardware to account for data scalability...

Re: SQL Databases Don't Scale

#104

Post by TheXenocide »

nettdata wrote: post #44
rjurney wrote: post #38

Earlier quoted context omitted.

SANS are too expensive. They don't scale cost-effectively compared to commodity PC hardware.

I'd beg to differ. Go take a look at Adam Leventhal's work with the Fishworks stuff. Specifically, go check out the Sun Storage 7310. It will scale HUGE, and is nowhere near the stupid cost of NetApp or EMC or the other major vendors. I'd love to see how a bunch of commodity PC's will scale to 100TB, and still be manageable, and have anywhere near the same feature sets. Again, if you're railing against something as u…

Maybe you should ask Google... they seem to be doing pretty well with commodity hardware and something's telling me that they've got better reliability, higher efficiency and lower costs. Also, unused features merely add unnecessary complexity to a system. Thanks for the info on the cheaper SAN though, I'll have to look into it.

Re: SQL Databases Don't Scale

#105

Post by TheXenocide »

gaius wrote: post #17
pj wrote: post #11

Okay, see... here's more of the no-SQL agenda at hacker news. This isn't a question of "Does SQL Scale or not?" but "How much do SQL Databases Scale?" I feel like the propaganda here is that because RDBMS doesn't scale to youtube or google scale they suck and that's not true. Like SQL is a waste of time because at some point, you're going to need to shard your database. Look, at that kind of scale, you're going to ha…

As others have pointed out, the "no SQL" crowd are invariably MySQL users who have run into the limitations of MySQL but for ideological reasons can't state that the problems they encounter are specific to MySQL. DB2, Teradata and Oracle users regularly tackle problems 100x larger than MySQL can handle.

I beg to differ; the company I'm at now had SAN representatives saying that they've never seen a non-SAN DB push as many IOPS as ours was before we started exploring scalability solutions and I'm still rather compelled by the potential of "no SQL" solutions, thanks. Oh, and no, we're not using MySQL or any other open source DB for that matter, so maybe it's you that has the prejudice here...

Re: SQL Databases Don't Scale

#106

Post by rythie »

nailer wrote: post #86
rythie wrote: post #83

Earlier quoted context omitted.

Whilst I can believe that Oracle probably does scale, when it costs $17.5k per CPU in licensing then it's not for everyone. It would seem that problems of a bank with a large already established userbase and lucrative, stable business model is very different to a start-up with no tested business model, may never become popular enough to need to scale and may not survive.

CPU core. Not CPU (which is generally taken to mean CPU die).

Really? That's 4 times worse than I thought then

Re: SQL Databases Don't Scale

#107

Post by TheXenocide »

psadauskas wrote: post #8
wildwood wrote: post #3

"Sharding kills most of the value of a relational database." I can appreciate the point the author makes about partitioning schemes requiring heavy integration with the business logic, but I disagree with the claim that sharding doesn't work. It's probably more accurate to say that sharding only works well if you design your database very carefully, or just get lucky about how your data model maps to sharding schemes…

If you data model maps well to sharding, it maps well to distributed databases, and you're not gaining much by being on an RDBMS. I think the point they're trying to make is if you're going through the contortions to shard a relational database, you're better off just doing it in a distributed database in the first place.

There's more than one way to skin a mongoose and in this case the more common shard solutions are fundamentally different from distributed DB in that a distributed DB often times contains large amounts of a a subset of data that has to be merged with data on other DBs whereas a shard can have entire schema duplication on different shards where you can use relational technology just fine and even if the query is distributed you can just add rows to the end of a dataset rather than joining the data columns manually (implementing your own relational joins). In many cases you partition your sharded data in such a way that the majority of your queries are not distributed so that they can continue to leverage the relational model you had before, only performing a distributed query for more complex actions like reporting and such.

Also going back to the first place isn't really an option, shards usually come of a system that has grown, not ground up design. Ultimately I still support the distributed model myself, but a shard model does support relational data much more so than a distributed DB (at least out of the box).

Re: SQL Databases Don't Scale

#108

Post by nailer »

rythie wrote: post #106
nailer wrote: post #86

Earlier quoted context omitted.

CPU core. Not CPU (which is generally taken to mean CPU die).

Really? That's 4 times worse than I thought then

Yep. Oracle are famous for this - at my office we often talk about counting the molecules to work out Oracle licensing.

Re: SQL Databases Don't Scale

#109

Post by itgoon »

moe wrote: post #96
itgoon wrote: post #91

Earlier quoted context omitted.

FWIW, I've built one of those SuperMicro machines, with eight drives, around the beginning of the year. Allowing for redundancy (RAID 5 on six disks for data, and mirrored the boot drives), we ended up with 6.5TB of storage at about $3500 (that includes i7 proc and 12Gb RAM, and a fancy controller). My labor added some more on top of the $3500. Mostly, we've been happy. It is more sensitive to heat issues than our HP…

Yep, that spec would be over the top for a storage node, though. A large chunk likely went into the controller and the i7 whereas in a storage-node most cheapo controllers (i.e. 2x8 is cheaper than 1x16) and whatever old xeon/core2 will do. Wrt cooling, for actual servers (i.e. not storage nodes) we've had good success with Sun XFires, the 4100+ range. They are very well built and the markup over the SuperMicro junk…

Most of the money went to the hard drives themselves. They've come down a lot since (1.5TB had just been announced), but it was around 70% of the cost. Everything else was relatively cheap.

The 2.5 drives are good, and seem to becoming typical, because they run much cooler. We've got some G5s in the same closet, with just as many drives, and they don't run nearly as warm.

Re: SQL Databases Don't Scale

#110

Post by nettdata »

TheXenocide wrote: post #100
nettdata wrote: post #23

I've been an Oracle database architect for almost 20 years. His whole concept of "SQL doesn't scale" is the typical crap I always hear from people that are either not database experts, are using the wrong database technologies, or don't know what they're doing. More than likely a combination of all three. And just because you can create an object model, and a simplistic data model, does not make you an architect of l…

For future reference, cold hard facts are much more useful than posting your resume. I'm not trying to say you're unqualified or unintelligent in any way, just that rationalizing an opinion with a job history is far less compelling evidence than factual examples. It's also interesting that you would immediately associate a well-made and supported argument (for those of us who have any experience scaling with technolo…

Just to clarify, I never said Oracle was the be-all end-all solution for everything.

And I didn't rationalize anything with a "job history", but rather with large systems that actually have been built and are working.

I'd be very interested to see just how many people in this discussion actually have actual experience building large, scalable systems?

The original article made an asinine, generic statement without any context, or mention of cost, and I said it was silly, and pointed to the obvious (and easiest) reason why it was silly, and that was Oracle.

All of a sudden a bunch of people started making statements and assumptions about the scenario the article was probably talking about, that weren't actually made in the article, to discount Oracle as an option. Cost, commodity hardware, etc., etc. Then people started to point out other large websites that actually HAVE scaled, without the use of Oracle, as if that disproves Oracle's abilities... and yet it totally disproves the statement of the original article.

If you want to get into some context-specific details of why certain specific SQL technologies don't scale well (or at all), at a certain price-point, then that's a whole other discussion that I'd be happy to enter into, and would probably agree with.

It's also interesting to note that most of the "DB technologies" that are being used to scale those sites aren't DB's at all, but rather various levels of data caching that are employed to reduce the load on the databases, and are only applicable to the general read-only and non-transactional nature of social sites.

The whole reason I brought up banking sites in the first place is that they are one of the few, more obvious scenarios where most of your end-user interactions are actually hitting the database in real-time, and all data must be current and consistent. There is no real option for caching to save your ass, except at the DB layer itself, via such mechanisms as Oracle's Cache Fusion technology.

Social sites generally don't have any of those real-time, consistent constraints, and are therefore much easier to scale larger, because the nature of the site and the data allows for so much more technology to be used in front of the database.

The plain and simple fact of the matter is that building a large, scalable system is hard work. It requires that you analyze and design ALL aspects of the entire system to scale, not just the database. (Network, caching servers, application, database, hardware, etc).

Post reply on HN