Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQL Databases Don't Scale

adam.heroku.com

Post reply on HN
Print view View original

11–20 of 34 posts

Re: SQL Databases Don't Scale

#11

Post by sophacles »

gregjor wrote: post #3

I can't count how many Rails fanboys have written this same article. What do they think big companies, banks, government agencies -- organizations with huge databases -- use? What kind of database do the RoR developers think the real world runs on? Claiming that SQL databases don't scale, especially using MySQL as the example, is simply ignorant. Oracle, DB2, SQL Server all scale to handle databases orders of magnitu…

What do they think big companies, banks, government agencies -- organizations with huge databases -- use? What kind of database do the RoR developers think the real world runs on?

CICS and TPF on z/OS?

Re: SQL Databases Don't Scale

#13

Post by jasonjei »

Wow, a lot of dislike of SQL, and a bit biased in their description of scaling SQL and sharding. Again, use the right tool for the right purpose, not what you think is intrinsically better.

Our app does double entry accounting, and considering the relational structure of accounting ledgers as well as that every entry needs to be posted in an ACID manner, we don't feel comfortable building out, say, a banking system in NoSQL.

Yes, SQL is painful to scale, but it can be manageable. We partition it in a scoped schema way (e.g each company account/tenant has its own set of tables on one of the shards with the server of the fewest consumed resources, or if it's a tie, a coin toss). Scaling it in an arbitrary manner such as by names beginning A to M, N to Q, etc simply seems like a bad way to do it. If your app is like a 37 Signals app where it doesn't typically require access to another account's data, this might be a good way to do it.

One of the programmers I hired was dead set on trying to use CouchDB for everything. I told him don't try to use it for everything--it's good to use for the audit trail and document revisions on invoices, orders, etc, but not on actual double entry journals.

Re: SQL Databases Don't Scale

#14

Post by sunjain »

Oracle RAC may come closest to addressing most of these issues. Of course it is pricey. However it fulfills pretty much all the constraints mentioned here - application transparency while scaling(sharding does not involve changing the app), horizontal scalability(add nodes as needed), failover for both read & write transactions. It sounds like sale-pitch for RAC but it seems to geared to deal with these kind of scenarios.

Re: SQL Databases Don't Scale

#16

Post by jsr »

It's useful to differentiate between "SQL Databases do not scale" and "SQL Databases do not _cost effectively_ scale". The second argument is more accurate.

Vertical scaling of a DB is definitely an option for many people and has been used to scale many applications. However, the cost curve associated with buying bigger and bigger hardware is super-linear; doubling CPU & Memory in a single system leads to more than doubling the hardware cost. This can be problematic for many businesses whose revenue growth is exceeded by cost growth of the database.

Sharding is also an option for scaling, leveraged to great success by Facebook, Yahoo, and many others. However as the article points out, sharding prevents the developer from using many of the features that make a relational database a productive development environment. There are lots of foot guns that emerge in a sharded SQL environment and if you have not set up your development constraints appropriately, you can slow the pace of development considerably. This again leads to a cost problem because the incremental costs of adding features grows as you add more things like sharding around your database.

SQL is not useless and not hopeless. In a large number of cases, SQL is the right solution. However the techniques used to scale SQL tend to be options only to very large budget organisations. NoSQL solutions tend to be more cost effective in their scaling approach (scale out vs. scale up) without crippling the developers productivity. For these reasons, NoSQL solutions tend to be the better choice for the cost-conscious.

Re: SQL Databases Don't Scale

#17

Post by steveklabnik »

gregjor wrote: post #3

I can't count how many Rails fanboys have written this same article. What do they think big companies, banks, government agencies -- organizations with huge databases -- use? What kind of database do the RoR developers think the real world runs on? Claiming that SQL databases don't scale, especially using MySQL as the example, is simply ignorant. Oracle, DB2, SQL Server all scale to handle databases orders of magnitu…

Hundreds of tiny systems that all integrate together in various complex ways? I mean, it's not like you can "mysqldump bankofamerica_production > backup.dump" or anything, right?

Re: SQL Databases Don't Scale

#19

Post by gregjor »

kolektiv wrote: post #9
gregjor wrote: post #3

I can't count how many Rails fanboys have written this same article. What do they think big companies, banks, government agencies -- organizations with huge databases -- use? What kind of database do the RoR developers think the real world runs on? Claiming that SQL databases don't scale, especially using MySQL as the example, is simply ignorant. Oracle, DB2, SQL Server all scale to handle databases orders of magnitu…

Well, they don't scale indefinitely, and you do run in to Brewers Theorem at some point. While you're right in saying that if you have enough cash and horsepower to throw at SQL databases you can get a very long way, there is some point where anything which must be consistent is going to have an impact on availability. This gets especially critical once you stop assuming that data can be centralised. Once your requir…

Nothing scales indefinitely, not even RoR. But that wasn't what the article complained about. If the author had written "SQL Databases Don't Scale Indefinitely" I wouldn't have any complaint about it.

My point is that there are plenty of real-world examples of SQL databases that are scaled way beyond any RoR web app's needs, so flatly claiming they don't scale is wrong.

The solution banks (and other large organizations) have arrived at -- using multiple databases with different transactional and availability requirements -- IS a solution to scaling large databases. There's no requirement that SQL databases be monolithic. Even the original article describes using a MySQL master for updates and one or more slaves for reads. That's a valid scalability solution if your application can live with the possibility of stale data in the slaves.

Re: SQL Databases Don't Scale

#20

Post by SigmundA »

yuvadam wrote: post #4

I think TFA gets it all wrong. It's not that SQL databases do not scale per se , they just can't arbitrarily scale. Most applications today manage pretty well with nothing more than sharding and replication. Sure, it's not the optimal solution, and it's not easy - but it gets the job done. Facebook, at the very core (behind all the layers of cache), run, arguably, the largest MySQL landscape in the world - and they s…

"It's not that SQL databases do not scale per se, they just can't arbitrarily scale."

Tell that to Clustrix, Vertica, and VoltDB.

SQL is a query language, Relational is a data model, neither one prevents scaling out.

Post reply on HN