Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQL Databases Don't Scale

adam.blog.heroku.com

Post reply on HN
Print view View original

51–60 of 112 posts

Re: SQL Databases Don't Scale

#51

Post by akeefer »

alexgartrell wrote: post #47
alexgartrell wrote: post #12

Crazy idea: Write your data-handling code in it's own tight little module. Use good abstractions so that the rest of your app doesn't give a crap HOW it's happening. When it's time to update to millions of clustered servers running bigTable, you rewrite the module, and you're done. Software Engineering saves the day!

3 upvotes and 2 downvotes without a single response. As a junior member of Hacker News, I demand an education when I'm downvoted! :)

I wasn't one of the ones who upvoted or downvoted you, but just because you asked . . . while abstractions around your data-handling code are always a good idea, on some level they always leak. (For example, see the classic post http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Compute... describing how ORM, one of the most common approaches to encapsulating data-handling code, tends to fail).

Most of your code is data-handling code on some fundamental level, so it's pretty inevitable that it's going to matter to your application how that data is stored and what sorts of operations you can do on it, and that's especially true if you care about performance (which you generally always do after some level). So even if you have a nice abstraction of your query layer, the kinds of questions you can ask the database efficiently depend heavily on the underlying storage mechanism, and so your application logic has to be built so it only asks the right kinds of questions.

Even abstracting away the differences between, say, Oracle and MS SQLServer and MySQL is difficult enough, because of the different capabilities and performance characteristics. When you do that, you basically end up coding to the lowest common denominator, which can often limit what your applications does.

Trying to come up with an abstraction that can encapsulate the difference between a row- versus column-oriented database, or between a relational database and some other kind of storage, is pretty much a losing proposition: they're too fundamentally different in terms of what kind of data you can store, how you can store it, how you can query it, what kinds of transactional guarantees you get, and what operations are fast and which ones are slow.

So you really do kind of have to take your best shot at it, choose an approach, and if you choose wrong and have to change, it's just going to hurt. A lot. Good encapsulation and abstraction will ease some of the pain, but it's more like drinking whiskey before your leg gets sawed off than it is like general anesthesia.

Re: SQL Databases Don't Scale

#52

Post by ianso »

blasdel wrote: post #29
nettdata wrote: post #23

I've been an Oracle database architect for almost 20 years. His whole concept of "SQL doesn't scale" is the typical crap I always hear from people that are either not database experts, are using the wrong database technologies, or don't know what they're doing. More than likely a combination of all three. And just because you can create an object model, and a simplistic data model, does not make you an architect of l…

You totally ignore the background of the author -- his company Heroku : Ruby/Rack :: Google AppEngine : Python/WSGI But there's a huge problem -- AppEngine succeeds at seamless multi-tenant truly-distributed clustered hosting thanks to BigTable. Heroku needs to support standard Rails apps, so Postgres is the best they can do, and it's a huge hole in their offering. You just can't make Postgres (or Oracle) scale on an…

WRT exposing Postgres, you're right that you can't simply expose a database and then expect the DB to scale under random selects, inserts, etc. (RAC is another matter I guess.)

However, as I said in the first comment to the blog post, if you define an interface using stored procedures (a pattern familiar to many Oracle DBAs), then PL/Proxy (http://pgfoundry.org/projects/plproxy/) lets you do hash-based partitioning in a way that's more or less transparent to the end-user of a DB, again assuming that it has a defined interface. The PL/Proxy installs form a 'bus' between the DB and the user.

Self-promotion: I'm currently working on hacking PL/Proxy into something that can be used to auto-scale a Postgres cluster on-demand, which has interfaces as a pre-requisite, along with some other things. The end-goal is to do exactly what you say: to scale-out Postgres like any other internet service on an ideal horizontal. Link: http://code.google.com/p/hotrepart

(Hum, I'm gonna get accused of spamming for repeating myself so often :-)

Re: SQL Databases Don't Scale

#53

Post by trezor »

derefr wrote: post #46
nostrademons wrote: post #43

Earlier quoted context omitted.

People can still make money charging for them? Open-sourcing is the last step in the technology lifecycle, after the technology has become widely understood and commoditized. When people can still make money off something, they will. What's the incentive for them to give it away?

Usually, an ideological one. The first group of Linux kernel developers (past the "toy" stage) all hated Microsoft with a passion, and wished to deprive it of as much revenue as possible. They wanted to give people a "free alternative", and lower the total investment people [that is, they] would have to make in owning a computer. I could imagine an analogous situation with some developers and Oracle.

Being ideologically for open software and an open operating system is not the same as "hating Microsoft with a passion". Mixing these two different concepts into the same bag is misleading, as the first one represents an ideological conviction and the other merely childish spite.

And if you insist on being cheap, don't be surprised when it turns out that your free database was indeed some cheap stuff which doesn't come fully featured.

Re: SQL Databases Don't Scale

#54

Post by gaius »

dflock wrote: post #6
stuff4ben wrote: post #4

I thought it was a good summary of the problems facing SQL databases as organizations grow. However I would have appreciated some solutions to the problem.

I think one of the points made at the end was that there kinda isn't one - people have been trying to come up with the perfect solution to scaling RDBMS's for ever and the current state of the art (pretty much) is as described in the article - i.e, not great. People with really massive MySQL setups (eBay, for example) basically just do a huge amount of the stuff described here, but with lots of message-queue type stu…

the current state of the art (pretty much) is as described in the article - i.e, not great

That simply isn't true.

For me, thousands of transactions per second and 10s of terabytes of data on a single database is normal. It's unremarkable, it's everyday, it's what we do, we have done it for years. And I know of installations handling 10x that. It's only people who's only "experience" is websites that whinge about how RDBMS can't handle their tiny datasets.

Re: SQL Databases Don't Scale

#55

Post by trezor »

Summary of an article about how "SQL Databases Don't Scale":

  1. Mentions RAID, not SAN
  2. Mentions MySQL and only MySQL (with the exception of PostgreSQL once).
  3. Mentions Master-slave replication as a killer scalability feature.
I suggest he rename the piece to "A $500 server and MySQL Don't Scale" and then we can all agree and get along.

Re: SQL Databases Don't Scale

#57

Post by bayareaguy »

ianso wrote: post #52
blasdel wrote: post #29

Earlier quoted context omitted.

You totally ignore the background of the author -- his company Heroku : Ruby/Rack :: Google AppEngine : Python/WSGI But there's a huge problem -- AppEngine succeeds at seamless multi-tenant truly-distributed clustered hosting thanks to BigTable. Heroku needs to support standard Rails apps, so Postgres is the best they can do, and it's a huge hole in their offering. You just can't make Postgres (or Oracle) scale on an…

WRT exposing Postgres, you're right that you can't simply expose a database and then expect the DB to scale under random selects, inserts, etc. (RAC is another matter I guess.) However, as I said in the first comment to the blog post, if you define an interface using stored procedures (a pattern familiar to many Oracle DBAs), then PL/Proxy ( http://pgfoundry.org/projects/plproxy/ ) lets you do hash-based partitioning…

While Skype's PL/Proxy is a great way to make PostgreSQL more scalable, it doesn't do much to refute the basic argument that SQL databases aren't scalable since the SQL it helps you scale is limited to short RPC style operations.

Re: SQL Databases Don't Scale

#58

Post by derefr »

trezor wrote: post #53
derefr wrote: post #46

Earlier quoted context omitted.

Usually, an ideological one. The first group of Linux kernel developers (past the "toy" stage) all hated Microsoft with a passion, and wished to deprive it of as much revenue as possible. They wanted to give people a "free alternative", and lower the total investment people [that is, they] would have to make in owning a computer. I could imagine an analogous situation with some developers and Oracle.

Being ideologically for open software and an open operating system is not the same as "hating Microsoft with a passion". Mixing these two different concepts into the same bag is misleading, as the first one represents an ideological conviction and the other merely childish spite. And if you insist on being cheap, don't be surprised when it turns out that your free database was indeed some cheap stuff which doesn't co…

It's a causal relationship. If you believe that all software should be free (the GNU folks—those "first kernel developers"), then you must at least dislike any company which tries to profit from the creation of artificial scarcity of software. The two groups (the idealists and the "haters"), which now have very little overlap, originally started much the same.

> Don't be surprised when it turns out that your free database was indeed some cheap stuff which doesn't come fully featured.

But if being "fully featured" is the goal, then it would be very surprising indeed if what you considered a "competitor" was not, in fact, fully featured. It would not, then, by definition, be a competitor. Or, at least, it would not be worth calling version 1.0 yet.

To refine that: Oracle currently has no FOSS competitors, because there are no FOSS databases that are trying to compete with Oracle. They may be trying to take parts of Oracle's market share, but this is a different thing—optimizing their fit for a situation where Oracle itself is a bad fit.

Re: SQL Databases Don't Scale

#59

Post by prodigal_erik »

Confusion wrote: post #21

So while sharding is a form of horizontal scaling, it fails point #2: it is not transparent to the business logic of the application. I do not believe this is usually true, for two reasons. One may be nitpicking, but 'where to get the data' is not part of the business logic: it's pure application logic, dependent on your solution of the problem. In that sense, his argument is wrong. The other reason definitely isn't…

I don't see a good reason for "which server has my data" to be in-your-face application logic to code, when "which sector of which disk has my data" has long since been delegated to the platform and forgotten.

Re: SQL Databases Don't Scale

#60

Post by rjurney »

bayareaguy wrote: post #50
rjurney wrote: post #37

Earlier quoted context omitted.

It was the failure of Oracle to scale cost-effectively that led them to create HIVE.

You can't be talking about Hive[1], the SQL system for structuring and quering Hadoop datasets. That would never scale... 1- http://hadoop.apache.org/hive/

SQL atop an unstructured datastore, with minimal metadata. Something of an innovation, not a traditional RDBMS, but yes - still SQL :)

The nice thing about HIVE is - you're not limited to just SQL, though. You can still analyze your files on HDFS any whichaway, with Pig, with your own MapReduce jobs, whatever. Personally though, I look forward to Apache Pig getting SQL, and being able to run SQL queries on any intermediate state of a Pig script.

I don't think that anyone is really complaining about SQL. SQL is a swell query language for certain kinds of data. They're complaining about static schemas in relational dbs and having to store objects via SQL - reasonable complaints.

Post reply on HN