Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQL Databases Don't Scale

adam.blog.heroku.com

Post reply on HN
Print view View original

61–70 of 112 posts

Re: SQL Databases Don't Scale

#61

Post by bayareaguy »

rjurney wrote: post #60
bayareaguy wrote: post #50

Earlier quoted context omitted.

You can't be talking about Hive[1], the SQL system for structuring and quering Hadoop datasets. That would never scale... 1- http://hadoop.apache.org/hive/

SQL atop an unstructured datastore, with minimal metadata. Something of an innovation, not a traditional RDBMS, but yes - still SQL :) The nice thing about HIVE is - you're not limited to just SQL, though. You can still analyze your files on HDFS any whichaway, with Pig, with your own MapReduce jobs, whatever. Personally though, I look forward to Apache Pig getting SQL, and being able to run SQL queries on any interm…

I don't think that anyone is really complaining about SQL. They're complaining about relational dbs.

Actually I suspect Mr. Wiggins is really only complaining about problems with MySQL and PostgreSQL. He would be less inaccurate if he admitted as much rather than making sweeping uninformed claims about SQL and relational systems.

Re: SQL Databases Don't Scale

#62

Post by jacquesm »

JulianMorrison wrote: post #56

Scale like what? If you aren't IBM, Facebook or Google, why do you even care?

ouch, I'm not IBM, Facebook or Google but SQL scalability issues are a good part of my daily workload.

(the other part is made up from file system scalability issues).

Once you you get past a certain level these are non-trivial problems and anybody out there that is busy solving them has my interest.

Re: SQL Databases Don't Scale

#63

Post by trapper »

nettdata wrote: post #23

I've been an Oracle database architect for almost 20 years. His whole concept of "SQL doesn't scale" is the typical crap I always hear from people that are either not database experts, are using the wrong database technologies, or don't know what they're doing. More than likely a combination of all three. And just because you can create an object model, and a simplistic data model, does not make you an architect of l…

I've just read all your comments. Do you have a blog?

Re: SQL Databases Don't Scale

#64

Post by jacquesm »

trezor wrote: post #26
gaius wrote: post #17

Earlier quoted context omitted.

As others have pointed out, the "no SQL" crowd are invariably MySQL users who have run into the limitations of MySQL but for ideological reasons can't state that the problems they encounter are specific to MySQL. DB2, Teradata and Oracle users regularly tackle problems 100x larger than MySQL can handle.

Agreed. I also fund it hard to sympathize with these startups having ambitions to scale to Google-like sizes, yet are unwilling to pony up cash for a proper database system like Oracle or SQL Server. Seeing the amount of ugly hacks people are willing to come up with and employ and features they are willing to cut, just to handle trivial loads, kinda makes me think that MySQL can only be considered free if your time i…

yes, because we all know google runs on a 'proper' database system like Oracle or SQL server.

Google is one of those excellent examples of a non-sql datastore that works just fine and seems to blow the socks of anything the competition has come up with to date.

Re: SQL Databases Don't Scale

#65

Post by Confusion »

prodigal_erik wrote: post #59
Confusion wrote: post #21

So while sharding is a form of horizontal scaling, it fails point #2: it is not transparent to the business logic of the application. I do not believe this is usually true, for two reasons. One may be nitpicking, but 'where to get the data' is not part of the business logic: it's pure application logic, dependent on your solution of the problem. In that sense, his argument is wrong. The other reason definitely isn't…

I don't see a good reason for "which server has my data" to be in-your-face application logic to code, when "which sector of which disk has my data" has long since been delegated to the platform and forgotten.

If you are bound to SQL databases, either for historical reasons or because the no-SQL database make other problems harder, then 'scaling the application' is a pretty good reason for writing that logic. There is no silver bullet.

Re: SQL Databases Don't Scale

#66

Post by ntoshev »

nettdata wrote: post #36
blasdel wrote: post #29

Earlier quoted context omitted.

You totally ignore the background of the author -- his company Heroku : Ruby/Rack :: Google AppEngine : Python/WSGI But there's a huge problem -- AppEngine succeeds at seamless multi-tenant truly-distributed clustered hosting thanks to BigTable. Heroku needs to support standard Rails apps, so Postgres is the best they can do, and it's a huge hole in their offering. You just can't make Postgres (or Oracle) scale on an…

For what it's worth, you can most definitely do that with Oracle. I can't speak to Postgres scalability, but it is my DB of preference. (I guess it might have something to do with the fact it's as close to Oracle as you're going to get in Open Source). We design apps to treat the datasource as a network service, and have no problems load balancing DB connections across our database cluster, and adding new DB nodes as…

Maintaining connections in the face of failure is a relatively little concern, what I perceive as the big problem here is having the database handle an arbitrary application with zero application-specific administration.

Google's datastore promises that if you write your app for their platform and it works on the small scale, it will scale out without problems. Datastore latency is constant regardless of how many records you have there. How will you scale a join between two tables, each being partitioned between multiple different servers, transparently for the app?

Re: SQL Databases Don't Scale

#67

Post by ntoshev »

nettdata wrote: post #45
nostrademons wrote: post #42

Earlier quoted context omitted.

"For example, Facebook is, for the most part, a read-only, fairly static site." FaceBook's actually one of the more dynamic sites out there. I'd bet that the average FaceBook user makes many more updates to their FaceBook profile than they make financial transactions. The real difference in requirements is that FaceBook can - and does - drop updates on the floor. If your friend throws a sheep at you and you don't get…

I agree... never meant to imply that Facebook had fewer hits/etc than a banking site in raw numbers, just that the ratio of DB reads (not considered a transaction, very easy to cache, etc) to DB writes, makes the site much more read-only than write-only. Facebook has the ability to do a TON of edge caching, with very few (relatively speaking) operations having to go to the database to perform an actual write operatio…

My guess is apps hosted on Heroku (like the majority of the internet applications) are a lot like Facebook and little like online banking: they don't really need reliable transactions, they need to be flexible, scalable and cost-effective.

Re: SQL Databases Don't Scale

#68

Post by rjurney »

nettdata wrote: post #44
rjurney wrote: post #38

Earlier quoted context omitted.

SANS are too expensive. They don't scale cost-effectively compared to commodity PC hardware.

I'd beg to differ. Go take a look at Adam Leventhal's work with the Fishworks stuff. Specifically, go check out the Sun Storage 7310. It will scale HUGE, and is nowhere near the stupid cost of NetApp or EMC or the other major vendors. I'd love to see how a bunch of commodity PC's will scale to 100TB, and still be manageable, and have anywhere near the same feature sets. Again, if you're railing against something as u…

You can use Hadoop to scale to 100TB using Commodity PCs and still be manageable, much easier and cheaper than you can use Oracle to do same. The featureset could easily DWARF those available using Oracle, as you can mapreduce all the data using your own hadoop jobs, and it can hold whatever kind of data you want it to - you're not limited to a static schema and precomputed summaries. HIVE is running on commodity PCs at facebook on more than 100TB, and they prefer it to their enormous, overly expensive Oracle OLAP system. So there's your answer, for one use case.

But of course, if you're updating the data often - you wouldn't use hadoop (or if you were, you would run HBase or some such on top of it). But there are many use cases where you only write once at that scale. And in those cases, from my perspective - its much nicer to scale on commodity hardware than on big iron.

Re: SQL Databases Don't Scale

#69

Post by ianso »

bayareaguy wrote: post #57
ianso wrote: post #52

Earlier quoted context omitted.

WRT exposing Postgres, you're right that you can't simply expose a database and then expect the DB to scale under random selects, inserts, etc. (RAC is another matter I guess.) However, as I said in the first comment to the blog post, if you define an interface using stored procedures (a pattern familiar to many Oracle DBAs), then PL/Proxy ( http://pgfoundry.org/projects/plproxy/ ) lets you do hash-based partitioning…

While Skype's PL/Proxy is a great way to make PostgreSQL more scalable, it doesn't do much to refute the basic argument that SQL databases aren't scalable since the SQL it helps you scale is limited to short RPC style operations.

That's a good point, and it brings to light an unspoken assumption underlying my post, which is that the use-case is a web-based, read-heavy OLTP-style system. If you were thinking of things like OLAP and data warehousing, then I'd agree with you absolutely.

However, under my re-qualified assertion :-) for large, complicated commits, the logic would either have to be at the database level for it to be in the same transaction, or a solution using temporary tables could be put together for more convoluted calls.

Neither of these are elegant, I'll grant you, but the two basic approaches - longer transactions over logic closely coupled to the datastore, or staged writes - are what most DBAs on high-end databases end up doing anyway, and would probably be reproduced in some form or another in any ACID system, no? Either way you'd still have a database that scales.

Re: SQL Databases Don't Scale

#70

Post by khaless »

trezor wrote: post #55

Summary of an article about how "SQL Databases Don't Scale": 1. Mentions RAID, not SAN 2. Mentions MySQL and only MySQL (with the exception of PostgreSQL once). 3. Mentions Master-slave replication as a killer scalability feature. I suggest he rename the piece to "A $500 server and MySQL Don't Scale" and then we can all agree and get along.

Yeah, there certainly are a few more tricks with regards to scaling RDBMS than the author covered. Depending on what your demands are there are different techniques and protocols which best suite you and can go very far to solving your problem. But with that in mind there may be better ways to solve your problem, and we should not forget about those.

If You start sharding you may get write gains, but you have to work very hard to keep things consistent (depending) and you may have to duplicate shards to make them highly available ($$$). Oh - and later down the track your schema might change in ways which your sharding scheme is just not flexible enough to deal with and depending on who you are that may be too much of a risk.

Besides the issue is really with availability, consistency and performance. It is very hard to scale all three of these together and even your cashcow solutions will hit their limits (although some of their limits are quite high :))

Post reply on HN