Live data from Hacker News

I Can't Wait for NoSQL to Die

teddziuba.com

21–30 of 80 posts

Re: I Can't Wait for NoSQL to Die

#21
post #19
post #16

Earlier quoted context omitted.

If Rails == ActiveRecord == SQL, then NoSQL implies !Rails. (This isn't true of course, but we're already dealing with oversimplifications here.) Or maybe he's just saying that irrational NoSQL hype has replaced irrational Rails hype.

The Rails message is "you don't need a DBA, you don't need to know SQL, just have your developers do the default install of MySQL and we'll do the rest". That is also the NoSQL message.

Speaking as a member of the rails core team, if you heard someone say that then he made it up.

Rails until 3.0 was heavily invested into SQL simply because it's what everyone used. However, it was swimming against the stream in that rails declared DBs to be something which is incrementally developed by the software through migrations instead of setup by DBAs per change tickets. This has been enormously successfully, there is probably not a single web framework left that pretends we still live in a DBA dominated world. Part of the shrapnel of this decision is that rails did away with triggers, db constrains and stored procedures but this is simply because most high volume sites don't use these things anyways because they get very hard to scale later on.

Rails past 3.0 will work natively with any data store that you can imagine. It ships something called ActiveModel which is a tiny interface that you can implement on top of Mongo, Cassanda, Redis. ActiveRecord is just the SQL incarnation of this interface.

Very high quality libraries based on ActiveModel already exist. Have a look at Cassandra Object as an example.

Re: I Can't Wait for NoSQL to Die

#22
post #6

NoSQL will never die, but it will eventually get marginalized, like how Rails was marginalized by NoSQL. Is it just me, or does this statement make absolutely no sense whatsoever?

I don't know if this is what he was trying to say, but it makes a little more sense if you change it to NoSQL's hype will never die, but it will eventually get marginalized, like how Rails' hype was marginalized by NoSQL's hype.

[deleted]

Re: I Can't Wait for NoSQL to Die

#23
I'm getting tired of both sides of this argument, I'll be happy when the whole back and forth dies :). Rarely do you see a balanced opinion. Sometimes it's people that are fanatical about the new-ish NoSQL idea. Other times, like this, it's someone so stuck in their ways they think that everything but what they like is a fad and nothing will ever change.

One of the key things I look for when I interview developers is that they can recognize the right tool for the job. Potentials that get married to a technology or language are shown the door pretty quickly.

Also, as others have pointed out, this particular article seems to not quite understand the decisions involved, to the point of getting some things backwards.

Re: I Can't Wait for NoSQL to Die

#24
NoSQL might be hype. Let's get specific. Cassandra eliminates the SQL database single point of failure and hard to replace masters via a lose sync, "eventually consistent" protocol.

Is there some startup offering a web service that doesn't need that?

And have you ever tried to deploy an SQL database capable of thousands of miles apart syncing?

Eventually consistent is quite a different model than ACID. If you accept that, and accept that you can't rely on networks to always be up, you'll live comfortably and cost effectively.

Re: I Can't Wait for NoSQL to Die

#25
post #21
post #19

Earlier quoted context omitted.

The Rails message is "you don't need a DBA, you don't need to know SQL, just have your developers do the default install of MySQL and we'll do the rest". That is also the NoSQL message.

Speaking as a member of the rails core team, if you heard someone say that then he made it up. Rails until 3.0 was heavily invested into SQL simply because it's what everyone used. However, it was swimming against the stream in that rails declared DBs to be something which is incrementally developed by the software through migrations instead of setup by DBAs per change tickets. This has been enormously successfully,…

this is simply because most high volume sites don't use these things anyways because they get very hard to scale later on.

If you start with the assumption that "you don't need a DBA" then that's probably true.

Just to give you an idea of my background, I work on a system using a commercial RDBMS that "scales" to thousands of commits/sec and tens of terabytes of data. We expect to take it to tens of thousands of commits (we already do that many reads!) and hundreds of teras with no major structural changes. One thing you learn in this game is that database agnosticism is a wild goose chase. To really scale, you need to intelligently choose a technology and use its features to the fullest and just accept that you will be "locked in". We couldn't port to another RDBMS if we tried because certain things, like our chosen database's locking strategy for example, are baked in to the way we do things. It's not a matter of SQL syntax. We'd be starting again from scratch, we'd need new algorithms. But we can do things, we take things for granted, that most of the Internet peanut gallery takes to be impossible, because they start from assumption that abstracting the database actually helps anything.

Re: I Can't Wait for NoSQL to Die

#26
post #8
post #5

Earlier quoted context omitted.

"Sometimes you just want to store data on the disk in a safe and language agnostic way." You mean, kind of like a file?

Well, since you need a safe way, you'll need a locking mechanism, too. And since it needs to be language agnostic, you can't just dump the internal representation of your object to disk, so you need some sort of serialization. At that point, it's probably easier to go with an already existent object or document store.

XML is nice for this because it supports multiple schema versions, validation, and has support in just about every language. My chief complaint with json as a cross-language serialization/interchange format is lack of a great way to validate your format. Most of the json schema definitions i've seen require your schema to follow a certain convention, which seems backwards and wrong to me.

Files can be locked if your OS supports it.

Re: I Can't Wait for NoSQL to Die

#27
post #12

Earlier quoted context omitted.

Absolutely right. He has it backwards. You use NoSQL to 'get shit done'. When you have a billion rows, then worry about schemas. By that point you will have a much better idea, a - what said schema should look like, b - what the architecture of the Postgres, or mysql, or Oracle should look like, and c - how much money you will have to solve the problem.

I don't really see the point of starting with NoSQL and then going for Oracle. NoSQL DBs are real tools you know, not toys you through after your app gets traction. It's actually the contrary that has happened so far.

Maybe I should have been clear. You only go to SQL RDBMS if you need it. Which, in the vast majority of cases you will not. Further, you only go to SQL where you need it.

For instance, right now someone is developing a street car racing game for Facebook. The XBox kind, not the FarmVille kind. At any rate, one of the features is obviously, playback. Now keeping all of those physics updates in an SQL is pointless. And figuring out a schema for that data would have only gotten in the way of them getting that out the door. Throw the physics messages in a queue and write them to Cassandra. If you have even 10000 MAUs, you will easily generate billions of rows. It's just not data that really needs to go into mysql.

I don't think Cassandra is a toy. I think you should 'get it out the door' with everything in Cassandra, and then slowly, move the business stuff off. User names, what cars they bought for instance. Stuff that is not read often. But at first, get it out the door. Don't stop to figure out a perfectly normalized schema with balanced indexes.

Re: I Can't Wait for NoSQL to Die

#28
post #5
post #3

The author makes some very valid points, but I will retort that RDBMS are overused as well. Sometimes you just want to store data on the disk in a safe and language agnostic way. You don't care about relations. In that case, many "NoSQL" engines are really great.

"Sometimes you just want to store data on the disk in a safe and language agnostic way." You mean, kind of like a file?

Filesystem is the original document store. Unfortunately, most filesystems really suffer when you put $LOTS of files in the same folder, so you end up implementing a nested folder structure and that complicates your code. Now, if this is more complex than running an entire "object" store depends on the application. Also, the filesystem's addressing may not be granular enough (and so waste a lot of space,) if you are going to store < $BLOCK_SIZE files.

Re: I Can't Wait for NoSQL to Die

#29
AdWords implemented on top of MySQL? Perhaps the CRM portion of AdWords (i.e., where the advertisers submit their ads and publishers view their balances) is -- it's fairly easy to partition by functionality and doesn't have extremely tight latency bounds. This isn't where real time auctions (what really distinguishes AdWords from what came before) happen.

You can be sure, however that the data used for real time ad auctions is extracted out of MySQL and into a highly customized data store (likely, a pure in memory one). It's all about using a right tool for the job. You can also be sure that you'll never see a paper on that data store, as that's their competitive edge. If you could duplicate it with off the shelf components (whether MySQL or Cassandra), Google would be toast.

Likewise, I am sure Amazon uses Oracle for their billing system and catalog submission interface, but they use specialized systems for search, shopping cart and recommendations.

For a business app that only needs to scale to the amount of paying customers (i.e., advertisers, account managers and customer support) and has no real time constraints -- but on the other hand involves complex and frequently changing business logic (e.g., where altering tables may be required) an RDBMS is the right tool for the job.

Where latency matters, data grows much faster than Moore's law (in relational to main memory size), Amdahl's law starts to matter in regards to computation (computation work load needs to be partitioned to take advantage of parallelism), and traditional caching strategies simply don't work, something else is. That situation is starting to become more and more common across web companies. You can also be sure that places like Wallmart and the like employ plenty of non-relational technologies (my personal bet would be is that they're likely using Coherence or Terracotta): usually, however, they're expensive and are built/configured by field-engineers to be custom tailored for their workloads. When you employ a world-class engineering team, "build" starts to make more sense than buy when you're solving a very specific and constrained problem (e.g., fault tolerant shopping cart system).

You don't need to be of Google's size to be at that stage. Talking about scalability and performance without taking the workloads into account (e.g., "Google Facebook or Amazon" as if e-commerce, search and social networking were compatible) is also an anti-pattern: I am sure engineers at Google would laugh when you compare Facebook's scale to theirs; likewise Facebook's engineers would laugh when you compare the real time aggregation that happens on their site to what happens at Amazon; Amazon's engineers would likely tell you holiday season pager duty horror stories that would scare Facebook or Google engineers.

Re: I Can't Wait for NoSQL to Die

#30
post #12

Earlier quoted context omitted.

Absolutely right. He has it backwards. You use NoSQL to 'get shit done'. When you have a billion rows, then worry about schemas. By that point you will have a much better idea, a - what said schema should look like, b - what the architecture of the Postgres, or mysql, or Oracle should look like, and c - how much money you will have to solve the problem.

I once worked in a Notes shop. Notes has no schema for documents and nothing to enforce migrating data from older documents to the current format. After a few years of customers manipulating data with various versions of the code, they had documents in such bizzare combinations of states that it was no longer possible for anyone on our dev team to inspect them and say which behavior would be right for the workflow. S…

That's because you waited years to address the problem. Not only that, you also rewrote the code, as I advise. But you did not take the opportunity to address the structural data issues you were having, contrary to my advice.

My strategy is to rewrite the code, if needed, but with an eye towards addressing structural data issues. After a few months use of a web app you have a good idea of any surprising usage patterns that may appear. Readjust at that point when you are 'talking with data'.

This advice is for small startups of the HN variety, where 'customers' are a lot more important than 'authoritative' data stores initially. NoSQL systems are useful tools for mitigating the danger of doing too much engineering upfront. Many tech entrepreneurs fall victim to doing too much upfront engineering in the hopes of their data store not 'decaying to garbage', only to find that no one wants to use their product. NoSQL makes it easy to go back and migrate off the data you want to store 'on the move'. When you have a better idea of how much of it there is, and how it is used.

Post reply on HN