Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How NoSQL forced the evolution of a scalable relational database

blog.memsql.com

Post reply on HN
Print view View original

81–90 of 211 posts

Re: How NoSQL forced the evolution of a scalable relational database

#81

Post by mmt »

matwood wrote: post #80
189723954 wrote: post #38

Earlier quoted context omitted.

>I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript. I chose Mongo for a project recently. I have many years experience with SQL databases, and I think SQL will become more of a niche in the future. Here are the reasons: Everybody uses an ORM with a SQL database. You can pretend they don't and everyone is writing raw sql, but they aren't. This is basically a b…

> Everybody uses an ORM with a SQL database. I'm ripping ORMs out of any backend code and using things like jOOQ. > I was just thinking to myself "I can write this in code in a few minutes instead of an hour or more".... > Joins are slow and complicated You know what's really slow? Creating joins in code. > This is not true at all. Programming languages are vastly superior at querying a database. Just go write a comp…

> It makes zero sense to pull a bunch of records back from the db using multiple network calls to join and then filter. Let the db do its job.

But then it wouldn't be distributed processing! :)

Seriously, though, consider it for a moment.. this pattern has similar features to something like Hadoop. The data comes from storage nodes (database server and, hopefully, their read replicas) and goes to processing nodes (app server) to have the work done and is then new data is written back out over the network to storage nodes and replicated across the network (to the replica/slave database servers).

If the data volume is particularly low or the compute load (CPU and/or RAM) is particularly high, the distributed method would make intuitive sense. I haven't seen it yet, however.

Re: How NoSQL forced the evolution of a scalable relational database

#82

Post by kapv89 »

sametmax wrote: post #40
keithnz wrote: post #22

I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript. While scaling is one of the advantages, I'd be super surprised if many people actually need scaling capabilities ( other than because their design is super inefficient ). Recently I've been inspired to play around with kicking out as many layers between a relational db and a REST Api, largely because I've bee…

> it is super easy to program with javascript. It would be more honest to say that the weird js design, its lack of features and ridiculously small stdlib made it harder to do anything complicated at the time (no ES6 or webpack, remember ?). Hence, not good tooling existed for relational databases: no decent abstraction layer, migration lib or ORM. Lost of languages had a 2 liner to pop an sqlite db without needing t…

True that Node.JS didn't have, and still doesn't have a 2 liner to pop an sqlite db. But what it did have(a few years after its launch) was a pure-JS sqlite client which can leverage V8's event loop and libuv to make parallel execution of db-queries a trivial operation.

There is a lot of mettle in Node.js for server side programming.

PS: have you looked at TypeORM(http://typeorm.io/)

Re: How NoSQL forced the evolution of a scalable relational database

#83

Post by charlesvdv »

threeseed wrote: post #36
antirez wrote: post #31

Redis is not mentioned here at all so maybe the author is thinking mostly of other NoSQL software here, but well, in the case of Redis the whole point was not just the in-memory performance part, but the data model as well. My claim is that you can't really exploit the advantage of using memory if you perpetuate in using the memory to represent the same data model that you were using with relational databases. For in…

This is the problem with people that use the term NoSQL. Do you mean Cassandra (BigTable), MongoDB (Document), Riak (Key/Value), Redis (Mix), Kafka (Log Structured) ? There are dozens of fundamentally different systems many of which are closer to an RDBMS than their NoSQL peers. And many of them have rigid schemas so it definitely isn't that either.

You can also add Neo4j (graph) as part of the NoSQL family, right?

Re: How NoSQL forced the evolution of a scalable relational database

#84

Post by Sjenk »

lotsofcows wrote: post #72
Sjenk wrote: post #39

Earlier quoted context omitted.

Not only easy in Javascript, it is easy in any language that has easy serliazation to JSON. I always use it for my own hobby projects because I dont have that much SQL knowledge. Which makes Nosql easy choice for me, just serialize your object into json and push it to your DB.

That's not a choice. To make a choice you need to be aware of alternatives. You use NOSQL because it was easy to achieve your goal.

Dont mean to troll. But how choosing for the easiest way to achieve your goal not a choice?

Re: How NoSQL forced the evolution of a scalable relational database

#85

Post by matwood »

189723954 wrote: post #44
logophobia wrote: post #6

Earlier quoted context omitted.

I would say the the rise of nosql options is driven by the need for scalability by a few large companies that desperately need it, and cargo-culting by developers that don't actually need it, but want to be like google and don't want to take the time to properly learn and optimize sql. There's a genuine need for good scalable options, and with stuff like google spanner, those don't necessarily need to be "eventually…

>don't want to take the time to properly learn and optimize sql. This sounds like an old-wives tail at this point.

> This sounds like an old-wives tail at this point.

Except it's not. I've seen it in person, over and over. I've gone into systems where people were complaining about the 'database is slow', but there were no indexes. I've seen systems pull back 100s of thousands of records and then sort on app server and take the first 50. I've seen what would have been a simple join with exists clause turned into many round trips to the db with loops and app code complexities where a query would have 100x easier to reason about.

It's not even about optimization yet, but taking the time to learn even the surface capabilities of the tool.

Re: How NoSQL forced the evolution of a scalable relational database

#86

Post by matwood »

sqlcook wrote: post #58
logophobia wrote: post #55

Earlier quoted context omitted.

I've met developers like that. Although honestly, sql is more than 40 years old and can be really nasty. Stored procedures, each database has a different dialect, nullability comparisons, CTEs. It's very different from any other programming language. It's not that hard to imagine people finding it hard to learn, especially front-end web developers who'd rather not touch the backend. If people don't know how indexes w…

It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL. It's a lot harder to write "nasty" sql than "nasty" js, and if you're a front-end dev struggling with sql, then you probably should not be doing back end work. Yes not all databases are equal, thats why there are ORMs and ANSI standards. From the list of bad practices you've "seen" people do in SQL, I would guess their comfort z…

> It actually IS quite hard to imagine solid engineers finding it hard to learn simple SQL.

I used to think this because I learned SQL right along with all my other coding. I've realized though it is a mindset shift to go from imperative to declarative, and to think mostly in set operations. That shift can be hard for otherwise good developers.

Re: How NoSQL forced the evolution of a scalable relational database

#87

Post by matwood »

uncle_d wrote: post #10

My own experience chimes with this somewhat, having used MongoDB on my last project - quite often application data storage requirements are relatively trivial - and it is a boon to do away with the ORM layer and be able to vary any given object schema without touching the database (although in practice a .js data migration script may be involved, so this is moot). As a sidenote local cultural issue, the fact that we…

> As a sidenote local cultural issue, the fact that we can operate the MongoDB servers ourselves, whereas RDBMS instances are with a central team and buried under a layer of bureaucracy was also probably an operational consideration.

This is clearly an oversight in the organization and will get fixed. I don't see it as neutral in the argument for or against some system.

> But where we experienced pain was when the business decided they wanted to do live reporting. We ended up piping the data into a SQL Server instance and using SSRS.

This is common even in RDBMs. You have a database designed to handle the running of whatever application. That design may not be the most efficient for reporting off that application. Flattening the data, and pre-aggregating is usually a good first step. Many data warehouse solutions can do some of this automagically if set up with a common schema type (like star).

Re: How NoSQL forced the evolution of a scalable relational database

#88

Post by matwood »

mmt wrote: post #81
matwood wrote: post #80

Earlier quoted context omitted.

> Everybody uses an ORM with a SQL database. I'm ripping ORMs out of any backend code and using things like jOOQ. > I was just thinking to myself "I can write this in code in a few minutes instead of an hour or more".... > Joins are slow and complicated You know what's really slow? Creating joins in code. > This is not true at all. Programming languages are vastly superior at querying a database. Just go write a comp…

> It makes zero sense to pull a bunch of records back from the db using multiple network calls to join and then filter. Let the db do its job. But then it wouldn't be distributed processing! :) Seriously, though, consider it for a moment.. this pattern has similar features to something like Hadoop. The data comes from storage nodes (database server and, hopefully, their read replicas) and goes to processing nodes (ap…

Touché. You're right, at some point the data must be joined and filtered. My point was to let the tool do its job :)

Re: How NoSQL forced the evolution of a scalable relational database

#89

Post by siscia »

ddorian43 wrote: post #76
siscia wrote: post #30

Earlier quoted context omitted.

Honestly, I believe that for small workload you can definitely use RediSQL in production, it will happily contain your cache or it will be a great SQL database. However, I need a way to cut it between people just using the free product and people actually supporting the project, so provide as paying feature something that the big company will require it seemed to me the only way to go. Unfortunately, I don't have the…

To be honest, I fail to see what I could use your product for so I'm out of the target audience. Assuming nosql is for something very efficient or very scalable, I need some space to use it before I have to shell $$. There are many products where I have to pay before going on production.

It really depends on what you are building.

If I were building a fast prototype I would not use a postgres box anymore but just a redis one.

If you need to cache data in a way more complex than just key->value you don't have too many alternatives at the moment.

If you want an easy and fast way to have an SQL engine in memory, again is not going to be simple.

If you need a separated database for every of your user there are no many alternatives that I am aware of.

It is definitely not a revolutionary product, but it has it's niche, any of the problems that I mentioned can be solved in a different way, but those different ways are quite complex.

Re: How NoSQL forced the evolution of a scalable relational database

#90

Post by 189523954 »

matwood wrote: post #80
189723954 wrote: post #38

Earlier quoted context omitted.

>I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript. I chose Mongo for a project recently. I have many years experience with SQL databases, and I think SQL will become more of a niche in the future. Here are the reasons: Everybody uses an ORM with a SQL database. You can pretend they don't and everyone is writing raw sql, but they aren't. This is basically a b…

> Everybody uses an ORM with a SQL database. I'm ripping ORMs out of any backend code and using things like jOOQ. > I was just thinking to myself "I can write this in code in a few minutes instead of an hour or more".... > Joins are slow and complicated You know what's really slow? Creating joins in code. > This is not true at all. Programming languages are vastly superior at querying a database. Just go write a comp…

>You know what's really slow? Creating joins in code.

Ahh, you have combined two separate points into a new point that I didn't make and took it out of context. As I mentioned, I was writing queries for analytics software. This software needed fresh data once per day. There was no requirement for speed. I was referring to speed of development, not speed of the query.

As for the other point, yes joins are slow, and yes, multiple queries can scale better, even if they are not the fastest when you time them in an isolated one-off scenario. Multiple queries are simpler in code and scale better. Nobody is using joins at scale and every large company that started their scaling journey in a SQL database started by performing multiple queries on a distributed database.

Then there is the argument that "well you won't be operating at that scale". Then in that case, you don't have to worry about the minute differences between a single network call and multiple calls. There are so many approaches and realities that make your argument just a theoretical argument rather than a practical one. There is caching, denormalizing, and then there is the fact that people use a vast array of languages from slow as hell like python and upwards.

As for joins in code in particular, they are slower than in a database that is often written in c++ sure. It depends on the specific situation as to how fast things need to be. Did you know that many ORMs join in code? This is because of all the redundant data that the join creates. So you have to factor in whether the larger amount of data is slower to send than smaller data in multiple network calls.

From django:

However, to avoid the much larger result set that would result from joining across a ‘many’ relationship, select_related is limited to single-valued relationships - foreign key and one-to-one.

prefetch_related, on the other hand, does a separate lookup for each relationship, and does the ‘joining’ in Python.

Post reply on HN