How NoSQL forced the evolution of a scalable relational database • Hacker News Board

Re: How NoSQL forced the evolution of a scalable relational database

#111

Post by matwood » Fri, Jul 13, 2018, 2:02 PM UTC

threeseed wrote: post #71

davidgould wrote: post #62

Earlier quoted context omitted.

I think you misunderstood the parents use of the term "model". All the things you describe here can be described in the relational model. The parent was arguing that the model is general enough to cover almost all collections of data. You appear to be arguing that for specific use cases specific implementations perform badly. These arguments pass each other unseen like ships in the night.

Actually wide tables for example can't be done at all. And technically we can use Microsoft Excel as a photo editor. But that's just ridiculous. Just like it's ridiculous to say you can use relational models for any data structure even if the queries will never return.

There may be other limitations, but MSSQL supports[1]: Columns per nonwide table 1,024 Columns per wide table 30,000

I think a lot of this RDBMs can't do X, came from using a MySQL. It has improved some, but was garbage until recently.

I'm not sure how wide you're taking about, but PG support up to 1600 depending on the datatypes (there is a page size maximum). Oracle supports 1000 IIRC.

[1] https://docs.microsoft.com/en-us/sql/sql-server/maximum-capa...

Re: How NoSQL forced the evolution of a scalable relational database

#112

Post by jeffreyrogers » Fri, Jul 13, 2018, 2:02 PM UTC

CuriousSkeptic wrote: post #68

189723954 wrote: post #38

Earlier quoted context omitted.

>I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript. I chose Mongo for a project recently. I have many years experience with SQL databases, and I think SQL will become more of a niche in the future. Here are the reasons: Everybody uses an ORM with a SQL database. You can pretend they don't and everyone is writing raw sql, but they aren't. This is basically a b…

Linq works for trivial left join project style of queries. But for anything actually using the features of modern sql-databases it comes short quickly. As a example try to express something like “sum(x) over (partition by y order by z)” I do agree that SQL the syntax leaves a lot to be desired, and a proper relational language with a syntax optimized for actual development, and even better, optimized for 6NF style da…

My understanding is everyone uses 3NF in practice though, so what would a 6NF database get you practically?

I see SQL the way I see things like Linux: it's got some weird things that you'd design differently if you could do it over again, but overall it's pretty good and probably not worth the effort to switch.

Re: How NoSQL forced the evolution of a scalable relational database

#113

Post by jandrewrogers » Fri, Jul 13, 2018, 2:03 PM UTC

mmt wrote: post #81

matwood wrote: post #80

Earlier quoted context omitted.

> Everybody uses an ORM with a SQL database. I'm ripping ORMs out of any backend code and using things like jOOQ. > I was just thinking to myself "I can write this in code in a few minutes instead of an hour or more".... > Joins are slow and complicated You know what's really slow? Creating joins in code. > This is not true at all. Programming languages are vastly superior at querying a database. Just go write a comp…

> It makes zero sense to pull a bunch of records back from the db using multiple network calls to join and then filter. Let the db do its job. But then it wouldn't be distributed processing! :) Seriously, though, consider it for a moment.. this pattern has similar features to something like Hadoop. The data comes from storage nodes (database server and, hopefully, their read replicas) and goes to processing nodes (ap…

Doing distributed joins correctly requires an architectural/technical capability that most distributed database engines don't have: decentralized parallel orchestration. If you have this, you can do joins even with very high data volumes efficiently given good parallel scheduling algorithms. Most databases are designed such that there is a single point of control that declaratively schedules all data flows required to execute the query; this scales poorly for operations like joins, never mind recursive joins, which is why you don't see it.

Letting individual database nodes dynamically schedule and orchestrate their own data flows with each other, essentially allowing each node in the parallel system construct its own execution plan in relation to other nodes as it goes along, does not fit within the "giant distributed file system" paradigm that most distributed systems are based on. People who design codes for supercomputers are often familiar with parallel orchestration idioms that work at extremely large scales but it hasn't crossed over into ordinary distributed database engines. (This is also a good litmus test for what makes a database "parallel" as distinct from "distributed".)

Most distributed database architectures are much more centralized than they need to be, particularly around control of execution planning, and this limits their expressiveness. It is quite difficult to hack together a distributed join that performs better than a centralized one without good support for parallel orchestration.

Re: How NoSQL forced the evolution of a scalable relational database

#114

Post by orf » Fri, Jul 13, 2018, 2:03 PM UTC

189723954 wrote: post #38

keithnz wrote: post #22

I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript. While scaling is one of the advantages, I'd be super surprised if many people actually need scaling capabilities ( other than because their design is super inefficient ). Recently I've been inspired to play around with kicking out as many layers between a relational db and a REST Api, largely because I've bee…

>I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript. I chose Mongo for a project recently. I have many years experience with SQL databases, and I think SQL will become more of a niche in the future. Here are the reasons: Everybody uses an ORM with a SQL database. You can pretend they don't and everyone is writing raw sql, but they aren't. This is basically a b…

> The fact is that programming languages are much better at querying databases than SQL.

They really are not though. Sure, perhaps for simple single table scans, bu nothing beats the pure optimizing potential of SQL. Try joining three tables in JS neatly and fast.

It's a bit terse, but you're lying to yourself if you think hodgepodge mess of ad-hoc JavaScript written to poorly replicate a single specific query is in any way better than what we have now.

Re: How NoSQL forced the evolution of a scalable relational database

#115

Post by sytse » Fri, Jul 13, 2018, 2:05 PM UTC

_Codemonkeyism wrote: post #109

keithnz wrote: post #22

I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript. While scaling is one of the advantages, I'd be super surprised if many people actually need scaling capabilities ( other than because their design is super inefficient ). Recently I've been inspired to play around with kicking out as many layers between a relational db and a REST Api, largely because I've bee…

Same here, using Mongo helped us a lot to iterate fast, JSON is much more agile than SQL. We now migrate to PG using 'core' attributes in tables and 'flexible' attributes in JSONB to get the best of two worlds. I agree to the API problem, JDBC and PG/SQL is not as nice as a interace compared to Mongo libs.

When you say JSON is more agile is that because you don't need to write migrations? Do you use a framework that supports migrations like active record or south?

Re: How NoSQL forced the evolution of a scalable relational database

#116

Post by jasonlotito » Fri, Jul 13, 2018, 2:05 PM UTC

189723954 wrote: post #44

logophobia wrote: post #6

Earlier quoted context omitted.

I would say the the rise of nosql options is driven by the need for scalability by a few large companies that desperately need it, and cargo-culting by developers that don't actually need it, but want to be like google and don't want to take the time to properly learn and optimize sql. There's a genuine need for good scalable options, and with stuff like google spanner, those don't necessarily need to be "eventually…

>don't want to take the time to properly learn and optimize sql. This sounds like an old-wives tail at this point.

Just read through these threads. There are people literally saying these things.

Re: How NoSQL forced the evolution of a scalable relational database

#117

Post by dang » Fri, Jul 13, 2018, 2:15 PM UTC

189723954 wrote: post #50

richmarr wrote: post #45

Earlier quoted context omitted.

> This is basically a big, complicated, and slow piece of software that tries to make a SQL database into something else. Just to nitpick a little, but I think it's a worthwhile distinction, ORMs don't try to make a SQL database something else... either literally or philosophically. To use them in any more than a trivial way you still to understand RDBMSs. They just make the queries less verbose and the output more c…

You are just defining the limits of "something else" and then saying it doesn't do that. If your data looks completely differently to how the database outputs it, I don't think you can say it is just a trivial difference. Both the structure of a query and its output use different concepts to sql. Using linq, a query will look like: db.Products.Include(p => p.manufacturers).Include(p => p.parts).where(p => p.name == '…

Could you please stop creating new accounts for every few comments you post? We ban accounts that do this, and it's in the site guidelines: https://news.ycombinator.com/newsguidelines.html. It's particularly abusive that you used multiple accounts in this same thread.

HN is a community. Obviously you don't have to use your real name, but if users don't have some consistent identity for others to relate to, we may as well have no usernames and no community at all. That would be quite a different kind of forum. Anonymity is fine, and throwaways for a specific purpose are ok—just not routinely.

https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...

Re: How NoSQL forced the evolution of a scalable relational database

#118

Post by ben509 » Fri, Jul 13, 2018, 2:15 PM UTC

keithnz wrote: post #22

I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript. While scaling is one of the advantages, I'd be super surprised if many people actually need scaling capabilities ( other than because their design is super inefficient ). Recently I've been inspired to play around with kicking out as many layers between a relational db and a REST Api, largely because I've bee…

> I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript.

NoSQL caught on because of what it doesn't do: attempt to implement the relational model.

A lot of people were using MySQL in infamous LAMP stacks to set up websites with virtually no business logic.

So you had:

1. An application that didn't need the relational model 2. A DBMS that had a very poor implementation of SQL, which itself is a poor implementation of the relational model 3. A large community who didn't know anything about database theory.

When you're using MySQL as a glorified hashtable, you get none of the benefits of the relational model, and all the downsides.

From their understanding of the technology, throwing all that out made a lot of sense. It's pretty reasonable to use a distributed hashtable if what you want is a distributed hashtable.

I think the backlash now is that NoSQL marketed themselves as a capable alternative to SQL DBMSs. But they had no query optimizer, no transaction support, no reliable backups, and no schemas. The last was supposedly a selling point, but the reality is that your data always has a schema, and code will crash if the data doesn't match it.

Great marketing, but people were sold half a car and then told building the rest themselves was a feature.

> One of the problems of many stacks is that the frameworks wrap general purpose languages over SQL, which, is not really a good idea, SQL is a vastly more capable language for dealing with relational data and layers built over the top often dumb down the database.

Another problem is that object-orientation is fairly broken, and ORMs tried to figure out how to shoehorn relational databases into inheritance hierarchies. If current notions like value classes had been around, a more straightforward mapping would have been possible.

Re: How NoSQL forced the evolution of a scalable relational database

#119

Post by lmm » Fri, Jul 13, 2018, 2:19 PM UTC

Amezarak wrote: post #93

189523954 wrote: post #90

Earlier quoted context omitted.

>You know what's really slow? Creating joins in code. Ahh, you have combined two separate points into a new point that I didn't make and took it out of context. As I mentioned, I was writing queries for analytics software. This software needed fresh data once per day. There was no requirement for speed. I was referring to speed of development, not speed of the query. As for the other point, yes joins are slow, and ye…

> Nobody is using joins at scale and every large company that started their scaling journey in a SQL database started by performing multiple queries on a distributed database. I've used joins "at scale" in multiple jobs in (conventional relational) databases up to a few TB with tens of thousands of transactions per second and never experienced any performance problem that was the result of a join, unless you count cr…

> never experienced any performance problem that was the result of a join, unless you count cross joins with no filters.

An unindexed join will have major performance problems, and look exactly like an indexed join.

> I have never seen an instance where doing a join was faster in code and I don't see how it could be except perhaps in some unusual edge cases.

Faster to do the same thing? No. Easier to see which things are fast and which things are slow? Yes. Also easier to avoid the deadlocks that traditional databases' overzealous application of ACID can easily lead to.

> Can you go into more detail? In my experience, the "slow" part of writing any analytic query is deciding exactly what you want to know and making sure you understand that the data means what you think it means. Once you have that, the only thing slowing you down is your typing speed, and I don't think a more compact syntax would really make a difference.

I find the pseudo-English syntax of SQL is always very hard to follow - it slips into a kind of uncanny valley - the grammar of what goes where can be backwards from what I'd expect. Tooling is also rather limited compared to a "real" programming language. Just basic things like unit testing your queries are much harder than they should be.

Re: How NoSQL forced the evolution of a scalable relational database

#120

Post by uptownJimmy » Fri, Jul 13, 2018, 2:20 PM UTC

keithnz wrote: post #22

I think NoSql, especially things like Mongo, got popular because it is super easy to program with javascript. While scaling is one of the advantages, I'd be super surprised if many people actually need scaling capabilities ( other than because their design is super inefficient ). Recently I've been inspired to play around with kicking out as many layers between a relational db and a REST Api, largely because I've bee…

Dapper is a great tool for getting an app started and up to speed, and even later once the app gets bigger if sometimes it makes sense to bypass the ORM and hit the DB straight-on.