Why SQL is beating NoSQL, and what this means for the future of data • Hacker News Board

Re: Why SQL is beating NoSQL, and what this means for the future of data

#101

Post by anarazel » Tue, Sep 26, 2017, 4:46 AM UTC

pjungwir wrote: post #73

craigkerstiens wrote: post #53

I don't think the article could have said it much better. SQL is super powerful and makes much sense in so many ways. Nearly all apps have a relational structure to them and SQL is a reasonable way to interact with them. Some of my favorite conversations from the Postgres community 5-6 years back were when they were talking about a time when Postgres was being disrupted. The gray bearded DBAs (Hi Berkus and others) w…

Hi Craig! Postgres refers to itself as an "object-relational database", but do you think the "object" part is meaningful today, or just a label left over from the 90s? As best as I can tell, it just means that Postgres has table inheritance, which is great for partitioning but has never been all that feature-ful. (I guess it is improving a lot with pg 10 though!) Is there any theory or academic research on what an "o…

> Postgres refers to itself as an "object-relational database", but do you think the "object" part is meaningful today, or just a label left over from the 90s?

I think it's largely a leftover - the things that made it "object" back then are partially not used much, partially deprecated, partially removed, partially repurposed (inheritance as a partitioning mechanism).

You could perhaps just do a s/object/multi-modal with a focus on/. But that's a bit awkward ;). There's a bit document store (json, xml), arguably some graph (recursive queries), plenty people just use it for KV, ...

Re: Why SQL is beating NoSQL, and what this means for the future of data

#102

Post by ianamartin » Tue, Sep 26, 2017, 4:53 AM UTC

ChicagoDave wrote: post #87

curun1r wrote: post #75

Earlier quoted context omitted.

> Reports require a static schema This may be true, but reports also normally require a different schema from the OLTP database. Data warehousing with star schemas is a thing and it doesn't matter whether your OLTP database is an RDBMS or a NoSQL solution...you're going to have to push data to your warehouse to run those reports. The last thing you want to be doing is running reporting queries against a database that…

The level of effort to move data from DynamoDB to a data-warehouse (which is what Redshift is), is not trivial. It may be that the Data Pipeline tools within AWS are still opaque and incomplete, but but I've seen the same problem with MongoDB implementations. Reporting is nearly always an afterthought in system architecture and it should be a first-class requirement.

The reason that moving data from a transactional platform to a warehouse platform is not trivial is because you actually have to have ETL processes in place to transform data from one architecture to another.

A data warehouse has nothing to do with the mechanism that stores it. It is a design that comes with its own challenges just like transactional systems do. Transactional systems are optimized for writing. Warehouses are optimized for reading. They way you choose to execute those optimizations is up to you. But getting data from one of those systems to the other is never going to be trivial, and there is no general purpose tool to get things from one format to another because the warehouse structure is too entity-specific to generalize beyond a certain point (and that point of generalization is really the process of creating the design. See the Kimball group's books for more information.)

Redshift is not a data warehouse anymore than Postgres is a data warehouse. In fact, Redshift is a fork of Postgres that leaves out the functionality, data types, and indexes that make it difficult to scale horizontally. Postgres also isn't a data warehouse. Nor is it transactional system. It's just a tool that you can use to build either one of those things with.

Re: Why SQL is beating NoSQL, and what this means for the future of data

#103

Post by unclebucknasty » Tue, Sep 26, 2017, 4:55 AM UTC

craigkerstiens wrote: post #53

I don't think the article could have said it much better. SQL is super powerful and makes much sense in so many ways. Nearly all apps have a relational structure to them and SQL is a reasonable way to interact with them. Some of my favorite conversations from the Postgres community 5-6 years back were when they were talking about a time when Postgres was being disrupted. The gray bearded DBAs (Hi Berkus and others) w…

I think I'm one of those graybeards.

I see it in so many things tech. It's a pattern, and once you've seen it repeat a half-dozen times and also gain a depth of experience over that time, you can actually recognize when something represents genuine progress vs yet another passing fad. Spoiler alert: those that are most rabidly promoted are often the latter.

But, if you try to raise the point in the midst of the latest fad, you generally get shouted down. So, you wait until the less-jaded figure it out...again. It was plainly obvious for NoSQl, just as it now is for SPAs (or at least our current approach).

Don't believe me? Wait 5 years.

And, the thing with SQL that's so much like other tech is that we largely figured it out years ago. Improvements have been incremental at best and, frequently, not improvements at all.

Re: Why SQL is beating NoSQL, and what this means for the future of data

#104

Post by craigkerstiens » Tue, Sep 26, 2017, 4:59 AM UTC

unclebucknasty wrote: post #103

craigkerstiens wrote: post #53

I don't think the article could have said it much better. SQL is super powerful and makes much sense in so many ways. Nearly all apps have a relational structure to them and SQL is a reasonable way to interact with them. Some of my favorite conversations from the Postgres community 5-6 years back were when they were talking about a time when Postgres was being disrupted. The gray bearded DBAs (Hi Berkus and others) w…

I think I'm one of those graybeards. I see it in so many things tech. It's a pattern, and once you've seen it repeat a half-dozen times and also gain a depth of experience over that time, you can actually recognize when something represents genuine progress vs yet another passing fad. Spoiler alert: those that are most rabidly promoted are often the latter. But, if you try to raise the point in the midst of the lates…

Oh, I generally agree.

I didn't mean the graybeard in a bad way at all.

I feel a bit 50-50 on some of the reactions to JSON as a new thing at the time. But, also the practicality of how JSONB finally came to exist, the foundation it was built on and the new index types[1] in Postgres to leverage it felt like it came in the right time and in the right way.

[1] http://www.craigkerstiens.com/2017/03/12/getting-started-wit...

Re: Why SQL is beating NoSQL, and what this means for the future of data

#105

Post by manigandham » Tue, Sep 26, 2017, 5:00 AM UTC

There's really no such thing as NoSQL first of all. Relational, graph, key/value (including wide column), document-store, etc. There are lots of database types but it turns out relational works 95% of the time and we're getting better at recognizing the use-cases for the others.

SQL is also just a query language, that's literally the name. Any database can implement it, not just relational. Is SQL a great interface for both OLTP and OLAP? Yes, it's proven itself over decades. Is it the only valid interface? No. Does it work with other newer/different data systems? Yes, Spark, Hadoop, Kafka, and even these new distributed relational databases are all examples of such.

It would be far better for the industry if we can get past these ideological and faulty "debates" and move on to more important things.

Re: Why SQL is beating NoSQL, and what this means for the future of data

#106

Post by takeda » Tue, Sep 26, 2017, 5:02 AM UTC

Great article, but I think the author puts too much emphasis on SQL instead on the fact that the data being stored in a relational way.

Personally I actually would like an alternative language that would be capable to be integrated with the language in such way that it could also fall with type checking.

So for example if I rename a column in a database, the type checker would highlight all places in my code that were broken by this change.

JOOQ[1] seems to do something like that, but it's only for Java.

Also, looks like QUEL[2] would be a bit easier to be integrated with a language, too bad it died.

[1] http://www.jooq.org/

[2] https://en.wikipedia.org/wiki/QUEL_query_languages

Re: Why SQL is beating NoSQL, and what this means for the future of data

#107

Post by wjossey » Tue, Sep 26, 2017, 5:04 AM UTC

cachemiss wrote: post #57

wjossey wrote: post #22

Earlier quoted context omitted.

As someone who helped design, then manage a massive NoSQL store which hundreds of terabytes of data, of which most of it was hot, I’ll humbly disagree and say it’s not necessarily as bad as you make it sound. Did we have someone who managed that system? Sometimes, but mostly it just did it’s own thing. We ironically invested way more time on our MySQL database over the years because we couldn’t get that to scale the…

Caveat, this comment isn't directed at you (I agree with your comment), but rather the points around what you are saying. One thing that helps is if people stop referring to things as SQL / NoSQL as what ends up happening is various things get conflated. When talking about stores, it's important to be explicit about a few things: 1. Storage model 2. Distribution model 3. Access model 4. Transaction model 5. Maturity…

Wonderful response. Thank you. Wish I could give multiple upvotes. I’ll add some of those points to my thought process going forward.

Re: Why SQL is beating NoSQL, and what this means for the future of data

#108

Post by wolco » Tue, Sep 26, 2017, 5:09 AM UTC

craigkerstiens wrote: post #104

unclebucknasty wrote: post #103

Earlier quoted context omitted.

I think I'm one of those graybeards. I see it in so many things tech. It's a pattern, and once you've seen it repeat a half-dozen times and also gain a depth of experience over that time, you can actually recognize when something represents genuine progress vs yet another passing fad. Spoiler alert: those that are most rabidly promoted are often the latter. But, if you try to raise the point in the midst of the lates…

Oh, I generally agree. I didn't mean the graybeard in a bad way at all. I feel a bit 50-50 on some of the reactions to JSON as a new thing at the time. But, also the practicality of how JSONB finally came to exist, the foundation it was built on and the new index types[1] in Postgres to leverage it felt like it came in the right time and in the right way. [1] http://www.craigkerstiens.com/2017/03/12/getting-started-w…

It is greybeard or graybeard?

Re: Why SQL is beating NoSQL, and what this means for the future of data

#109

Post by labster » Tue, Sep 26, 2017, 5:12 AM UTC

If SQL is to databases as Javascript is to browsers — ubiquitous and largely standard across the market — are there any languages that transpile to SQL?

People in this thread have commented that SQL is clunky, and JS definitely fit that description for a while. So I'm wondering if there are any alternatives that would prove SQL's clunkiness to me.

Re: Why SQL is beating NoSQL, and what this means for the future of data

#110

Post by ww520 » Tue, Sep 26, 2017, 5:12 AM UTC

pjungwir wrote: post #73

craigkerstiens wrote: post #53

I don't think the article could have said it much better. SQL is super powerful and makes much sense in so many ways. Nearly all apps have a relational structure to them and SQL is a reasonable way to interact with them. Some of my favorite conversations from the Postgres community 5-6 years back were when they were talking about a time when Postgres was being disrupted. The gray bearded DBAs (Hi Berkus and others) w…

Hi Craig! Postgres refers to itself as an "object-relational database", but do you think the "object" part is meaningful today, or just a label left over from the 90s? As best as I can tell, it just means that Postgres has table inheritance, which is great for partitioning but has never been all that feature-ful. (I guess it is improving a lot with pg 10 though!) Is there any theory or academic research on what an "o…

Mike Stonebraker started Ingres to build a relational database. After Ingres, he and Larry Rowe wanted to see what could be done to add the OO aspect and other complex data types in addition to the relational model; OO was starting to take off back then. They started the Postgres project (Post-Ingres) in UCB to research on object-relational database. It turned out the object-relational part was better done on the client side, evidenced with the legions of client side object-relational libraries. The relational part sticks around and becomes the better part of the PostgreSQL database.

Edit: the object part was to build complex data types from simple type in the relational model, and had the database to manage the storage and querying of the complex object. E.g. build Location object from (x: int, y: int), and array of Location[].