Live data from Hacker News

How NoSQL forced the evolution of a scalable relational database

blog.memsql.com

191–200 of 211 posts

Re: How NoSQL forced the evolution of a scalable relational database

#191

Earlier quoted context omitted.

It wasn't just devs that didn't know how to properly optimize SQL. Programming w a NoSQL db was a better experience because there was no object impedance mismatch. ORMs fill the gap of course and SQL w JSON types are really nice. So I agree with your last paragraph but the rise was due to speed of db and speed for the developer. I cranked out a lot of apps under short deadlines w MongoDB way back in the day. And I mo…

I've seen some of those applications. It might be easier & faster to start with a schema-less json database, but I saw some serious maintenance issues with applications like that: * Using the stored data outside the use cases imagined by the original developer is harder than it should be. (Oh you want a customer dashboard with this and this data correlated, ..) * "Migrations" can be rather error-prone if developers d…

Some apps aren't subject to new use-cases and some aren't meant to live that long or have another developer either. And SQL isn't great for the use-cases that solutions like Elasticsearch solves.

Migrations are less of an issue if you use an object document mapper which is just a leaner ORM. You can enforce your "schema" in your code while being schema-less. But yes if the type changes it could alter your queryability or you might have to write a script to update all records to reflect the new type. That can be a pain of course.

Viewing JSON collections in NoSQL is nice compared to SQL. Most your data is already composed as a sensible document and if you have subcollections the clients usually allow you to fold them or expand and show all. That's a win over SQL IMO.

Regarding JSON columns in SQL - I tend to store config/settings there and it's better than what it used to be for that stuff (serialized data or base64 encoded). In Navicat JSON columns are viewable in single record mode but pretty print format would be ideal.

Re: How NoSQL forced the evolution of a scalable relational database

#192

Earlier quoted context omitted.

Linq works for trivial left join project style of queries. But for anything actually using the features of modern sql-databases it comes short quickly. As a example try to express something like “sum(x) over (partition by y order by z)” I do agree that SQL the syntax leaves a lot to be desired, and a proper relational language with a syntax optimized for actual development, and even better, optimized for 6NF style da…

My understanding is everyone uses 3NF in practice though, so what would a 6NF database get you practically? I see SQL the way I see things like Linux: it's got some weird things that you'd design differently if you could do it over again, but overall it's pretty good and probably not worth the effort to switch.

6NF gives flexibility since every thing is decoupled until you bring it in. It makes evolving the schema easier, makes optimizations easier. I also have a hunch that with such a focus it would be easier to find an interesting design space for new relational programming models since it’s, in a sense, “purer”.

But I guess it depends on the use case. For normal oltp style save/fetch entity 3NF makes sense. But it kind of makes sense in the way an ORM would make sense. Which makes you question if you really need relational database at all.

Re: How NoSQL forced the evolution of a scalable relational database

#193

Earlier quoted context omitted.

It's been a pleasure working with MongoDB in TypeScript as well

I could see that. But I can’t tell from briefly perusing the docs, does Typescript give you transpile time checking of objects that it will allow you to put in Mongo? Using a dynamically type language with a schemaless database isn’t something I would want to do for a large project.

I think it depends on your implementation.

For example, I tend to use Mongoose as the interface to Mongo, with which you're required to set a Schema for each collection of documents. You can easily set every single field type to `Mixed` (analogous to `any`, `auto`, etc), but if you're the least bit disciplined you can set any fields to any variation of:

    * String
    * Number
    * Date
    * Buffer
    * Boolean
    * Mixed
    * ObjectId
    * Array
    * Decimal128
    * Map
(http://mongoosejs.com/docs/guide.html)

And there are TypeScript typings supplied for Mongoose, so I typically recreate the models as TS interfaces for wherever I use them in other parts of my code. It's extremely intuitive and quick to develop. It's a tad redundant defining the Schema types, and then a TS interface, but I usually don't mind. I include them in the same file for coherence so that I'm always importing from the same module for anything relating to the model.

For (a basic) example:

    /* MyCollection.ts */

    import {
        Document,
        Model,
        model,
        Schema,
        Types
    } from 'mongoose';
    
    export interface IMySchemaFields
    {
        _id?:   Types.ObjectId; // †
        name:   string;
        age?:   number;
        fungi:  boolean;
        ET?:    "phone home" | "owwww";
    }

    export type TMySchemaDocument = Document & IMySchemaFields;
 
    export type TMySchemaModel = Model;

    export const MySchema: Schema = new Schema(
        {
            _id:    Schema.Types.ObjectId // † NOTE
            name:   String,
            age:    Number,
            fungi:  Boolean,
            ET:     String,
        });

    export const MyCollection: TMySchemaModel = model('myCollection', MySchema);
† NOTE: there are a couple of quirks. Mongoose uses constructors to define types whereas TypeScript types are a different symbol understood only by the compiler— so in this case they use different aliases of the same object...

So when I move to make any queries via the ODM, I import and use the associated types. The types won't allow you to reference fields that don't exist on the schema that way. You can force it by cheating the compiler and casting any arbitrary non-existent field to ``, but I try to reserve that for hungover POC cases. It does suggest immutability of the object received from the query, but again— there are many ways to ignore that if you wanted to, or just didn't care.

Example ODM query:

    /* MyController.ts */

    import * as mongoose from 'mongoose';
    import {
        MyCollection,
        IMySchemaFields
    } from '../models/MyCollection.ts'

    /* ... pretend there's relevant stuff happening here */

        MyCollection.findOne({ _id: '436rgerr25gw4gfdhdsf', },
            (error: Error, collection: IMyCollectionFields, }) : any => {

             // Do something with your `collection` object.
             // Just don't try to access, modify, or add fields
             // that don't exist on the referenced interface.
        });

     /* More things probably happen here */

As for TypeScript for a large project... I think we get into different kinds of discussions here (eg, is Node or Deno the right tool, etc etc). The kind of applications I've used this design on have not had to face significant scaling or load problems. They're very applied circumstances that face small-to-medium/large traffic. I haven't benchmarked for anything that operated on a real significant scale. We will be testing for around the ~1k-2k concurrent user mark later this summer. But that doesn't involve consistent read/writes— it will involve regular reads, but seldom writes. I might try to write about it later, but it's kind of a fast-track thing so I'm a bit buried in it.

Other projects following a similar design tend to be internal tooling for editorial and other teams in media and so don't see significant traffic or structures/changes beyond initial versions.

I can understand many peoples' reservations and concerns. I definitely think it's a matter of what's right for the job, considering the circumstances and resources available.

Re: How NoSQL forced the evolution of a scalable relational database

#194
post #167

Earlier quoted context omitted.

I hired a couple junior devs and they did fine. There really aren't that many edge cases. If you're changing data, you're in a transaction and that's always strongly consistent. If you're querying data, nobody really cares if it's 10s behind (and usually it was 1-2s). On the other hand, counting a few thousand things is a pretty major PITA. And doing bulk updates requires map/reduce. So there's a cost... but it's not…

Some of your projects looks like are in e-commerce space so say if I look at my cart and it is missing products I added say 5 sec ago that would be an issue.

It's trivial to define your entity groups in Google Cloud Datastore such that you have strongly consistent carts. It's how you would naturally build an app, and it doesn't require http sessions (which I've never used on GAE and never expect to in the future). I've built multiple ecommerce sites this way and they work great.

What's hard is to do strongly consistent queries across carts. But that isn't something anyone cares about.

Re: How NoSQL forced the evolution of a scalable relational database

#195

NoSQL is an excellent technology for rapid R&D, but once a domain “settles”, data should be modeled and data stores should be switched to relational or graph backends. If a system already has a well-defined domain, NoSQL adds little value. Of course you could leverage AWS DynamoDB and reduce cost, but you still have downstream implications for things like reporting, which requires a known schema with relational parad…

I don’t see how NoSQL could be better for rapid dev when something like Postgres can migrate its schema and data at once while enforcing constraints. That’s clearly better precisely when your schema is changing.

In R&D scenarios, NoSQL wins hands down because devs are jamming schema changes hourly. That level of frequency in a relational model is a massive drag on time and it’s wasted.

If it’s a redevelopment of a well-known domain, depending on reporting requirements, I’d slightly side with relational.

But AWS DynamoDB can be a huge cost savings, so as any good architect will say, “It depends.”

Re: How NoSQL forced the evolution of a scalable relational database

#196

Earlier quoted context omitted.

I think you misunderstood the parents use of the term "model". All the things you describe here can be described in the relational model. The parent was arguing that the model is general enough to cover almost all collections of data. You appear to be arguing that for specific use cases specific implementations perform badly. These arguments pass each other unseen like ships in the night.

Actually wide tables for example can't be done at all. And technically we can use Microsoft Excel as a photo editor. But that's just ridiculous. Just like it's ridiculous to say you can use relational models for any data structure even if the queries will never return.

As I recall, MonetDB claims "practically unlimited" number of columns per table. And in this particular use case you're probably looking for an RDBMS specialized in analytics, not a general-purpose one like Postgres. But relational model will handle it certainly better than any other.

Re: How NoSQL forced the evolution of a scalable relational database

#197
post #115

Earlier quoted context omitted.

Same here, using Mongo helped us a lot to iterate fast, JSON is much more agile than SQL. We now migrate to PG using 'core' attributes in tables and 'flexible' attributes in JSONB to get the best of two worlds. I agree to the API problem, JDBC and PG/SQL is not as nice as a interace compared to Mongo libs.

When you say JSON is more agile is that because you don't need to write migrations? Do you use a framework that supports migrations like active record or south?

We do not use migrations with JSON.

Adding and removing happens on the application layer with using Option[_] monads in the data model. Add a field means add Option[_], removing a field means making e.g. String an Option[String] - or fill with default values on read. If we drop a field over time we ignore it during writes.

   case class Person(name:String)
adding a job field after some time becomes

   case class Person(name:String, job:Option[String])
and application code needs to deal with it.

Works for us, compared to my previous jobs where we used SQL and migrations.

Re: How NoSQL forced the evolution of a scalable relational database

#198
It's hard to understand the point of this article. First of all this is a good example of a completely biased ranting.

Secondly after that it's bashing NoSQL through many chapters later it is using those same features that NoSQL was capable of in the first place as a selling point for MemSQL... Seriously? This is a really biased marketing opinion piece at best.

Not to mention that it talks about new types of analytics but it completely ignores (or at least it doesn't mention any of) the category of low latency streaming analytics applications like Apache Storm, Apache Spark, Apache Kakfka et al.

> "To do this requires a new breed of analytics systems that can scale to hundreds of concurrent queries, deliver fast queries without pre-aggregation, and ingest data as it is created. On top of that, they want to expose data to customers and partners, requiring an operational SLA, security capabilities, performance, and scale not possible with current data stores"

Guess what! This is what streaming analytics was invented for.

Re: How NoSQL forced the evolution of a scalable relational database

#199
> "To give an analogy, imagine libraries saying they are doing away with the Dewey Decimal System and just throwing the books into a big hole in the ground and declaring it a better system because it is way less work for the librarians"

No, to give an analogy imagine a library where the librarian only accepts books which are below a given size, have a specific color and weight, and he rejects any books which have even a cm bigger size, a different color or it's just a gram heavier. Also from time to time (let's say each month) the librarians are sitting together to decide what other books shall they accept (sometimes extending (and by that inherently limiting) the acceptable ranges to other properties like title or smell). Also to give a good analogy the librarians also reject those books which are returned without their softcover. That's RDBMS for you. It works, but not everyone is satisfied with such a service.

Re: How NoSQL forced the evolution of a scalable relational database

#200

Perfect! We at Shippable moved from NoSQL MongoDB to PostgreSQL for several reasons. Here is a small story, It started with small problems... Even though we had the ability to add features at a lightening pace, we started seeing occasional downtimes which always seemed to come down to MongoDB. For instance: > We were very happy to have 24x7 availability with primary and secondary instances of MongoDB. However, our pe…

> "And then came the knockout punch!"

I am trying to understand what you say. So basically this was not clear for you from the beginning?

Post reply on HN