Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How NoSQL forced the evolution of a scalable relational database

blog.memsql.com

Post reply on HN
Print view View original

11–20 of 211 posts

Re: How NoSQL forced the evolution of a scalable relational database

#11

Post by pavbelshippable »

Perfect! We at Shippable moved from NoSQL MongoDB to PostgreSQL for several reasons. Here is a small story,

It started with small problems...

Even though we had the ability to add features at a lightening pace, we started seeing occasional downtimes which always seemed to come down to MongoDB. For instance:

> We were very happy to have 24x7 availability with primary and secondary instances of MongoDB. However, our perf suddenly deteriorated one day and retrieval started taking more than a second per document. We tried using many tools and profilers, but could not figure out what was happening. Finally, we rebuilt a new server, switched that over as primary, and rebuilt our secondary. Retrieval times dropped to 150ms again. This is still an unsolved mystery! > Our Mongo instance reached 4TB and we were proud of our growing adoption. Due to the lack of tooling around managing large DBs, we relied on indexes to keep the search times low. When NoSQL DBs first became popular, there was no way to create uniqueness, so these features were built as an afterthought. Some of the bloating of our MongoDB was actually due to indexes, but rebuilding them was primitive and the entire DB would lock down. > At one point, we needed to reboot our DB server and it took MongoDB 4 hours to come back online. This led to an extended downtime for our service, and we had very little visibility into the MongoDB process or status.

And then came the knockout punch! The biggest advantage, and disadvantage, of MongoDB is that it has a flexible schema. This means that documents in the same collection (aka table in the old world) do not need to have the same set of fields or structure, and common fields in a collection's documents may hold different types of data. In a nutshell, there are no strict schema rules and this opens it up to a lot of cowboy tinkering. While many developers love the flexibility, it also puts a very high degree of responsibility on their shoulders to get things right.

The whole experience we have written on our blog http://blog.shippable.com/why-we-moved-from-nosql-mongodb-to...

Re: How NoSQL forced the evolution of a scalable relational database

#12

Post by exclusiv »

logophobia wrote: post #6
stephenr wrote: post #3

> NoSQL came into existence because the databases at the time couldn’t handle the scale required. Arguably the first "NoSQL" database was a DIT, generally accessed via the LDAP protocol. OpenLDAP is one of the better known open source instances, but Novell eDirectory was there in the early part of this century, as was MS AD. In the case of Directory Services, it's a completely different approach to both SQL and the c…

I would say the the rise of nosql options is driven by the need for scalability by a few large companies that desperately need it, and cargo-culting by developers that don't actually need it, but want to be like google and don't want to take the time to properly learn and optimize sql. There's a genuine need for good scalable options, and with stuff like google spanner, those don't necessarily need to be "eventually…

It wasn't just devs that didn't know how to properly optimize SQL. Programming w a NoSQL db was a better experience because there was no object impedance mismatch.

ORMs fill the gap of course and SQL w JSON types are really nice. So I agree with your last paragraph but the rise was due to speed of db and speed for the developer.

I cranked out a lot of apps under short deadlines w MongoDB way back in the day. And I moved to Elasticsearch and CouchDB awhile ago. I'll still use those where appropriate but now in MariaDB for most. The competition was good.

Re: How NoSQL forced the evolution of a scalable relational database

#13

Post by elvinyung »

I feel like this topic was covered better by Stonebraker :p https://homes.cs.washington.edu/~billhowe/mapreduce_a_major_...

In the late 90s/early 00s Eric Brewer also wrote a bunch of papers about his experiences scaling stateful systems at Inktomi. The most famous one is probably the one that introduced/codified the CAP theorem: https://pdfs.semanticscholar.org/5015/8bc1a8a67295ab7bce0550...

Re: How NoSQL forced the evolution of a scalable relational database

#14

Post by siscia »

sqlcook wrote: post #9
rickn wrote: post #2

My name is Rick Negrin. I run the Product Management team at MemSQL, a scalable relational database. I recently wrote a blog on my thoughts regarding NoSQL vs. Relational Databases and I'd love to hear the community’s thoughts on this.

Hi Rick, I've been following Memsql for a few years now, are there any plans to release "community" edition? Last time I checked about 1.5 years ago json support was very basic and EE pricing (dont remember exact #s) was rather high. Thanks

What are you looking for in a "community" edition?

Re: How NoSQL forced the evolution of a scalable relational database

#15

Post by icebraining »

nostalgeek wrote: post #5

Well, this is exactly why competition is good, in every domain. That's why PostgreSQL got its column mode and JSON type, because there was a clear use case for performance for the former and schema-less data for the latter. EAV pattern is a plague and I'm glad documents are replacing it.

Postgres has had hstore for schema-less data since 2003.

Re: How NoSQL forced the evolution of a scalable relational database

#16

Post by ddorian43 »

siscia wrote: post #14
sqlcook wrote: post #9

Earlier quoted context omitted.

Hi Rick, I've been following Memsql for a few years now, are there any plans to release "community" edition? Last time I checked about 1.5 years ago json support was very basic and EE pricing (dont remember exact #s) was rather high. Thanks

What are you looking for in a "community" edition?

Something that you can use on production based on what your db advertises as it's advantages (high availability + sharding).

Re: How NoSQL forced the evolution of a scalable relational database

#17

Post by lyqwyd »

uncle_d wrote: post #10

My own experience chimes with this somewhat, having used MongoDB on my last project - quite often application data storage requirements are relatively trivial - and it is a boon to do away with the ORM layer and be able to vary any given object schema without touching the database (although in practice a .js data migration script may be involved, so this is moot). As a sidenote local cultural issue, the fact that we…

The fact that you could operate the mongo instance, but not the DB is definitely a cultural issue, or more likely a managerial issue. Definitely NOT a DB issue. I’ve worked at places where we could manage the DB ourselves, and other places where we couldn’t eve talk to the DBAs without beuraucracy. It’s all about the management in those scenarios. And the company results were as you would expect.

Re: How NoSQL forced the evolution of a scalable relational database

#18

Post by pjmlp »

mikekchar wrote: post #8
stephenr wrote: post #3

> NoSQL came into existence because the databases at the time couldn’t handle the scale required. Arguably the first "NoSQL" database was a DIT, generally accessed via the LDAP protocol. OpenLDAP is one of the better known open source instances, but Novell eDirectory was there in the early part of this century, as was MS AD. In the case of Directory Services, it's a completely different approach to both SQL and the c…

There were NoSQL databases well before LDAP was a thing. We called them "Object databases". The idea was that doing object relational mapping was a PITA and wouldn't it be better to store your domain objects directly in the database? (As an aside, in case you are curious, the answer is: No. Generally it's not a good idea) Of course if your data is not relational, you can't do SQL. :-)

Memories of POET and Gemstone spring to mind...

Re: How NoSQL forced the evolution of a scalable relational database

#19

Post by siscia »

ddorian43 wrote: post #16
siscia wrote: post #14

Earlier quoted context omitted.

What are you looking for in a "community" edition?

Something that you can use on production based on what your db advertises as it's advantages (high availability + sharding).

I was asking because I am the creator of RediSQL[1] -- SQL steroids for Redis -- which is a less sophisticated product than MemSQL but still has its own use cases.

And maybe for parent was enough, or if not it would be very interesting to know what is missing.

[1]: http://redbeardlab.tech/rediSQL/

Re: How NoSQL forced the evolution of a scalable relational database

#20

Post by sqlcook »

siscia wrote: post #14
sqlcook wrote: post #9

Earlier quoted context omitted.

Hi Rick, I've been following Memsql for a few years now, are there any plans to release "community" edition? Last time I checked about 1.5 years ago json support was very basic and EE pricing (dont remember exact #s) was rather high. Thanks

What are you looking for in a "community" edition?

EE features without support. I hate to bring up Mongo as example, but something similar...where support, additional software/plugins and cloud hosting are where the $ is made.

I did thorough testing of Memsql two years ago but went with Aurora instead. Would love to see how the product evolved since (Spark and Streaming integration was just being rolled out at the time), but something tells me pricing will be a deal breaker.

Post reply on HN