Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why some NoSQL DBs only let you perform transactions on a single data item

dbmsmusings.blogspot.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Why some NoSQL DBs only let you perform transactions on a single data item

#31

Post by beering »

fiatjaf wrote: post #16

What is the percentage of web applications that really need all that scaling? How many database instances does Hypothes.is need? What about Airbnb? Duolingo? Feedly?

A lot of services would be fine with some big traditional databases - StackOverflow is one example that scaled vertically instead of horizontally.

But a lot of services would also get by with a distributed DB with weak integrity guarantees. Does it really matter if a few upvotes get lost on a busy post? By the time you notice sporadic integrity issues in your DB, the startup will probably have failed or raised enough money to not care.

Re: Why some NoSQL DBs only let you perform transactions on a single data item

#32

Post by s_kilk »

zurn wrote: post #7
mooreds wrote: post #2

This was very helpful. I am in my first project with MongoDb after a lot of experience with RDBMs and I do miss transactions. There are other things I miss more: foreign keys (dobt use dbrefs!), joins, and SQL, though. Unfortunately this application is new, so we haven't yet seen the benefit of schemaless data changes.

PostgreSQL + JSON gets you working transactions and schemaless.

Shameless plug here, but my side-project (http://bedquiltdb.github.io) aims to provide a nice mongo-a-like programmatic API on top of PostgreSQL and JSONB.

In theory there's no reason why we shouldn't be able to support a nice transactions API in the client drivers too, I just haven't gotten around to it yet.

Re: Why some NoSQL DBs only let you perform transactions on a single data item

#33

Post by Roboprog »

Tenhundfeld wrote: post #26
mooreds wrote: post #2

This was very helpful. I am in my first project with MongoDb after a lot of experience with RDBMs and I do miss transactions. There are other things I miss more: foreign keys (dobt use dbrefs!), joins, and SQL, though. Unfortunately this application is new, so we haven't yet seen the benefit of schemaless data changes.

If this new application is something that will be in production and important to a business, I would seriously question the decision to use MongoDB over PostgreSQL. I'm not saying it's a bad decision or that MongoDB is bad in general. I'm saying it's a decision that should not be taken lightly. If you've done the analysis and feel strongly that MongoDB's the perfect fit, then cool. However, I think people routinely d…

Plus, you can make "hybrid" tables in PostgreSQL that have some fields in separate indexable primary and foreign key columns (or other uses, as needed), but remaining fields shoveled into a JSON CLOB "column".

The E/R information between tables remains exposed and enforced, but the content details can be made fungible where desired. There is an extension to include JSON fields in queries, as well, though it won't be as fast as using a real column with an index on it.

Re: Why some NoSQL DBs only let you perform transactions on a single data item

#34

Post by Roboprog »

Rezo wrote: post #11
radicalbyte wrote: post #9

Earlier quoted context omitted.

Plus it scales well and doesn't randomly lose data. Oh, and the project leaders are professionals who are honest and transparent about the capabilities of their product.

It scales, to a point. In my opinion however, any type of sharding/clustering is still a total mess in PostgreSQL. Most projects will however probably never need to.

How DARE you, sir, imply that we are not the next "Amazon"!

Kidding aside, I suspect with a little discipline, one could design a PG DB that could be ported to Oracle (Exadata???) should the need, and budget ($$$$$), for that sort of scaling arise. Cheap and safe now, Expensive and safe later (if needed)

Re: Why some NoSQL DBs only let you perform transactions on a single data item

#35

Post by Roboprog »

Rezo wrote: post #11
radicalbyte wrote: post #9

Earlier quoted context omitted.

Plus it scales well and doesn't randomly lose data. Oh, and the project leaders are professionals who are honest and transparent about the capabilities of their product.

It scales, to a point. In my opinion however, any type of sharding/clustering is still a total mess in PostgreSQL. Most projects will however probably never need to.

Also, I seem to recall that PG scales up on multiprocessor hardware fairly well (vs say MySQL), so one can explore exhausting that limit first before more drastic measures are taken.

Re: Why some NoSQL DBs only let you perform transactions on a single data item

#36

Post by jerven »

arosenbaum wrote: post #30

Misleading title: Abadi specifically called out particular NoSQL DB's, he wasn't generalizing to all NoSQL DB's. MarkLogic ( http://www.marklogic.com ) is an Enterprise Database used by many large organizations as a system of record for information traditionally stored in RDBMS or Mainframe. Joe Hellerstein gave a wonderful keynote at ACM SoCC last year that is a great read for all who are interested in this subject:…

And it's a graph database as well with sparql support. Maklogic is really flexible that way.

Re: Why some NoSQL DBs only let you perform transactions on a single data item

#37

Post by joeclark77 »

mooreds wrote: post #2

This was very helpful. I am in my first project with MongoDb after a lot of experience with RDBMs and I do miss transactions. There are other things I miss more: foreign keys (dobt use dbrefs!), joins, and SQL, though. Unfortunately this application is new, so we haven't yet seen the benefit of schemaless data changes.

The idea is that you won't need database-wide transactions if your data is structured into "natural aggregates" such as orders, articles, profiles, etc. In a relational database, something as simple as an "order" has components in lots of different tables -- customer, product, order, line_item, shipping_method, etc -- so you have to lock all of those tables when doing an INSERT or UPDATE.

In the ideal MongoDB implementation, an "order" would be a document of its own, and all the other parts would be attributes or nested structures. So you only really need a transaction for the document itself. Other documents could be inserted/updated at the same time without creating a conflict. That's the theory, at least. Martin Fowler has a good Youtube on "Introduction to NoSQL" that I sometimes show to my students: https://youtu.be/qI_g07C_Q5I

The other thing about NoSQL is that it's often used for write-once, then read-only applications. Twitter, Facebook, or Instagram would be good examples. When 1% of your queries are INSERTs and 99% are SELECTS (with UPDATE and DELETE almost never occurring), you really don't run as much risk of anomalies. I would not recommend NoSQL for something like a banking application where you're doing lots of updates and need to guarantee consistency.

Re: Why some NoSQL DBs only let you perform transactions on a single data item

#38

Post by shin_lao »

eis wrote: post #20
shin_lao wrote: post #15

Sorry to say but we have something that is truly transactional and distributed ( https://www.quasardb.net ). It slows down a bit writes (because of the commit) but it scales well. And before us FoundationDB delivered the same feature (although implemented a bit differently). The truth is that there is a market for databases with "unreliable" writes and there are easier to design. That's why you see a lot of them.

What are you using LevelDB for? I hope not the main KV store. Do you use 2PC or something like Raft to handle distributed transactions? From the docs it sounds like everytime a node joins/leaves the cluster goes into a (brief?) unstable mode and failures can happen, that doesn't sound great.

I'm happy to see you've read the documentation thoroughly!

We indeed stopped using LevelDB. It has been used for a while but we couldn't work around some speed issues. LevelDB is good for many scenarii when properly configured, though.

The database is multilayered, LevelDB was the last layer, most parallel operation were managed by a massively parallel in-memory database which used LevelDB as a backend.

We do 2PC to handle distributed transactions. Raft breaks our consistency model.

When a node joins/leaves the cluster you might have some transient failures which are generally absorbed by the protocol.

I hope I answered your questions and feel free to give it a spin next week, we're about to release a major update!

Re: Why some NoSQL DBs only let you perform transactions on a single data item

#39

Post by teraflop »

eis wrote: post #20
shin_lao wrote: post #15

Sorry to say but we have something that is truly transactional and distributed ( https://www.quasardb.net ). It slows down a bit writes (because of the commit) but it scales well. And before us FoundationDB delivered the same feature (although implemented a bit differently). The truth is that there is a market for databases with "unreliable" writes and there are easier to design. That's why you see a lot of them.

What are you using LevelDB for? I hope not the main KV store. Do you use 2PC or something like Raft to handle distributed transactions? From the docs it sounds like everytime a node joins/leaves the cluster goes into a (brief?) unstable mode and failures can happen, that doesn't sound great.

Is there something particularly wrong with LevelDB? It may not have all of the bells and whistles you could possibly want from an on-disk key-value store, but it seems like a perfectly serviceable building block to put a database on top of.
Post reply on HN