Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The thrill of a new technology: CouchDB

tempe.st

Post reply on HN
Print view View original

21–30 of 48 posts

Re: The thrill of a new technology: CouchDB

#21

Post by jrockway »

moe wrote: post #14
jrockway wrote: post #9

No, graph databases will rule the world. Couch only supports trees, which is annoying if you want to have relationships between "documents". Also, Couch's implementation of map/reduce arbitrarily limits the kinds of queries you can run. If you actually want to ditch your relational database, take a look at things like KiokuDB, Elephant, AllegroCache, and so on. They may not have exciting web 2.0 screencasts, but the…

Well, just to state the obvious, no particular database will "rule the world". They are different tools for different tasks. RDMBS will not go away in data warehousing for a while. Key/Value stores are useful for many webapps where they match the access pattern. Graph DBs have yet their own area of application. There is not "one tool to rule them all". Albeit it would be an interesting project to wrap up all those en…

Well, just to state the obvious, no particular database will "rule the world".

I don't think this either, I was just parodying the overly-editorialized title.

Re: The thrill of a new technology: CouchDB

#22

Post by mattmcknight »

I have run into serious issues with the document based approach. Imagine a domain where you have several different types of documents. You still end up doing joins. In the example the author gives, doing a query to get you all of the unique tags can be slow. Or, if they are dates, trying to get a max(date) can be tricky. The biggest problem though, is document size. For a complex domain, there are big tradeoffs between keeping the entire graph in the document, which is slow to read, or creating a variety of document types, which brings you back to joining.

A hybrid approach can give you a little of both. For most of the kind of applications I build, I use relational databases, but I dump a denormalized view of the data into Lucene for searching and such and use a caching layer for most reads. This way I get some document type performance on many operations, but still have the normalization and simple transactions of a relational model.

Re: The thrill of a new technology: CouchDB

#23

Post by jrockway »

dasil003 wrote: post #19
jrockway wrote: post #15

Earlier quoted context omitted.

This is probably why it isn't ruling the world. Technical merit and popularity are only occasionally related in the computer world, so I won't comment on this. I will elaborate further on the use case of graphs versus a relational database. Basically, I see relational databases as especially useful for cases where you have a big pile of data, and have no idea what it really means. You do queries to learn what you hav…

I sort of see where you're coming from in terms of the parity mismatch between RDBMSs and typical application code. Certainly we can avoid some coding headaches by just dumping things into a data store that is optimized for what we want to do with that data right now. But then you say that most applications have a well-defined data model and rarely run "queries". This is where I think you've gone terribly terribly wr…

Well, let's think about this in more detail.

Let's say you have two types of data, customers and orders. Customers have many orders, an order belongs to a customer. This is easy enough with a typical relational database. You have a customers table and an orders table. You can join the tables and ask questions like "how many customers spent more than $300 last year?"

Now let's consider the graph/object database equivalent. You have two classes, Order and Customer. A Customer has a set of Orders, and the Order has a customer. (Cycles are fine, this is a graph, not a tree.) Creating an order works with some method like $customer->new_order_for('some pants'). You store this in the database, and the graph structure is stored and indexed. (Usually, object databases index on class name, but you can always specify other conditions. This makes it basically equivalent to the relational database.) Note that this structure works very well; the in-memory relationship is the same as the in-storage relationship. You can also write the same query as with the relational database. Get all customers, find their orders from this year, and sum the totals. (Instead of writing SQL, you would just write a script here. You can index things like the order year to speed up the query, as well. Otherwise, it's O(n), but so is the relational database without an index. There is no magic after all.)

Anyway, there is no lack of flexibility with the graph database. If you want to query your data, you can. It's just less convenient, since you have to write a program to do it, instead of letting your database management engine do it. (This is actually not true in general, AllegroGraph has a querying engine based on prolog.)

Back when I did data warehousing, we had to move all our data from the web app servers to a warehousing server in a specialized schema so that some GUI software could manipulate the data. Even though we used a relational database, we had to convert anyway. Using an object database would have made the app code simpler, and the warehousing code equally complex. So I think that would be a gain, not a loss.

Re: The thrill of a new technology: CouchDB

#24

Post by hendler »

jrockway wrote: post #9

No, graph databases will rule the world. Couch only supports trees, which is annoying if you want to have relationships between "documents". Also, Couch's implementation of map/reduce arbitrarily limits the kinds of queries you can run. If you actually want to ditch your relational database, take a look at things like KiokuDB, Elephant, AllegroCache, and so on. They may not have exciting web 2.0 screencasts, but the…

I agree, in principle that the most dominant, important technology will be graph databases. The reason I think so is because of experience working in technologies related to the semantic web.

However, there are reasons one might a Distributed Document Store to implement a graph store - it's the index that matters. MySQL is not the same thing as InnoDB/Sphynx. Graphdb product X could be built on mapreduce +CouchDB+Lucene(+somegraphindex) .

The only thing I'm sure of, is that if anything stands out above the rest, Oracle and IBM (and maybe MS) will try to buy it.

Re: The thrill of a new technology: CouchDB

#25

Post by dagheti »

jrockway wrote: post #23
dasil003 wrote: post #19

Earlier quoted context omitted.

I sort of see where you're coming from in terms of the parity mismatch between RDBMSs and typical application code. Certainly we can avoid some coding headaches by just dumping things into a data store that is optimized for what we want to do with that data right now. But then you say that most applications have a well-defined data model and rarely run "queries". This is where I think you've gone terribly terribly wr…

Well, let's think about this in more detail. Let's say you have two types of data, customers and orders. Customers have many orders, an order belongs to a customer. This is easy enough with a typical relational database. You have a customers table and an orders table. You can join the tables and ask questions like "how many customers spent more than $300 last year?" Now let's consider the graph/object database equiva…

The flexibility of the relational model is built on its bare simplicity: values + sets + logic. If you propose adding to this and giving up the benefits provided by these simplifications (simplicity of reasoning, ability to change the physical implementation without affecting the logical one [those not-magic indexes], declarative constraints, declarative queries), you need to have a really good reason.

The network model if it means anything is letting you add pointers to the mix and changing how you reason about your data from sets to graphs. This makes it harder to declare constraints, check integrity, update sets of data, access your data in different ways, and reason about your queries. An imperative script is in no way as safe a program as a declarative query.

The most clear benefit in my view to the network model it lets you use your object code fairly seamlessly. Ok.

The problem is that this is a good trade off for programs where you don't care primarily about your data, but about your code. I'd argue that isn't true for the majority of programs that have databases at all. The issues that kill your system years down the line and give you nightmares are not code issues, they are data issues. And the code you write to fix those problems... will end up re-implementing all those annoying "heavy" bits of RDBMSs that coders seems to hate.

Re: The thrill of a new technology: CouchDB

#26

Post by jrockway »

dagheti wrote: post #25
jrockway wrote: post #23

Earlier quoted context omitted.

Well, let's think about this in more detail. Let's say you have two types of data, customers and orders. Customers have many orders, an order belongs to a customer. This is easy enough with a typical relational database. You have a customers table and an orders table. You can join the tables and ask questions like "how many customers spent more than $300 last year?" Now let's consider the graph/object database equiva…

The flexibility of the relational model is built on its bare simplicity: values + sets + logic. If you propose adding to this and giving up the benefits provided by these simplifications (simplicity of reasoning, ability to change the physical implementation without affecting the logical one [those not-magic indexes], declarative constraints, declarative queries), you need to have a really good reason. The network mo…

And the code you write to fix those problems... will end up re-implementing all those annoying "heavy" bits of RDBMSs that coders seems to hate.

Or you won't. I have many important KiokuDB applications in production. And they are not toy Web 2.0 things, they are important sites that perform important data analyses. I don't think any of the logic I had to write to support was particualarly difficult, and it was fewer lines of code than I would need to define my ORM classes.

The flexibility of the relational model is built on its bare simplicity: values + sets + logic.

Simplicity is good. However, software is complex, and complexity has to live somewhere. Look at git, for example. The underlying model is simple and beautiful, blobs, trees, and commits. Wonderful. But, to make that beautiful model into a revision control system, thousands of lines of code had to be written. So while simplicity makes the bottom part of the system simpler, it didn't do much for the overall simplicity of the entire system.

I feel the same way about object databases. They are more complex than a relational database (unless you count things like replication and embedded scripting and ... as relational database features, which I don't), but they help me decrease the complexity of the code I see.

As an example, when I use an RDBMS, I have to maintain:

SQL schema + upgrades

ORM classes

Logic classes (for abstractions over multiple tables or data stores)

Random scripts to query the database and give me munged reports

With an object database, I only have to write the logic classes and the random scripts. The random scripts are slightly more complicated, but not significantly more. The main app is much simpler, and much easier to test. I also don't have to hack my data model into the relational model. (Ever represent a tree in a relational database? It's a hack.)

Re: The thrill of a new technology: CouchDB

#27

Post by Tichy »

jshen wrote: post #16
Tichy wrote: post #3

Hm, flat files will rule the world? No more messy SQL, just iterate over the file's entries with a for loop - even Java programmers can understand that. Sorry, but I am not convinced yet.

You may not be convinced yet, nor am I, but your strawman argument makes you look ... well a little silly.

That is the way I understood it. What do you mean by "it creates an index for each view"? I just would like to avoid reinventing the wheel, and relational databases abstract away some of the problems of dealing with large data sets.

Edit: OK, read the CouchDB introduction now. Did not know that the "views" are server side. I have a past as a Java developer, and I have actually come across the "select all, then compute in Java" pattern several times in the real world.

Re: The thrill of a new technology: CouchDB

#28

Post by dagheti »

jrockway wrote: post #26
dagheti wrote: post #25

Earlier quoted context omitted.

The flexibility of the relational model is built on its bare simplicity: values + sets + logic. If you propose adding to this and giving up the benefits provided by these simplifications (simplicity of reasoning, ability to change the physical implementation without affecting the logical one [those not-magic indexes], declarative constraints, declarative queries), you need to have a really good reason. The network mo…

And the code you write to fix those problems... will end up re-implementing all those annoying "heavy" bits of RDBMSs that coders seems to hate. Or you won't. I have many important KiokuDB applications in production. And they are not toy Web 2.0 things, they are important sites that perform important data analyses. I don't think any of the logic I had to write to support was particualarly difficult, and it was fewer…

Ok so let's clarify this:

(SQL schema & upgrades + ORM classes + Logic classes, random scripts) - (logic classes + random scripts) = SQL Schema & upgrades + ORM classes

The question then becomes how do you handle the trade-off between being able to declare constraints (that evil schema) and the necessity to "map" how you access your data to your programming language.

If you "program" your constraints in your host language, you will need to either recreate the declarative system provided by a RDBMs or else create a system that will be fragile. Sure your code protects you against inserting bad data right? Well what about updating? What about when you delete? What if your constraint references another value in your database? What if that one changes? Integrity is best declared, not guarded by your program.

If your network model implements declarative constraints, then how do you define them? Those "random scripts" again? Suddenly all those performance benefits of navigation disspear when you're updating that data and checking those constraints.

I've seen many improtant production systems that used the network data model and as a warning: it usually ends badly. Be it COBOL, or whatever fresh re-invention of that navigation database wheel.

Re: The thrill of a new technology: CouchDB

#29

Post by jshen »

Tichy wrote: post #27
jshen wrote: post #16

Earlier quoted context omitted.

You may not be convinced yet, nor am I, but your strawman argument makes you look ... well a little silly.

That is the way I understood it. What do you mean by "it creates an index for each view"? I just would like to avoid reinventing the wheel, and relational databases abstract away some of the problems of dealing with large data sets. Edit: OK, read the CouchDB introduction now. Did not know that the "views" are server side. I have a past as a Java developer, and I have actually come across the "select all, then comput…

You should phrase your comments in the form of a question rather than passing judgement on things you haven't looked into.

Re: The thrill of a new technology: CouchDB

#30

Post by Tichy »

intinig wrote: post #4
Tichy wrote: post #3

Hm, flat files will rule the world? No more messy SQL, just iterate over the file's entries with a for loop - even Java programmers can understand that. Sorry, but I am not convinced yet.

Tichy, give it a spin. The ease of execution (and the sheer speed, even over giant amounts of data) is impressive.

Reading CouchDB documentation now and can't stop reading. It sounds like a lot of fun. Haven't understood yet how to handle the performance problems, but would like to use it just for the fun of it.
Post reply on HN