Live data from Hacker News

Rich Hickey: Deconstructing the Database

youtube.com

51–60 of 93 posts

Re: Rich Hickey: Deconstructing the Database

#51
post #9

I recall Datomic making a bit of a splash on HN when it was announced 6+ months ago, but basically crickets since then. Anybody build something cool that took advantage of Datomic's unique design?

People tend to be more conservative about data than they are about the other parts of their stack -- probably for a good reason. I don't think datomic (or it's kin) will have that huge of an influence until years from now.

If you want to be conservative about your data, it looks like Datomic is an excellent choice. It preserves the entire history of your data instead of allowing it to be modified in place.

Re: Rich Hickey: Deconstructing the Database

#52
post #51

Earlier quoted context omitted.

People tend to be more conservative about data than they are about the other parts of their stack -- probably for a good reason. I don't think datomic (or it's kin) will have that huge of an influence until years from now.

If you want to be conservative about your data, it looks like Datomic is an excellent choice. It preserves the entire history of your data instead of allowing it to be modified in place.

Conservative generally means using tried and tested technologies that has run stably for years on thousands of servers and where every corner case is well documented and well understood. Not using some new largely untested technology that sounds awesome on paper, but has yet to truly prove itself in the real world.

Disclaimer: I'm a huge fan of clojure and Hickey and I think Datomic sounds amazing. I'm currently looking for an excuse to play with Datomic and learn more about it, so don't take what I said as any indication that I'm somehow against Datomic.

Re: Rich Hickey: Deconstructing the Database

#53
Watching this talk has so far (I'm halfway through, and now giving up) been very disappointing, primarily because many of the features and implementation details ascribed to "traditional databases" are not true of the common modern SQL databases, and almost none of them are true of PostgreSQL. As an initial trivial example, many database systems allow you to store arrays. In the case of PostgreSQL, you can have quite complex data types, from dictionaries and trees to JSON, or even whatever else you want to come up with, as it is a runtime extensible system.

However, it really gets much deeper than these kinds of surface details. As a much more bothersome example that is quite fundamental to the point he seems to be taking with this talk, at about 15:30 he seriously says "in general, that is an update-in-place model", and then has multiple slides about the problems of this data storage model. Yet, modern databases don't do this. Even MySQL doesn't do this (anymore). Instead, modern databases use MVCC, which involves storing all historical versions of the data for at least some time; in PostgreSQL, this could be a very long time (when a manual VACUUM occurs; if you want to store things forever, this can be arranged ;P).

http://en.wikipedia.org/wiki/Multiversion_concurrency_contro...

This MVCC model thereby directly solves one of the key problems he spends quite a bit of time at the beginning of his talk attempting to motivate: that multiple round-trips to the server are unable to get cohesive state; in actuality, you can easily get consistent state from these multiple queries, as within a single transaction (which, for the record, is very cheap under MVCC if you are just reading things) almost all modern databases (Oracle, PostgreSQL, MySQL...) will give you an immutable snapshot of what the database looked like when you started your transaction. The situation is actually only getting better and more efficient (I recommend looking at PostgreSQL 9.2's serializable snapshot isolation).

At ~20:00, he then describes the storage model he is proposing, and keys in on how important storing time is in a database; the point is also made that storing a timestamp isn't enough: that the goal should be to store a transaction identifier... but again, this is how PostgreSQL already stores its data: every version (as again: it doesn't delete data the way Rich believes it does) stores the transaction range that it is valid for. The only difference between existing SQL solutions and Rich's ideal is that it happens per row instead of per individual field (which could easily be modeled, and is simply less efficient).

Now, the point he makes at ~24:00 actually has some merit: that you can't easily look up this information using the presented interfaces of databases. However, if I wanted to hack that feature into PostgreSQL, it would be quite simple, as the fundamental data model is already what he wants: so much so that the indexes are still indexing the dead data, so I could not only provide a hacked up feature to query the past but I could actually do so efficiently. Talking about transactions is even already simple: you can get the identifier of a transaction using txid_current() (and look up other running transactions if you must using info tables; the aforementioned per-row transaction visibility range is even already accessible as magic xmin and xmax columns on every table).

Re: Rich Hickey: Deconstructing the Database

#54
post #53

Watching this talk has so far (I'm halfway through, and now giving up) been very disappointing, primarily because many of the features and implementation details ascribed to "traditional databases" are not true of the common modern SQL databases, and almost none of them are true of PostgreSQL. As an initial trivial example, many database systems allow you to store arrays. In the case of PostgreSQL, you can have quite…

In another talk he addresses your point specifically. He said, and I'm paraphrasing:

"It doesn't matter if you're using append only data-structures if your view of the world is update in place".

PostgreSQL exposes a view to the world of an update in place database, no matter what it's doing underneath. You could create a new interface to PostgreSQL's internals that doesn't and if you did, it would look a lot like datomic.

Re: Rich Hickey: Deconstructing the Database

#55
post #53

Watching this talk has so far (I'm halfway through, and now giving up) been very disappointing, primarily because many of the features and implementation details ascribed to "traditional databases" are not true of the common modern SQL databases, and almost none of them are true of PostgreSQL. As an initial trivial example, many database systems allow you to store arrays. In the case of PostgreSQL, you can have quite…

In another talk he addresses your point specifically. He said, and I'm paraphrasing: "It doesn't matter if you're using append only data-structures if your view of the world is update in place". PostgreSQL exposes a view to the world of an update in place database, no matter what it's doing underneath. You could create a new interface to PostgreSQL's internals that doesn't and if you did, it would look a lot like dat…

First, there is a major difference between MVCC and update-in-place that you can detect as a client, and that difference is that the problems that Rich outlines at the beginning of his talk do not happen: if one client edits something in the database, other transactions do not get an inconsistent view because the data on disk has already been permanently and irrevocably "updated in place". (Which, to be clear, means that modern SQL databases do not "expose a view to the world of an update in place database".)

Second, if all that is required to get his model is to add a command to an existing database (such as PostgreSQL, as I feel I know enough about how it works to be confident that this would be a reasonably simple task) "mark the current transaction read-only and pretend that it is as old as transaction X" (something that can be implemented quite rapidly in an existing system like PostgreSQL) we really aren't talking about something that is either very new, or that totally reinvents the "traditional database".

Re: Rich Hickey: Deconstructing the Database

#56
post #53

Watching this talk has so far (I'm halfway through, and now giving up) been very disappointing, primarily because many of the features and implementation details ascribed to "traditional databases" are not true of the common modern SQL databases, and almost none of them are true of PostgreSQL. As an initial trivial example, many database systems allow you to store arrays. In the case of PostgreSQL, you can have quite…

The fundamental problem is even though it uses MVCC internally, Postgres doesn't expose that model. Instead, it exposes an update-in-place model.

Re: Rich Hickey: Deconstructing the Database

#57
post #53

Watching this talk has so far (I'm halfway through, and now giving up) been very disappointing, primarily because many of the features and implementation details ascribed to "traditional databases" are not true of the common modern SQL databases, and almost none of them are true of PostgreSQL. As an initial trivial example, many database systems allow you to store arrays. In the case of PostgreSQL, you can have quite…

The fundamental problem is even though it uses MVCC internally, Postgres doesn't expose that model. Instead, it exposes an update-in-place model.

Please see the response I left from ten minutes ago to DanWaterworth's similar comment (the one where I point out that, as a client, you can tell the difference between those two models by testing for some of the specific problems that are mentioned near the beginning of this talk; yadda yadda).

Re: Rich Hickey: Deconstructing the Database

#58
post #9

I recall Datomic making a bit of a splash on HN when it was announced 6+ months ago, but basically crickets since then. Anybody build something cool that took advantage of Datomic's unique design?

At least I avoid using it (even though I really like its design) because it isn't open source. I will not trust my data to something I can't even attempt to maintain myself if necessary.

Re: Rich Hickey: Deconstructing the Database

#59
post #55

Earlier quoted context omitted.

In another talk he addresses your point specifically. He said, and I'm paraphrasing: "It doesn't matter if you're using append only data-structures if your view of the world is update in place". PostgreSQL exposes a view to the world of an update in place database, no matter what it's doing underneath. You could create a new interface to PostgreSQL's internals that doesn't and if you did, it would look a lot like dat…

First, there is a major difference between MVCC and update-in-place that you can detect as a client, and that difference is that the problems that Rich outlines at the beginning of his talk do not happen: if one client edits something in the database, other transactions do not get an inconsistent view because the data on disk has already been permanently and irrevocably "updated in place". (Which, to be clear, means…

AFAIK there was a Postgres extension that did something like that. But than it became deprecated and was eventualy removed.

Re: Rich Hickey: Deconstructing the Database

#60
post #53

Watching this talk has so far (I'm halfway through, and now giving up) been very disappointing, primarily because many of the features and implementation details ascribed to "traditional databases" are not true of the common modern SQL databases, and almost none of them are true of PostgreSQL. As an initial trivial example, many database systems allow you to store arrays. In the case of PostgreSQL, you can have quite…

The fundamental problem is even though it uses MVCC internally, Postgres doesn't expose that model. Instead, it exposes an update-in-place model.

To elaborate further on what saurik has already pointed out: Hickey's "update-in-place" characterization of relational databases places blame in the wrong place. This is more about how people think about the data models; they think of their primary keys as identifying places, and updating rows as modifying those places. The relational model itself does not encourage this mode of thinking, although SQL arguably does, unless you think of UPDATE as shorthand for DELETE followed by INSERT. It's not that uncommon to build data models, or parts of models, that have the characteristic of never deleting facts. It's true that time-based as-of queries in such models are unwieldy, but that's a problem of the query languages that can be addressed by how queries are constructed, without redesigning the entire database system. There is research on "temporal databases" that addresses this, although I'm not up to date on it.

What I could not understand from the talk or from googling for more information on Datomic afterwards, is how it supposedly simplifies anything about the consistency issues that he talks about, with respect to the read/decide/write sequence. You read; time passes; other people make changes; when you write, the real world (modeled in your "transactor" as the single common store and arbiter of what is) will have changed.

Post reply on HN