Earlier quoted context omitted.
Datomic allows one to get a consistent basis for multiple queries, separated by arbitrary amounts of time, outside of any transactions . Having to group queries motivated and conducted by different parts of your system into a single transaction in order to get a consistent basis is a source of coupling.
Given that I need to share that common basis among the different parts of my system that need the consistent view, I am already taking the hit on that coupling: I am going to need to make certain that all of those parts all have access to the shared basis, whether it be a timestamp, a transaction identifier, or an active connection. If the concern is coupling across space, you can store the transaction on the server…
Rich Hickey: Deconstructing the Database
81–90 of 93 posts
Re: Rich Hickey: Deconstructing the Database
#82Earlier quoted context omitted.
In your last paragraph, I feel like you are mischaracterizing my overall thesis. I am not claiming the design is simple: MVCC took many lives in sacrifice to its specification and discovery, and I certainly am not claiming "anyone could have thought that up". Instead, my primary issue is that this is a talk about databases and database design that is providing motivation vs a strawman: specifically, the way Rich seem…
I agree with most of what you say, but I don't think that MVCC is really what this is about. The qualities you describe are a feature of ACID. MVCC is just a way of implementating ACID so that it requires less locking. More importantly, I think, there are issues with some data structures that are not well supported by postgres or any other DBMS (relational or otherwise). I do a lot of text analytics work and there ar…
Instead, we would have had contention: in the case of my example walkthrough, to implement the repeatable read semantics that I requested, the first connection would have taken a share lock on the rows it queried, causing the second connection to block on the update until after the first connection committed.
This means that you would not have been able to have the semantics where the first connect and the second connection were seeing different things at the same time (which, to be again clear, is due to none of the data being destroyed: MVCC is providing the semantics of a snapshot).
(As for your text analytics work, I am curious: are you using gin and trigrams at all? There are a bunch of things I dislike about PostgreSQL's implementation of both, but if you haven't looked at them yet you really should: if your use case fits into them they are amazing, and if not the entire point of PostgreSQL is to let you build your own data types and indexes using the ones it comes with as examples.)
Re: Rich Hickey: Deconstructing the Database
#83I'm always puzzled when the Datomic folks speak of reads not being covered under a transaction. This is dangerous. Here's the scenario, that in a conventional update-oriented store, is termed as a "lost update". "A" reads object.v1, "B" reads the same version, "B" adds a fact to the object making it v2, then "A" comes along and writes obj.v3 based on its own _stale_ knowledge of the object. In effect, it has clobbere…
Re: Rich Hickey: Deconstructing the Database
#84Earlier quoted context omitted.
Given that I need to share that common basis among the different parts of my system that need the consistent view, I am already taking the hit on that coupling: I am going to need to make certain that all of those parts all have access to the shared basis, whether it be a timestamp, a transaction identifier, or an active connection. If the concern is coupling across space, you can store the transaction on the server…
I would characterize the level of coupling involved in sharing "the basis is 12345" as categorically different from having to nest database access within the same transaction. Consider, e.g. the ease of moving the former to different processes.
EDIT: You can find documentation of this at http://www.postgresql.org/docs/9.2/static/sql-set-transactio...
If you in PostgreSQL would save all historical snapshots and disable vacuum you could communicate any point in time simply with a snapshot number. Now this probably wont be very efficient since PostgreSQL is not optimized for this kind of use.
Re: Rich Hickey: Deconstructing the Database
#85Earlier quoted context omitted.
I agree with most of what you say, but I don't think that MVCC is really what this is about. The qualities you describe are a feature of ACID. MVCC is just a way of implementating ACID so that it requires less locking. More importantly, I think, there are issues with some data structures that are not well supported by postgres or any other DBMS (relational or otherwise). I do a lot of text analytics work and there ar…
If you implement ACID using the "normal" locking semantics (such as the ones the SQL standard authors who defined the isolation levels used in the language were assuming) you can tell the difference because old values are not preserved. Instead, we would have had contention: in the case of my example walkthrough, to implement the repeatable read semantics that I requested, the first connection would have taken a shar…
But you're right, it might be a good idea to look into the postgres extension mechanism. I've never seriously done that.
Re: Rich Hickey: Deconstructing the Database
#86I'm always puzzled when the Datomic folks speak of reads not being covered under a transaction. This is dangerous. Here's the scenario, that in a conventional update-oriented store, is termed as a "lost update". "A" reads object.v1, "B" reads the same version, "B" adds a fact to the object making it v2, then "A" comes along and writes obj.v3 based on its own _stale_ knowledge of the object. In effect, it has clobbere…
Addressed here: http://news.ycombinator.com/item?id=4448351
Re: Rich Hickey: Deconstructing the Database
#87Earlier quoted context omitted.
To elaborate further on what saurik has already pointed out: Hickey's "update-in-place" characterization of relational databases places blame in the wrong place. This is more about how people think about the data models; they think of their primary keys as identifying places, and updating rows as modifying those places. The relational model itself does not encourage this mode of thinking, although SQL arguably does,…
Re: read/decide/write - Datomic supports transaction functions (written in Java or Clojure) which run within the transaction, on the transactor, and are true functions of the prior state of the database (with full access to the query API). They can be used to do all of the transformative work (traditional read/modify/write), or merely to confirm that preconditions of work done prior to transaction submission still ho…
Re: Rich Hickey: Deconstructing the Database
#88Earlier quoted context omitted.
The reason I claim this would be simple is that PostgreSQL is almost already doing this. The way the data is stored on disk, every row has two transactions identifiers, xmin and xmax, which represent the transaction when that row was inserted the the transaction that row was deleted; rows, meanwhile, are never updated in place, so the old data stays around until it is deleted by a vacuum. To demonstrate more tangibly…
Thanks for taking the time to elaborate, very interesting. I wonder if the sql db vendors or open source projects will take the next step to make querying against a transaction ID possible given the underlying implementation details bring it pretty close. I also see Rich has made some interesting points elsewhere in this thread about consistent views being available outside of transactions and without need for coordi…
(edit: Ah, but the feature as implemented actually saves a file to disk and thereby has a lot of server-side state: the way I've gone ahead and implemented it does not have this complexity; I simply take a single integer and store nothing on the server.)
http://news.ycombinator.com/item?id=4448472
> I wonder if the sql db vendors or open source projects will take the next step to make querying against a transaction ID possible given the underlying implementation details bring it pretty close.
For the hell of it, I just went ahead and implemented the "backdate a transaction" feature; I didn't solve the vacuum guarantees problem, however: I only made it so that a transaction can be backdated to another point in time.
To demonstrate, I will start with a very similar sequence of events to before. However, I am going to instead use txid_current_snapshot(), which returns the range (and an exception set that will be unused for this example) of transaction identifiers that are valid.
Connection 1:
demo=# create table q (data int);
CREATE TABLE
demo=# begin; select txid_current_snapshot();
BEGIN
710:710:
demo=# insert into q (data) values (0); commit;
INSERT 0 1
COMMIT
demo=# begin; select txid_current_snapshot();
BEGIN
711:711:
demo=# select xmin, xmax, data from q;
710|0|0
Connection 2: demo=# begin; select txid_current_snapshot();
BEGIN
711:711:
demo=# update q set data = 1; commit;
UPDATE 1
COMMIT
demo=# select xmin, xmax, data from q;
711|0|1
Connection 1: demo=# select xmin, xmax, data from q;
710|711|0
demo=# begin; select txid_current_snapshot();
BEGIN
712:712:
demo=# select xmin, xmax, data from q;
711|0|1
So far, this is the same scenario as before: I have two connections that are seeing different visibility to the same data, based on these snapshots. Now, however, I'd like to "go back in time": I want our first connection to be able to use the same basis for its consistency that we were using in the previous transaction.Connection 1:
demo=# set snapshot_txid = 711;
SET
demo=# select txid_current_snapshot();
711:711:
demo=# select xmin, xmax, data from q;
710|711|0
This new variable, snapshot_txid, is something I created: it gets the current transaction's active snapshot and modifies it to be a range from that transaction id to that same id (I think a better version of this would take the exact same string value that is returned by txid_current_snapshot()).From that previous basis, the row with the value 0 is visible, not the row with the value 1. I can, of course, go back to the future snapshot if I wish, in order to view the new row. (I am not yet certain what this will do to things writing to the database; this might actually be sufficient, however I feel like I might need to either mess with more things or mark the transaction read-only.)
Connection 1:
demo=# set snapshot_txid = 712;
SET
demo=# select txid_current_snapshot();
712:712:
demo=# select xmin, xmax, data from q;
711|0|1Re: Rich Hickey: Deconstructing the Database
#89Earlier quoted context omitted.
I would characterize the level of coupling involved in sharing "the basis is 12345" as categorically different from having to nest database access within the same transaction. Consider, e.g. the ease of moving the former to different processes.
Agreed, but this is due to MVCC databases not saving the snapshots of committed transactions. If you have an active transaction in PostgreSQL you can start a new transaction using the same snapshot from another process. (This feature is exposed in SQL in version 9.2, and was motivated by the plan to implement parallel pg_dump.) EDIT: You can find documentation of this at http://www.postgresql.org/docs/9.2/static/sql-…
Not knowing this existed, I spent the last hour implementing this feature in a way that just requires getting a single integer out (the txid) and then restoring it as the snapshot being viewed (but not changing the txid of the running transaction, which solves a lot of the "omg what would that mean" problems).
http://news.ycombinator.com/item?id=4448767
With this implementation, you can just use txid_current() to save a transaction snapshot and you can restore it using my new variable (which correctly installs this functionality only into the currently executing transaction's snapshot). (In a more ideal world, I'd re-parse the string from txid_current_snapshot().)
I didn't also solve the vacuum problems, but I think there might be some reasonable ways to do that. Regardless, I imagine that most of the interesting usages of this are not "restore a snapshot from three days ago" but more "share a snapshot between multiple processes and machines for a few seconds".
However, if you can get even one of those processes to hold open a transaction, you can copy its snapshot to other transactions in other sessions, and then even this naive implementation should be guaranteed to have access to the old data.
If you are using it only for these short periods of time, for purposes of "distributed and decoupled consistency", we also don't need to worry about the overhead of never running a vacuum: the vacuum process runs as normal, as we are only holding on to data very temporarily.
That said, in practice, you really can go for quite a while without running a vacuum on your database without issues, and I imagine any alternative system is going to run into similar problems anyway (log structured merge trees, for example, get screwed on this after a while, as the bloom filters become less selective).
You can't, however, with PostgreSQL, make this work "forever" (if you wanted to store data going back until the beginning of time) due to 32-bit txid wraparound problems :(. (This also should affect my silly implementation, as I should save the epoch in addition to the txid.)
Re: Rich Hickey: Deconstructing the Database
#90Earlier quoted context omitted.
The model of consistency envisioned by Datomic is one in which consistency normally available only within a transaction is available outside of any transactions , and without any central authority. Consistent views can be reconstituted the next hour, day or week. Consistent points in time can be efficiently communicated to other processes. Nothing about MVCC gives you any of that. MVCC is an implementation detail tha…
First of all, thank you very much for the reply: you really didn't need to bother, as despite being a Clojure user who stores a lot of data, I'm probably simply not in your target market segment ;P. For the record, I do not believe that you have explicitly stated this is revolutionary, although I believe various other people on HN in various threads on Datomic have. However, my specific reactions in the comment you a…
Other then that, the true genius is to recogniced that a system like that would be worthwhile. Just pointing out that one could theoreticly do that with something else is kind of pointless if nobody has ever done it.