Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Immutability, MVCC, and garbage collection

xaprb.com

Post reply on HN
Print view View original

41–48 of 48 posts

Re: Immutability, MVCC, and garbage collection

#41

Post by corresation »

jasonwatkinspdx wrote: post #28
RyanZAG wrote: post #16

Earlier quoted context omitted.

What? Come on, really? My database - only the latest version of all the data that gets updated frequently - has TB of data. My whole git repo with full history is about 500mb and that's excluding large assets. It's a completely different ball game. Sure, if I could buy 100TB disks cheaply and set it up so I could access data across hundreds of them in real time then there wouldn't be a problem. Obviously, I can't do…

When talking about architecture at this level, you need to take a longer view. Once upon a time, a gigabyte database was mammoth beyond thought. Also, you're taking it as given that your data has its present storage footprint even though alternative architectures might have very different properties. It's worth noting that a purely additive database can be very aggressive about compression. But also, petabyte dataset…

petabyte datasets are downright cheap to deal with these days

Are you from the future? In no universe is a petabyte database remotely close to being cheap, even for large organizations. At this point I am seriously questioning your industry knowledge.

Re: Immutability, MVCC, and garbage collection

#42

Post by empthought »

corresation wrote: post #41
jasonwatkinspdx wrote: post #28

Earlier quoted context omitted.

When talking about architecture at this level, you need to take a longer view. Once upon a time, a gigabyte database was mammoth beyond thought. Also, you're taking it as given that your data has its present storage footprint even though alternative architectures might have very different properties. It's worth noting that a purely additive database can be very aggressive about compression. But also, petabyte dataset…

petabyte datasets are downright cheap to deal with these days Are you from the future? In no universe is a petabyte database remotely close to being cheap, even for large organizations. At this point I am seriously questioning your industry knowledge.

Sure it is, a few thousand entries pointing to 10-100GB pirated HD video blobs. What's the problem?

/s

Re: Immutability, MVCC, and garbage collection

#43

Post by corresation »

nickik wrote: post #40
corresation wrote: post #38

Earlier quoted context omitted.

By way of simple argument: would anyone advise using a source control system that only kept the last few hundreds of commits? Why do we treat our data differently? All mainstream database products allow you to retain the entire history of every change ever made in the form of transaction logs. With those logs you can recover your database to any specific point in time, and to unwind specific transactions. Of course m…

Well, I have worked in a few companys with big SQL databases. You say the only care about the current state, but inreally EVER big database I have seen actually impnents some kind of backup thingy. Mostly handcrafted stuff. The diffrence between Datomic and just storing your transaction log is that datomic gives you first class access to your hole history, you can work with it as easly as with the current data. I hav…

you can work with it as easily as with the current data. I have at least not seen that in any other database.

At the cost that working with your current data is as hard as working with the historical data. This does not come for free, and the only way to have such a versioned history is at a significant cost to generalized query performance -- sure certain things (like a single scalar lookup) can be fast, but generalized queries will be terrible. I'm sure there will be some map/reduce claimed solution.

For me at least Datomic is the new standard and only if I have a special use case I would go to something else.

This is an incredible and rather ridiculous statement.

Re: Immutability, MVCC, and garbage collection

#44

Post by gbog »

jasonwatkinspdx wrote: post #27
jeltz wrote: post #12

Earlier quoted context omitted.

Git is not append only though. It has a stop the world GC which compacts data and removes objects not referenced by any current branch, tag or similar object. So even if you want to keep the history you may not want to keep everything.

It's important to make a distinction between semantic and physical datasets. Git is append only (or more properly, strictly accumulative in an information sense) from a semantic perspective. From a user perspective we don't really care what the packfiles etc look like. Git also will forget timelines (transitive closure of a reference) you don't care about if you tell it to. So it's important to distinguish purely per…

Your comment triggered again a thought of mine: What if we had a perfect data store? What would it look like? I think it would have these properties:

- instant access to the most current data from anywhere in the world (no need for replication)

- infinite storage space (no need for gc)

- instant access to any value in the dataset (any joins are ok, indexing not needed)

Obviously we are not there, but let's suppose we had such a tool, I think going the append-only way would make sense, and data deletion would only happen proactively (e.g. for legal or privacy reasons).

Also, in fact a good database is trying to mimic this perfect storage, and some of its users are hitting the boudaries. Anyway, when you feel that a database is not perfect in this sense, and need to adjust your usage, it means you met a boundary, which may not be there anymore in a few years.

Another angle is that maybe there is a physical limit to data transfer that will stop the evolution of datastores. A bit like speed of light, or would the speed of light already put noticable boundaries to this hypothetical perfect datastore?

Re: Immutability, MVCC, and garbage collection

#45

Post by lectrick »

jasonwatkinspdx wrote: post #27
jeltz wrote: post #12

Earlier quoted context omitted.

Git is not append only though. It has a stop the world GC which compacts data and removes objects not referenced by any current branch, tag or similar object. So even if you want to keep the history you may not want to keep everything.

It's important to make a distinction between semantic and physical datasets. Git is append only (or more properly, strictly accumulative in an information sense) from a semantic perspective. From a user perspective we don't really care what the packfiles etc look like. Git also will forget timelines (transitive closure of a reference) you don't care about if you tell it to. So it's important to distinguish purely per…

> wouldn't you like it if when you get an exception notification instead of just a text blob of a stack trace you can resume a continuation of the state of the system as the user saw it at the time of the error?

this, along with capturing headers (perhaps via webserver logs) as well as POST data would actually be incredibly powerful. There is a certain difficult class of bugs that this would greatly help mitigate.

Re: Immutability, MVCC, and garbage collection

#46

Post by jeffdavis »

jasonwatkinspdx wrote: post #29
jeffdavis wrote: post #6

Earlier quoted context omitted.

I think you are making a mistake by reading it as a critique of immutability in general. Rather, I think the author is criticizing an overly-simplisitic approach to using immutability. When he describes traditional databases: "What’s happening in such a database is a combination of short-term immutability, read and write optimizations to save and/or coalesce redundant work, and continuous “compaction” and reuse of di…

I strongly disagree with the idea that temporal databases will be forced to revise themselves to the implementation designs of Postgres, MySQL, etc. In fact, I'd make the firm prediction that the paged window CoW strategy of LMDB will become dominant for these more traditional database designs, and that a new class of temporal databases will improve upon that architecture.

It's not so much that everything will look exactly like postgres. It's that the biggest difference right now between the designs is complexity (the kind that comes with maturity).

Every mature general-purpose database system makes use of both CoW/immutable design principles as well as locking and update-in-place in various contexts. It remains to be seen how "temporal databases" (as you call them) will adapt to real-world demands, but it might not look as radically different as you think.

Also, for anyone who thinks that CoW is the be-all-end-all design principle, I have my doubts. The main reason is that it doesn't compose very cleanly -- a design that involves CoW over CoW over CoW starts to look wasteful after a while.

That being said, I would not be surprised if some of these new databases found some new trade-offs that worked very nicely and got adopted by more traditional systems. Storage is not nearly as much of a concern for many kinds of problems, and it seems natural that such a shift would lead to some design changes.

Re: Immutability, MVCC, and garbage collection

#47

Post by richhickey »

I'd ignored this, trying to have a nice holiday with my family (without having to contend with wrongness on the internet :), but would like to introduce some facts. Datomic is not an append-only b-tree, and never has been.

There's a presumption in the article that "immutable" implies "append-only b-tree", specifically, an argument against append-only b-trees is used to disparage immutability. That's pretty naive. One only need to look at e.g. the BigTable paper for an example of how non-append immutability can be used in a DB. Such architectures are now pervasive. Datomic works similarly.

One can't make a technical argument against keeping information. It's obviously desirable (git) and often legally necessary. Our customers want and need it. Yeah, it's also hard, but update-in-place systems (including MVCC ones) that discard information can't possibly be considered to have "solved" the same problem.

Prefixing a polemic with "apparently" doesn't get one off the hook for spreading a bunch of misinformation.

Re: Immutability, MVCC, and garbage collection

#48

Post by stuarthalloway »

corresation wrote: post #43
nickik wrote: post #40

Earlier quoted context omitted.

Well, I have worked in a few companys with big SQL databases. You say the only care about the current state, but inreally EVER big database I have seen actually impnents some kind of backup thingy. Mostly handcrafted stuff. The diffrence between Datomic and just storing your transaction log is that datomic gives you first class access to your hole history, you can work with it as easly as with the current data. I hav…

you can work with it as easily as with the current data. I have at least not seen that in any other database. At the cost that working with your current data is as hard as working with the historical data. This does not come for free, and the only way to have such a versioned history is at a significant cost to generalized query performance -- sure certain things (like a single scalar lookup) can be fast, but general…

"The only way to have such a versioned history is ..." is incorrect, and trivially so.

Datomic provides one existence proof of this: Datomic's history data is kept in distinct data structures, so it is "pay as you go" -- querying history is more expensive, because there is more stuff. Querying the present is cheaper.

Datomic queries are datalog, and do not require writing map/reduce jobs.

Post reply on HN