Immutability, MVCC, and garbage collection • Hacker News Board

Re: Immutability, MVCC, and garbage collection

#11

Post by greatsuccess » Sun, Dec 29, 2013, 3:59 AM UTC

Id like to keep immutability and databases separate in this comment. Immutability the supposed "big advantage" of functional languages, is a discussion which is one-way.

No one ever discusses the implications of a system which is constantly, needlessly, insanely doing nothing but MAKING COPIES OF DATA.

This is not how systems are/should be designed and Im sure as hell not going to use a functional language until functional understands basic common sense. A system that is constantly copying data for no reason will do nothing else.

Re: Immutability, MVCC, and garbage collection

#12

Post by jeltz » Sun, Dec 29, 2013, 3:59 AM UTC

homerowilson wrote: post #9

jasonwatkinspdx wrote: post #2

Pretty disappointing critique. There are manifold differences between couchdb, datomic, rethinkdb and more traditional sql databases, but the author can't see past his pet issue. He doesn't seem to understand the use cases, or the infrastructure differences. In terms of use cases, there are plenty of analytically datasets that are strictly monotonic. There is no opportunity to reclaim overwritten storage in this case…

"By way of simple argument: would anyone advise using a source control system that only kept the last few hundreds of commits? Why do we treat our data differently?" Well said!

Git is not append only though. It has a stop the world GC which compacts data and removes objects not referenced by any current branch, tag or similar object. So even if you want to keep the history you may not want to keep everything.

Re: Immutability, MVCC, and garbage collection

#13

Post by catnaroek » Sun, Dec 29, 2013, 4:24 AM UTC

greatsuccess wrote: post #11

Id like to keep immutability and databases separate in this comment. Immutability the supposed "big advantage" of functional languages, is a discussion which is one-way. No one ever discusses the implications of a system which is constantly, needlessly, insanely doing nothing but MAKING COPIES OF DATA. This is not how systems are/should be designed and Im sure as hell not going to use a functional language until func…

> Immutability the supposed "big advantage" of functional languages

No, immutability is just the means. The objective is writing programs that are easier to reason about.

> No one ever discusses the implications of a system which is constantly, needlessly, insanely doing nothing but MAKING COPIES OF DATA.

Just because copy assignment requires, well, copying in languages like C++, it does not mean things work the same way in functional languages. Precisely because data structures are immutable, you can internally just pass around a pointer to the data structure and pretend you have copied it (e.g., when you pass it as parameter to another function).

Re: Immutability, MVCC, and garbage collection

#14

Post by talaketu » Sun, Dec 29, 2013, 4:45 AM UTC

greatsuccess wrote: post #11

Id like to keep immutability and databases separate in this comment. Immutability the supposed "big advantage" of functional languages, is a discussion which is one-way. No one ever discusses the implications of a system which is constantly, needlessly, insanely doing nothing but MAKING COPIES OF DATA. This is not how systems are/should be designed and Im sure as hell not going to use a functional language until func…

> No one ever discusses the implications of a system which is constantly, needlessly, insanely doing nothing but MAKING COPIES OF DATA.

see persistent data structure

Re: Immutability, MVCC, and garbage collection

#15

Post by yresnob » Sun, Dec 29, 2013, 5:06 AM UTC

The author needs to read up way more on datomic... and clojure for that matter then try again...

Re: Immutability, MVCC, and garbage collection

#16

Post by RyanZAG » Sun, Dec 29, 2013, 5:21 AM UTC

homerowilson wrote: post #9

jasonwatkinspdx wrote: post #2

Pretty disappointing critique. There are manifold differences between couchdb, datomic, rethinkdb and more traditional sql databases, but the author can't see past his pet issue. He doesn't seem to understand the use cases, or the infrastructure differences. In terms of use cases, there are plenty of analytically datasets that are strictly monotonic. There is no opportunity to reclaim overwritten storage in this case…

"By way of simple argument: would anyone advise using a source control system that only kept the last few hundreds of commits? Why do we treat our data differently?" Well said!

What? Come on, really? My database - only the latest version of all the data that gets updated frequently - has TB of data. My whole git repo with full history is about 500mb and that's excluding large assets. It's a completely different ball game. Sure, if I could buy 100TB disks cheaply and set it up so I could access data across hundreds of them in real time then there wouldn't be a problem. Obviously, I can't do that.

As the author of the blog post points out: the little thing called reality gets in the way of such nice and perfect arguments and ideas.

Re: Immutability, MVCC, and garbage collection

#17

Post by dschiptsov » Sun, Dec 29, 2013, 5:38 AM UTC

Very rare example of an old-school thinking and reasoning.

Indeed, industrial-strength databases are hard, and as an Informix DBA, I could only agree, that there is no silver bullet, and that just "check-point early, check-point often" strategy has its trade-offs in terms of concurrent query performance. And, of course, unpredictable stop-the-world GC pauses are unacceptable. Informix have polished their Dynamic server for a decade and it is still one of the best (if not the best) solutions available.

The more subtle notion is that one could never avoid compactiom/GC pauses, so the strategy should be to partition the data and perform "separate" compaction/GC, blocking/freezing only some "snapshot" of the data, without stopping the whole world, like some modern FS do.

Of course, implementation is hard, and here the simplicity of persistent, append-only data-structures could be beneficial for the implementer of "partitioned/partial GC".

There must be some literature, because these ideas were well researched in old times, when design of Lisp's GC, or CPU caches were hot topics.

Re: Immutability, MVCC, and garbage collection

#18

Post by chongli » Sun, Dec 29, 2013, 6:14 AM UTC

greatsuccess wrote: post #11

Id like to keep immutability and databases separate in this comment. Immutability the supposed "big advantage" of functional languages, is a discussion which is one-way. No one ever discusses the implications of a system which is constantly, needlessly, insanely doing nothing but MAKING COPIES OF DATA. This is not how systems are/should be designed and Im sure as hell not going to use a functional language until func…

constantly, needlessly, insanely doing nothing but MAKING COPIES OF DATA.

That's simply not the case. With persistent data structures (used in languages such as Clojure, upon which Datomic is built), copying is kept to a minimum. If you've got a million-element hash-map and you want to associate a new key/value pair to it, you copy a path to the root of the tree -- an O(log32N) operation -- and you get a new map with 1,000,001 key/value pairs. Just for reference, log32 of 1,000,000 is ~4. This growth in complexity is so negligible that it ought to be considered constant for most practical purposes.

Re: Immutability, MVCC, and garbage collection

#19

Post by thirsteh » Sun, Dec 29, 2013, 6:17 AM UTC

RyanZAG wrote: post #16

homerowilson wrote: post #9

Earlier quoted context omitted.

"By way of simple argument: would anyone advise using a source control system that only kept the last few hundreds of commits? Why do we treat our data differently?" Well said!

What? Come on, really? My database - only the latest version of all the data that gets updated frequently - has TB of data. My whole git repo with full history is about 500mb and that's excluding large assets. It's a completely different ball game. Sure, if I could buy 100TB disks cheaply and set it up so I could access data across hundreds of them in real time then there wouldn't be a problem. Obviously, I can't do…

You're right, but the vast majority of databases aren't that big. You are very explicitly recommended against using something like Datomic if you think you will need to purge data with regular intervals.

Re: Immutability, MVCC, and garbage collection

#20

Post by wging » Sun, Dec 29, 2013, 6:19 AM UTC

greatsuccess wrote: post #11

Id like to keep immutability and databases separate in this comment. Immutability the supposed "big advantage" of functional languages, is a discussion which is one-way. No one ever discusses the implications of a system which is constantly, needlessly, insanely doing nothing but MAKING COPIES OF DATA. This is not how systems are/should be designed and Im sure as hell not going to use a functional language until func…

Immutability doesn't mean large amounts of copying. You can exploit the fact that things will never change in clever ways.

http://hypirion.com/musings/understanding-persistent-vector-...