Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Immutability, MVCC, and garbage collection

xaprb.com

Post reply on HN
Print view View original

31–40 of 48 posts

Re: Immutability, MVCC, and garbage collection

#31

Post by lmm »

greatsuccess wrote: post #11

Id like to keep immutability and databases separate in this comment. Immutability the supposed "big advantage" of functional languages, is a discussion which is one-way. No one ever discusses the implications of a system which is constantly, needlessly, insanely doing nothing but MAKING COPIES OF DATA. This is not how systems are/should be designed and Im sure as hell not going to use a functional language until func…

Maybe you make more copies. But that can be done in parallel. Immutability makes it much easier to do fully concurrent garbage collection. If I make my program use 3x more (parallelized) CPU time but avoid 500ms pauses, that's a good tradeoff - and in my actual job solving real problems I've seen Haskell beat even carefully written C++ by a factor of 5.

Re: Immutability, MVCC, and garbage collection

#32

Post by jasonwatkinspdx »

jasonwatkinspdx wrote: post #2

Pretty disappointing critique. There are manifold differences between couchdb, datomic, rethinkdb and more traditional sql databases, but the author can't see past his pet issue. He doesn't seem to understand the use cases, or the infrastructure differences. In terms of use cases, there are plenty of analytically datasets that are strictly monotonic. There is no opportunity to reclaim overwritten storage in this case…

s/analytically/analytic/

Re: Immutability, MVCC, and garbage collection

#33

Post by RyanZAG »

jasonwatkinspdx wrote: post #28
RyanZAG wrote: post #16

Earlier quoted context omitted.

What? Come on, really? My database - only the latest version of all the data that gets updated frequently - has TB of data. My whole git repo with full history is about 500mb and that's excluding large assets. It's a completely different ball game. Sure, if I could buy 100TB disks cheaply and set it up so I could access data across hundreds of them in real time then there wouldn't be a problem. Obviously, I can't do…

When talking about architecture at this level, you need to take a longer view. Once upon a time, a gigabyte database was mammoth beyond thought. Also, you're taking it as given that your data has its present storage footprint even though alternative architectures might have very different properties. It's worth noting that a purely additive database can be very aggressive about compression. But also, petabyte dataset…

We're obviously coming from different backgrounds here - do you work at Google or similar ?

'Downright cheap' is a Heroku instance or two. 'Affordable' is 20 or so servers with a few TB of drive space each in a cluster with a few backups. Petabytes of data needs a huge number of servers and dedicated personal to look after - I don't know how that can be regarded as 'downright cheap'.

If you're in a position to handle petabytes of data you're probably going to roll your own data center with custom software ala Google's bigtable. This definitely isn't the market that these little startups are targeting as far as I can see. The OPs critique is very well founded.

Re: Immutability, MVCC, and garbage collection

#34

Post by mgkimsal »

thirsteh wrote: post #19
RyanZAG wrote: post #16

Earlier quoted context omitted.

What? Come on, really? My database - only the latest version of all the data that gets updated frequently - has TB of data. My whole git repo with full history is about 500mb and that's excluding large assets. It's a completely different ball game. Sure, if I could buy 100TB disks cheaply and set it up so I could access data across hundreds of them in real time then there wouldn't be a problem. Obviously, I can't do…

You're right, but the vast majority of databases aren't that big. You are very explicitly recommended against using something like Datomic if you think you will need to purge data with regular intervals.

"but the vast majority of databases aren't that big" probably because they don't keep all copies of everything forever. Even as basic blog system would grow huge just based on spam comments alone.

Re: Immutability, MVCC, and garbage collection

#35

Post by webmaven »

To me, what was most striking about the article (once I got past the pot-shots), was how much the descriptions of Datomic and RethinkDB reminded me of the ZODB, the object database built into the Zope framework: http://www.zodb.org

ZODB is an append-only object database, and one of the data structures commonly used for persistence in it are BTrees: http://www.zodb.org/en/latest/documentation/guide/modules.ht...

Several disadvantages noted in the OP (infinitely growing data, needing to pack the DB to discard old object versions, needing 2x disk space to do the packing) definitely apply to ZODB, but others do not, as it is ACID-compliant and implements MVCC.

Re: Immutability, MVCC, and garbage collection

#36

Post by thirsteh »

mgkimsal wrote: post #34
thirsteh wrote: post #19

Earlier quoted context omitted.

You're right, but the vast majority of databases aren't that big. You are very explicitly recommended against using something like Datomic if you think you will need to purge data with regular intervals.

"but the vast majority of databases aren't that big" probably because they don't keep all copies of everything forever. Even as basic blog system would grow huge just based on spam comments alone.

But you don't have to keep redundant copies around forever. Immutability means nothing in the past can change, which means it's perfectly fine for future entries to reference old values. New rows can essentially be deltas, much like version control systems.

I disagree that Datomic necessarily grows huge. I'm just saying don't use it if you know you're going to be storing terabytes and you don't have enough disk space to handle it without needing to truncate data with regular intervals.

Re: Immutability, MVCC, and garbage collection

#37

Post by pkolaczk »

greatsuccess wrote: post #11

Id like to keep immutability and databases separate in this comment. Immutability the supposed "big advantage" of functional languages, is a discussion which is one-way. No one ever discusses the implications of a system which is constantly, needlessly, insanely doing nothing but MAKING COPIES OF DATA. This is not how systems are/should be designed and Im sure as hell not going to use a functional language until func…

Copying is in many cases faster than mutating data in-place by several threads, requiring blocking and cache synchronization. Therefore, it is easier to scale.

Re: Immutability, MVCC, and garbage collection

#38

Post by corresation »

jasonwatkinspdx wrote: post #2

Pretty disappointing critique. There are manifold differences between couchdb, datomic, rethinkdb and more traditional sql databases, but the author can't see past his pet issue. He doesn't seem to understand the use cases, or the infrastructure differences. In terms of use cases, there are plenty of analytically datasets that are strictly monotonic. There is no opportunity to reclaim overwritten storage in this case…

By way of simple argument: would anyone advise using a source control system that only kept the last few hundreds of commits? Why do we treat our data differently?

All mainstream database products allow you to retain the entire history of every change ever made in the form of transaction logs. With those logs you can recover your database to any specific point in time, and to unwind specific transactions.

Of course most discard transaction logs because the volumes tend to be huge in many mainstream, operating databases. I worked on one system where the main database barely pushed 10GB, but there were 100s of GBs of transaction logs generated daily.

However the core reason that mainstream databases discard with historical data is performance, not size -- if your current users overwhelmingly only care about the current state of the data, and it is rare that you need to go back into the past (which is what transaction logs and snapshots provide), paying an extremely high performance penalty to retain all of that historical data does not pay off. And there are no immutable products that offer similar performance to mainstream products under most usage scenarios, instead forcing you into extremely confined uses.

It'd be nice to not suffer scornful surface criticisms from people who base their career around the status quo.

This is a garbage non sequitur, as an aside. It is a disgraceful attempt -- as so commonly happens in such discussion -- to try to attach agenda to an opinion that one doesn't like.

Re: Immutability, MVCC, and garbage collection

#39

Post by corresation »

jasonwatkinspdx wrote: post #27
jeltz wrote: post #12

Earlier quoted context omitted.

Git is not append only though. It has a stop the world GC which compacts data and removes objects not referenced by any current branch, tag or similar object. So even if you want to keep the history you may not want to keep everything.

It's important to make a distinction between semantic and physical datasets. Git is append only (or more properly, strictly accumulative in an information sense) from a semantic perspective. From a user perspective we don't really care what the packfiles etc look like. Git also will forget timelines (transitive closure of a reference) you don't care about if you tell it to. So it's important to distinguish purely per…

What bugged me most about the OP was the attitude of "we're done, shut up with this other stuff."

The OP didn't say that at all, but instead observed (in a manner that you have quite overwhelmingly failed to refute at all, instead relying upon emotives and non sequiturs, betraying a bizarre and completely unsupported defensiveness that seems to be some variation "but it's new, man! It's new!") that everything old is new again over and over again.

People have been trying these things for literally decades, and many of the current products that lead categories have some elements of those designs. If someone is pitching a very old idea as (r)evolutionary, it is worthy of discussion, whether that offends your sensibilities or not.

And it's odd that you mentioned MySQL, given that yes, it did face to "criticisms" because it made the same old mistakes that so many products before made. FUD? Do you think since then Oracle became more like MySQL, or the opposite? I'll answer that for you -- MySQL abandoned all of the supposed "advantages" that were birthed largely in ignorance and adopted the designs of the products that came before.

Re: Immutability, MVCC, and garbage collection

#40

Post by nickik »

corresation wrote: post #38
jasonwatkinspdx wrote: post #2

Pretty disappointing critique. There are manifold differences between couchdb, datomic, rethinkdb and more traditional sql databases, but the author can't see past his pet issue. He doesn't seem to understand the use cases, or the infrastructure differences. In terms of use cases, there are plenty of analytically datasets that are strictly monotonic. There is no opportunity to reclaim overwritten storage in this case…

By way of simple argument: would anyone advise using a source control system that only kept the last few hundreds of commits? Why do we treat our data differently? All mainstream database products allow you to retain the entire history of every change ever made in the form of transaction logs. With those logs you can recover your database to any specific point in time, and to unwind specific transactions. Of course m…

Well, I have worked in a few companys with big SQL databases. You say the only care about the current state, but inreally EVER big database I have seen actually impnents some kind of backup thingy. Mostly handcrafted stuff.

The diffrence between Datomic and just storing your transaction log is that datomic gives you first class access to your hole history, you can work with it as easly as with the current data. I have at least not seen that in any other database.

So for me at least it seams a database who just stores everything and gives access to it is what most people should use for most problems. For me at least Datomic is the new standard and only if I have a special use case I would go to something else.

Post reply on HN