So Rich's talk about Datomic's architecture is taking place at Oracle? I love the radical openness. There's something cheeky about going to your competitor's office and inviting their engineers to learn about how your product works and how they can make better databases.
Rich Hickey on Datomic (2012) [video]
11–20 of 39 posts
Re: Rich Hickey on Datomic (2012) [video]
#12I love Datomic, think it can be revolutionary and use it in my side projects. I understand their need to make a living off of their (brilliant, unique) labor and so keep it closed source. But the big bananas at my company are reluctant to try something so different. With Clojure I can push them over that hump since it's open source. But they are not willing to introduce proprietary lock-in of such a fundamental piece…
While it isn't the same https://juxt.pro/crux/docs/index.html is a good compromise in the space and is open source albeit a very freshly released project. One of the major blockers I had to adoption, needing Kafka, is no longer an issue with the release of https://github.com/juxt/crux/tree/master/crux-jdbc . A great starting point if anyone wants to check Crux out at a distance is https://nextjournal.com/crux-tutoria…
Could you say anything more at all about how Crux is a compromise compared to Datomic?
It seems to have a lot of what the latter has, as well as some features that it doesn't (e.g. it's bitemporal).
(I'm reading https://juxt.pro/crux/docs/faq.html but if there's anything you've seen that'd be super helpful)
Re: Rich Hickey on Datomic (2012) [video]
#13I love Datomic, think it can be revolutionary and use it in my side projects. I understand their need to make a living off of their (brilliant, unique) labor and so keep it closed source. But the big bananas at my company are reluctant to try something so different. With Clojure I can push them over that hump since it's open source. But they are not willing to introduce proprietary lock-in of such a fundamental piece…
It's kind of incredible how, over the course of two decades, being open source went from making something radioactive to business clients to being a requirement. I've seen it over and over again.
Re: Rich Hickey on Datomic (2012) [video]
#14Earlier quoted context omitted.
While it isn't the same https://juxt.pro/crux/docs/index.html is a good compromise in the space and is open source albeit a very freshly released project. One of the major blockers I had to adoption, needing Kafka, is no longer an issue with the release of https://github.com/juxt/crux/tree/master/crux-jdbc . A great starting point if anyone wants to check Crux out at a distance is https://nextjournal.com/crux-tutoria…
Could you say anything more at all about how Crux is a compromise compared to Datomic? It seems to have a lot of what the latter has, as well as some features that it doesn't (e.g. it's bitemporal). (I'm reading https://juxt.pro/crux/docs/faq.html but if there's anything you've seen that'd be super helpful)
The biggest thing I have noticed, granted I don't have a ton experience with either system, is point 1 outlined in the "Other Differences" section.
>Datomic’s datom model provides a very granular and comprehensive interface for expressing novelty through the assertion and retraction of facts. Crux instead uses documents (i.e. schemaless EDN maps) which are atomically ingested and processed as groups of facts that correspond to top-level fields with each document. This design choice simplifies bitemporal indexing (i.e. the use of valid time + transaction time coordinates) whilst satisfying typical requirements and improving the ergonomics of integration with other document-oriented systems. Additionally, the ordering of fields using the same key in a document is naturally preserved and can be readily retrieved, whereas Datomic requires explicit modelling of order for cardinality-many attributes. The main downside of Crux’s document model is that re-transacting entire documents to update a single field can be considered inefficient, but this could be mitigated using lower-level compression techniques and content-addressable storage. Retractions in Crux are implicit and deleted documents are simply replaced with empty documents
So what this means in pratice is that in Datomic you don't really "have" an entity/document (this isn't entirely true but feel free to dig into the details https://docs.datomic.com/on-prem/entities.html) but rather a set of facts (rows/tuples) at any given time, or across time, generally gathered up under one id. You can query on one of those facts and get exactly that back, one or more specific facts. Crux has this ability to query parts/attributes of a document but in terms of updating its all or nothing. There are other, more niche, considerations. One example I can can cite is looking at an entity through time in Datomic can be alien at first whereas with Crux its just a document, but overall this is the biggest thing I have noticed.
Oh and Datomic is pretty well battle tested and has the full support of Cognitect behind it whereas Crux is a newer OSS product by Juxt. That is both a pro and con for both camps, just depends on your needs.
Re: Rich Hickey on Datomic (2012) [video]
#15have always wondered why this didn't succeed on a larger scale
Maybe in a v0 release, most applications can get away with created & modified fields. Maybe the storage overhead of immutability scares people for some applications.
Maybe datomic is something that should be part of a powerful DB plugins ecosystem rather than its own database, or as an external audit log that can be written separately while keeping your main DB in SQL.
Re: Rich Hickey on Datomic (2012) [video]
#16I love Datomic, think it can be revolutionary and use it in my side projects. I understand their need to make a living off of their (brilliant, unique) labor and so keep it closed source. But the big bananas at my company are reluctant to try something so different. With Clojure I can push them over that hump since it's open source. But they are not willing to introduce proprietary lock-in of such a fundamental piece…
It's kind of incredible how, over the course of two decades, being open source went from making something radioactive to business clients to being a requirement. I've seen it over and over again.
It's still radioactive, just to a different group of people now and weaponized (see Mongo, Elastic, et al).
I also think GPL compliance is a looming major problem for organizations and most haven't realized it yet. We have seen a few folks try to monetize GPL compliance the last few years...netfilter, I'm looking at you.
And this is why many of us prefer open source software to free software.
Re: Rich Hickey on Datomic (2012) [video]
#17have always wondered why this didn't succeed on a larger scale Maybe in a v0 release, most applications can get away with created & modified fields. Maybe the storage overhead of immutability scares people for some applications. Maybe datomic is something that should be part of a powerful DB plugins ecosystem rather than its own database, or as an external audit log that can be written separately while keeping your m…
It seems to me that you need to be all-in on Datomic to get value from it. So that knocks out anyone working with a legacy system.
You also can’t really hire Datomic people in bulk so that knocks out anyone trying to scale their eng org.
I can see it working well for companies who are trying to do a lot with a small very bright staff, but that’s a small category. And even then it would only be places where the staff has consensus around immutability as a value.
Which... immutability, like many forms of strictness, is both a PITA and not actually that helpful unless you go all in.
It’s just a major commitment and for what benefit?
Maybe for a product where you need rewind everywhere.
Maybe for a product where you need extreme auditability.
I don’t know... what’s the killer feature that would make someone sacrifice everything else to use it?
Personally I think Datomic needs to be paired with a hosting system, debugging tools, basically some kind of integrated development environment that’s immutable-first in every respect.
Something like that would be powerfully differentiated.
Re: Rich Hickey on Datomic (2012) [video]
#18have always wondered why this didn't succeed on a larger scale Maybe in a v0 release, most applications can get away with created & modified fields. Maybe the storage overhead of immutability scares people for some applications. Maybe datomic is something that should be part of a powerful DB plugins ecosystem rather than its own database, or as an external audit log that can be written separately while keeping your m…
It seems to me that you need to be all-in on Datomic to get value from it. So that knocks out anyone working with a legacy system. You also can’t really hire Datomic people in bulk so that knocks out anyone trying to scale their eng org. I can see it working well for companies who are trying to do a lot with a small very bright staff, but that’s a small category. And even then it would only be places where the staff…
You don't need to be all-in on Datomic, it's unsuitable for some kinds of data (eg binary blobs, or generally text due to the 4k string limit, PII due to missing delete support, etc). It seems designed to be used along with ancillary data stores.
I think the benefits have been pretty well laid out and are quite attractive, your restatement of them is just damnation with faint praise :)
Re: Rich Hickey on Datomic (2012) [video]
#19I'm using the AWS hosted version of this (Datomic ions) to talk with the Monzo API it's been a lot of fun so far
Re: Rich Hickey on Datomic (2012) [video]
#20I love Datomic, think it can be revolutionary and use it in my side projects. I understand their need to make a living off of their (brilliant, unique) labor and so keep it closed source. But the big bananas at my company are reluctant to try something so different. With Clojure I can push them over that hump since it's open source. But they are not willing to introduce proprietary lock-in of such a fundamental piece…
It's kind of incredible how, over the course of two decades, being open source went from making something radioactive to business clients to being a requirement. I've seen it over and over again.
Actually it is more of having a generation that doesn't want to pay for their tooling, while expecting nonetheless to be paid for their work.
I keep working for customers where paying for tools is business as usual.