Live data from Hacker News

Datomic: Look at all the things I'm not doing

augustl.com

31–40 of 48 posts

Re: Datomic: Look at all the things I'm not doing

#31
post #4

This will be a tongue in cheek comment, but there's another thing Datomic isn't making you do either: GDPR compliance

Gpdr compliance is as much as not choosing to retrieve data after a certain point as it is not possessing that data.

It's a bit of an existential point, but if your logic allows the disclusion of data before a certain point of time, is that not effectively the same as not possessing it?

Re: Datomic: Look at all the things I'm not doing

#32
post #5
post #4

This will be a tongue in cheek comment, but there's another thing Datomic isn't making you do either: GDPR compliance

Well, they do support extinctions?

That is certainly something an add-on or external tool could provide.

Immutable is only immutable if you don't have write access to the lowest layer of storage.

Re: Datomic: Look at all the things I'm not doing

#33

One thing though: you still have to do data modeling. No matter how much you pretend that you don't have to do it, you still need relationships, and unique references, getting and querying ranges. Hell, even data access restrictions.

But quite often your first two or five models will be wrong. Usually the exact problem is not clear early on...

Delaying data architecture decisions is usually better than attempting to plan for them up front.

Re: Datomic: Look at all the things I'm not doing

#34
post #26
post #7

Earlier quoted context omitted.

OP here :) True, in that there's no solution out of the box. I kind of want to make a "look at all the things I have to do", and GDPR compliance is one of those things where there are probably many ways to do it wrong with Datomic, and a few ways of doing it right. One thing you can do, is to have a separate database for each person/user. It is trivial to join across multiple databases in Datomic, and you can even do…

IANAL, but I‘m pretty sure it is acceptable to delete data within 30 days of receiving a request to do so.

Forgot to add. The built-in excision API is similar to crypto shredding, in that all the metadata is retained, it's just the values themselves (in Datomic's EAVT structure) that are removed. So a lot of metadata is still retained.

Re: Datomic: Look at all the things I'm not doing

#35
post #5
post #4

This will be a tongue in cheek comment, but there's another thing Datomic isn't making you do either: GDPR compliance

Well, they do support extinctions?

Yes. They call this excision[0] and it leaves an Audi table hole in the timeline which lets you attach metadata like why there is missing data while still removing the data.

[0] https://docs.datomic.com/on-prem/excision.html

Re: Datomic: Look at all the things I'm not doing

#36
post #33

One thing though: you still have to do data modeling. No matter how much you pretend that you don't have to do it, you still need relationships, and unique references, getting and querying ranges. Hell, even data access restrictions.

But quite often your first two or five models will be wrong. Usually the exact problem is not clear early on... Delaying data architecture decisions is usually better than attempting to plan for them up front.

I whole heartedly disagree. Data doesn't care about your application.

Re: Datomic: Look at all the things I'm not doing

#37

For those that don't know, Datomic is a hosted database, similar to how Clojure is hosted. Datomic can be hosted on MySQL or PostgreSQL (and maybe others?). It's basically a 2 column table, so yes, Datomic inherits transaction safety. According to people I know that use Datomic, an early lesson is indexing, which apparently is often done way later than it should. Datomic also inherits the speed of indexing a huge tab…

An interesting point to add is that Datomic can also write to Riak. But since riak lacks an atomic CAS, zookeeper is used alongside Riak.

What do you refer to when you say that Datomic inherits the speed of indexing a huge table? The query engine works by fetching blocks of data from storage and placing the blocks in a lazy immutable tree structure that are the indices of Datomic. They are also covering (the indices is where the actual data is stored). So it's not like the query engine runs queries on the huge table stored in mysql or postgres, it's used more like a key/value store. Case in point: you can put memcached between the datomic client (peer) and any of the supported storage engines. So Datomic piggybacks on very little other than the actual capabiliy to safely store data.

Re: Datomic: Look at all the things I'm not doing

#38
post #32
post #5

Earlier quoted context omitted.

Well, they do support extinctions?

That is certainly something an add-on or external tool could provide. Immutable is only immutable if you don't have write access to the lowest layer of storage.

It's difficult to implement this by hand, though. What you'll have in storage are encoded blobs of chunks of some kind of sorted set tree structure. So if you are to poke the data directly like that, I would assume it is much easier to just do an offline backup and a restore, and filter the data in the restore process somehow, which I believe is possible.

Re: Datomic: Look at all the things I'm not doing

#39
post #33

Earlier quoted context omitted.

But quite often your first two or five models will be wrong. Usually the exact problem is not clear early on... Delaying data architecture decisions is usually better than attempting to plan for them up front.

I whole heartedly disagree. Data doesn't care about your application.

I guess we all have our personal experiences to go by. My 24 years have thought me that perhaps 25% of the real need is know up front. And usually there is some pattern that one's problem seems to fit into. Then run with it until more details appear. Then adapt or add...

On the other hand, I have worked with people who like to try to plan for all possibilities. That usually results in enormous data models and extra layers of abstraction that never prove their value.

Re: Datomic: Look at all the things I'm not doing

#40

For those that don't know, Datomic is a hosted database, similar to how Clojure is hosted. Datomic can be hosted on MySQL or PostgreSQL (and maybe others?). It's basically a 2 column table, so yes, Datomic inherits transaction safety. According to people I know that use Datomic, an early lesson is indexing, which apparently is often done way later than it should. Datomic also inherits the speed of indexing a huge tab…

It is not clear to me how you could enforce invariants that relate multiple database states. Allow me to give a silly toy example. Suppose you have to design a system where

(0) Every bank account must admit a linear transaction history.

(1) Given a causally related collection of transactions, all involved parties must agree on the relative order of these transactions.

How exactly do you get to enforce invariants like these without transaction coordination?

Post reply on HN