Datomic: Look at all the things I'm not doing • Hacker News Board

Re: Datomic: Look at all the things I'm not doing

#41

Post by augustl » Sat, Apr 21, 2018, 6:34 PM UTC

DonbunEf7 wrote: post #16

augustl wrote: post #8

Earlier quoted context omitted.

I am regularly alarmed by the fact that nobody has tried to make an open source clone yet..

Well, why would they? Clones would be using relatively old technology. The current bleeding-edge research here is on more-structured, more-formal knowledge representation, stuff like FQL/AQL [0] which is open-source from the first day. Maybe Datomic's not sufficiently better than Pg or SQLite to motivate anybody. [0] http://categoricaldata.net/fql.html

That looks very interesting! Do you have a recommended point of entry for a non-academic practitioner like myself? :)

Re: Datomic: Look at all the things I'm not doing

#42

Post by dizzystar » Sat, Apr 21, 2018, 7:05 PM UTC

blunte wrote: post #39

dizzystar wrote: post #36

Earlier quoted context omitted.

I whole heartedly disagree. Data doesn't care about your application.

I guess we all have our personal experiences to go by. My 24 years have thought me that perhaps 25% of the real need is know up front. And usually there is some pattern that one's problem seems to fit into. Then run with it until more details appear. Then adapt or add... On the other hand, I have worked with people who like to try to plan for all possibilities. That usually results in enormous data models and extra l…

It's been my experience that people who design their database around an app, or people that just dump data to fix for later, tend to have much greater issues in the near and far future.

Of course things can change, but most tables have fairly immutable designs.

Re: Datomic: Look at all the things I'm not doing

#43

Post by dizzystar » Sat, Apr 21, 2018, 7:10 PM UTC

augustl wrote: post #37

dizzystar wrote: post #30

For those that don't know, Datomic is a hosted database, similar to how Clojure is hosted. Datomic can be hosted on MySQL or PostgreSQL (and maybe others?). It's basically a 2 column table, so yes, Datomic inherits transaction safety. According to people I know that use Datomic, an early lesson is indexing, which apparently is often done way later than it should. Datomic also inherits the speed of indexing a huge tab…

An interesting point to add is that Datomic can also write to Riak. But since riak lacks an atomic CAS, zookeeper is used alongside Riak. What do you refer to when you say that Datomic inherits the speed of indexing a huge table? The query engine works by fetching blocks of data from storage and placing the blocks in a lazy immutable tree structure that are the indices of Datomic. They are also covering (the indices…

I don't know the full details, but I'm lead to understand that indexing a Datomic database too late is time-consuming and will end up causing considerable downtime.

I'll admit that I'm guessing about how much of this depends on the database and Datomic itself, so I shouldn't be making a sweeping statement.

Re: Datomic: Look at all the things I'm not doing

#44

Post by augustl » Sat, Apr 21, 2018, 7:46 PM UTC

dizzystar wrote: post #43

augustl wrote: post #37

Earlier quoted context omitted.

An interesting point to add is that Datomic can also write to Riak. But since riak lacks an atomic CAS, zookeeper is used alongside Riak. What do you refer to when you say that Datomic inherits the speed of indexing a huge table? The query engine works by fetching blocks of data from storage and placing the blocks in a lazy immutable tree structure that are the indices of Datomic. They are also covering (the indices…

I don't know the full details, but I'm lead to understand that indexing a Datomic database too late is time-consuming and will end up causing considerable downtime. I'll admit that I'm guessing about how much of this depends on the database and Datomic itself, so I shouldn't be making a sweeping statement.

Ah, that makes sense. I've never added an index myself, but essentialy Datomic has to rewrite the main index tree. In other words, all of the stored data, since there is no data stored outside of the covering indies. So I wouldn't be surpised if that causes some down time if you have lots of data in your database.

Re: Datomic: Look at all the things I'm not doing

#45

Post by dustingetz » Sat, Apr 21, 2018, 8:42 PM UTC

dizzystar wrote: post #30

For those that don't know, Datomic is a hosted database, similar to how Clojure is hosted. Datomic can be hosted on MySQL or PostgreSQL (and maybe others?). It's basically a 2 column table, so yes, Datomic inherits transaction safety. According to people I know that use Datomic, an early lesson is indexing, which apparently is often done way later than it should. Datomic also inherits the speed of indexing a huge tab…

Datomic Cloud indexes automatically and persists to DynamoDB; indexing happens in a background process

Re: Datomic: Look at all the things I'm not doing

#46

Post by Ryan1516 » Sat, Apr 21, 2018, 8:44 PM UTC

emidln wrote: post #35

davty wrote: post #5

Earlier quoted context omitted.

Well, they do support extinctions?

Yes. They call this excision[0] and it leaves an Audi table hole in the timeline which lets you attach metadata like why there is missing data while still removing the data. [0] https://docs.datomic.com/on-prem/excision.html

Excision is not support in Datomic Cloud

Re: Datomic: Look at all the things I'm not doing

#47

Post by ajss » Sat, Apr 21, 2018, 9:15 PM UTC

dizzystar wrote: post #43

augustl wrote: post #37

Earlier quoted context omitted.

An interesting point to add is that Datomic can also write to Riak. But since riak lacks an atomic CAS, zookeeper is used alongside Riak. What do you refer to when you say that Datomic inherits the speed of indexing a huge table? The query engine works by fetching blocks of data from storage and placing the blocks in a lazy immutable tree structure that are the indices of Datomic. They are also covering (the indices…

I don't know the full details, but I'm lead to understand that indexing a Datomic database too late is time-consuming and will end up causing considerable downtime. I'll admit that I'm guessing about how much of this depends on the database and Datomic itself, so I shouldn't be making a sweeping statement.

I think "late indexing" maybe referring to an implementation detail. The indexes aren't updated after each transaction. When you query the db, the result is a merge between the index and the transactions that haven't been indexed yet. The index is updated in batches.

Re: Datomic: Look at all the things I'm not doing

#48

Post by always_good » Sun, Apr 22, 2018, 12:46 AM UTC

blunte wrote: post #39

dizzystar wrote: post #36

Earlier quoted context omitted.

I whole heartedly disagree. Data doesn't care about your application.

I guess we all have our personal experiences to go by. My 24 years have thought me that perhaps 25% of the real need is know up front. And usually there is some pattern that one's problem seems to fit into. Then run with it until more details appear. Then adapt or add... On the other hand, I have worked with people who like to try to plan for all possibilities. That usually results in enormous data models and extra l…

On the other hand I think that's just a misnomer. That because you went through the trouble of writing a schema, it's somehow worse when you're wrong.

But people don't often look at it from the opposite perspective: that when you are explicit about the data in your system, then you can respond explicitly and immediately and actionably when it turns out you're wrong. The alternative is to not know when or how you're wrong.

For example, not having to lift a finger when you're wrong about the data is like suggesting your codebase might be more unintentionally useful after removing static analysis.