Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rich Hickey's new project: datomic.com

datomic.com

Post reply on HN
Print view View original

101–110 of 111 posts

Re: Rich Hickey's new project: datomic.com

#101

Post by weavejester »

danieljomphe wrote: post #98
weavejester wrote: post #96

Earlier quoted context omitted.

I don't believe there's such a thing as local writes in Datomic. All writes appear to go through the transactor to maintain atomicity.

Right. So the only way to make Peers resilient to network partitions is to install a middleman between them and the DB/Transactor. One whose responsibility is to ensure this Peer's app always has durable access to everything it's ever going to need to be able to read for its queries, and always has durable access to some local write log that doesn't exist in the current implementation. Thus my question is: is introdu…

I don't believe Datomic is designed to operate in a scenario where Peers don't have network connectivity. The local cache Peers keep is to cut down on network traffic and improve performance, not as a reliable "offline mode".

Re: Rich Hickey's new project: datomic.com

#102

Post by JulianMorrison »

This strikes me as yet another NoSQL with a niche in which it will be great. In this case, it's good for a read heavy application with minimal writing, where its working set is a small subset of the total data set and you care a lot about write consistency. It would fail in a smoking heap under heavy write load (single global lock, and the need to push every write to every client cache). It would blow the cache if you tried to do a range scan.

Re: Rich Hickey's new project: datomic.com

#103

Post by nickik »

JulianMorrison wrote: post #102

This strikes me as yet another NoSQL with a niche in which it will be great. In this case, it's good for a read heavy application with minimal writing, where its working set is a small subset of the total data set and you care a lot about write consistency. It would fail in a smoking heap under heavy write load (single global lock, and the need to push every write to every client cache). It would blow the cache if yo…

The Idea is that the transactor does a very small amount of work an does can scale much better then other "single point bottelneck". The problem is still there but smaller that way.

Read the comment here, it gives some information on the problem your discribing: http://blog.fogus.me/2012/03/05/datomic/

Re: Rich Hickey's new project: datomic.com

#104

Post by nickik »

weavejester wrote: post #101
danieljomphe wrote: post #98

Earlier quoted context omitted.

Right. So the only way to make Peers resilient to network partitions is to install a middleman between them and the DB/Transactor. One whose responsibility is to ensure this Peer's app always has durable access to everything it's ever going to need to be able to read for its queries, and always has durable access to some local write log that doesn't exist in the current implementation. Thus my question is: is introdu…

I don't believe Datomic is designed to operate in a scenario where Peers don't have network connectivity. The local cache Peers keep is to cut down on network traffic and improve performance, not as a reliable "offline mode".

Seams to be true but the intressting part is that peers can be made parallel and if one datacenter explodes you can go to an other without losing information. The only "Single Point of Failure" is the transactor and only for reads.

Re: Rich Hickey's new project: datomic.com

#105

Post by danieljomphe »

weavejester wrote: post #101
danieljomphe wrote: post #98

Earlier quoted context omitted.

Right. So the only way to make Peers resilient to network partitions is to install a middleman between them and the DB/Transactor. One whose responsibility is to ensure this Peer's app always has durable access to everything it's ever going to need to be able to read for its queries, and always has durable access to some local write log that doesn't exist in the current implementation. Thus my question is: is introdu…

I don't believe Datomic is designed to operate in a scenario where Peers don't have network connectivity. The local cache Peers keep is to cut down on network traffic and improve performance, not as a reliable "offline mode".

Notwithstanding what it's initially designed for, I think it may be quite good at supporting an "offline mode" as long as:

1. the app developer can confidently predict which queries the app will need through its lifespan, and

2. the app developer is willing to program and configure a layer that can persist and make durable a cache that spans all the data needed to run those queries (thus, persisting locally what amounts to a dynamic shard of the DB), and

3. the app developer is willing to program a layer that can persist and make durable all writes intended for the Transactor, and synchronize those to the Transactor when the app recuperates from a network partition, and

3.1. the app developer is willing to plan-or resolve-potential conflicts in advance of-or when-eventual conflicts, thus he's willing to sacrifice global consistency in the event of a network partition, in order to obtain availability, and

4. the app developer is willing to plug into the query engine in such a way that queries will include the local write log when there's a network partition.

Now, solution-wise:

1. depends on the requirements but most small to medium apps can predict the queries they'll need;

2. seems to be quite easy for small to medium apps:

2.1 run all possible queries at regular times, and

2.2 use a durable key-value store to keep the db values;

3. (1) make sure you're subscribed to events on partition and recovery; (2) coordinate writes over the same key-value store, probably using Clojure's STM and/or Avout; (3) on network recovery, replay those writes not present in the central DB;

3.1 due to the immutable nature of things and total ordering of the DB transactions, I expect to see no issue regarding eventual consistency when write logs are replayed centrally after a local Peer recovers from a network partition;

4. considering how Datalog works and is integrated into the Peer, this seems like a piece of cake.

So isn't this quite feasible to support the highly distributed case for apps in which each local Peer represents its own logical, dynamic and relatively natural and autonomous shard of the database?

Re: Rich Hickey's new project: datomic.com

#106

Post by gtani »

DanWaterworth wrote: post #91

This is pretty cool, it's very similar to a project I'm working on: Siege, a DBMS written in Haskell [1]. Siege uses roughly the same approach; I didn't know anyone else was working on a distributed immutable DBMS, so this is really exciting. [1] https://github.com/DanielWaterworth/siege

this was mentioned in /r/cloj

http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-17...

Re: Rich Hickey's new project: datomic.com

#107

Post by alkby »

programnature wrote: post #79
anamax wrote: post #77

Earlier quoted context omitted.

> a time (denoted by the transaction number that added it the database). Do transaction numbers have total order or just partial order? Total order is serializing. (And no, using real time as the transaction number doesn't help because it's impossible to keep an interesting number of servers time-synched.) Partial order is "interesting".

It is totally ordered. The transactor is a single point of failure. However, since its only job is doing the transactions, the idea is it can be faster than a database server that does both the transactions and the queries.

I think their statement about ACID is too bold.

How does somebody do read-"modify" style of transactions ?

Say I want to bump some counter. So I delete old fact and I establish new fact. But new fact needs to be exactly 1 + old value of counter. With transactions as simple "add this and remove that" you seemingly cannot do that. So it's not ACID. Right?

Re: Rich Hickey's new project: datomic.com

#108

Post by weavejester »

nickik wrote: post #104
weavejester wrote: post #101

Earlier quoted context omitted.

I don't believe Datomic is designed to operate in a scenario where Peers don't have network connectivity. The local cache Peers keep is to cut down on network traffic and improve performance, not as a reliable "offline mode".

Seams to be true but the intressting part is that peers can be made parallel and if one datacenter explodes you can go to an other without losing information. The only "Single Point of Failure" is the transactor and only for reads.

You mean only for writes :)

Re: Rich Hickey's new project: datomic.com

#109

Post by danieljomphe »

alkby wrote: post #107
programnature wrote: post #79

Earlier quoted context omitted.

It is totally ordered. The transactor is a single point of failure. However, since its only job is doing the transactions, the idea is it can be faster than a database server that does both the transactions and the queries.

I think their statement about ACID is too bold. How does somebody do read-"modify" style of transactions ? Say I want to bump some counter. So I delete old fact and I establish new fact. But new fact needs to be exactly 1 + old value of counter. With transactions as simple "add this and remove that" you seemingly cannot do that. So it's not ACID. Right?

From what I remember, compare-and-swap semantics are in place for that kind of case.

If that was not the case, you could still model such an order-dependent update as the fact that the counter has seen one more hit. Let the final query reduce that to the final count, and let the local cache implementation optimize that cost away for all but the first query, and then incrementally optimize the further queries when they are to see an increased count.

That said, I'm pretty sure I've seen the simpler CAS semantics support. (The CAS-successful update, if CAS is really supported, is still implemented as an "upsert", which means old counter values remain accessible if you query the past of the DB.)

Re: Rich Hickey's new project: datomic.com

#110

Post by richhickey »

alkby wrote: post #107
programnature wrote: post #79

Earlier quoted context omitted.

It is totally ordered. The transactor is a single point of failure. However, since its only job is doing the transactions, the idea is it can be faster than a database server that does both the transactions and the queries.

I think their statement about ACID is too bold. How does somebody do read-"modify" style of transactions ? Say I want to bump some counter. So I delete old fact and I establish new fact. But new fact needs to be exactly 1 + old value of counter. With transactions as simple "add this and remove that" you seemingly cannot do that. So it's not ACID. Right?

Transactions are not limited to add/retract. There are also things we call data functions, which are arbitrary, user-written, expansion functions that are passed the current value of the db (within the transaction) and any arbitrary args (passed in the call), and that emit a list of adds/retracts and/or other data function calls. This result gets spliced in place of the data function call. This expansion continues until the resulting transaction is only asserts/retracts, then gets applied. With this, increments, CAS and much more are possible.

We are still finalizing the API for installing your own data functions. The :db.fn/retractEntity call in the tutorial is an example of a data function. (retractEntity is built-in).

This call:

    [:db.fn/retractEntity entity-id]
must find all the in- and out-bound attributes relating to that entity-id (and does so via a query) and emit retracts for them. You will be able to write data functions of similar power. Sorry for the confusion, more and better docs are coming.
Post reply on HN