Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rich Hickey's new project: datomic.com

datomic.com

Post reply on HN
Print view View original

61–70 of 111 posts

Re: Rich Hickey's new project: datomic.com

#61

Post by swalsh »

edwardw wrote: post #55
spitfire wrote: post #45

Okay, I don't quite get this. The processing gets moved to the client. But what if the dataset involved is too large for the client to hold?

Get from server through network, of course. In the meantime, peers cache "facts" using LRU replacement policy.

So lets suppose I have several billion integers sitting in a data store, and I want to sort, count, and sum them. Do I have to collect all this data to my local cache first? What if millions of people are using my application who want the same value?

Re: Rich Hickey's new project: datomic.com

#62

Post by lukev »

mark_l_watson wrote: post #54

Reading through the site reminds me of append-only CouchDB (or even better, BigCouch) both Datomic datoms and CouchDB documents have time stamps so the state of data is available for different times. This looks new: local query peers that cache enough data to perform queries (I don't understand how that works, but it looks like indices might be local, with some data also cached locally). Also interesting that it seem…

It can use DynamoDB under the hood, as a data store.

The data storage system is strongly decoupled from the transactor and the peer, so there are a number of options, ranging from a filesystem to S3 to DynamoDB.

Re: Rich Hickey's new project: datomic.com

#63

Post by lukev »

swalsh wrote: post #61
edwardw wrote: post #55

Earlier quoted context omitted.

Get from server through network, of course. In the meantime, peers cache "facts" using LRU replacement policy.

So lets suppose I have several billion integers sitting in a data store, and I want to sort, count, and sum them. Do I have to collect all this data to my local cache first? What if millions of people are using my application who want the same value?

Remember, the 'peer' doesn't have to be embedded in your front-edge application (even though that's one use case). You could have a single 'peer' which sits on it's own beefy server expressly for this kind of calculation.

Re: Rich Hickey's new project: datomic.com

#64

Post by gfodor »

One question I have is the cold start problem. How can I ensure dropping in a new peer is not going to have a large negative effect on response times? With memcache, you can just prewarm a new node or have clients only round-robin it a few times per request to warm it up. It seems like pre-warming here is going to be more cumbersome since it's not a simple k-v store but will require you to pre-emptively run queries to get there. (Similar to Lucene.)

Edit: Rich's response here:

http://blog.fogus.me/2012/03/05/datomic/comment-page-1/#comm...

Seems to imply that non-cached performance won't be so bad anyway. Looking forward to seeing some benchmarks.

Re: Rich Hickey's new project: datomic.com

#65

Post by nickik »

swalsh wrote: post #60
nickik wrote: post #36

Earlier quoted context omitted.

What does this have to do with clojure? This system could have been build in any language, clojure only uses some of the same ideas (working with values). I cant help you with your question, sorry.

From my understanding the same guy wrote both. My point was that I personally view Clojure as a bit of genius.

Right. Agree.

Re: Rich Hickey's new project: datomic.com

#66

Post by spitfire »

edwardw wrote: post #55
spitfire wrote: post #45

Okay, I don't quite get this. The processing gets moved to the client. But what if the dataset involved is too large for the client to hold?

Get from server through network, of course. In the meantime, peers cache "facts" using LRU replacement policy.

So I need some sort of storage area network of clients. Okay, so I'm chopping up my database server somewhat, Making it easier to scale horizontally. I can live with this, but I wish it were stated explicitly.

The SaaS model they're offering won't work for the sorts of things I'm interested in.

Re: Rich Hickey's new project: datomic.com

#67

Post by lukev »

stevelosh wrote: post #56
lukev wrote: post #50

Earlier quoted context omitted.

Correct, it's not write-scalable in the same way it is read-scalable. The transactor is a bottleneck for writes. However, that doesn't mean it has slow writes - it should still do writes at least on a par with any traditional transactional database, and probably a good deal faster since it's append-only.

I'm more concerned with what happens when the transactor goes down, or gets silently partitioned from some of the clients. I assume reads will continue to work but all writes will break? I'd also like to know more about how the app-side caching works. If I've got a terabyte of User records and want to query for all users of a certain type, does a terabyte of data get sent over the wire, cached, and queried locally? O…

1. You're correct, however, the architecture does allow you to run a hot backup for fast failover.

2. The database is oriented around 'datoms', which are an entity/attribute/value/time. Each of these has its own (hierarchical) indexes, so you only end up pulling the index segments you need to fulfill a given query. You'd only pull 1TB if your query actually encompassed all the data you had.

Re: Rich Hickey's new project: datomic.com

#69

Post by swalsh »

lukev wrote: post #63
swalsh wrote: post #61

Earlier quoted context omitted.

So lets suppose I have several billion integers sitting in a data store, and I want to sort, count, and sum them. Do I have to collect all this data to my local cache first? What if millions of people are using my application who want the same value?

Remember, the 'peer' doesn't have to be embedded in your front-edge application (even though that's one use case). You could have a single 'peer' which sits on it's own beefy server expressly for this kind of calculation.

Thanks! This thing is actually a bit of genius.
Post reply on HN