Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asami: A flexible graph store in Clojure

github.com

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Asami: A flexible graph store in Clojure

#11

Post by packetlost »

harryvederci wrote: post #10
packetlost wrote: post #7

Earlier quoted context omitted.

It's not like datomic claims to be lightening fast, it's not. The real problem is if you want to support the historical queries "as_of()", which is very difficult to do efficiently with only B-tree indexes. So you're either left maintaining your own indices as tables or externalizing it to something else. I think the latter would actually be a pretty ok option if your dataset fits entirely in memory (though... for hi…

I'm talking about Datalog, I mentioned Datomic purely in the context of syntax. And to sum up what I know about Datalog, it's basically just this article (not reference material, but it got me interested in the concept), plus some light wikipedia-ing: https://www.instantdb.com/essays/datalogjs Edit: I think you added the part with your git repo after I wrote this. Looks cool, could you add a license file to it?

Got it. Yeah, datalog is cool. I really like it (if that wasn't obvious from my attempt at implementing it). While not built on SQLite, this project has caught my attention recently: https://github.com/cozodb/cozo

It's built on RocksDB and has slightly different query syntax (supposedly to be more similar to Python's, as a primary target usecase is within Jupyter notebooks). Check it out!

Re: Asami: A flexible graph store in Clojure

#12

Post by packetlost »

quoll wrote: post #9
packetlost wrote: post #7

Earlier quoted context omitted.

It's not like datomic claims to be lightening fast, it's not. The real problem is if you want to support the historical queries "as_of()", which is very difficult to do efficiently with only B-tree indexes. So you're either left maintaining your own indices as tables or externalizing it to something else. I think the latter would actually be a pretty ok option if your dataset fits entirely in memory (though... for hi…

Not sure what you mean here? as_of() is easy to do efficiently, so long as you have tree indexes (it doesn't really matter what type of tree, though fractal trees would not be good). But you need direct access to the tree, rather than having an index that's built using a tree, since then you can't build the phases.

Right, if you have direct access to the tree it's possible and not really inefficient at all (a modified B+Tree would be my first pick, but I digress). Part of my problem was I was trying to offload as much of the work to SQLite as possible. I think if I were to go back and try again, I would make it a bit further :)

Re: Asami: A flexible graph store in Clojure

#13

Post by fiddlerwoaroof »

mark_l_watson wrote: post #3

I just spent ten minutes looking through the repo. I like how the query language is similar enough to Datomic’s to have an easy learning curve. I wish I could push a magic button and have this in Common Lisp (where I usually just use SQLite as an easy to use data store).

Do you happen to know of any good resources for implementing datalog? I’ve thought about writing something similar in Common Lisp several times, but it’s hard to find a good introduction here.

Re: Asami: A flexible graph store in Clojure

#14

Post by dms92n1bx »

harryvederci wrote: post #6
mark_l_watson wrote: post #3

I just spent ten minutes looking through the repo. I like how the query language is similar enough to Datomic’s to have an easy learning curve. I wish I could push a magic button and have this in Common Lisp (where I usually just use SQLite as an easy to use data store).

An idea that pops up more and more often in my head is to create a SQLite wrapper library that basically treats SQLite like Datalog. I know way too little about Datalog, but it goes something like this: 1. create helper functions to create/alter/drop tables. 2. create helper functions for insert/select/update/delete queries, etc. Probably with the same syntax as in Datomic / Asami / etc. 3. each table its name would…

There is (now unmaintained) project called Mentat [0] from Mozilla.

[0] https://github.com/mozilla/mentat

Re: Asami: A flexible graph store in Clojure

#16

Post by zh217 »

packetlost wrote: post #11
harryvederci wrote: post #10

Earlier quoted context omitted.

I'm talking about Datalog, I mentioned Datomic purely in the context of syntax. And to sum up what I know about Datalog, it's basically just this article (not reference material, but it got me interested in the concept), plus some light wikipedia-ing: https://www.instantdb.com/essays/datalogjs Edit: I think you added the part with your git repo after I wrote this. Looks cool, could you add a license file to it?

Got it. Yeah, datalog is cool. I really like it (if that wasn't obvious from my attempt at implementing it). While not built on SQLite, this project has caught my attention recently: https://github.com/cozodb/cozo It's built on RocksDB and has slightly different query syntax (supposedly to be more similar to Python's, as a primary target usecase is within Jupyter notebooks). Check it out!

More precisely, Cozo currently has a RocksDB backend. The development version also has SQLite backend (for easier compilation on mobile platforms), in-memory backend, etc. It now can run as a web assembly module as well (see https://cozodb.github.io/wasm-demo/)

With respect to the SQLite backend, Cozo uses SQLite purely as a KV store with range scan capabilities, with both keys and values stored as bytes. This actually results in faster queries than offloading some stuff to the SQLite engine for the (very few) cases we tested.

Disclaimer: I'm the author of Cozo.

Re: Asami: A flexible graph store in Clojure

#17

Post by packetlost »

zh217 wrote: post #16
packetlost wrote: post #11

Earlier quoted context omitted.

Got it. Yeah, datalog is cool. I really like it (if that wasn't obvious from my attempt at implementing it). While not built on SQLite, this project has caught my attention recently: https://github.com/cozodb/cozo It's built on RocksDB and has slightly different query syntax (supposedly to be more similar to Python's, as a primary target usecase is within Jupyter notebooks). Check it out!

More precisely, Cozo currently has a RocksDB backend. The development version also has SQLite backend (for easier compilation on mobile platforms), in-memory backend, etc. It now can run as a web assembly module as well (see https://cozodb.github.io/wasm-demo/ ) With respect to the SQLite backend, Cozo uses SQLite purely as a KV store with range scan capabilities, with both keys and values stored as bytes. This actua…

Oh wow, I didn't catch that. Awesome work man, I'm fan of Cozo and hope to make contributions in the future!

Re: Asami: A flexible graph store in Clojure

#18

Post by zh217 »

packetlost wrote: post #7
harryvederci wrote: post #6

Earlier quoted context omitted.

An idea that pops up more and more often in my head is to create a SQLite wrapper library that basically treats SQLite like Datalog. I know way too little about Datalog, but it goes something like this: 1. create helper functions to create/alter/drop tables. 2. create helper functions for insert/select/update/delete queries, etc. Probably with the same syntax as in Datomic / Asami / etc. 3. each table its name would…

It's not like datomic claims to be lightening fast, it's not. The real problem is if you want to support the historical queries "as_of()", which is very difficult to do efficiently with only B-tree indexes. So you're either left maintaining your own indices as tables or externalizing it to something else. I think the latter would actually be a pretty ok option if your dataset fits entirely in memory (though... for hi…

What you described can be implemented efficiently with B-trees. The problem is, the required operation is low-level and not usually exposed by the database engines. Basically you need to be able to walk the tree, up and down.

Say your table is `[data, timestamp]`, with data sorted ascendingly and timestamp sorted descendingly in the tree, and you want to scan for values for `as_of(T)`. Assume you have found `[data1, T1]` as a valid row. To get the next row, instead of the usual scanning, you seek to the value greater than `[data1, neg_inf]`. Now assume the value you get for the seek is `[data2, T2]`. If `T2 It is doable in SQLite, but you must use its VM directly, as you cannot do tree-walking with SQL.

Assume you have `M` keys and that for every key you have `N` timestamped data points. The above algorithm cuts down the as-of query time from linear complexity in `N` (`MN log(M)`) to logarithmic complexity in `N` (`M log(MN)`), which I think is the best we can do.

Re: Asami: A flexible graph store in Clojure

#20

Post by sitkack »

mark_l_watson wrote: post #3

I just spent ten minutes looking through the repo. I like how the query language is similar enough to Datomic’s to have an easy learning curve. I wish I could push a magic button and have this in Common Lisp (where I usually just use SQLite as an easy to use data store).

My first thought was a Truffle CommonLisp so you could integrate via the GraalVM.

https://github.com/charig/TCLisp

https://github.com/armedbear/abcl/issues/62

Post reply on HN