This is pretty cool, it's very similar to a project I'm working on: Siege, a DBMS written in Haskell [1]. Siege uses roughly the same approach; I didn't know anyone else was working on a distributed immutable DBMS, so this is really exciting.
Rich Hickey's new project: datomic.com
91–100 of 111 posts
Re: Rich Hickey's new project: datomic.com
#92Earlier quoted context omitted.
So lets suppose I have several billion integers sitting in a data store, and I want to sort, count, and sum them. Do I have to collect all this data to my local cache first? What if millions of people are using my application who want the same value?
Remember, the 'peer' doesn't have to be embedded in your front-edge application (even though that's one use case). You could have a single 'peer' which sits on it's own beefy server expressly for this kind of calculation.
But ten what is then what us the benefit over just using Postgres there? And tossing a transaction ID sequence number onto your rows?
Re: Rich Hickey's new project: datomic.com
#93Earlier quoted context omitted.
Remember, the 'peer' doesn't have to be embedded in your front-edge application (even though that's one use case). You could have a single 'peer' which sits on it's own beefy server expressly for this kind of calculation.
But ten what is then what us the benefit over just using Postgres there? And tossing a transaction ID sequence number onto your rows?
It's about writing software which is simpler and more flexible.
If you write a traditional shared-nothing web app client with a traditional bag-o-sprocs database server, you'll probably be good as long as your workload doesn't change much. Assuming your write volume never exceeds that of a single (as beefy as necessary) server (which seems to be working out so far for Hacker News!) then you're ok.
However, products/services evolve and requirements change. Let's assume, for example, that you want to do some heavy duty number crunching. This number crunching involves some critical business logic calculations. Some of those calculations are in sprocs, but some of them are in your application code's native language. How do you offload that work to another server? You may have to juggle logic across that sproc/app boundary back and forth. It's pretty rigid; change is hard.
You can think of Datomic as a way of eliminating your sproc language and moving the query engine, indexes, and data itself into your process. Basically, you get everything you need to write your own database server. Furthermore, you can write specialized database servers for specialized needs... as long as you agree to allow a single Transactor service to coordinate your writes.
Back to the big number crunching. You've got the my-awesome-app process chugging along & you don't want to slow it down with your number crunching, so you spin up a my-awesome-cruncher peer & the data gets loaded up over there. Now you have the full power of your database engine in (super fast) memory and you can take your database-client-side business logic with you!
Now let's say you're finding that you're spending a lot of CPU time doing HTML templating and other web-appy like things. Well, you can trivially make additional my-awesome-app peers to share the work load.
You can do all this from a very simple start: One process on one machine. Everything in memory. Plain-old-java-objects treated the same as datums. No data modeling impedance mismatch. No network latency to think about. You can punt on a lot of very hard problems without sacrificing any escape hatches. You get audit trails and recovery mechanisms virtually for free.
Again, all this assumes the write-serialization trade offs are acceptable. Considering the prevalence and success of single-master architectures in the wild, that's not a hugely unacceptable tradeoff. Furthermore, the append-only model may enable even higher write speeds than something like Postgres' more traditional approach.
I hope my rant is helpful :-)
Re: Rich Hickey's new project: datomic.com
#94Earlier quoted context omitted.
> a time (denoted by the transaction number that added it the database). Do transaction numbers have total order or just partial order? Total order is serializing. (And no, using real time as the transaction number doesn't help because it's impossible to keep an interesting number of servers time-synched.) Partial order is "interesting".
It is totally ordered. The transactor is a single point of failure. However, since its only job is doing the transactions, the idea is it can be faster than a database server that does both the transactions and the queries.
Isn't this the same compromise we would already have had to make if we just used postgres?
Re: Rich Hickey's new project: datomic.com
#95If I read correctly, it is pretty expensive. $0.10 / connection (peer) / hour, plus dynamodb and transactor instance charges. For 100 clients, and not including the dynamodb or transactor instance(s), this makes it a hair more per year then a quad core oracle instance.
The $0.10 price tag is for the transactor, which just ensures that writes are atomic (but doesn't do any writing itself). I guess we'll need to do some benchmarking, but I suspect that $0.10 per peer per hour won't be as expensive as it first seems.
Re: Rich Hickey's new project: datomic.com
#96Thus Datomic would be very great for centrally-operated systems, but not so much with highly distributed systems where many peers are often partitioned out because, for example, they have no Internet connectivity for a few days, and they still need to operate within their limited universe. So if such a highly distributed system was to use Datomic, it would be harder to guarantee that each peer can work both for reads…
I don't believe there's such a thing as local writes in Datomic. All writes appear to go through the transactor to maintain atomicity.
Re: Rich Hickey's new project: datomic.com
#97As a developer, I find datomic easy to use. The getting started, running examples, tutorial, reference, the in-memory environment, the downloadable appliance - everything is so smooth. Last time I had a similar feeling was when I tried CloudFoundry.
Things should be like this - intuitive, some seed data and kickstart code w/ just enough documentation for when you get stuck.
Re: Rich Hickey's new project: datomic.com
#98Thus Datomic would be very great for centrally-operated systems, but not so much with highly distributed systems where many peers are often partitioned out because, for example, they have no Internet connectivity for a few days, and they still need to operate within their limited universe. So if such a highly distributed system was to use Datomic, it would be harder to guarantee that each peer can work both for reads…
I don't believe there's such a thing as local writes in Datomic. All writes appear to go through the transactor to maintain atomicity.
Right. So the only way to make Peers resilient to network partitions is to install a middleman between them and the DB/Transactor. One whose responsibility is to ensure this Peer's app always has durable access to everything it's ever going to need to be able to read for its queries, and always has durable access to some local write log that doesn't exist in the current implementation.
Thus my question is: is introducing such a middleman in the system going to denaturate Datomic?
Re: Rich Hickey's new project: datomic.com
#99Earlier quoted context omitted.
It is totally ordered. The transactor is a single point of failure. However, since its only job is doing the transactions, the idea is it can be faster than a database server that does both the transactions and the queries.
Hmm... presumably an application can act in read-only mode in the absence of a transactor. That's an interesting thought :-)
The transactor is only used for writes, so if the transactor went down, you could still run queries.
Re: Rich Hickey's new project: datomic.com
#100Earlier quoted context omitted.
It is totally ordered. The transactor is a single point of failure. However, since its only job is doing the transactions, the idea is it can be faster than a database server that does both the transactions and the queries.
Isn't this the same compromise we would already have had to make if we just used postgres?
It's actually slightly better than a SQL database. If your master SQL database gets fried, there's a chance you could lose some data. Datomic's transactor only handles atomicity, not writes, so if the transactor dies, nothing written to the database will be lost.