Now that people are considering NOSQL will more people consider no-DB

I am working on a team that is building an in-memory SQL database. It features a custom language that makes it possible to push the time-critical, data-processing parts of the application directly to the database, which allows for the same speed as this no-DB approach. But you don't have to build your own DB and do everything yourself (correct persistence, backup, transactions...)

Correct. For the fraction of a millisecond the transaction is executing, anyhow. Since transactions only deal with data hot in RAM, transactions are very fast.

> No, I mean like "I already wrote some data, but now a constraint has been violated so I need to undo it".

That shouldn't happen, and I've used two approaches to make sure. One is do all your checking before you change anything. The other is to make in-command reversion easy, which is basically the same way you'd make commands undoable.

Basically, instead of solving the problem with very complicated technology (arbitrary rollback), you solve it with some modest changes in coding style. Since you never have to worry about threading issues, I've found it pretty easy.

As far as I can see, they seem to be comparable to b+tree indexed key value stores. A key would e.g. be "/home/user/test.txt". Thanks to the B+Tree "indexation" you can do a prefix scan and list folders (e.g. "ls /home/user/"--> all keys starting with "/home/user/").

In the case of e.g. ReiserFS they actually use B+Trees. They have a caching layer managed by the OS. Most of them have journaling which would be the equivalent of a "write ahead log".

Map reduce based "view" generation can easily be done by pipes and utilities like grep. We might be even able to do some sort of simplistic filtering/views/relations using symlinks.

I guess the main difference is that they aren't optimized for this database-like behavior from a performance standpoint and that the network interfaces to them are SMB/AFP/NFS.

If that's just saying that startup time can be an issue, I agree. There are a variety of techniques to mitigate that, though. The simplest is to compress snapshots and/or put them on RAID, boosting read speed. The most complicated is just to have mirrored servers and only restart the one not in use right now.

> I'm talking about having more data than fits in an reasonable amount of RAM (say 1TB).

For something where you need transactions across all of that? This architecture's probably not a reasonable approach, then. The basic precondition is that everything fits in RAM. However, sharding is certainly possible if you can break your data into domains across which you don't require consistent transactions.

> So it's going to at the speed of the disk.

Sort of.

Because it's just writing to a log, mutations go at the speed of streaming writes, which is very fast on modern disks. And there are a variety of techniques for speeding that up, so I'm not aware of a NoDB system for which write speed is the major problem.

Regardless, it's a lot better for writes than the performance of an SQL database on the same hardware.

> My other question is how much of a pain in the ass is it to debug such a system?

It seemed fine. A big upside is that you have a full log of every change, so there's no more "how did X get like Y"; if you want to know you just replay the log until you see it change.

Last I did this we used BeanShell to let us rummage through the running system. It was basically like the Rails console.

Basically instead of making a transaction between 2 entities, you send a message to the first reserving some data, a message to the second reserving the data and once you get confirmation from both (or however many entities are involved in the transaction) you send a commit to them.

These reservations can be revoked though. Your rollback has to be managed by an "activity".

Ex: Bank transfers. You have the activity called BankTransfer. It manages the communication between entities and the overall workflow. It starts by sending messages to entities Account#1 with 100$ in it and Account#2 also with 100$. To #1 it says debit 500$. To #2 it says credit 500$. #2 responds first and says Done. #1 responds second and says Insufficient Funds. BankTransfer sends another message to #2 saying Cancel event id 100 (the crediting).

Other activities that want to read the state of number 1 will see 100$ in it but the transfer (as yet unconfirmed) had been of 50 rather than 500$ and another debit of 75$ comes in it would respond insufficient funds. At this point it's the activity's job to decide what to do. Wait and try again? Fail entirely and notify any other entities relevant to the workflow? That's up to the business rules. Also, since the credit has not yet been confirmed, reading the balance on #2 would still say 100, not 600$.

Of course, depending on your use case you may want the read to return the balance with unconfirmed transactions. That's entirely up to the application code and business rules but the example should be explanatory as to how rollback is implemented.

Eventual consistency is the only scalable way to go for very large systems.