81–90 of 147 posts
After reading the article and all the comments here, and from my own experience, I just don't think it's possible to not have a DB. At best, you write your own basic DB, because you don't need anything fancy. For example, you write S-expressions to files like Hacker News does. This is clever, because the file system has some of the features of a database system, and files and S-expressions are abstractions that alrea…
In many applications, data outlives code. This is certainly the case in enterprise applications, where data can sometimes migrate across several generations of an application. Data may also be more valuable to the organization than the code that processes it. While I'm no fan of databases, one obvious advantage is that they provide direct access to the data in a standard way that is decoupled from the specific applic…
When people set out to design a SQL database, they usually end up updating and deleting records. This is bad because it destroys history, and nothing that you can add to your SQL architecture will fix it at a fundamental level. By basing your system on a journaled event stream, you start with a foundation of complete history retention, and you can build exactly the sort of reporting views you need at any time (say, b…
That leaves tracking the remaining inserts and updates, which is a well understood problem. It's called Auditing. Here is a simple script that will auto-audit a SQL Server database... variations in other SQL dialects are likely just as straighforward.
http://www.geekzilla.co.uk/ViewECBC0CC3-1C7E-4E7E-B243-F2F25...
Earlier quoted context omitted.
I can't speak to all systems like this, but the Prevayler approach is pretty straightforward. Most importantly, there are no simultaneous transactions: changes happen one at a time. That seems crazy if you're used to dealing with disk-backed databases, but if everything is hot in RAM, then it's not a problem. In that context, it's pretty easy: when you start executing a change you verify all your constraints before d…
"Most importantly, there are no simultaneous transactions... but if everything is hot in RAM, then it's not a problem" Uh, OK. So, you're happy with single-core boxes then, I take it? Actually, you're regressing to even before that, when there was no pre-emptive multitasking. When the program is done doing something, it yields control to some other task. Also, I'd like to point out that just because you aren't explic…
Not at all. You use only one core for the core execution of write transactions, but that's a small part of any real system. All cores can read simultaneously. All cores can also do all sorts of other work, including preparing transactions to execute, deserialization of requests, rendering responses, logging, and anything else your app needs to get up to.
The limit is also one core per transactional domain. If you can split your data up into lumps between which you never need transactions, you can happily run one core on each.
> Also, I'd like to point out that just because you aren't explicitly doing I/O doesn't mean that you aren't doing I/O.
Actually, it does explicitly do I/O. You do it just before every command executes.
> The OS might have paged out some stale data.
I guess that's possible, which would indeed cause a momentary pause, but this approach is typically used with dedicated servers and plenty of RAM, so it's never been a problem in practice for me.
> I just want to clarify: so when you encounter a problem, you do some rollback, which automatically moves the state to the last snapshot and rolls forward to the previous transaction, right?
You mean a bug in our code that causes a problem? Depends, on the system, I suppose. Prevayler had an automatic rollback. It just kept two copies of the data model in RAM; if a transaction blew up it would throw out the possibly tainted one. But there are a number of ways to solve this, so I don't advocate anything in particular. Other than heavy unit testing, so that things don't blow up much.
Earlier quoted context omitted.
Ever hear of ANSI SQL?
Which is not turing complete. You need database specific extensions to get that. Edit: Also, not all databases follow the standard very closely
It stores data just fine and is not vendor specific.
For those who want to take this kind of approach (object prevalence) in Common Lisp see http://common-lisp.net/project/cl-prevalence/ Sven Van Caekenberghe (the author of cl-prevalence) and I used this approach to power the back-end/cms of a concert hall back in 2003. A write-up of our experiences can be found at http://homepage.mac.com/svc/RebelWithACause/index.html The combination of a long-running Lisp image with…
Earlier quoted context omitted.
"Most importantly, there are no simultaneous transactions... but if everything is hot in RAM, then it's not a problem" Uh, OK. So, you're happy with single-core boxes then, I take it? Actually, you're regressing to even before that, when there was no pre-emptive multitasking. When the program is done doing something, it yields control to some other task. Also, I'd like to point out that just because you aren't explic…
> Uh, OK. So, you're happy with single-core boxes then, I take it? Not at all. You use only one core for the core execution of write transactions, but that's a small part of any real system. All cores can read simultaneously. All cores can also do all sorts of other work, including preparing transactions to execute, deserialization of requests, rendering responses, logging, and anything else your app needs to get up…
Unless someone is writing, of course, in which case you have to worry about isolation. So a single writer would block all readers, right?
"You mean a bug in our code that causes a problem?"
No, I mean like "I already wrote some data, but now a constraint has been violated so I need to undo it".
To the extent that you could actually write your program as a pure function of its past input history — ideally, one whose only O(N) part (where N was the length of the history) was a fold, so the system could update it incrementally as new events were added — you could get schema upgrade and decentralization "for free". However, to get schema upgrade and decentralization, your program would need to be able to cope with "impossible" input histories — e.g. the same blog post getting deleted twice, or someone commenting on a post they weren't authorized to read — because of changes in the code over the years and because of distribution.
I called this "rumor-oriented programming", because the propagation of past input events among the nodes resembles the propagation of rumors among people: http://lists.canonical.org/pipermail/kragen-tol/2004-January...
I wrote a bit more on a possible way of structuring web sites as lazily-computed functions of sets of REST resources, which might or might not be past input events: http://lists.canonical.org/pipermail/kragen-tol/2005-Novembe...
John McCarthy's 1998 proposal, "Elephant", takes the idea of writing your program as a pure function of its input history to real-time transaction processing applications: http://www-formal.stanford.edu/jmc/elephant/elephant.html
The most advanced work in writing interactive programs as pure functions of their input history is "functional reactive programming", which unfortunately I don't understand properly. The Fran paper http://conal.net/papers/icfp97/ is a particularly influential, and there's a page on HaskellWiki about FRP: http://www.haskell.org/haskellwiki/Functional_Reactive_Progr...
Why not go for NoDisk solution too - just RAID your memory and back it up with ultracapacitors?
answering directly to the subject: i do hope so. SQL too often introduces only a layer of complexity between the server-side application and the storage, while most of times an application could be designed to just use the filesystem, which is a database on its own by the way: it's a big, usually efficient, lookup table that maps keys (file paths) to values (file contents). why store passwords through SQL when a serv…
Also, see maildir and various things associated with qmail.
Earlier quoted context omitted.
> In my view, direct access to data decoupled from application code is a bug, not a feature. With multiple code bases touching the same data, schema improvements become nearly impossible. It's unclear that multiple applications with direct access to said data make schema improvements any easier. The obvious solution, copying the data for applications that are using the new schema, pretty much guarantees that one or m…
> How do you guarantee that all of the apps that touch that data use the current version of said code? In a NoDB app? It's very easy: only one code base ever directly touches the data, because the data lives in the RAM allocated to the app. You give external access via an API, so integrity is very easy to enforce.