71–80 of 147 posts
For those who want to take this kind of approach (object prevalence) in Common Lisp see http://common-lisp.net/project/cl-prevalence/ Sven Van Caekenberghe (the author of cl-prevalence) and I used this approach to power the back-end/cms of a concert hall back in 2003. A write-up of our experiences can be found at http://homepage.mac.com/svc/RebelWithACause/index.html The combination of a long-running Lisp image with…
I've always wanted "no-DB" to the level of it being part of the platform/language. I've always thought that software-transactional memory and persistent distributed heaps would get us there. Unfortunately the nearest things have been Redis and Terracotta plugged into Clojure. It should be: Insert? new an object. Delete? dispose an object. Look up? Hash table. Solved problems that just require persistence.
There are some neat little libraries that help with this in Clojure. The basic idea is that when you introduce changes through transactions, the actual transaction (code) is appended to disk (which is very fast), and this becomes your "database" file. So, what is persisted is just a list of state changes that are "replayed" to restore state. The individual transactions could also easily be distributed to multiple nod…
Earlier quoted context omitted.
Yes, you can definitely do it either way. Years ago as a demo a friend built the heart of a financial exchange in stored procedures. It was very fast, and very reliable. But the same is true about the LMAX system that Fowler describes. Personally, though, I'd much rather do my important coding in a real programming language. Better tools, more libraries, bigger communities, and no vendor lock-in.
Hmmm. Modern SQL dialects are Turing complete and frankly pretty rich dialects. I know MS SQL Server best so can't speak in detail for others, but the community around that is certainly very substantial. Library support, well, doesn't work quite the same way (yet!) but there's plenty of libraries of code samples available for adapting. Vendor lock-in is a tricky one; by the time you've got to a certain scale of appli…
I'm sure the developer community for MS SQL Server is reasonably large, but it is much, much smaller than the Ruby community or the Java community. The same is true for library code for each environment.
I think the Facebook example cuts the other way. If they had implemented all their core application logic in MS SQL Server, they would have been well and truly screwed if the performance wasn't enough. With PHP, at least they could write their own compiler; trying to reverse-engineer MS SQL Server is orders of magnitude harder.
Regarding the "volumes of data being passed around" part, that works well with a hot-in-RAM system; no data is passed anywhere. As with doing it all in stored procedures, no data leaves the server.
I do agree that a lot of people don't get full value out of their database. Sometimes that's a very reasonable business choice: vendor lock-in is extremely expensive at scale. But it does often come from ignorance. On the other hand, almost every developer has written a few database-backed applications, but very few have written anything in the NoDB style. Many can't even conceive of coding something without a database. I'd love to see both sorts of ignorance reduced.
Earlier quoted context omitted.
"I also think data integrity is easier to maintain with a system like this." If you are in the middle of a transaction and you realize that some constraint is being violated, how do you roll it back without interfering with the other transactions?
I can't speak to all systems like this, but the Prevayler approach is pretty straightforward. Most importantly, there are no simultaneous transactions: changes happen one at a time. That seems crazy if you're used to dealing with disk-backed databases, but if everything is hot in RAM, then it's not a problem. In that context, it's pretty easy: when you start executing a change you verify all your constraints before d…
Uh, OK. So, you're happy with single-core boxes then, I take it?
Actually, you're regressing to even before that, when there was no pre-emptive multitasking. When the program is done doing something, it yields control to some other task.
Also, I'd like to point out that just because you aren't explicitly doing I/O doesn't mean that you aren't doing I/O. The OS might have paged out some stale data (quite likely, since you aren't managing I/O yourself), and you might be holding the giant lock while it's paging it in.
"I can't speak to all systems like this, but the Prevayler approach is pretty straightforward."
I just want to clarify: so when you encounter a problem, you do some rollback, which automatically moves the state to the last snapshot and rolls forward to the previous transaction, right? No manual steps?
I hope you have a recent snapshot or that will be a long wait (while holding the big lock, I might add).
No matter how skilled I become as a developer, there is always something lurking around the corner to make me feel more naive than ever. As I was reading this article, I realized that my whole career and knowledge about the way applications work is based around the one core idea that when non-binary data needs to be persisted, you use a database. The idea that you can reliably use event sourcing in memory to persist…
You're already familiar with a couple of things that can be built this way: word processors and multiplayer game servers. In both cases SQL databases are too slow and too awkward. Financial trading is another area where databases are too slow. I know of one place that uses this approach to keep pricing data hot in RAM for their financial models. And Fowler previously documented using this for a financial exchange: ht…
Of course, Fowler talks about 'snapshotting' the data, which, again, makes me wonder if playing with all of this resident memory and the systems needed to make that happen haven't already been solved by...um...databases.
Earlier quoted context omitted.
The goal is not to replace databases altogether. The goal is to solve some particular problems very well. Last time I used this approach, for example, we mirrored a bunch of data in a traditional SQL store for reporting and ad-hoc querying, things that databases are great at. In my view, direct access to data decoupled from application code is a bug, not a feature. With multiple code bases touching the same data, sch…
> In my view, direct access to data decoupled from application code is a bug, not a feature. With multiple code bases touching the same data, schema improvements become nearly impossible. It's unclear that multiple applications with direct access to said data make schema improvements any easier. The obvious solution, copying the data for applications that are using the new schema, pretty much guarantees that one or m…
In a NoDB app? It's very easy: only one code base ever directly touches the data, because the data lives in the RAM allocated to the app. You give external access via an API, so integrity is very easy to enforce.
What about transaction isolation? How do you keep one transaction from seeing partial results from a concurrent transaction? Sounds like a recipe for a lot of subtle bugs.
And all of the assumptions you need to make for this no-DB approach to be feasible (e.g. fits easily in memory) might hold at the start of the project, but might not remain valid in a few months or years. Then what? You have the wrong architecture and no path to fix it.
And what's the benefit to all of this? It's not like DBMS do disk accesses just for fun. If your database fits easily in memory, a DBMS won't do I/O, either. They do I/O because either the database doesn't fit in memory or there is some better use for the memory (virtual memory doesn't necessarily solve this for you with the no-DB approach; you need to use structures which avoid unnecessary random access, like a DBMS does).
I think it makes more sense to work with the DBMS rather than constantly against it. Try making simple web apps without an ORM. You might be surprised at how simple things become, particularly changing requirements. Schema changes are easy unless you have a lot of data or a lot of varied applications accessing it (and even then, often not as bad as you might think) -- and if either of those things are true, no-DB doesn't look like a solution, either.
This has limited use because of Maintenance: I can easily give a 10% raise to everyone with a single SQL statement. Fowler's method requires that I first create an entire infrastructure (transaction processing, ACID properties) in code for this particular application. And it had better be as reliable as the transaction processing available in modern relational databases (so says my boss) or I'll be looking for a new…
As to creating the infrastructure and worries about reliability, there are a number of frameworks for this. E.g., Prevayler. It gives you all the ACID guarantees, but has about three orders of magnitude less code than a modern database.
Supporting it could definitely be a problem. That's true for anything novel, so I'd only do this where the (major) performance benefits outweigh the support cost.
Some kinds of statistics are easier with this. For example, if you want to keep a bunch of up-to-date stats on stocks (latest price, highs, lows, and moving averages for last hour, day, and week) it is almost trivially easy in a NoDB system, and much, much faster than with a typical SQL system.
For other stats and reporting, though, dumping to an SQL database is great. For many systems you don't want to use your main database for statistics anyhow, so a NoDB approach mainly means you start using some sort of data warehouse a little earlier.
Wouldn't this system have a bunch of drawbacks: - Long startup times as the entire image needs to be loaded and prepared. - It would be hard to distribute the state across multiple nodes - What happens in case of a crash? How fault tolerant would this be? - Does this architecture essentially amount to building in a sort-of-kind-of datastore into your already complex application? Without a well-defined well-tested exi…
- The startup times can be a problem if you have a lot of data. Modern disks are pretty fast for streaming reads, though, and you can split the deserialization load across multiple processors. - Mirroring state is easy; you just pipe the serialized commands to multiple boxes. - It's very fault tolerant. Because every change is logged before being applied, you just load the last snapshot and replay the log. - It didn'…
> - The startup times can be a problem if you have a lot of data. Modern disks are pretty fast for streaming reads, though, and you can split the deserialization load across multiple processors.
Reading data, at even a GB/second from disk (which is currently not possible) is going to mean a second spent of a GB of data, just to read, let alone deserialize. That's with reading a snapshot, not replaying old transactions.
> - Mirroring state is easy; you just pipe the serialized commands to multiple boxes.
That's not distributing the load. I'm talking about having more data than fits in an reasonable amount of RAM (say 1TB). Also mirroring is nice for when you want read-only access to your data. You'll have the same problem as any other data store when you want multiple writers. Also, is replication synchronous or asynchronous (which end of CAP do you fall on)?
>- It's very fault tolerant. Because every change is logged before being applied, you just load the last snapshot and replay the log.
So it's going to at the speed of the disk then (http://smackerelofopinion.blogspot.com/2009/07/l1-l2-ram-and...). Don't get me wrong, this is still faster than writing to the network, but then writes are way slower than reads.
My other question is how much of a pain in the ass is it to debug such a system? I suppose if you have a nice offline API to look at your data, change something, revert back, etc, it would work well, but if it's deep within your normal application, it could become nightmarish.
Earlier quoted context omitted.
- The startup times can be a problem if you have a lot of data. Modern disks are pretty fast for streaming reads, though, and you can split the deserialization load across multiple processors. - Mirroring state is easy; you just pipe the serialized commands to multiple boxes. - It's very fault tolerant. Because every change is logged before being applied, you just load the last snapshot and replay the log. - It didn'…
- Mirroring state is easy; you just pipe the serialized commands to multiple boxes. What? No, that's ridiculous. That's how inconsistencies crop up. Unless you plan on locking the entire system during each command.
That seems ridiculous if you are thinking like databases do, in terms of taking away the pain of all those disk seeks needed to write something. But if everything is hot in RAM, executing a command is extremely fast. Much faster than a database.
If that still isn't fast enough, you can split your data graph into chunks that don't require simultaneous locking and have one lock per. For example, if you are making a stock exchange like the LMAX people were, you can have one set of data (and one lock) per stock.