Live data from Hacker News

Now that people are considering NOSQL will more people consider no-DB

martinfowler.com

61–70 of 147 posts

Re: Now that people are considering NOSQL will more people consider no-DB

#61

No matter how skilled I become as a developer, there is always something lurking around the corner to make me feel more naive than ever. As I was reading this article, I realized that my whole career and knowledge about the way applications work is based around the one core idea that when non-binary data needs to be persisted, you use a database. The idea that you can reliably use event sourcing in memory to persist…

I don't know if it uses "event sourcing" per se, but doesn't HN use in-memory & serialized Lisp data structures instead of a DB?

As did Viaweb.

Re: Now that people are considering NOSQL will more people consider no-DB

#62
post #57
post #49

Earlier quoted context omitted.

Yes, you can definitely do it either way. Years ago as a demo a friend built the heart of a financial exchange in stored procedures. It was very fast, and very reliable. But the same is true about the LMAX system that Fowler describes. Personally, though, I'd much rather do my important coding in a real programming language. Better tools, more libraries, bigger communities, and no vendor lock-in.

Ever hear of ANSI SQL?

Which is not turing complete. You need database specific extensions to get that.

Edit: Also, not all databases follow the standard very closely

Re: Now that people are considering NOSQL will more people consider no-DB

#63
After reading the article and all the comments here, and from my own experience, I just don't think it's possible to not have a DB. At best, you write your own basic DB, because you don't need anything fancy.

For example, you write S-expressions to files like Hacker News does. This is clever, because the file system has some of the features of a database system, and files and S-expressions are abstractions that already exist. You do have to manage what data is in memory and what data is on disk at any given time, but the complexity and amount of code are low.

The idea that "event sourcing" somehow keeps you from needing a DB is ridiculous. By the time you've defined the event format, and written the software to replay the logs, etc., which if you're smart will be fairly general and modular, congrats, you've just written a database. At best, you keep complexity low, and it's another example of a small custom DB for a case where you don't need a fancy off-the-shelf DB. Maybe it's the perfect solution for your app, but it's still a database.

"Memory images," as a completely separate matter, are an abstraction that saves you some of the work of making a DB. Just as S-expressions can save you from defining a data model format, and files can save you from a custom key-value store, memory images as in Smalltalk could save you from having to deal with persistence. And if your language has transactions built in, maybe that saves you from writing your own transaction system. In general, though, it's very hard to get the DB to disappear, as there is a constellation of features important to data integrity that you need one way or another. It's usually pretty clear that you're using a DB, writing a DB, or using a system that already has a DB built in. If you think there's no DB, there's a high chance you're writing one. Again, that could be fine if you don't need all the features and properties of a robust general-purpose DB.

Funnily enough, in EtherPad's case, we had full versioned history of documents, and did pretty much everything in RAM and application logic -- a pretty good example of what the article is talking about -- and yet we used MySQL as a "dumb" back-end datastore for persistence. Believe me, we tried not to; we spent weeks trying alternatives, and trying to write alternatives. Perhaps if every last aspect of the data model had been event-based, we could have just logged the events to a text file and avoided SQL. More likely, I think, we would use something like Redis now.

Re: Now that people are considering NOSQL will more people consider no-DB

#64
post #57

Earlier quoted context omitted.

Ever hear of ANSI SQL?

Which is not turing complete. You need database specific extensions to get that. Edit: Also, not all databases follow the standard very closely

Aren't vendors finally starting to abandon their proprietary crap in favor of SQL/PSM?

Re: Now that people are considering NOSQL will more people consider no-DB

#67
I spent about a year as a maintainer of FlockDB, Twitter's social graph store. If you don't know, it's basically a sharded MySQL setup. One of the key pain points was optimizing the row lock over the follower count. Whenever a Charlie Sheen joins, or someone tries to follow spam us, one particular row would get blasted with concurrent updates.

Doing this in-memory in java via someAtomicLong.incrementAndGet() sounds appealing.

Re: Now that people are considering NOSQL will more people consider no-DB

#68
post #36

Earlier quoted context omitted.

The goal is not to replace databases altogether. The goal is to solve some particular problems very well. Last time I used this approach, for example, we mirrored a bunch of data in a traditional SQL store for reporting and ad-hoc querying, things that databases are great at. In my view, direct access to data decoupled from application code is a bug, not a feature. With multiple code bases touching the same data, sch…

"I also think data integrity is easier to maintain with a system like this." If you are in the middle of a transaction and you realize that some constraint is being violated, how do you roll it back without interfering with the other transactions?

I can't speak to all systems like this, but the Prevayler approach is pretty straightforward. Most importantly, there are no simultaneous transactions: changes happen one at a time. That seems crazy if you're used to dealing with disk-backed databases, but if everything is hot in RAM, then it's not a problem. In that context, it's pretty easy: when you start executing a change you verify all your constraints before doing anything.

Re: Now that people are considering NOSQL will more people consider no-DB

#69

In many applications, data outlives code. This is certainly the case in enterprise applications, where data can sometimes migrate across several generations of an application. Data may also be more valuable to the organization than the code that processes it. While I'm no fan of databases, one obvious advantage is that they provide direct access to the data in a standard way that is decoupled from the specific applic…

When people set out to design a SQL database, they usually end up updating and deleting records. This is bad because it destroys history, and nothing that you can add to your SQL architecture will fix it at a fundamental level. By basing your system on a journaled event stream, you start with a foundation of complete history retention, and you can build exactly the sort of reporting views you need at any time (say, b…

The storage system of Postgres, one of the earliest rdbms, uses MVCC. The early version even allows you to roll back or query historical data.

http://wiki.postgresql.org/wiki/MVCC

Re: Now that people are considering NOSQL will more people consider no-DB

#70
post #12

Wouldn't this system have a bunch of drawbacks: - Long startup times as the entire image needs to be loaded and prepared. - It would be hard to distribute the state across multiple nodes - What happens in case of a crash? How fault tolerant would this be? - Does this architecture essentially amount to building in a sort-of-kind-of datastore into your already complex application? Without a well-defined well-tested exi…

- The startup times can be a problem if you have a lot of data. Modern disks are pretty fast for streaming reads, though, and you can split the deserialization load across multiple processors. - Mirroring state is easy; you just pipe the serialized commands to multiple boxes. - It's very fault tolerant. Because every change is logged before being applied, you just load the last snapshot and replay the log. - It didn'…

- Mirroring state is easy; you just pipe the serialized commands to multiple boxes.

What? No, that's ridiculous. That's how inconsistencies crop up. Unless you plan on locking the entire system during each command.

Post reply on HN