101–110 of 147 posts
After reading the article and all the comments here, and from my own experience, I just don't think it's possible to not have a DB. At best, you write your own basic DB, because you don't need anything fancy. For example, you write S-expressions to files like Hacker News does. This is clever, because the file system has some of the features of a database system, and files and S-expressions are abstractions that alrea…
After reading the article and all the comments here, and from my own experience, I just don't think it's possible to not have a DB. At best, you write your own basic DB, because you don't need anything fancy. For example, you write S-expressions to files like Hacker News does. This is clever, because the file system has some of the features of a database system, and files and S-expressions are abstractions that alrea…
This sort of reminds me of some of the schema involved in OLAP databases for tracking changes over time and the ability to retrieve results and data for any specific point in time. Albiet with a less constricted implementation.
I am 100% agreeing with the article, with one caveat. Database engines are not just for storing - each is basically a "utility knife" of data retrieval - indexing, sorting and filtering are available via (relatively) simple SQL constructs. If your app uses an index right now, ditching the DB will mean re-implementing it manually. It's not hard, but it's extra code. So basically, the DB engine might still be a necessa…
The SQL constructs are great, but the biggest advantage to relational databases is that the engine handles your data consistency issues for you. Consistency isn't just about rolling the datastore back to a specific moment in time -- you have to handle locking, concurrent reads/writes, etc. If you're building a trading platform that handles 6M transactions/second, you have the money to handle this in the application l…
Standard tools are useful because you can get to working code fast ... this is why LAMP is still such a powerful framework upon which to build. While it may make sense to consider adding a search indexer (Solr) or key-value cache (Redis), for almost every use case, rewriting data storage is a waste.
Also, to paraphrase Ted Dziuba, it probably doesn't matter if your product doesn't scale, because nobody cares, or will ever use it. So I think it is better to get something up and running quickly to see if anyone cares before you bother trying to optimize for the rare case where your product turns out to be the next Twitter.
Earlier quoted context omitted.
When people set out to design a SQL database, they usually end up updating and deleting records. This is bad because it destroys history, and nothing that you can add to your SQL architecture will fix it at a fundamental level. By basing your system on a journaled event stream, you start with a foundation of complete history retention, and you can build exactly the sort of reporting views you need at any time (say, b…
When people set out to design a data driven application, they usually end up updating and deleting records. FTFY... It's not hard to build history into a SQL table design. You can even store events in a...wait for it... SQL database. I have built numerous systems backed by SQL databases that have complete history retention. Answering questions like 'who had id 'X' on this date 3 years ago' are easily solvable with ba…
If your system operates in this journaled/event-sourcing way at the most basic level then you have the ultimate future-proof storage layer. You could decide to completely change the way the data is stored and represented (in-memory or otherwise) at any time, as long as you have that raw history.
I spent about a year as a maintainer of FlockDB, Twitter's social graph store. If you don't know, it's basically a sharded MySQL setup. One of the key pain points was optimizing the row lock over the follower count. Whenever a Charlie Sheen joins, or someone tries to follow spam us, one particular row would get blasted with concurrent updates. Doing this in-memory in java via someAtomicLong.incrementAndGet() sounds a…
Just for fun, in Clojure:
(def current-id (atom (long 0)))
(defn get-id [] (swap! current-id inc))That sounds interesting. However, as soon as you have several distinct applications that share e.g. the same master data, how do do interface them? You will have to design the in-memory transactional store as a kind of global component. Then, not much is left until you end up with a real database. I am working on a team that is building an in-memory SQL database. It features a custom language that makes it possible t…
Earlier quoted context omitted.
When people set out to design a data driven application, they usually end up updating and deleting records. FTFY... It's not hard to build history into a SQL table design. You can even store events in a...wait for it... SQL database. I have built numerous systems backed by SQL databases that have complete history retention. Answering questions like 'who had id 'X' on this date 3 years ago' are easily solvable with ba…
It's not hard, no, but it usually doesn't happen in the average application. That's the issue: it's not built in, it's not standardized, and every SQL database is fully mutabile by default. If your system operates in this journaled/event-sourcing way at the most basic level then you have the ultimate future-proof storage layer. You could decide to completely change the way the data is stored and represented (in-memor…
Earlier quoted context omitted.
The goal is not to replace databases altogether. The goal is to solve some particular problems very well. Last time I used this approach, for example, we mirrored a bunch of data in a traditional SQL store for reporting and ad-hoc querying, things that databases are great at. In my view, direct access to data decoupled from application code is a bug, not a feature. With multiple code bases touching the same data, sch…
> In my view, direct access to data decoupled from application code is a bug, not a feature. With multiple code bases touching the same data, schema improvements become nearly impossible. It's unclear that multiple applications with direct access to said data make schema improvements any easier. The obvious solution, copying the data for applications that are using the new schema, pretty much guarantees that one or m…
> How do you guarantee that all of the apps that touch that data use the current version of said code?
An approach that may be worth considering is to not require all apps to use the current version: allow multiple versions. For some cases this would work, say if the semantics of the newer version are backwards compatible with the semantics of the older version. If the data semantics are preservable, transforming a schema could happen while each data access request to the schema is actively transformed.
But it clearly wouldn't work in all cases. More work to handle that would be needed.
> Code normalization is as important as data normalization.
True, this is the near show-stopper really. In that case, the best one can hope for is preparing the state of the new version (new data in new schema) and carefully coordinating a quick restart of the old version for the new version.
I would love to hear your thoughts on this. We have been working towards that direction with ChronicDB (http://chronicdb.com) and would welcome feedback.
Earlier quoted context omitted.
> In my view, direct access to data decoupled from application code is a bug, not a feature. With multiple code bases touching the same data, schema improvements become nearly impossible. It's unclear that multiple applications with direct access to said data make schema improvements any easier. The obvious solution, copying the data for applications that are using the new schema, pretty much guarantees that one or m…
> How do you guarantee that all of the apps that touch that data use the current version of said code? In a NoDB app? It's very easy: only one code base ever directly touches the data, because the data lives in the RAM allocated to the app. You give external access via an API, so integrity is very easy to enforce.
But one has to wonder, what happens when you need to upgrade the app? Shutting down the process and destroying the memory image doesn't seem like the best option:
- First, it disrupts connected applications since the process is killed, introducing downtime.
- Second, when starting up again in say version 2, the data that will be loaded in memory still needs to be transformed in the format expected by version 2. This transformation can take time on large data, introducing further downtime.
The challenge would be to eliminate this downtime by combining a solution for both client disruption and state transfer. A data abstraction like a database using SQL can simplify such a solution.
While this article wants to establish additional layers above the filesystem, I always wondered how comparable modern filesystems are to key-value datastores. As far as I can see, they seem to be comparable to b+tree indexed key value stores. A key would e.g. be "/home/user/test.txt". Thanks to the B+Tree "indexation" you can do a prefix scan and list folders (e.g. "ls /home/user/"--> all keys starting with "/home/us…