Datomic
The concept is even older than that.
Turning the database inside-out (2015)
41–50 of 97 posts
Re: Turning the database inside-out (2015)
#42> Databases are global, shared, mutable state. [...] However, most self-respecting developers have got rid of mutable global variables in their code long ago. So why do we tolerate databases as they are? Because the world itself is a global, shared, mutable state (which, incidentally, is also a single source of truth) and databases were invented to mirror it (well, relevant parts of it) 1-to-1, or close to it. This s…
Just get rid of side effects, man! (Turns out side effects are the whole reason we write code, whoops)
Re: Turning the database inside-out (2015)
#43> Databases are global, shared, mutable state. [...] However, most self-respecting developers have got rid of mutable global variables in their code long ago. So why do we tolerate databases as they are? Because the world itself is a global, shared, mutable state (which, incidentally, is also a single source of truth) and databases were invented to mirror it (well, relevant parts of it) 1-to-1, or close to it. This s…
The "world" can just as easily be conceived of as being composed of discrete immutable facts rather than global mutable state. Either way, I kind of don't think that unfalsifiable ontological premises should be an important factor in system design.
Re: Turning the database inside-out (2015)
#44Does any one have some resources where a real, practical example is implemented? Because I can only find fairly theoretical resources but not real world examples.
Say, like how some simple CMS would work with a datastore like this. What does the event to update the headline of an article look like? How are integrity constraints enforced, e.g. an article can't reference an author that doesn't exist? Things like that.
Re: Turning the database inside-out (2015)
#45> Databases are global, shared, mutable state. [...] However, most self-respecting developers have got rid of mutable global variables in their code long ago. So why do we tolerate databases as they are? Because the world itself is a global, shared, mutable state (which, incidentally, is also a single source of truth) and databases were invented to mirror it (well, relevant parts of it) 1-to-1, or close to it. This s…
The "world" can just as easily be conceived of as being composed of discrete immutable facts rather than global mutable state. Either way, I kind of don't think that unfalsifiable ontological premises should be an important factor in system design.
While the world can tentatively be conceived to be like that, it is not nearly "just as easily". Lots of those "immutable facts" can't be realistically discovered: e.g. good luck recovering "this blackboard had a drawing of a cat and a bird until five minutes ago when it was cleared with a wet sponge" if you weren't there in time. The approach with mutable state fosters this destructible nature of many things upon you and makes you cope with it, somehow.
Re: Turning the database inside-out (2015)
#46https://materialize.com/ provides another approach, based on "timely dataflow" (https://timelydataflow.github.io/timely-dataflow/) - originated at MS.
Re: Turning the database inside-out (2015)
#47We did this style on top of plain MSSQL. Each event would have a SQL table which is the primary storage. Then we have workers that listens to new data in tables and updates projections we needed. (Sometimes DB triggers but mostly async workers.) The main issue is "listening to new data in a SQL table". I wrote this code to achieve it in MSSQL (perhaps it is somehow built into postgres?): https://github.com/vippsas/ms…
I always thought that the most flexible approach was:
- good old mutable relational tables
- a separate db to store immutable events (could be the same kind of db you use for business transactions or something fancy like big query)
I feel like mixing both into one has more disadvantages
Re: Turning the database inside-out (2015)
#48Earlier quoted context omitted.
Seeing the world as mutable is a matter of perspective, if you explicitly model time as a dimension it can instead be seen as a sequence of transitions from immutable state to immutable state, an accumulation of events over time, which fits the log abstraction perfectly.
> if you explicitly model time as a dimension it can instead be seen as a sequence of transitions from immutable state to immutable state, an accumulation of events over time, which fits the log abstraction perfectly I worked with a feature that used this approach once. It even made sense for the feature (an immutable history log of patient chart data). It was absolute hell to work with. Querying current state, which…
You can use snapshotting to increase performance of the sequence of transitions approach.
Re: Turning the database inside-out (2015)
#49This is Aurora...
https://pages.cs.wisc.edu/~yxy/cs764-f20/papers/aurora-sigmo...
Re: Turning the database inside-out (2015)
#50We did this style on top of plain MSSQL. Each event would have a SQL table which is the primary storage. Then we have workers that listens to new data in tables and updates projections we needed. (Sometimes DB triggers but mostly async workers.) The main issue is "listening to new data in a SQL table". I wrote this code to achieve it in MSSQL (perhaps it is somehow built into postgres?): https://github.com/vippsas/ms…
I would love to know if other people in the industry (beside hickey/datomic) use the immutable log/stream + integrators. From my small experience in enterprise app: auditability and time travelling are always bolted on good old sql tables/snapshots after the fact and the pain is already baked in.
Depends which industry. If you look at a lot of non-tech industry then they'll use a commercial DB with all those features in place already, rather than hacking up their own data layer. A few years ago I spent some time in the enterprise finance space, and learned some unfashionable tech you don't see talked about on Hacker News much. It left me with a new appreciation for what goes on there. A staggering amount of time spent in tech startups is spent on solving and resolving problems that you can buy off the shelf solutions for and have been able to for a long time.
After all, this talk is now 10 years old but appears to be describing features that have been around for much longer. Take your average bank - it will have a bunch of Oracle databases in it. Those already have every feature discussed in this thread and in the talk:
• Incremental materialized view maintenance (with automatic query rewrite to use it, so users don't have to know it exists).
• Exposing logical commit logs as an API, with tooling (e.g. GoldenGate, LogMiner, query change notifications).
• Time travelling SELECT (... AS OF).
• Lots of audit features.
• Integrated transactional and scalable MQ (no need for Kafka).
My experience was that faced with a data processing problem, enterprise devs will tend to just read the user guide for their corporation's database, or ask for advice from a graybeard who already did so. They go write some SQL or an Excel plugin or something old school, ship it, close the ticket, go home. Then a few years later you look at HN and find there's a whole startup trying to sell the same feature.