Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turning the database inside-out (2015)

martin.kleppmann.com

Post reply on HN
Print view View original

61–70 of 97 posts

Re: Turning the database inside-out (2015)

#61

Post by mrkeen »

Swizec wrote: post #34
reubenmorais wrote: post #31

Earlier quoted context omitted.

Seeing the world as mutable is a matter of perspective, if you explicitly model time as a dimension it can instead be seen as a sequence of transitions from immutable state to immutable state, an accumulation of events over time, which fits the log abstraction perfectly.

> if you explicitly model time as a dimension it can instead be seen as a sequence of transitions from immutable state to immutable state, an accumulation of events over time, which fits the log abstraction perfectly I worked with a feature that used this approach once. It even made sense for the feature (an immutable history log of patient chart data). It was absolute hell to work with. Querying current state, which…

> I worked with a feature that used this approach once.

You work with many features that use this approach.

  https://www.dbi-services.com/blog/oracle-log-writer-and-write-ahead-logging/
  https://git-scm.com/docs/git-reflog
  https://dev.mysql.com/blog-archive/mysql-8-0-new-lock-free-scalable-wal-design/
  https://www.postgresql.org/docs/current/wal-intro.html
  https://www.sqlite.org/wal.html
  https://www.amazon.science/blog/lessons-learned-from-10-years-of-dynamodb

Re: Turning the database inside-out (2015)

#62

Post by agumonkey »

082349872349872 wrote: post #60
agumonkey wrote: post #29

Earlier quoted context omitted.

I would love to know if other people in the industry (beside hickey/datomic) use the immutable log/stream + integrators. From my small experience in enterprise app: auditability and time travelling are always bolted on good old sql tables/snapshots after the fact and the pain is already baked in.

thoughts from 1992 (Gray+Reuter): https://news.ycombinator.com/item?id=42829878

very interesting, every generation foresees the same solutions somehow

Re: Turning the database inside-out (2015)

#63

Post by reubenmorais »

crabbone wrote: post #59
reubenmorais wrote: post #31

Earlier quoted context omitted.

Seeing the world as mutable is a matter of perspective, if you explicitly model time as a dimension it can instead be seen as a sequence of transitions from immutable state to immutable state, an accumulation of events over time, which fits the log abstraction perfectly.

> perfectly That's a joke, right? If you tried to live in this world, as a human being, not in some abstract database-building sense, you'd be completely lost in the first second of your existence, and would probably die in minutes because your body parts would "forget" how to function properly. We make sense of the world because we have "object permanence", which requires the concept of larger things made up of comp…

> If all you can do is transitions, you don't know upon transitioning if you still have a keyboard you are typing on or not. This also means that if you want to be able to function somehow in this world, then after each transition, you'd have to reassess all properties of all interesting aspects of the world just to make sure they are still there.

You're assuming that at every timestep all transitions are equally valid or equally likely. That's not a given, you can and of course should carefully model which transitions your system allows or not. In real life this model is Schrödinger's equation, as best as we know it. In your information system it can be whatever you design it to be.

Re: Turning the database inside-out (2015)

#64

Post by EvanAnderson »

dagss wrote: post #5

We did this style on top of plain MSSQL. Each event would have a SQL table which is the primary storage. Then we have workers that listens to new data in tables and updates projections we needed. (Sometimes DB triggers but mostly async workers.) The main issue is "listening to new data in a SQL table". I wrote this code to achieve it in MSSQL (perhaps it is somehow built into postgres?): https://github.com/vippsas/ms…

> The main issue is "listening to new data in a SQL table".

You may want to take a look at Service Broker[0]. It's the idiomatic messaging and queuing bit of SQL Server. It's a bit of an obscure feature and has a bit of a steep learning curve. If I were trying to implement what you're doing it would be the tool I'd reach for.

[0] https://learn.microsoft.com/en-us/sql/database-engine/config...

Re: Turning the database inside-out (2015)

#66

Post by agentultra »

This is also known as event sourcing [0] and is a common pattern used inside of databases, in git, in lots of popular software.

I don't generally recommend it for every application as the tooling is not as well integrated as it is in an RDBMS and the data model doesn't fit every use-case.

However, if you have a system that needs to know "when" something happened in an on-going process, it can be a very handy architecture... although with data-retention laws it can get tricky quickly (among other reasons).

[0] https://martinfowler.com/eaaDev/EventSourcing.html

Re: Turning the database inside-out (2015)

#67

Post by bubbleRefuge »

agumonkey wrote: post #29
dagss wrote: post #5

We did this style on top of plain MSSQL. Each event would have a SQL table which is the primary storage. Then we have workers that listens to new data in tables and updates projections we needed. (Sometimes DB triggers but mostly async workers.) The main issue is "listening to new data in a SQL table". I wrote this code to achieve it in MSSQL (perhaps it is somehow built into postgres?): https://github.com/vippsas/ms…

I would love to know if other people in the industry (beside hickey/datomic) use the immutable log/stream + integrators. From my small experience in enterprise app: auditability and time travelling are always bolted on good old sql tables/snapshots after the fact and the pain is already baked in.

In large scale business intergration platforms/apps, you have operational systems like SAP and and Oracle Service Cloud generate/stream raw or business events which are published to message brokers in topics ( orders, incidents, suppliers, logistics, etc). There the data is published , validated, transformed (filtered, routed, formatted, enriched, aggregated, etc) into other downstream topics which can be used to egress to other apps or enterprise data stores/data lakes. Data governance apps control who has access. Elastic search or Splunk for data lineage and debugging. you also have sbservability systems sandwiched in there as well.

Re: Turning the database inside-out (2015)

#68

Post by timacles »

mike_hearn wrote: post #50
agumonkey wrote: post #29

Earlier quoted context omitted.

I would love to know if other people in the industry (beside hickey/datomic) use the immutable log/stream + integrators. From my small experience in enterprise app: auditability and time travelling are always bolted on good old sql tables/snapshots after the fact and the pain is already baked in.

Depends which industry. If you look at a lot of non-tech industry then they'll use a commercial DB with all those features in place already, rather than hacking up their own data layer. A few years ago I spent some time in the enterprise finance space, and learned some unfashionable tech you don't see talked about on Hacker News much. It left me with a new appreciation for what goes on there. A staggering amount of t…

Who besides Oracle offers this stuff though?

Yeah Oracle has a bunch of nice features, it also costs a gajillion dollars that no one besides a large enterprise can afford.

Re: Turning the database inside-out (2015)

#69

Post by oa335 »

dagss wrote: post #5

We did this style on top of plain MSSQL. Each event would have a SQL table which is the primary storage. Then we have workers that listens to new data in tables and updates projections we needed. (Sometimes DB triggers but mostly async workers.) The main issue is "listening to new data in a SQL table". I wrote this code to achieve it in MSSQL (perhaps it is somehow built into postgres?): https://github.com/vippsas/ms…

“ The main issue is "listening to new data in a SQL table". I wrote this code to achieve it in MSSQL (perhaps it is somehow built into postgres?): https://github.com/vippsas”

Postgres uses “publications” for this purpose. Clients can subscribe to a publication that gets updates to a given table.

Re: Turning the database inside-out (2015)

#70

Post by agumonkey »

timacles wrote: post #68
mike_hearn wrote: post #50

Earlier quoted context omitted.

Depends which industry. If you look at a lot of non-tech industry then they'll use a commercial DB with all those features in place already, rather than hacking up their own data layer. A few years ago I spent some time in the enterprise finance space, and learned some unfashionable tech you don't see talked about on Hacker News much. It left me with a new appreciation for what goes on there. A staggering amount of t…

Who besides Oracle offers this stuff though? Yeah Oracle has a bunch of nice features, it also costs a gajillion dollars that no one besides a large enterprise can afford.

time based snapshots are in datomic and also possible in other dbms (maybe via extensions)

for the rest i don't know

Post reply on HN