Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every System is a Log: Avoiding coordination in distributed applications

restate.dev

Post reply on HN
Print view View original

101–110 of 157 posts

Re: Every System is a Log: Avoiding coordination in distributed applications

#101

Post by withinboredom »

hinkley wrote: post #90
TuringTest wrote: post #76

Excuse me for sounding rough, but - isn't this reinventing comp-sci, one step at a time? I learned about distributed incrementally -monotonic logs back at the late 90s, with many other ways to do guaranteed transactional database actions. And I'm quite certain these must have been invented in the 50s or 60s, as these are the problems that early business computer users had: banking software. These are the techniques t…

When I was in school I had an optional requirement. You had to take one out of 2 or 3 classes to graduate. That was compiler design, which was getting terrible reviews from my peers who were taking it the semester before me, or distributed computing. Might have been a third but if so it was unmemorable. So I took distributed computing. Which ended up being one of the four classes that satisfied the 80/20 rule for my…

So many people I work with don't "get" distributed systems and how they interplay and cause problems. Most people don't even know that the ORDER you take potentially competing (distributed) locks even matters -- which is super important if you have different teams taking the same locks in different services!

The article is well written, but they still have a lot of problems to solve.

Re: Every System is a Log: Avoiding coordination in distributed applications

#102

Post by anarazel »

ianburrell wrote: post #97
hinkley wrote: post #87

Earlier quoted context omitted.

I just want a recognized standard format for write ahead logs. Start with replicating data between OLTP and OLAP databases with minimal glue code, and start moving other systems to a similar structure, like Kafka, then new things we haven’t thought of yet.

The structure for the write head logs needs to different between systems. For Postgres, the WAL is a record of writes with new blocks. It can't be used without knowing the Postgres disk format. I don't think it can be used to construct logical changes. Using a standard format, converting things into logical data, would be significantly slower. It is important that WAL be fast because it is the bottleneck in transacti…

> I don't think it can be used to construct logical changes.

It can: https://www.postgresql.org/docs/current/logicaldecoding.html

It's not entirely from the WAL though, some catalog accesses are necessary for metadata (shape and name of tables etc).

Re: Every System is a Log: Avoiding coordination in distributed applications

#103

Post by inopinatus »

ianburrell wrote: post #100
inopinatus wrote: post #92

Earlier quoted context omitted.

I’ve always concurred with the Helland/Kleppman observation mentioned viz. that the transaction log of a typical RDBMS is the canonical form and all the rows & tables merely projections. It’s curious that over those projections, we then build event stores for CQRS/ES systems, ledgers etc, with their own projections mediated by application code. But look underneath too. The journaled filesystem on which the database r…

The table data in database is the canonical form. You can delete the transaction logs, and temporarily lose some reliability. It is very common to delete the transaction logs when not needed. When databases are backed up, they either dump the logical data or take snapshot of the data. Then can take stream of transaction logs for syncing or backup until the next checkpoint. I'm pretty sure journalled filesystem recycl…

Sorry, this is mistaking the operational for the fundamental.

If a transaction log is replayed, then an identical set of relations will be obtained. Ergo, the log is the prime form of the database.

It’s that simple.

Re: Every System is a Log: Avoiding coordination in distributed applications

#104

Post by withinboredom »

zellyn wrote: post #72

sewen (et al) This is lovely and I'm itching to try it. One question: We have a use case where a location gets cut off completely from the internet at large. In that case, it makes sense for the local hardware (typically Android and/or iOS tablets or equivalent) to take over as a log owner: even though you're cut off, if you're willing to swallow the risk (and hence cost) of offline payments, you should be able to cr…

I'm actually working on a database implementation for this exact use-case... It's a distributed edge database and still quite a long ways to go -- https://github.com/bottledcode/atlas-db if you want to give it a star.

It's mostly based on wpaxos (wide-area consensus), spaxos, fpaxos and pretty neat. The repo above is a productionization of several proof of concepts to get there.

> One could also imagine ensuring that > n/2 consensus members are always located inside the restaurant/hardware store/etc., so if you go offline, you can still proceed.

This is what annoys me to no end about RAFT. It's a great protocol, don't get me wrong, but its too simple for these types of problems. RAFT fails when it doesn't have consensus and because the consensus is non-deterministic, it must have an odd number of nodes. PAXOS, while far more complex than RAFT in terms of "grok", is deterministic so you don't need an odd number of nodes.

If you throw in some flexible quorums, you can do some really neat stuff, like how Atlas handles a "region" (ie, areas connected via the internet instead of the same network) becoming disconnected; but I'm not ready yet. There's still a long way to go!

Re: Every System is a Log: Avoiding coordination in distributed applications

#105

Post by magicalhippo »

inopinatus wrote: post #103
ianburrell wrote: post #100

Earlier quoted context omitted.

The table data in database is the canonical form. You can delete the transaction logs, and temporarily lose some reliability. It is very common to delete the transaction logs when not needed. When databases are backed up, they either dump the logical data or take snapshot of the data. Then can take stream of transaction logs for syncing or backup until the next checkpoint. I'm pretty sure journalled filesystem recycl…

Sorry, this is mistaking the operational for the fundamental. If a transaction log is replayed, then an identical set of relations will be obtained. Ergo, the log is the prime form of the database. It’s that simple.

At work we need to distribute daily changes to a dataset, so we have a series of daily deltas. If a new client is brought up, they need to apply all the deltas to get the current dataset.

This is time consuming, so we optimized it by creating "base versions" every month. So a client only needs to download the latest base version and the apply the deltas since then...

Re: Every System is a Log: Avoiding coordination in distributed applications

#106

Post by isignal »

inopinatus wrote: post #103
ianburrell wrote: post #100

Earlier quoted context omitted.

The table data in database is the canonical form. You can delete the transaction logs, and temporarily lose some reliability. It is very common to delete the transaction logs when not needed. When databases are backed up, they either dump the logical data or take snapshot of the data. Then can take stream of transaction logs for syncing or backup until the next checkpoint. I'm pretty sure journalled filesystem recycl…

Sorry, this is mistaking the operational for the fundamental. If a transaction log is replayed, then an identical set of relations will be obtained. Ergo, the log is the prime form of the database. It’s that simple.

The transaction log maintained from time 0 would be equivalent but too expensive to store compared to the tables.

Re: Every System is a Log: Avoiding coordination in distributed applications

#107

Post by inopinatus »

isignal wrote: post #106
inopinatus wrote: post #103

Earlier quoted context omitted.

Sorry, this is mistaking the operational for the fundamental. If a transaction log is replayed, then an identical set of relations will be obtained. Ergo, the log is the prime form of the database. It’s that simple.

The transaction log maintained from time 0 would be equivalent but too expensive to store compared to the tables.

On the contrary, I’ve known plenty of sites that keep their logs.

Often written to tape, for obvious reasons.

Re: Every System is a Log: Avoiding coordination in distributed applications

#108

Post by Animats »

magicalhippo wrote: post #105
inopinatus wrote: post #103

Earlier quoted context omitted.

Sorry, this is mistaking the operational for the fundamental. If a transaction log is replayed, then an identical set of relations will be obtained. Ergo, the log is the prime form of the database. It’s that simple.

At work we need to distribute daily changes to a dataset, so we have a series of daily deltas. If a new client is brought up, they need to apply all the deltas to get the current dataset. This is time consuming, so we optimized it by creating "base versions" every month. So a client only needs to download the latest base version and the apply the deltas since then...

Which is what accountants call "closing the books". Once all ledgers have been reconciled, old ledgers can be archived and you go forward from the last closing.

Forensic accounting, incidentally, is when something went badly wrong and outside accountants have to go back through the old ledgers, and maybe old invoices and payments and reconstruct the books. FTX had to do that after the bankruptcy to find out where the money went and where it was supposed to go.

Re: Every System is a Log: Avoiding coordination in distributed applications

#109

Post by fellowniusmonk »

Animats wrote: post #91

This is a basic concept in accounting. The general ledger is an immutable log of transactions. Other accounting documents are constructed from the general ledger, and can, if necessary, be rebuilt from it. This is the accepted way to do money-related things. Synchronization is called "reconcilation" in accounting terminology. The computer concept is that we have a current state, and changes to it come in. The databas…

This is why EG-Walker is so important, diamond types adoption and a solid TS port can't come soon enough for distributed systems.

Re: Every System is a Log: Avoiding coordination in distributed applications

#110

Post by 10000truths »

isignal wrote: post #106
inopinatus wrote: post #103

Earlier quoted context omitted.

Sorry, this is mistaking the operational for the fundamental. If a transaction log is replayed, then an identical set of relations will be obtained. Ergo, the log is the prime form of the database. It’s that simple.

The transaction log maintained from time 0 would be equivalent but too expensive to store compared to the tables.

If you relax your constraint to "retain logs for the past N days", you can accumulate the logs from T=0 to T=(today - N) into tables and still benefit from having snapshots from that cutoff onwards.
Post reply on HN