Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed transactions in Go: Read before you try

threedots.tech

Post reply on HN
Print view View original

21–30 of 58 posts

Re: Distributed transactions in Go: Read before you try

#21

Post by natdempk »

ekidd wrote: post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

Pretty great advice!

I think the one thing you can run into that is hard is once you want to support different datasets that fall outside the scope of a transaction (think events/search/derived-data, anything that needs to read/write to a system that is not your primary transactional DB) you probably do want some sort of event bus/queue type thing to get eventual consistency across all the things. Otherwise you just end up in impossible situations when you try to manage things like doing a DB write + ES document update. Something has to fail and then your state is desynced across datastores and you're in velocity/bug hell. The other side of this though is once you introduce the event bus and transactional-outbox or whatever, you then have a problem of writes/updates happening and not being reflected immediately. I think the best things that solve this problem are stuff like Meta's TAO that combines these concepts, but no idea what is available to the mere mortals/startups to best solve these types of problems. Would love to know if anyone has killer recommendations here.

Re: Distributed transactions in Go: Read before you try

#22

Post by jrockway »

ekidd wrote: post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

I agree with this.

Having worked at FAANG, I'm always excited by the fanciness that is required. Spanner is the best database I've ever used.

That said, my feeling is that in the real world, you just pick Postgres and forget about it. Let's Encrypt issues every TLS cert on the Internet with one beefy Postgres database. Computers are HUGE these days. By the time a 128 core machine isn't good enough for your app, you will have sold your shares and will be living on your own private island or whatever. If you want to wrap sqlite in raft over the weekend for some fun, sure, do that. But don't put it in prod.

Re: Distributed transactions in Go: Read before you try

#23

Post by natdempk »

MuffinFlavored wrote: post #16
ekidd wrote: post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

> For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. I don't think this handles "what if my app is a wrapper on external APIs and my own database". You don't get automatic rollbacks with API calls the same way you do database transactions. What to do then?

You have a distributed system on your hands at that point so you need idempotent processes + reconciliation/eventual consistency. Basically thinking a lot about failure, resyncing data/state, patterns like transactional outboxes, durable queues, two-phase commit, etc etc. It just quickly gets into the specifics of your task/system/APIs so hard to give general advice. Most apps do not solve these problems super well for a long time unless they are in critical places like billing, and even then it might just mean weird repair jobs, manual APIs to resync stuff, audits, etc. Usually an event-bus/queue or related DB table for the idempotent work + some async process validating that table can go a long way though.

Re: Distributed transactions in Go: Read before you try

#26

Post by kgeist »

wavemode wrote: post #14
atombender wrote: post #11

I find the "forwarder" system here a rather awkward way to bridge the database and Pub/Sub system. A better way to do this, I think, is to ignore the term "transaction," which overloaded with too many concepts (such as transactional isolation), and instead to consider the desired behaviour, namely atomicity: You want two updates to happen together, and (1) if one or both fail you want to retry until they are both suc…

I think attempting to automatically "undo" partially failed distributed operations is fraught with danger. 1. It's not really safe, since another system could observe the updated data B (and perhaps act on that information) before you manage to roll it back to A. 2. It's not really reliable, since the sort of failures that prevent you from completing the operation, could also prevent you from rolling back parts of it…

>I think attempting to automatically "undo" partially failed distributed operations is fraught with danger.

We once tried it, and decided against it because

1) in practice rollbacks were rarely properly tested and were full of bugs

2) we had a few incidents when a rollback overwrote everything with stale data

Manual intervention is probably the safest way.

Re: Distributed transactions in Go: Read before you try

#27

Post by [deleted] »

wavemode wrote: post #14
atombender wrote: post #11

I find the "forwarder" system here a rather awkward way to bridge the database and Pub/Sub system. A better way to do this, I think, is to ignore the term "transaction," which overloaded with too many concepts (such as transactional isolation), and instead to consider the desired behaviour, namely atomicity: You want two updates to happen together, and (1) if one or both fail you want to retry until they are both suc…

I think attempting to automatically "undo" partially failed distributed operations is fraught with danger. 1. It's not really safe, since another system could observe the updated data B (and perhaps act on that information) before you manage to roll it back to A. 2. It's not really reliable, since the sort of failures that prevent you from completing the operation, could also prevent you from rolling back parts of it…

[deleted]

Re: Distributed transactions in Go: Read before you try

#28

Post by Scubabear68 »

If I had a nickel for all the clients I’ve seen with micro services everywhere, and 90% of the code is replicating an RDBMS with hand coded in memory joins.

What could have been a simple SQL query in a sane architecture becomes N REST calls (possibly nested with others downstream) and manually stitching together results.

And that is just the read only case. As the author notes updates add another couple of levels of horror.

Re: Distributed transactions in Go: Read before you try

#29

Post by codethief »

cletus wrote: post #13

To paraphase [1]: > Some people, when confronted with a problem, think “I know, I'll use micro-services.” Now they have two problems. As soon as I read this example where there's users and orders microservices, you've already made an error (IMHO). What happens when the traffic becomes such an issue that you need to shard your microservices? Now you've got session and load-balancing issues. If you ignore them, you may…

> Even if the Netflix microservices architecture is an objectively good idea (which I honestly have no opinion on

I have no opinion on that either, but at least this[0] story by ThePrimeagen didn't make it sound all too great. (Watch this classic[1] before for context, unless you already know Wingman, Galactus, etc.)

[0]: https://youtu.be/s-vJcOfrvi0?t=319

[1]: https://m.youtube.com/watch?v=y8OnoxKotPQ

Re: Distributed transactions in Go: Read before you try

#30

Post by ljm »

natdempk wrote: post #21
ekidd wrote: post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

Pretty great advice! I think the one thing you can run into that is hard is once you want to support different datasets that fall outside the scope of a transaction (think events/search/derived-data, anything that needs to read/write to a system that is not your primary transactional DB) you probably do want some sort of event bus/queue type thing to get eventual consistency across all the things. Otherwise you just…

I think the question is if you need the entire system to be strongly consistent, or just the core of it?

To use ElasticSearch as an example: do you need to add the complexity of keeping the index up to date in realtime, or can you live with periodic updates for search or a background job for it?

As long as your primary DB is the source of truth, you can use that to bring other less critical stores up to date outside of the context of an API request.

Post reply on HN