Distributed transactions in Go: Read before you try • Hacker News Board

Re: Distributed transactions in Go: Read before you try

#41

Post by latchkey » Sun, Oct 06, 2024, 6:31 PM UTC

Back in the early 2000's, I was working for the largest hardcore porn company in the world, serving tons of traffic. We built a cluster of 3 Dell 2950 servers with JBoss4. We were using Hibernate and EJB2 entities, with a MySQL backend. This was all before "cloud" allowed porn on their own systems, so we had to do it ourselves.

Once configured correctly and all the multicast networking was set up, distributed 2PC transactions via jgroups worked flawlessly for years. We actually only needed one server for all the traffic, but used 3 for redundancy and rolling updates.

¯\_(ツ)_/¯, kids these days

Re: Distributed transactions in Go: Read before you try

#42

Post by danenania » Sun, Oct 06, 2024, 6:48 PM UTC

foobiekr wrote: post #36

ekidd wrote: post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

Great advice. Microservices also open the door to polyglot, so you lose the ability to even arrive that everyone uses/has access to/understands the things in a common libCompany that make it possible for anyone to at least make sense of code. When I talk to people who did microservices, I ask them "why is this a service separate from this?" I have legitimately - and commonly - gotten the answer that the dev I'm talki…

> Microservices also open the door to polyglot

While I see your point about the downsides of involving too many languages/technologies, I think the really key distinction is whether a service has its own separate database.

It's really not such a big problem to have "microservices" that share the same database. This can bring many of the benefits of microservices without most of the downsides.

Imo it would be good if we had some common terminology to distinguish these approaches. It seems like a lot of people are creating services with their own separate databases for no reason other than "that's how you're supposed to do microservices".

Re: Distributed transactions in Go: Read before you try

#43

Post by Andys » Sun, Oct 06, 2024, 7:02 PM UTC

jrockway wrote: post #22

ekidd wrote: post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

I agree with this. Having worked at FAANG, I'm always excited by the fanciness that is required. Spanner is the best database I've ever used. That said, my feeling is that in the real world, you just pick Postgres and forget about it. Let's Encrypt issues every TLS cert on the Internet with one beefy Postgres database. Computers are HUGE these days. By the time a 128 core machine isn't good enough for your app, you w…

Agreed - A place I worked needed 24/7 uptime for financial transactions, and we still managed to keep scaling a standard MySQL database, despite it getting hammered, over the course of 10 years up to something like 64 cores and 384GB of RAM on a large EC2 instance.

We did have to move reporting off to a non-SQL solution because it was too slow to do in realtime, but that was a decision based on evidence of the need.

Re: Distributed transactions in Go: Read before you try

#44

Post by ebiester » Sun, Oct 06, 2024, 7:03 PM UTC

latchkey wrote: post #41

Back in the early 2000's, I was working for the largest hardcore porn company in the world, serving tons of traffic. We built a cluster of 3 Dell 2950 servers with JBoss4. We were using Hibernate and EJB2 entities, with a MySQL backend. This was all before "cloud" allowed porn on their own systems, so we had to do it ourselves. Once configured correctly and all the multicast networking was set up, distributed 2PC tra…

Different problems have different solutions.

You likely mostly had very simple business logic in 90% of your system. If your system is automating systems for a cross-domain sector (think payroll), you're likely to have a large number of developers on a relatively small amount of data and speed is secondary to managing the complexity across teams.

Microservices might not be a great solution, and distributed monoliths will always be an anti-pattern, but there are reasons for more complex setups to enable concurrent development.

Re: Distributed transactions in Go: Read before you try

#45

Post by latchkey » Sun, Oct 06, 2024, 7:20 PM UTC

ebiester wrote: post #44

latchkey wrote: post #41

Back in the early 2000's, I was working for the largest hardcore porn company in the world, serving tons of traffic. We built a cluster of 3 Dell 2950 servers with JBoss4. We were using Hibernate and EJB2 entities, with a MySQL backend. This was all before "cloud" allowed porn on their own systems, so we had to do it ourselves. Once configured correctly and all the multicast networking was set up, distributed 2PC tra…

Different problems have different solutions. You likely mostly had very simple business logic in 90% of your system. If your system is automating systems for a cross-domain sector (think payroll), you're likely to have a large number of developers on a relatively small amount of data and speed is secondary to managing the complexity across teams. Microservices might not be a great solution, and distributed monoliths…

Due to the unwillingness for corporations to work with us, we had to develop our own cross-TLD login framework, payments system, affiliate tracker, micro-currency for live pay per minute content, secure image/video serving across multiple CDN's, and a whole ads serving network. It took years to build it all and was massively complicated.

The point I was making is that the tooling for all of this has existed for ages. People keep reinventing it. Nothing wrong with that, but these sorts of blog posts are entertaining to watch history repeat itself and HN to argue over the best way to do things.

Re: Distributed transactions in Go: Read before you try

#46

Post by mekoka » Sun, Oct 06, 2024, 7:39 PM UTC

ekidd wrote: post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

> 1. Design for 10x your current scale, not 1000x.

I'd even say that the advice counts double for early stage startups. That is, at that scale, it should be design for 5x.

You could spend years building a well architected, multi-tenanted, microserviced system, whose focus on sound engineering is actually distracting your team from building core solutions that address your clients' real problems now. Or, you could instead redirect that focus on first solving those immediate problems with simplistic, suboptimal, but valid engineering.

An early stage solopreneur could literally just clone or copy/paste/configure their monolith in a new directory and spawn a new database every time they have a new client. They could literally do this for their first 20+ clients in their first year. When I say this, some people look at me in disbelief. Some of them, having yet to make their first sale. Instead, they're working on solutions to counter the anticipated scalability issues they'll have in two years, when they finally start to sell and become a huge success.

For another few, copy/paste/createdb seems like a stroke of genius. But I'm not a genius, I'm just oldish. Many companies did basically this 20 years ago and it worked fine. The reason it's not even considered anymore seems to be a cultural amnesia/insanity that's made certain practices arcane, if not taboo altogether. So we tend to spontaneously reach for the nuclear reactor, when a few pieces of coal would suffice to fuel our current momentum.

Re: Distributed transactions in Go: Read before you try

#47

Post by latentsea » Sun, Oct 06, 2024, 9:22 PM UTC

relistan wrote: post #4

This is a good summary of building an evented system. Having built and run one that scaled up to 130 services and nearly 60 engineers, I can say this solves a lot of problems. Our implementation was a bit different but in a similar vein. When the company didn’t do well in the market and scaled down, 9 engineers were (are) able to operate almost all of that same system. The decoupling and lack of synchronous dependenc…

I would really hate to join a team of 9 engineers that owned 130 services.

Re: Distributed transactions in Go: Read before you try

#48

Post by misiek08 » Sun, Oct 06, 2024, 9:41 PM UTC

natdempk wrote: post #37

ljm wrote: post #30

Earlier quoted context omitted.

I think the question is if you need the entire system to be strongly consistent, or just the core of it? To use ElasticSearch as an example: do you need to add the complexity of keeping the index up to date in realtime, or can you live with periodic updates for search or a background job for it? As long as your primary DB is the source of truth, you can use that to bring other less critical stores up to date outside…

Well, the problem you run into is that you kind of want different datastores for different use-cases. For example search vs. specific page loads, and you want to try and make both of those consistent, but you don't have a single DB that can serve both use-cases (often times primary DB + ElasticSearch for example). If you don't keep them consistent, you have user-facing bugs where a user can update a record but not se…

The way I solved it once was trying to update ES synchronously and if it failed or timeouted - queue event to index the doc. Timeout wasn’t an issue, because double update wasn’t harmful.

Re: Distributed transactions in Go: Read before you try

#49

Post by misiek08 » Sun, Oct 06, 2024, 9:45 PM UTC

Looks like crypto ad for the library and showing probably worst, most over-engineered method for „solving” transactions in more diverse environment. Eventually consistency is big tradeoff not possible to accept in many payment and stock related areas. Working in company where all described problems exist and were solved the worst way possible I see this article as very misleading. You don’t want the events instead of transactions - if something has to be commited together - you need to reachitect system and that’s it. Of course people who were building this monster for years will block anyone from doing this. Over-engineered AF, because most of the parts where transactions are required could be handled by single database, even SQL and currently are split between dozens of separate Mongo clusters.

"Event based consistency" leaves us in state where you can’t restore system to a stable, safe, consistent state. And of course you have a lot more fun in debugging and developing, because you (we, here) can’t test locally anything. Hundreds of mini-clones of prod setup running, wasting resources and always out of sync are ready to see the change and tell you a little more than nothing. Great DevEx…

Re: Distributed transactions in Go: Read before you try

#50

Post by ekidd » Sun, Oct 06, 2024, 9:48 PM UTC

devjab wrote: post #31

ekidd wrote: post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

I disagree rather strongly with this advice. Mostly because I’ve spent almost a decade earning rather lucrative money on cleaning up after companies and organisations which did it. Part of what you say is really good advice, if you’re not Facebook then don’t build your infrastructure as though you were. I think it’s always a good idea to remind yourself that StackOverflow ran on a few IIS servers for a long while doi…

I wanted to respond to you, because you had some excellent points.

> Mostly because I’ve spent almost a decade earning rather lucrative money on cleaning up after companies and organisations which did it.

For many companies, this is actually a pretty successful outcome. They built an app, they earned a pile of money, they kept adding customers, and now they have a mess. But they can afford to pay you to fix their mess!

My rule of thumb of "design for 10x scale" is intended to be used iteratively. When something is slow and miserable, take the current demand, multiply it by 10, and design something that can handle it. Sometimes, yeah, this means you need to split stuff up or use a non-SQL database. But at least it's a real need at that point. And there's no substitute for engineering knowledge and good taste.

But as other people have pointed out, people who can't use an RDBMS correctly are going to have a bad time implementing distributed transactions across microservices.

So I'm going to stick with my advice to start with a single database, and to only pull things out when there's a clear need to scale something.