Distributed transactions in Go: Read before you try • Hacker News Board

Re: Distributed transactions in Go: Read before you try

#51

Post by pkhuong » Sun, Oct 06, 2024, 10:55 PM UTC

mekoka wrote: post #46

ekidd wrote: post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

> 1. Design for 10x your current scale, not 1000x. I'd even say that the advice counts double for early stage startups. That is, at that scale, it should be design for 5x . You could spend years building a well architected, multi-tenanted, microserviced system, whose focus on sound engineering is actually distracting your team from building core solutions that address your clients' real problems now . Or, you could i…

> spawn a new database every time they have a new client.

I've seen this work great for multitenancy, with sqlite (again, a single beefy server goes a long way). At some point though, you hit niche scaling issues and that's how you end up with, e.g., arcane sqlite hacks. Hopefully these mostly come from people who have found early success, or looked at by others who want reassurance that there is an escape hatch that doesn't involve rewriting everything for web scale.

Re: Distributed transactions in Go: Read before you try

#52

Post by kukkeliskuu » Mon, Oct 07, 2024, 1:47 AM UTC

ljm wrote: post #30

natdempk wrote: post #21

Earlier quoted context omitted.

Pretty great advice! I think the one thing you can run into that is hard is once you want to support different datasets that fall outside the scope of a transaction (think events/search/derived-data, anything that needs to read/write to a system that is not your primary transactional DB) you probably do want some sort of event bus/queue type thing to get eventual consistency across all the things. Otherwise you just…

I think the question is if you need the entire system to be strongly consistent, or just the core of it? To use ElasticSearch as an example: do you need to add the complexity of keeping the index up to date in realtime, or can you live with periodic updates for search or a background job for it? As long as your primary DB is the source of truth, you can use that to bring other less critical stores up to date outside…

I think a different framing for the question might be more helpful. What is your overall goal? You cannot have everything. In fact, if you try to have everything, you will get nothing.

I would say that 99% of time the implicit goal is to cut down development time. And the best way to cut development time on long-term is to cut down complexity.

To cut down complexity, we should avoid complex problems, use existing solutions to solve them or at least be able to contain them. Sometimes, the price is that you need to solve some easier problems yourself.

For example, microservice architectures promise that you need less coordination between teams, because parts of the systems can be deployed independently. The price is that you cannot use database transactions to guarantee integrity.

I think data integrity is almost always much more important problem to solve, partly because it is so difficult to solve by yourself. Actually it is often so difficult that most people just ignore it.

For example, if you adopt microservices architecture, you often just ignore data integrity, and call your system "eventually consistent". Practically this means that you push the data integrity problems to the sink system.

It is better to think of data integrity as a meta-feature, rather than a feature. Having data integrity helps you in making other features of your system more simple. For example, migrating schema changes in your system is much more manageable if you use a database which can handle the migration within a transaction.

In your example, there are various ways where system can be left in an inconsistent state after a crash, even if the database is the "source of truth". For example, do you always reconstruct the ES cache after a crash? If not, how do you know whether it contains inconsistencies? Whose job is it to initiate the reconstruction? etc.

Re: Distributed transactions in Go: Read before you try

#53

Post by devjab » Mon, Oct 07, 2024, 9:39 AM UTC

ekidd wrote: post #50

devjab wrote: post #31

Earlier quoted context omitted.

I disagree rather strongly with this advice. Mostly because I’ve spent almost a decade earning rather lucrative money on cleaning up after companies and organisations which did it. Part of what you say is really good advice, if you’re not Facebook then don’t build your infrastructure as though you were. I think it’s always a good idea to remind yourself that StackOverflow ran on a few IIS servers for a long while doi…

I wanted to respond to you, because you had some excellent points. > Mostly because I’ve spent almost a decade earning rather lucrative money on cleaning up after companies and organisations which did it. For many companies, this is actually a pretty successful outcome. They built an app, they earned a pile of money, they kept adding customers, and now they have a mess. But they can afford to pay you to fix their mes…

Well, I guess the side of my argument which is missing by my anecdotal experiences is that monoliths is what I work on because it was the trend. It’s probably easier to fix the complicated mess of a monolith than a complicated mess of micro-services done wrong.

Re: Distributed transactions in Go: Read before you try

#54

Post by devjab » Mon, Oct 07, 2024, 9:45 AM UTC

mamcx wrote: post #39

devjab wrote: post #31

Earlier quoted context omitted.

I disagree rather strongly with this advice. Mostly because I’ve spent almost a decade earning rather lucrative money on cleaning up after companies and organisations which did it. Part of what you say is really good advice, if you’re not Facebook then don’t build your infrastructure as though you were. I think it’s always a good idea to remind yourself that StackOverflow ran on a few IIS servers for a long while doi…

All that you say is true, and people who do that are THE LESS capable of becoming better at the MUCH harder challenges of microservices. I work in the ERP space and interact with dozens and I see the horrors that some only know as fair tales. Without exception, staying in an RDBMS is the best option of all. I have seen the cosmical horrors of what people that struggle with rdbms do when moved to nosql and such, and i…

What I dislike about single databases is that it’s too easy for people to build unnecessary relationships. You obviously don’t have to do it and there are a lot of great tools to separate data. That’s not what people are going to do on a Thursday afternoon after a day of horrible meetings though. They’re going to take shortcuts and mess things up if it’s easy to do so. Having multiple databases, and they can all be SQL (should if that’s what your developers know), in isolation is to protect you from yourself, not so much because it’s a great idea technically.

Re: Distributed transactions in Go: Read before you try

#55

Post by geodel » Mon, Oct 07, 2024, 2:04 PM UTC

ekidd wrote: post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

This advice is good for people at top VPs/CIO/CTOs. Because mandate for micro service is coming from up top. Doing anything else is either not enterprise architecture approved or need to be justified against much powerful higher ups.

Here I have services working at high performance, low resource usage, fewer errors but only feedback I have is how soon can we break it into micro services, how soon we can get into cloud.

Re: Distributed transactions in Go: Read before you try

#56

Post by mamcx » Tue, Oct 08, 2024, 4:47 PM UTC

devjab wrote: post #54

mamcx wrote: post #39

Earlier quoted context omitted.

All that you say is true, and people who do that are THE LESS capable of becoming better at the MUCH harder challenges of microservices. I work in the ERP space and interact with dozens and I see the horrors that some only know as fair tales. Without exception, staying in an RDBMS is the best option of all. I have seen the cosmical horrors of what people that struggle with rdbms do when moved to nosql and such, and i…

What I dislike about single databases is that it’s too easy for people to build unnecessary relationships. You obviously don’t have to do it and there are a lot of great tools to separate data. That’s not what people are going to do on a Thursday afternoon after a day of horrible meetings though. They’re going to take shortcuts and mess things up if it’s easy to do so. Having multiple databases, and they can all be S…

But that is the same if you have many databases. Only that the problem spread!

Maybe is because we are in different niches?. In mine, I have never seen microservices having ANY improvement over the norm, and most certainly are far more negatives.

However, what is more, the norm is making a 2/3-tier from a monolithic, and that could be better.

P.D: In the ERP/business space you can have many, whole apps, with ETL in the middle orchestrating. That may improve things because the quality of each app varies, but what is terrible is to split apps into micro services. That is itself a bridge too far.

Re: Distributed transactions in Go: Read before you try

#57

Post by ljm » Tue, Oct 08, 2024, 7:08 PM UTC

natdempk wrote: post #37

ljm wrote: post #30

Earlier quoted context omitted.

I think the question is if you need the entire system to be strongly consistent, or just the core of it? To use ElasticSearch as an example: do you need to add the complexity of keeping the index up to date in realtime, or can you live with periodic updates for search or a background job for it? As long as your primary DB is the source of truth, you can use that to bring other less critical stores up to date outside…

Well, the problem you run into is that you kind of want different datastores for different use-cases. For example search vs. specific page loads, and you want to try and make both of those consistent, but you don't have a single DB that can serve both use-cases (often times primary DB + ElasticSearch for example). If you don't keep them consistent, you have user-facing bugs where a user can update a record but not se…

In instances like that I tend to push back on the requirement, for example with this classic DB + Elasticsearch case:

1. How often is a user going to perform an update and then search for the exact same thing immediately after?

2. Suppose they did: if elasticsearch was updated in the background, is the queue/worker running fast enough such that the user won't even notice a latency of a second or two max?

It really depends on what you're doing, because if Elasticsearch is operating as its own source of truth with data that the primary DB doesn't have, then yeah, you're going to have trouble keeping both strongly consistent in a transactional manner without layering on complexity (like sagas with transactions and compensations). But if it's merely a search engine on top of your source of truth (for example, you search ES to get a list of primary keys and then fetch all the data from the DB), you've got some breathing room.

I mean, we're talking plucky upstart here and not enterprise FAANG, so there's definitely a case for 'less is more'.

Re: Distributed transactions in Go: Read before you try

#58

Post by foobiekr » Wed, Oct 09, 2024, 4:53 PM UTC

danenania wrote: post #42

foobiekr wrote: post #36

Earlier quoted context omitted.

Great advice. Microservices also open the door to polyglot, so you lose the ability to even arrive that everyone uses/has access to/understands the things in a common libCompany that make it possible for anyone to at least make sense of code. When I talk to people who did microservices, I ask them "why is this a service separate from this?" I have legitimately - and commonly - gotten the answer that the dev I'm talki…

> Microservices also open the door to polyglot While I see your point about the downsides of involving too many languages/technologies, I think the really key distinction is whether a service has its own separate database. It's really not such a big problem to have "microservices" that share the same database. This can bring many of the benefits of microservices without most of the downsides. Imo it would be good if…

Microservices with their own database are often a terrible design choice, but I will grant you it is one of the two dimensions that make sense:

1. is there a _very significant_ difference in the horizontal scaling footprint/model or demand (and demand is _only relevant_ if there is static footprint)? Home the function with a similar service, if there is one, otherwise yes, add a service.

2. is there a _genuine_ and _legitimate_ need for a different database, with completely independent schema, and not some horrible bullshit where you will end up doing cross-party transactions (and almost always failing to do so well)? Are you sure you need a different database or is your team just afraid of SQL and schema management (the usual mongodb garbage)? Is the issue that you don't understand how database security works? .. if all of these pass muster, then yes, ok, that's an OK reason.

Every architecture I've seen since 2013 at startups and big companies alike (since I do technical diligence professionally as a side gig) has been microservices or simple CRUD.

Almost all of the microservices ones were totally fucking wrong and had to be massively reworked, and usually multuple times, because they had no thesis at all for what they were doing and it was often - even mostly - an excuse not to have to learn their tools beyond tutorial level and/or a desire to play with new shiny or not read someone else's code. The CRUD guys were fine, they just did business until they needed to add caching, and so on, like real products.