Live data from Hacker News

Distributed transactions in Go: Read before you try

threedots.tech

31–40 of 58 posts

Re: Distributed transactions in Go: Read before you try

#31
post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

I disagree rather strongly with this advice. Mostly because I’ve spent almost a decade earning rather lucrative money on cleaning up after companies and organisations which did it. Part of what you say is really good advice, if you’re not Facebook then don’t build your infrastructure as though you were. I think it’s always a good idea to remind yourself that StackOverflow ran on a few IIS servers for a long while doing exactly what you’re recommending that people do. (Well almost anyway).

Using a single database always ends up being a mess. Ok, I shouldn’t say always because it’s technically possible for it not to happen. I’ve just never seen the OOP people not utterly fuck up the complexity in their models. It gets even worse when they’ve decided to use stores procedures or some magical ORM which not everyone understood the underlying workings of. I think you should definitely separate your data as much as possible. Even small scale companies will quickly struggle scaling their DBs if they don’t, and it’ll quickly become absolutely horrible if you have to remove parts of your business. Maybe they are unseeded, maybe they get sold off, whatever it is. With that said, however, I think you’re completely correct about not doing distributed transactions. I think that both you and the author are completely right that if you’re doing this, then you’re building complexity you should be building until you’re Facebook (or maybe when you’re almost Facebook).

A good micro-service is one that can live in total isolation. It’ll full-fill the need of a specific business domain, and it should contain all the data for this. If that leaves you with a monolith and a single shared database, then that is perfectly fine. If you can split it up. Say you have solar plants which are owned by companies but as far as the business goes a solar plant and a company can operate completely independently, then you should absolutely build them as two services. If you don’t, then you’re going start building your mess once you need to add wind plants or something different. Do note that I said that this depends on the business needs. If something like individual banking accounts of company owners is relevant to the greenfield workers and asset managers, then you probably can’t split up solar plants and companies. Keeping things separate like this will also help you immensely as you add on business intelligence and analytics.

If you keep everything in a single “model store”, then you’re eventually going end up with “oh, only John knows what that data does” while needing to pay someone like me a ridiculous amount of money to help your IT department get to a point where they are no longer hindering your company growth. Again, I’m sure this doesn’t have to be the case and I probably should just advise people to do exactly as you say. In my experience it’s an imperfect world and unless you keep things as simple as possible with as few abstractions as possible then you’re going to end up with a mess.

Re: Distributed transactions in Go: Read before you try

#32

Earlier quoted context omitted.

I agree with start with a monolith and a shared database. I’ve done that in the past quite successfully. I would just add that if scaling becomes an issue, I wouldn’t consider sharding my first option, it’s more of a last resort. I would prefer scaling vertically the shared database and optimizing it as much as possible. Also, another strategy I’ve adopted was avoiding doing `JOIN` or `ORDER BY`, as they stress your…

I don't understand how do you avoid JOIN and ORDER BY? Well, with ORDER BY, if your result set is not huge, sure, you can just sort it on the client side. Although sorting 100 rows on database side isn't expensive. But if you need, say, latest 500 records out of million (very frequent use-case), you have to sort it on the database side. Also with proper indices, database sometimes can avoid any explicit sort. Do you…

I think a better maxim would be to never have an un-indexed ORDER BY or JOIN.

A big part of what many "nosql" databases that prioritize scale are doing is simply preventing you from ever running an adhoc un-indexed query.

Re: Distributed transactions in Go: Read before you try

#33

If I had a nickel for all the clients I’ve seen with micro services everywhere, and 90% of the code is replicating an RDBMS with hand coded in memory joins. What could have been a simple SQL query in a sane architecture becomes N REST calls (possibly nested with others downstream) and manually stitching together results. And that is just the read only case. As the author notes updates add another couple of levels of…

In the good old days, if you did that, you would rightfully be labeled as an "amateur".

Re: Distributed transactions in Go: Read before you try

#34
post #25

Main source of pain with eventual consistency is lots of customer calls/emails "we did X but nothing happened". I'd also add that you should make it clear to the user that the action may not be instantenous.

That's another thing about a single database, if it's well-tuned and your squeel is good - in many cases you don't even need any kind of cache, removing an entire class of bugs.

Re: Distributed transactions in Go: Read before you try

#36
post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

Great advice. Microservices also open the door to polyglot, so you lose the ability to even arrive that everyone uses/has access to/understands the things in a common libCompany that make it possible for anyone to at least make sense of code.

When I talk to people who did microservices, I ask them "why is this a service separate from this?"

I have legitimately - and commonly - gotten the answer that the dev I'm talking to wanted their own service.

It's malpractice.

Re: Distributed transactions in Go: Read before you try

#37
post #30

Earlier quoted context omitted.

Pretty great advice! I think the one thing you can run into that is hard is once you want to support different datasets that fall outside the scope of a transaction (think events/search/derived-data, anything that needs to read/write to a system that is not your primary transactional DB) you probably do want some sort of event bus/queue type thing to get eventual consistency across all the things. Otherwise you just…

I think the question is if you need the entire system to be strongly consistent, or just the core of it? To use ElasticSearch as an example: do you need to add the complexity of keeping the index up to date in realtime, or can you live with periodic updates for search or a background job for it? As long as your primary DB is the source of truth, you can use that to bring other less critical stores up to date outside…

Well, the problem you run into is that you kind of want different datastores for different use-cases. For example search vs. specific page loads, and you want to try and make both of those consistent, but you don't have a single DB that can serve both use-cases (often times primary DB + ElasticSearch for example). If you don't keep them consistent, you have user-facing bugs where a user can update a record but not search for it immediately, or if you try to load everything from ES to provide consistent views to a user, then updates can disappear on refresh. Or if you try to write to both SQL + ES in an API request, they can desync on failure writing to one or the other. The problem is even less the complexity of keeping the index up to date in realtime, and more that the ES index isn't even consistent with the primary DB, and to a user they are just different parts of your app that kinda seem a little broken in subtle ways inconsistently. It would be great to be able to have everything present a consistent view to users, that updates together on-write.

Re: Distributed transactions in Go: Read before you try

#38
post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

Sure sure sure

...but micro services is used as a people organization technique.

Once you're there you'll run into a situation where you'll have to do transactions. Might as well get good at it.

Re: Distributed transactions in Go: Read before you try

#39
post #31
post #6

Let's assume you're not a FAANG, and you don't have a billion customers. If you're gluing microservices together using distributed transactions (or durable event queues plus eventual consistency, or whatever), the odds are good that you've gone far down the wrong path. For many applications, it's easiest to start with a modular monolith talking to a shared database, one that natively supports transactions. When this…

I disagree rather strongly with this advice. Mostly because I’ve spent almost a decade earning rather lucrative money on cleaning up after companies and organisations which did it. Part of what you say is really good advice, if you’re not Facebook then don’t build your infrastructure as though you were. I think it’s always a good idea to remind yourself that StackOverflow ran on a few IIS servers for a long while doi…

All that you say is true, and people who do that are THE LESS capable of becoming better at the MUCH harder challenges of microservices.

I work in the ERP space and interact with dozens and I see the horrors that some only know as fair tales.

Without exception, staying in an RDBMS is the best option of all. I have seen the cosmical horrors of what people that struggle with rdbms do when moved to nosql and such, and is always much worse than before.

And all that you say, that is true, hide the real (practical) solution: Learn how to use the RDBMS, use SQL, remove complexity, and maybe put the thing in a bigger box.

All that is symptoms that are barely related to the use of a single database.

Re: Distributed transactions in Go: Read before you try

#40
AGREE! The author's point is very well argued. Beginning a transaction is almost never a good idea. Design your data model so that if two pieces of data must be consistent, they are in the same row, and allow associated rows to be missing, handling nulls in the application. Inserts and updates should operate on a single table, because in the case of failure, nothing changed, and you have a simple error to deal with. In short, as explained in the article, embrace eventual consistency. There was a great post from the Github team about why they didn't allow transactions in their rails app, from around 2013, but I can't find it for the life of me.

I realize that you're staring at me in disbelief right now, but this is gospel!

Post reply on HN