Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anti-patterns in event-driven architecture

codeopinion.com

Post reply on HN
Print view View original

161–170 of 171 posts

Re: Anti-patterns in event-driven architecture

#161

Post by [deleted] »

setr wrote: post #120
ninkendo wrote: post #28

Earlier quoted context omitted.

Chiming in with another “no” here. We adopted a message bus/event-driven architecture when moving a very popular piece of software from the cloud, to directly on the user’s device… it was a disaster IMO. The core orchestration of the system was done via events on the bus, and nobody had any idea what was happening when a bug occurred. People would pass bugs around, “my code did the right thing given the event it got”…

> (Wild tangent: holy shit is hard to get iOS to accept “do process” in a sentence. I edited that paragraph at least 30 times, no joke, trying every trick I could to stop it correcting it to “due process”. I almost gave up. I used to defend autocorrect but holy shit that was a nightmare.) can you not just pick the original spelling in the autocomplete menu above the keyboard?

[deleted]

Re: Anti-patterns in event-driven architecture

#162

Post by syca_f13 »

therealdrag0 wrote: post #77
pudwallabee wrote: post #22

I have seen Kafka pulled out by its hairs and replaced with request based architecture. Event driven architecture, to me is itself an antipattern. It seems like a replacement for batch processing. Replayable messages are AWESOME. Until you encounter the complexity for a system to actually replay them consistently. As far as the authors video, while there was some truth in there, it was a little thin, compared to the…

What are the causes of data loss?

Jepsen has written a fantastic article on this issue. I'm not sure if it has been fixed since then. https://aphyr.com/posts/293-call-me-maybe-kafka

Re: Anti-patterns in event-driven architecture

#163

Post by Supermancho »

withinboredom wrote: post #148
Supermancho wrote: post #147

Earlier quoted context omitted.

Regarding monoliths...when there's an issue, now everyone who made a PR is subject to forensics to try to identify cause. I rather make a separate app that is infrequently changed, resulting in less faults and shorter investigations. Being on the hook to try to figure out when someone breaks "related" to my team's code, is also a waste of developer time. There is a middle ground for optimizing developer time, but put…

I'm not sure how you think microservices gets around that (it doesn't!). We didn't play a blame game though... your team was responsible for your slice of the world and that was it. Anyone could open a PR to your code and you could open a PR to anyone else's code. It was a pretty rare event unless you were working pretty deep in the stack (aka, merging framework upgrades from open source) or needing new API's in some…

> I'm not sure how you think microservices gets around that (it doesn't!).

Microservices get around potential dependency bugs, because of the isolation. Now there's an API orchestration between the services. That can be a point of failure. This is why you want BDD testing for APIs, to provide a higher confidence.

The tradeoff isn't complicated. Slightly more work up front for less maintenance long term; granted this approach doesn't scale forever. There's not any science behind finding the tipping point.

Re: Anti-patterns in event-driven architecture

#164

Post by trevor-e »

gnat wrote: post #61
trevor-e wrote: post #47

Earlier quoted context omitted.

Like anything it can be abused and sometimes folks go overboard with turning everything into an event. However, when done right, it is really amazing to work with. As an event producer as long as you follow reasonable backwards-compatibility best practices then you should be pretty safe from breaking things downstream. As a consumer, follow defensive programming and allow for idempotency in case you need to reprocess…

Do you have pointers to such best practices? Gratefully received etc.

Design for things to be easily failable. It should be trivial to have failed messages to go to a DLQ and then reprocess them later on, say after a bug fix.

Only make additive changes, don't change existing fields. For enums it's up to the consumer to ensure they don't fail when a new case is added.

Be very careful with including data (especially time/expiry stuff) in the message too. If you need to reprocess the event several hours later then it may no longer work or be stale. Rather than include the data in the message itself, we would include the database ID and then have the consumer query for that entry.

Re: Anti-patterns in event-driven architecture

#165

Post by ramchip »

mrkeen wrote: post #160
ramchip wrote: post #83

Earlier quoted context omitted.

I'm a bit confused by the story. Why did you disagree?

They asked how I would guarantee that a Postgres update and a Kafka event could both happen transactionally. (P, K) Which sounds like one of those classical impossibility proofs. Their solution was to introduce another part into the system, "the outbox": (P, O) K P and O can form a transaction, but that still leaves the original problem unanswered, how to include K in the transaction.

The goal of the outbox pattern is at-least-once publishing though, not only-once. You either get P + (eventually) at least one copy of K, or you get no P and no K.

Without the outbox you can get P without K or K without P, which lead to consumers out of sync with the producer.

This requires the consumer to be able to deal with repeated events to some extent, but you usually want that anyway, since an event can be processed twice even if it appears only once in the topic. For instance if the consumer crashes after processing but before updating the offset.

Re: Anti-patterns in event-driven architecture

#166

Post by mrkeen »

ramchip wrote: post #165
mrkeen wrote: post #160

Earlier quoted context omitted.

They asked how I would guarantee that a Postgres update and a Kafka event could both happen transactionally. (P, K) Which sounds like one of those classical impossibility proofs. Their solution was to introduce another part into the system, "the outbox": (P, O) K P and O can form a transaction, but that still leaves the original problem unanswered, how to include K in the transaction.

The goal of the outbox pattern is at-least-once publishing though, not only-once. You either get P + (eventually) at least one copy of K, or you get no P and no K. Without the outbox you can get P without K or K without P, which lead to consumers out of sync with the producer. This requires the consumer to be able to deal with repeated events to some extent, but you usually want that anyway, since an event can be pro…

> The goal of the outbox pattern is at-least-once publishing though, not only-once.

Right, which is why it's an unacceptable solution to 'transacting over postgres and Kafka', and why I wouldn't want to work for a company that wants me to believe differently.

And there's a better solution anyway: just K.

Re: Anti-patterns in event-driven architecture

#167

Post by ramchip »

mrkeen wrote: post #166
ramchip wrote: post #165

Earlier quoted context omitted.

The goal of the outbox pattern is at-least-once publishing though, not only-once. You either get P + (eventually) at least one copy of K, or you get no P and no K. Without the outbox you can get P without K or K without P, which lead to consumers out of sync with the producer. This requires the consumer to be able to deal with repeated events to some extent, but you usually want that anyway, since an event can be pro…

> The goal of the outbox pattern is at-least-once publishing though, not only-once. Right, which is why it's an unacceptable solution to 'transacting over postgres and Kafka', and why I wouldn't want to work for a company that wants me to believe differently. And there's a better solution anyway: just K.

I think you're just using "transaction" in a different sense than what the interviewer meant; "guarantee that an event happened in addition to a database update" sounds like at-least-once to me, and it's normally what you would want in this kind of system.

Re: Anti-patterns in event-driven architecture

#168

Post by YZF »

davedx wrote: post #158
YZF wrote: post #113

Earlier quoted context omitted.

I'm pretty sure different banks use vastly different architectures. Some run nightly batch jobs on mainframes that are written in COBOL. But alas, I've not worked for any bank, this is just a commonly used example. I am willing to bet the transaction log, or ledger, is indeed a very common approach since that's also common in accounting. EDIT: Also event sourcing would typically be eventually consistent. I imagine fo…

I’m sceptical, because I can’t find any examples of banks using it in production, just lots of blog posts by consultants and companies selling event sourcing solutions. I did work somewhere that used a Kafka stack in production. It wasn’t a compelling use case and they spent almost an entire year on infra and productionizing it. It left me extremely sceptical about anything “big data streams” related :)

Fair enough. But I've used event sourcing in production in two companies. One project was a large scale distributed object store and the other was network equipment (like switches) and network management. The banking example is a classical one but I can't tell you if it's actually used in production. What I do know is that accounting software follows the ledger approach which has a similar spirit, recording transactions, and my guess would be regardless of technology banks also are transaction/event based as their source of truth (even in a COBOL mainframe batch processing scenario).

Re: Anti-patterns in event-driven architecture

#169

Post by withinboredom »

Supermancho wrote: post #163
withinboredom wrote: post #148

Earlier quoted context omitted.

I'm not sure how you think microservices gets around that (it doesn't!). We didn't play a blame game though... your team was responsible for your slice of the world and that was it. Anyone could open a PR to your code and you could open a PR to anyone else's code. It was a pretty rare event unless you were working pretty deep in the stack (aka, merging framework upgrades from open source) or needing new API's in some…

> I'm not sure how you think microservices gets around that (it doesn't!). Microservices get around potential dependency bugs, because of the isolation. Now there's an API orchestration between the services. That can be a point of failure. This is why you want BDD testing for APIs, to provide a higher confidence. The tradeoff isn't complicated. Slightly more work up front for less maintenance long term; granted this…

> Microservices get around potential dependency bugs, because of the isolation.

How so? I'd buy that bridge if you could deliver, but you can't. Isolation doesn't protect you from dependency bugs and doesn't protect your dependents from your own bugs. If you start returning "payment successful" when it isn't; lots of people are going to get mad -- whether there is isolation or not.

> Now there's an API orchestration between the services

An API is simply an interface -- whether that is over a socket or in-memory, you don't need a microservice to provide an API.

> This is why you want BDD testing for APIs, to provide a higher confidence.

Testing is possible in all kinds of software architectures, but we didn't need testing just to make sure an API was followed. If you broke the API contract in the monolith, it simply didn't compile. No testing required.

> Slightly more work up front for less maintenance long term

I'm actually not sure which one you are pointing at here... I've worked with both pretty extensively in large projects and I would say the monolith was significantly LESS maintenance for a 20 year old project. The microservice architectures I've worked on have been a bit younger (5-10 years old) but require significantly more work just to keep the lights on, so maybe they hadn't hit that tipping point you refer to, yet.

Re: Anti-patterns in event-driven architecture

#170

Post by Salgat »

Spivak wrote: post #25
Salgat wrote: post #19

Earlier quoted context omitted.

https://www.eventstore.com/case-studies/insureon I can attest to this case study being 100% true. Our platform has been using EventStore as our primary database for 9 years going strong, and I'm still very happy with it. The key thing is that it needs to be done right from the very beginning; you can't do major architecture reworks later on and you need an architect who really knows what they're doing. Also, you can'…

Am I dumb or is this basically the binlog of your database but without the tooling to let you do efficient querying? Like I get the "message bus" architecture when you have a bunch of services emitting events and consumers for differing purposes but I don't think I would feel comfortable using it for state tracking. Especially when it seems really hard to enforce a schema / do migrations. CQRS also makes sense for th…

For events you include a version. When you're only adding properties to the event or removing properties (assuming you defensively write the projectors), no need for a new version, but if you're creating a breaking change in event schema, you'd increment the version for the event and update your projector to handle each version. It's simpler than you'd think.
Post reply on HN