Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anti-patterns in event-driven architecture

codeopinion.com

Post reply on HN
Print view View original

61–70 of 171 posts

Re: Anti-patterns in event-driven architecture

#61

Post by gnat »

trevor-e wrote: post #47
arwhatever wrote: post #13

I often hear the argument in favor of event-driven architecture that you can work on one part of a system in isolation without having to consider the other parts, and then I get assigned some task which requires me to consider the entire system operation, now with events that are harder to trace than function calls would have been. Now when people argue “because decoupling,” I hear, “You don’t get as much notificatio…

Like anything it can be abused and sometimes folks go overboard with turning everything into an event. However, when done right, it is really amazing to work with. As an event producer as long as you follow reasonable backwards-compatibility best practices then you should be pretty safe from breaking things downstream. As a consumer, follow defensive programming and allow for idempotency in case you need to reprocess…

Do you have pointers to such best practices? Gratefully received etc.

Re: Anti-patterns in event-driven architecture

#62

Post by mvc »

gnat wrote: post #61
trevor-e wrote: post #47

Earlier quoted context omitted.

Like anything it can be abused and sometimes folks go overboard with turning everything into an event. However, when done right, it is really amazing to work with. As an event producer as long as you follow reasonable backwards-compatibility best practices then you should be pretty safe from breaking things downstream. As a consumer, follow defensive programming and allow for idempotency in case you need to reprocess…

Do you have pointers to such best practices? Gratefully received etc.

Rich Hickey's talk about "growth" (as opposed to change) of software systems is a good one for this.

Tldr: ok to add things. Not ok to remove things or change things

Re: Anti-patterns in event-driven architecture

#63

Post by rswail »

candiddevmike wrote: post #3

Can someone share some long term event driven success stories? Almost everything you see online is written by consultants or brand new, greenfield implementations, curious how long these systems last.

Wrote a public transport ticketing system that processes 100-200K+ trips/day with sub-second push of notification to mobiles of trip/payments.

Event driven and CQRS "entities" made logic and processing much easier to create/test/debug.

Primary issues: 1. Making sure you focus on the "Nouns" (entities) not the "Verbs". 2. Kafka requiring polling for consumers sucks if you want to "scale to zero". 3. Sharding of event consumers can be complicated. 4. People have trouble understanding the concepts and keep wanting to write "ProcessX" type functions instead of state machines and event handlers. 5. Retry/replay is complicated, better to reverse/replay. Dealing with side effects in replay is also complicated (does a replay generate the output events which trigger state changes in other entities?)

Been running now for 6 years, minimal downtime except for maintenance/upgrades.

In the process of introducing major new entity and associated changes, most of the system unaffected due to the decoupling.

Re: Anti-patterns in event-driven architecture

#64

Post by The_Colonel »

hobs wrote: post #18
arwhatever wrote: post #13

I often hear the argument in favor of event-driven architecture that you can work on one part of a system in isolation without having to consider the other parts, and then I get assigned some task which requires me to consider the entire system operation, now with events that are harder to trace than function calls would have been. Now when people argue “because decoupling,” I hear, “You don’t get as much notificatio…

I think generally a lot of these types of problems were actually had by folks who grew out of single node systems and had a lot of interesting ideas to solve problems that were new in those domains, GIVEN they've already solved the stateful domain problems as well. When you've never grown out of a single node domain but you do event driven "because scaling" or whatever, you've shot yourself in the foot amazingly hard…

Yes, events, async, eventual consistency, decoupling represent a difficult/complex solution for some hard problems encountered when scaling high.

But people often forget there are trade-offs to everything and if you don't have these hard problems, you're giving yourself only headaches.

My pet-peeve is "decoupling" - it's treated as holy with only benefits and no downsides. But it's actually again a level of complexity - unless you need it, tightly coupled code will be easier to write, read, debug etc.

Re: Anti-patterns in event-driven architecture

#65

Post by throwaway24x7 »

tkiolp4 wrote: post #35
candiddevmike wrote: post #3

Can someone share some long term event driven success stories? Almost everything you see online is written by consultants or brand new, greenfield implementations, curious how long these systems last.

No. The usual pains are: - Producer and consumer are decoupled. That’s a good thing m right? Good luck finding the consumer when you need to modify the producer (the payload). People usually don’t document these things - Let’s use SNS/SQS because why not. Good luck reproducing producers and consumers locally in your machine. Third party infra in local env is usually an afterthought - Observability. Of rather the lack…

> Good luck finding the consumer when you need to modify the producer

It sounds like your alternative is a producer that updates consumers using HTTP calls. That pushes a lot of complexity to the producer and the team that has to sync up with all of the other teams involved.

> Let’s use SNS/SQS because why not. Good luck reproducing producers and consumers locally in your machine

At work we pull localstack from a shared repo and run it in the background. I almost forget that it's there until I need to "git pull" if another team has added a new queue that my service is interested in. Just like using curl to call your HTTP endpoints, you can simply just send a message to localstack with the standard aws cli

https://github.com/localstack/localstack

> Observability. Of rather the lack of it. It’s never out of the box, and so usually nobody cares about it until an incident happens

I think it depends on what type of framework you use. At work we use a trace-id field in the header when making HTTP calls or sending a message (sqs) which is propagated automatically downstream. This enables us to easily search logs and see the flow between systems. This was just configured once and is added automatically for all HTTP requests and messages that the service produces. We have a shared dependency that all services use that handles logging, monitoring and other "plumbing". Most of it comes out of the box from Spring, and the dependency just needs to configure it. The code imports a generic sns/http/jdbc producer and don't have to think about it

Re: Anti-patterns in event-driven architecture

#66

Post by junto »

arwhatever wrote: post #13

I often hear the argument in favor of event-driven architecture that you can work on one part of a system in isolation without having to consider the other parts, and then I get assigned some task which requires me to consider the entire system operation, now with events that are harder to trace than function calls would have been. Now when people argue “because decoupling,” I hear, “You don’t get as much notificatio…

You need to improve your telemetry to feel the benefits. I can trace all the way through multiple services easily on a simple detailed flame graph in our systems.

https://www.datadoghq.com/knowledge-center/distributed-traci...

Unless you have a single monolith, you’re going to face issues with versioning whether it’s event based or API based. In each case you can usually add new properties to a message, but you can’t remove properties or change their types. If you need that, create a new version.

The author does a lot of videos on the event sourcing topic. Event driven I get. It works well in several applications I’ve helped to build over the last 15 years. But event sourcing? I truly don’t get it. Yeah I get it’s nice in terms of auditing to see every change to an entity and who made it, or replay up to to change x on y date, but that really is a niche requirement.

Re: Anti-patterns in event-driven architecture

#67

Post by scubbo »

junto wrote: post #66
arwhatever wrote: post #13

I often hear the argument in favor of event-driven architecture that you can work on one part of a system in isolation without having to consider the other parts, and then I get assigned some task which requires me to consider the entire system operation, now with events that are harder to trace than function calls would have been. Now when people argue “because decoupling,” I hear, “You don’t get as much notificatio…

You need to improve your telemetry to feel the benefits. I can trace all the way through multiple services easily on a simple detailed flame graph in our systems. https://www.datadoghq.com/knowledge-center/distributed-traci... Unless you have a single monolith, you’re going to face issues with versioning whether it’s event based or API based. In each case you can usually add new properties to a message, but you can’t…

> I can trace all the way through multiple services easily on a simple detailed flame graph in our systems

I'm not sure what point is being made here. It's good that you can do that - but are you implying that that's not possible in an API-driven system?

Re: Anti-patterns in event-driven architecture

#68

Post by blowski »

_3u10 wrote: post #39
Spivak wrote: post #25

Earlier quoted context omitted.

Am I dumb or is this basically the binlog of your database but without the tooling to let you do efficient querying? Like I get the "message bus" architecture when you have a bunch of services emitting events and consumers for differing purposes but I don't think I would feel comfortable using it for state tracking. Especially when it seems really hard to enforce a schema / do migrations. CQRS also makes sense for th…

Pretty much. Also all your projectors need to be deterministic. eg. Your commands have to ALWAYS do the same thing else replaying the event log does not produce the same output and then you’re back to square one. It’s usually easier / more useful to just use an audit table.

Yes, if version 1.3 of Command handler X was the active version when an event happened, then it needs to be replayed with that version, even if you’re now on v4.5.

Re: Anti-patterns in event-driven architecture

#69

Post by okr »

junto wrote: post #66
arwhatever wrote: post #13

I often hear the argument in favor of event-driven architecture that you can work on one part of a system in isolation without having to consider the other parts, and then I get assigned some task which requires me to consider the entire system operation, now with events that are harder to trace than function calls would have been. Now when people argue “because decoupling,” I hear, “You don’t get as much notificatio…

You need to improve your telemetry to feel the benefits. I can trace all the way through multiple services easily on a simple detailed flame graph in our systems. https://www.datadoghq.com/knowledge-center/distributed-traci... Unless you have a single monolith, you’re going to face issues with versioning whether it’s event based or API based. In each case you can usually add new properties to a message, but you can’t…

A niche requirement? There are big accounting firms who organize payrolls, where the changes that you mentioned are an important part of their business.

There are also other companies, which do the typical snapshot and roll up to the current time, when they start the services, that need the data without having access to the database.

Re: Anti-patterns in event-driven architecture

#70

Post by richardw »

lanstin wrote: post #56
richardw wrote: post #40

Earlier quoted context omitted.

Banking, 7 ish years. Worked well for us in general. When I start needing increased confidence and truth the effort level goes way up but can be done. Definitely still worth it, has given us some solid benefits. When I say increased, I mean we want the best answer but there are some answers the bank can’t know. If someone has transferred money into your account from another bank but we don’t know that yet, optimising…

Or worse “hey you have a reward” but it doesn’t show up in the UI for three minutes. Twitter used to do this to me all the time.

Eventually consistent means just that, I guess. But I’m sure their reasoning was a lot more sophisticated or impacted by scale than most. Cool problem.
Post reply on HN