Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anti-patterns in event-driven architecture

codeopinion.com

Post reply on HN
Print view View original

91–100 of 171 posts

Re: Anti-patterns in event-driven architecture

#91

Post by lmm »

YZF wrote: post #84

I've worked in a large company where some variation of event driven architecture was used everywhere and treated as the word of G-d. Fairly successfully. Mostly in applications that ran on a single machine. I've ended up in a lot of arguments about this while we were building larger distributed systems because I've come from a more request/response oriented message passing architectures. I.e. more synchronous. What I…

> What I've found is that the event driven architecture did tend to lead to less abstractions and more leaked internal details. This isn't fundamental (you can treat events like an API)

I'd put it the other way: event driven architecture makes it safer to expose more internal details for longer, and lets you push back the point where you really need to fully decouple your API. I see that as an advantage; an abstract API is a means not an end.

> Another problem with distributed systems with persistent queues passing events is that if the consumer falls behind you start developing a lag.

Isn't that what you want? Whatever your architecture, fundamentally when you can't keep up either you queue or you start dropping some inputs.

> If you have a request/response system you can simulate events over that.

How? I mean you can implement your own eventing layer on top of a request/response system, but that's going to give you all the problems of both.

> If we look at things like consensus protocols or distributed/persistent queues then obviously we would need some underlying resources (e.g. you might need a database behind your request/response model).

Huh?

> Don't know if others have a similar experience but when one system is mandated people will invent workarounds that end up looking like the other paradigm, which makes things worse.

I agree that building a request/response system on top of an event sourcing system gives you something worse than using a native request/response system. But that's not a good reason to abandon the mandate, because building a true event-sourcing system has real advantages, and most of those advantages disappear once you start mixing the two. What you do need is full buyin and support at every level rather than a mandate imposed on people who don't want to follow it, but that's true for every development choice.

Re: Anti-patterns in event-driven architecture

#92

Post by soulofmischief »

lmm wrote: post #89
mrkeen wrote: post #71

Earlier quoted context omitted.

> now with events that are harder to trace than function calls would have been I don't know how this could be true. Events are things - nouns which can be backed-up, replicated, stored, queried, rendered, indexed and searched over.

There's no "find usages" for events, and it becomes harder to find out why something didn't happen. A function call can't simply "not return" - in the worst case you get an exception, or a stuck thread in the caller that will show up in a stack dump. But downstream event processing can very easily just not happen, for one of many different reasons, and out-of-the-box it's often difficult to investigate.

In JavaScript,

  const
    myEvent = 'myEvent', 
    target = new EventTarget()

  target.on( myEvent, () => {
    console.log( "It's easy to introspect well-organized code." )
  })

  target.dispatchEvent( new Event( myEvent ))

Re: Anti-patterns in event-driven architecture

#93

Post by lmm »

pudwallabee wrote: post #22

I have seen Kafka pulled out by its hairs and replaced with request based architecture. Event driven architecture, to me is itself an antipattern. It seems like a replacement for batch processing. Replayable messages are AWESOME. Until you encounter the complexity for a system to actually replay them consistently. As far as the authors video, while there was some truth in there, it was a little thin, compared to the…

> I believe that even though Kafka acts the part of "dumb pipe", it doesnt stay dumb for long

In my experience programmers are very happy to do everything in the application (something database people often complain about). What kind of problems do you see?

> If you work with Kafka long enough you’ll realize that data loss will become a problem you didnt think you had. You’ll have to hire people to “look into” data loss problems constantly with Kafka.

Not my experience at all, and I've used Kafka at a wide range of companies, from household-name scale to startups. Kafka is the boring just-works technology that everyone claims they're looking for.

I'm no fan of microservices, but Kafka is absolutely the right datastore most of the time.

Re: Anti-patterns in event-driven architecture

#94

Post by lmm »

Spivak wrote: post #25
Salgat wrote: post #19

Earlier quoted context omitted.

https://www.eventstore.com/case-studies/insureon I can attest to this case study being 100% true. Our platform has been using EventStore as our primary database for 9 years going strong, and I'm still very happy with it. The key thing is that it needs to be done right from the very beginning; you can't do major architecture reworks later on and you need an architect who really knows what they're doing. Also, you can'…

Am I dumb or is this basically the binlog of your database but without the tooling to let you do efficient querying? Like I get the "message bus" architecture when you have a bunch of services emitting events and consumers for differing purposes but I don't think I would feel comfortable using it for state tracking. Especially when it seems really hard to enforce a schema / do migrations. CQRS also makes sense for th…

> Am I dumb or is this basically the binlog of your database but without the tooling to let you do efficient querying?

Yes, and I honestly think a traditional database that exposed this stuff would be a winner (but I guess it's too hard, and meanwhile event-sourcing frameworks are building better alternatives). Separating the low-level part from the high-level part that does things like indexing and querying has a lot of advantages: you decouple data storage from validation so you can have validated data without having to drop invalid data on the floor, you decouple index updates from data writes so your scaling problems get way simpler, you can get sensible happens-before semantics without needing transactions that can deadlock and all the crazy stuff that traditional databases do (secret MVCC implementations while the database goes out of its way to present an illusion of a single "current" version of the data, snapshotting that you can't access directly, ...).

Re: Anti-patterns in event-driven architecture

#95

Post by lmm »

jgraettinger1 wrote: post #21
candiddevmike wrote: post #3

Can someone share some long term event driven success stories? Almost everything you see online is written by consultants or brand new, greenfield implementations, curious how long these systems last.

PostgreSQL. The WAL is an event log, and when you squint at its internal architecture, you’ll see plenty of overlap with distributed event sourcing.

Right. The hard part is already done. Which makes it infuriating that it's all "internal". Every serious RDBMS already contains an implementation of an event-sourcing system, but you're not allowed to actually use it.

Re: Anti-patterns in event-driven architecture

#96

Post by lmm »

soulofmischief wrote: post #92
lmm wrote: post #89

Earlier quoted context omitted.

There's no "find usages" for events, and it becomes harder to find out why something didn't happen. A function call can't simply "not return" - in the worst case you get an exception, or a stuck thread in the caller that will show up in a stack dump. But downstream event processing can very easily just not happen, for one of many different reasons, and out-of-the-box it's often difficult to investigate.

In JavaScript, const myEvent = 'myEvent', target = new EventTarget() target.on( myEvent, () => { console.log( "It's easy to introspect well-organized code." ) }) target.dispatchEvent( new Event( myEvent ))

Yeah, good luck remembering to do that up-front for every event handler. You missed one? Whoops, enjoy the information you wanted silently not being there when you need it.

Re: Anti-patterns in event-driven architecture

#97

Post by nick__m »

mrkeen wrote: post #72
nitwit005 wrote: post #54

Earlier quoted context omitted.

I've seen successful, but flawed usage. Every use I've seen sent events after database transactions, with the event not part of the transaction. This means you can get both dropped events, and out of order events. My current company has analytics driven by a system like that. I'm sure there's some corrupted data as a result. The main issue being people just don't know how to build and test distributed systems.

I had an interview where I was asked how I would guarantee that an event happened in addition to a database update (transactionally). It sounded kind of impossible, I said as much, and then proposed a different approach. The interviewer persisted and claimed that it could be done with 'the outbox pattern'. I disagreed and ended the interview there. Later when I was chatting about it with a former colleague, he said "…

You could two phase commit with XA compliant broker and database.

Re: Anti-patterns in event-driven architecture

#98

Post by YZF »

lmm wrote: post #91
YZF wrote: post #84

I've worked in a large company where some variation of event driven architecture was used everywhere and treated as the word of G-d. Fairly successfully. Mostly in applications that ran on a single machine. I've ended up in a lot of arguments about this while we were building larger distributed systems because I've come from a more request/response oriented message passing architectures. I.e. more synchronous. What I…

> What I've found is that the event driven architecture did tend to lead to less abstractions and more leaked internal details. This isn't fundamental (you can treat events like an API) I'd put it the other way: event driven architecture makes it safer to expose more internal details for longer, and lets you push back the point where you really need to fully decouple your API. I see that as an advantage; an abstract…

Everything is a means and not an end but decoupling via an explicit API makes change easier. Spreading state across your system via events (specifically synchronizing data across systems via events relating to how that data changes) creates coupling.

re: Huh. Sorry I was not clear there. What I meant is you can not create persistent queue semantics out of a request/response model without being able to make certain kinds of requests that access resources. Maybe that's an obvious statement.

re: mandate. I think I'm saying these sort of mandates inevitable result in poor design. even the purest of purest event sourcing systems actually use requests/response simply because that is the fundamental building block of systems. E.g. Kafka uses gRPCs from the client and waits for a response in order to inject something into a queue. The communication between Kafka nodes is based on messages. The basic building block of any distributed computer system is a packet (request) being sent from one machine to another, and a response being sent back (e.g. TCP control messages). A mandate that says though shall build everything on top of event sourcing is sort of silly in this context since it should be obvious the building blocks of event sourced systems use requests/response. Even without this nit-picking restricting application developers to only build on top of this abstraction inevitably leads to ugliness. IMO anyways and having seen this mandate at work in a large organizations. Use the right tool for your job is more or less what I'm saying or the other famous way of stating this is when all you have is a hammer everything looks like a nail.

re: isn't that what you want. well, if it is what you want then it is what you want, but many systems are ok with things just getting lost and not persisted. e.g. an HTTP GET request from a browser, in the absence of a network connection, is just lost, it's not persisted to be played later, and so there is no way to build a lagging queue with HTTP GET requests that are yet to be processed. Again, maybe an obvious statement.

Re: Anti-patterns in event-driven architecture

#99

Post by lmm »

YZF wrote: post #98
lmm wrote: post #91

Earlier quoted context omitted.

> What I've found is that the event driven architecture did tend to lead to less abstractions and more leaked internal details. This isn't fundamental (you can treat events like an API) I'd put it the other way: event driven architecture makes it safer to expose more internal details for longer, and lets you push back the point where you really need to fully decouple your API. I see that as an advantage; an abstract…

Everything is a means and not an end but decoupling via an explicit API makes change easier. Spreading state across your system via events (specifically synchronizing data across systems via events relating to how that data changes) creates coupling. re: Huh. Sorry I was not clear there. What I meant is you can not create persistent queue semantics out of a request/response model without being able to make certain ki…

> decoupling via an explicit API makes change easier. Spreading state across your system via events (specifically synchronizing data across systems via events relating to how that data changes) creates coupling.

An explicit API comes at a cost; the way I'd put it is that the inherently lower coupling of events (because e.g. you can publish the same events in multiple formats, whereas a request-response API generally needs to have a single response format) means that you have more slack to defer that cost for longer (i.e. it takes you longer to reach the point where the overall system coupling is too bad and you need to introduce those API layers).

I'm not sure I follow what you're saying about sharing events relating to how data changes. IMO if you need a shared view of "the current version of the data", the right solution is to publish that as events too.

> E.g. Kafka uses gRPCs from the client and waits for a response in order to inject something into a queue. The communication between Kafka nodes is based on messages. The basic building block of any distributed computer system is a packet (request) being sent from one machine to another, and a response being sent back (e.g. TCP control messages).

I don't know the details of kafka's low-level protocols, but it's certainly possible to build these systems based on one-way messaging all the way down; gRPC has one-way messages, plenty of protocols are built on one-way UDP rather than TCP...

> e.g. an HTTP GET request from a browser, in the absence of a network connection, is just lost, it's not persisted to be played later, and so there is no way to build a lagging queue with HTTP GET requests that are yet to be processed.

Right, because HTTP is a request-response protocol. Whereas Kafka does buffer messages and send them later if you lose your network connection for a short time (of course there is a point at which it will give up and call your error handler).

I don't think the fact that HTTP works that way means it's desirable to just abandon those requests - e.g. in fact these days if you navigate to a page with Chrome when you have no network connection it will make the request when you're back online and send you a notification that it's loaded the page that it couldn't load earlier.

Re: Anti-patterns in event-driven architecture

#100

Post by jesse__ »

tadfisher wrote: post #38
ninkendo wrote: post #28

Earlier quoted context omitted.

Chiming in with another “no” here. We adopted a message bus/event-driven architecture when moving a very popular piece of software from the cloud, to directly on the user’s device… it was a disaster IMO. The core orchestration of the system was done via events on the bus, and nobody had any idea what was happening when a bug occurred. People would pass bugs around, “my code did the right thing given the event it got”…

I think the true term for this phenomenon is "decoherence" rather than "decoupling". Your components are still as coupled as they ever were, but the coupling has moved from compile-time (e.g. function calls) to runtime. The component that "handles events" decoheres the entire system because it's now responsible for the entire messaging layer between components, rather than the individual components being responsible…

That's a great name for that property. I've always cringed when people say 'something something decoupling' because most of the time the end result is actually just as coupled, but ends up indirected or something. Now I have a more specific word for it, thanks!!
Post reply on HN